gorgon 0.2.0 → 0.3.0

data/.gitignore

@@ -5,4 +5,6 @@ tags
 TAGS
 pkg
 .idea
-.rbenv-version
+.rbenv-version
+*.log
+*.json

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    gorgon (0.2.0)
+    gorgon (0.3.0)
       amqp (~> 0.9.7)
       awesome_print
       bunny (~> 0.8.0)
@@ -30,14 +30,14 @@ GEM
     eventmachine (1.0.0)
     open4 (1.3.0)
     rake (0.9.2.2)
-    rspec (2.8.0)
-      rspec-core (~> 2.8.0)
-      rspec-expectations (~> 2.8.0)
-      rspec-mocks (~> 2.8.0)
-    rspec-core (2.8.0)
-    rspec-expectations (2.8.0)
-      diff-lcs (~> 1.1.2)
-    rspec-mocks (2.8.0)
+    rspec (2.11.0)
+      rspec-core (~> 2.11.0)
+      rspec-expectations (~> 2.11.0)
+      rspec-mocks (~> 2.11.0)
+    rspec-core (2.11.1)
+    rspec-expectations (2.11.3)
+      diff-lcs (~> 1.1.3)
+    rspec-mocks (2.11.3)
     ruby-progressbar (1.0.1)
     test-unit (2.5.2)
     uuidtools (2.1.3)

data/README.md

@@ -4,7 +4,7 @@ Gorgon
 About
 ---------------------
 
-Gorgon provides a method for distributing the workload of running a ruby test suites. It relies on amqp for message passing, and rsync for the synchronization of source code.
+Gorgon provides a method for distributing the workload of running ruby test suites. It relies on amqp for message passing, and rsync for the synchronization of source code.
 
 Usage
 ---------------------
@@ -19,18 +19,24 @@ Configuration
 ### gorgon.json
 This file contains project-specific settings for gorgon, such as:
 
-* A glob for generating the list of test files
 * The connection information for AMQP
-* Information about how clients can rsync the working directory
+* Information about how clients can rsync the working directory (optional)
+* Files that can be excluded by rsync
 * Files containing Ruby code to be used as callbacks
+* A glob for generating the list of test files
+* The file used for Originator's logs
+
+See [gorgon.json example](https://github.com/Fitzsimmons/Gorgon/blob/master/gorgon.json.sample) for more details.
 
 ### gorgon_listener.json
 This file contains the listener-specific settings, such as:
 
-* How many worker slots are provided by this listener
 * The connection information for AMQP
+* How many worker slots are provided by this listener
 * The file used for logs
 
+See [gorgon_listener.json example](https://github.com/Fitzsimmons/Gorgon/blob/master/gorgon_listener.json.sample) for more details.
+
 Architecture
 ---------------------
 
@@ -39,15 +45,33 @@ By running `bundle exec gorgon start`, the originating computer will publish a *
 * The rsync information with which to fetch the source tree
 * The name of a AMQP queue that contains the list of files that require testing
 * The name of a AMQP exchange to send replies to
-* Application-specific setup/teardown, either per-job or per-worker [scheduled for post-alpha]
+* Application-specific setup/teardown, either per-job or per-worker (callbacks)
 
 The job listener subscribes to the job publish event, and maintains its own queue of jobs. When a job has available *worker slots*, it will prepare the workspace:
 
 * Create a unique temporary workspace directory for the job
 * Rsync the source tree to the temporary workspace
-* Run per-job application-specific setup [scheduled for post-alpha]
-* Invoke *n* workers, where *n* is the number of available *worker slots*.
+* Run after_sync callback
+* Invoke a WorkerManager
+
+After WorkerManager starts, it will:
+* Run before\_creating\_workers callback
+* Fork *n* workers, where *n* is the number of available *worker slots*.
+* Subscribe to a queue where originator can send a cancel_job message
+
+Each Worker will:
+* Run before_start callback
+* Pop a file from file queue and run it until file queue is empty, or WorkerManager sends an INT signal. For each file, it post a 'start_message' and a 'finish_message' with the results to the *reply queue*
+* Run after_complete callback
 
-To invoke a job worker, the listener passes the name of the *file queue*, *reply queue*, and *listener queue* to the worker initialization. After all workers have been started, the listener will block until an event appears on the *listener queue*.
+To invoke the worker manager, the listener passes the name of the *file queue*, *reply queue*, and *listener queue* to the worker manager initialization, and then it will block until worker manager finishes.
+
+Contributors
+---------------------
+* Justin Fitzsimmons
+* Arturo Pie
+* Sean Kirby
+* Clemens Park
+* Victor Savkin
 
-The worker process will run any application-specific startup, start a test environment, and load a stub test file that dynamically pulls files out of the *file queue*. It runs the test, posts the results to the *reply queue*, and repeats until the *file queue* is empty. When the *file queue* becomes empty, the worker runs application-specific teardown, then reports its completion to the *listener queue*, and shuts down.
+Gorgon is funded by [Nulogy Corp](http://www.nulogy.com/).

data/bin/gorgon

@@ -3,6 +3,7 @@ require 'gorgon/originator'
 require 'gorgon/listener'
 require 'gorgon/worker_manager'
 require 'gorgon/ping_service'
+require 'gorgon/gem_service'
 require 'gorgon/version'
 
 WELCOME_MSG = "Welcome to Gorgon #{Gorgon::VERSION}"
@@ -31,11 +32,16 @@ def ping_listeners
   PingService.new.ping_listeners
 end
 
+def run_gem command
+  GemService.new.run command
+end
+
 def usage
   #print instructions on how to use gorgon
   puts "\tstart - remotely runs all tests specified in gorgon.json"
   puts "\tlisten - starts a listener process using the settings in gorgon_listener.json"
   puts "\tping - pings listeners and shows hosts and gorgon's version they are running"
+  puts "\tgem command [options...] - execute the gem command on every listener and shutdown listener. e.g. 'gorgon gem install --version 1.0.0'"
 end
 
 puts WELCOME_MSG
@@ -51,6 +57,9 @@ when "manage_workers"
   manage_workers
 when "ping"
   ping_listeners
+when "gem"
+  ARGV.shift
+  run_gem ARGV.join(' ')
 when "help"
   usage
 else

data/gorgon.json.sample

@@ -0,0 +1,29 @@
+{
+  "connection": {
+    "host": "localhost"
+  },
+
+  "job": {
+    "sync_exclude": [
+        "tmp",
+        "log",
+        "doc",
+        ".git",
+        ".rvmrc"
+    ],
+    "callbacks": {
+        "before_start": "test/gorgon_callbacks/before_start_callback.rb",
+        "after_complete": "test/gorgon_callbacks/after_complete.rb",
+        "before_creating_workers": "test/gorgon_callbacks/before_creating_workers.rb",
+        "after_sync": "test/gorgon_callbacks/after_sync.rb"
+    }
+  },
+
+  "files": [
+      "test/unit/**/*_test.rb",
+      "test/functional/**/*_test.rb",
+      "test/integration/**/*_test.rb"
+  ],
+
+  "originator_log_file": "log/gorgon-orginator.log"
+}

data/gorgon_listener.json.sample

@@ -0,0 +1,8 @@
+{
+    "connection": {
+        "host": "localhost"
+    },
+
+    "worker_slots": 3,
+    "log_file": "/tmp/gorgon-remote.log"
+}

data/lib/gorgon/colors.rb

@@ -1,4 +1,5 @@
 module Colors
   FILENAME = :light_cyan
   HOST = :light_blue
+  COMMAND = :yellow
 end

data/lib/gorgon/crash_reporter.rb

@@ -0,0 +1,19 @@
+module CrashReporter
+  OUTPUT_LINES_TO_REPORT = 70
+
+  def report_crash reply_exchange, info
+    stdout = `tail -n #{OUTPUT_LINES_TO_REPORT} #{info[:out_file]}`
+    stderr = `tail -n #{OUTPUT_LINES_TO_REPORT} #{info[:err_file]}` + \
+    info[:footer_text]
+
+    send_crash_message reply_exchange, stdout, stderr
+
+    "#{stdout}\n#{stderr}"
+  end
+
+  def send_crash_message reply_exchange, output, error
+    reply = {:type => :crash, :hostname => Socket.gethostname,
+      :stdout => output, :stderr => error}
+    reply_exchange.publish(Yajl::Encoder.encode(reply))
+  end
+end

data/lib/gorgon/gem_command_handler.rb

@@ -0,0 +1,46 @@
+require "socket"
+require "yajl"
+require "bunny"
+require 'open4'
+
+class GemCommandHandler
+  def initialize bunny
+    @bunny = bunny
+  end
+
+  def handle payload, configuration
+    reply_exchange_name = payload[:reply_exchange_name]
+    publish_to reply_exchange_name, :type => :running_command
+
+    gem = configuration[:bin_gem_path] || "gem"
+
+    cmd = "#{gem} #{payload[:body][:gem_command]} gorgon"
+    pid, stdin, stdout, stderr = Open4::popen4 cmd
+    stdin.close
+
+    ignore, status = Process.waitpid2 pid
+    exitstatus = status.exitstatus
+
+    output, errors = [stdout, stderr].map { |p| begin p.read ensure p.close end }
+
+    if exitstatus == 0
+      reply = {:type => :command_completed, :command => cmd, :stdout => output,
+        :stderr => errors}
+      publish_to reply_exchange_name, reply
+      @bunny.stop
+      exit     # TODO: for now exit until we implement a command to exit listeners
+    else
+      reply = {:type => :command_failed, :command => cmd, :stdout => output, :stderr => errors}
+      publish_to reply_exchange_name, reply
+   end
+  end
+
+  private
+
+  # TODO: factors this out to a class
+  def publish_to reply_exchange_name, message
+    reply_exchange = @bunny.exchange(reply_exchange_name, :auto_delete => true
+)
+    reply_exchange.publish(Yajl::Encoder.encode(message.merge(:hostname => Socket.gethostname)))
+  end
+end

data/lib/gorgon/gem_service.rb

@@ -0,0 +1,77 @@
+require 'gorgon/originator_protocol'
+require 'gorgon/originator_logger'
+require 'gorgon/configuration'
+
+class GemService
+  include Configuration
+
+  TIMEOUT = 3
+  def initialize
+    @configuration = load_configuration_from_file("gorgon.json")
+    @logger = OriginatorLogger.new @configuration[:originator_log_file]
+    @protocol = OriginatorProtocol.new @logger
+    @hosts_running = []
+    @started_running = 0
+    @finished_running = 0
+  end
+
+  def run command
+    EM.run do
+      @logger.log "Connecting..."
+      @protocol.connect @configuration[:connection],  :on_closed => proc {EM.stop}
+
+      @logger.log "Sending gem command #{command}..."
+      @protocol.send_message_to_listeners :gem_command, :gem_command => command
+
+      @protocol.receive_payloads do |payload|
+        @logger.log "Received #{payload}"
+
+        handle_reply(Yajl::Parser.new(:symbolize_keys => true).parse(payload))
+      end
+
+      EM.add_periodic_timer(TIMEOUT) { disconnect_if_none_running }
+   end
+  end
+
+  private
+
+  def disconnect_if_none_running
+    disconnect if @hosts_running.empty?
+  end
+
+  def handle_reply payload
+    hostname = payload[:hostname].colorize(Colors::HOST)
+    command = payload[:command].colorize(Colors::COMMAND) if payload[:command]
+
+    case payload[:type]
+    when "running_command"
+      puts "#{hostname} is running command #{payload[:command]}..."
+      @hosts_running << payload[:hostname]
+      @started_running += 1
+    when "command_completed"
+      puts "Command '#{command}' completed in #{hostname}"
+      command_finished payload
+    when "command_failed"
+      puts "Command '#{command}' failed in #{hostname}."
+      command_finished payload
+    else
+      puts "Unknown message received: #{payload}"
+    end
+  end
+
+  def command_finished payload
+    puts "Output:\n#{payload[:stdout]}#{payload[:stderr]}"
+    @hosts_running.delete payload[:hostname]
+    @finished_running += 1
+  end
+
+  def disconnect
+    @protocol.disconnect
+    print_summary
+  end
+
+  def print_summary
+    puts "#{@started_running} host(s) started running the command. #{@finished_running} host(s) reported they finished"
+    puts "Use 'gorgon ping' to check if all listeners are running the correct gorgon version."
+  end
+end

data/lib/gorgon/job_state.rb

@@ -30,7 +30,7 @@ class JobState
   end
 
   def file_started payload
-    raise_if_completed_or_cancelled
+    raise_if_completed
 
     if @state == :starting
       @state = :running
@@ -43,7 +43,7 @@ class JobState
   end
 
   def file_finished payload
-    raise_if_completed_or_cancelled
+    raise_if_completed
 
     @remaining_files_count -= 1
     @state = :complete if @remaining_files_count == 0
@@ -115,9 +115,9 @@ class JobState
     @failed_tests << payload
   end
 
-  def raise_if_completed_or_cancelled
+  def raise_if_completed
     raise "JobState#file_finished called when job was already complete" if is_job_complete?
-    raise "JobState#file_finished called after job was cancelled" if is_job_cancelled?
+    puts "NOTICE: JobState#file_finished called after job was cancelled" if is_job_cancelled?
   end
 
   def failed_test? payload

data/lib/gorgon/listener.rb

@@ -4,6 +4,9 @@ require 'gorgon/source_tree_syncer'
 require "gorgon/g_logger"
 require "gorgon/callback_handler"
 require "gorgon/version"
+require "gorgon/worker_manager"
+require "gorgon/crash_reporter"
+require "gorgon/gem_command_handler"
 
 require "yajl"
 require "bunny"
@@ -11,11 +14,11 @@ require "awesome_print"
 require "open4"
 require "tmpdir"
 require "socket"
-require "bundler"
 
 class Listener
   include Configuration
   include GLogger
+  include CrashReporter
 
   def initialize
     @listener_config_filename = Dir.pwd + "/gorgon_listener.json"
@@ -63,7 +66,9 @@ class Listener
     when "job_definition"
       run_job(payload)
     when "ping"
-      respong_to_ping payload[:reply_exchange_name]
+      respond_to_ping payload[:reply_exchange_name]
+    when "gem_command"
+      GemCommandHandler.new(@bunny).handle payload, configuration
     end
   end
 
@@ -79,9 +84,7 @@ class Listener
       return
     end
 
-    Bundler.with_clean_env do
-      fork_worker_manager
-    end
+    fork_worker_manager
 
     clean_up
   end
@@ -120,7 +123,7 @@ class Listener
     if @syncer.success?
       log "Command '#{@syncer.sys_command}' completed successfully."
     else
-      send_crash_message @syncer.output, @syncer.errors
+      send_crash_message @reply_exchange, @syncer.output, @syncer.errors
       log_error "Command '#{@syncer.sys_command}' failed!"
       log_error "Stdout:\n#{@syncer.output}"
       log_error "Stderr:\n#{@syncer.errors}"
@@ -131,10 +134,12 @@ class Listener
     @syncer.remove_temp_dir
   end
 
+  ERROR_FOOTER_TEXT = "\n***** See #{WorkerManager::STDERR_FILE} and #{WorkerManager::STDOUT_FILE} at '#{Socket.gethostname}' for more details *****\n"
   def fork_worker_manager
     log "Forking Worker Manager..."
     ENV["GORGON_CONFIG_PATH"] = @listener_config_filename
-    pid, stdin, stdout, stderr = Open4::popen4 "bundle exec gorgon manage_workers"
+
+    pid, stdin = Open4::popen4 "gorgon manage_workers"
     stdin.write(@job_definition.to_json)
     stdin.close
 
@@ -142,21 +147,27 @@ class Listener
     log "Worker Manager #{pid} finished"
 
     if status.exitstatus != 0
-      log_error "Worker Manager #{pid} crashed with exit status #{status.exitstatus}!"
-      error_msg = stderr.read
-      log_error "ERROR MSG: #{error_msg}"
+      exitstatus = status.exitstatus
+      log_error "Worker Manager #{pid} crashed with exit status #{exitstatus}!"
 
-      send_crash_message stdout.read, error_msg
+      msg = report_crash @reply_exchange, :out_file => WorkerManager::STDOUT_FILE,
+      :err_file => WorkerManager::STDERR_FILE, :footer_text => ERROR_FOOTER_TEXT
+
+      log_error "Process output:\n#{msg}"
     end
   end
 
-  def respong_to_ping reply_exchange_name
+  def respond_to_ping reply_exchange_name
     reply = {:type => "ping_response", :hostname => Socket.gethostname,
-      :version => Gorgon::VERSION}
+      :version => Gorgon::VERSION, :worker_slots => configuration[:worker_slots]}
+    publish_to reply_exchange_name, reply
+  end
+
+  def publish_to reply_exchange_name, message
     reply_exchange = @bunny.exchange(reply_exchange_name, :auto_delete => true)
 
-    log "Sending #{reply}"
-    reply_exchange.publish(Yajl::Encoder.encode(reply))
+    log "Sending #{message}"
+    reply_exchange.publish(Yajl::Encoder.encode(message))
   end
 
   def connection_information
@@ -166,10 +177,4 @@ class Listener
   def configuration
     @configuration ||= load_configuration_from_file("gorgon_listener.json")
   end
-
-  def send_crash_message output, error
-    reply = {:type => :crash, :hostname => Socket.gethostname,
-      :stdout => output, :stderr => error}
-    @reply_exchange.publish(Yajl::Encoder.encode(reply))
-  end
 end