divvy 1.0

data/COPYING

@@ -0,0 +1,18 @@
+Copyright (c) 2013 by Ryan Tomayko <http://tomayko.com/>
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to
+deal in the Software without restriction, including without limitation the
+rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+sell copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
+IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,120 @@
+divvy - parallel script runner
+==============================
+
+This is a (forking) parallel task runner for Ruby designed to be run in the
+foreground and to require no external infrastructure components (like redis or a
+queue server).
+
+Divvy provides a light system for defining parallelizable pieces of work and a
+process based run environment for executing them. It's good for running coarse
+grained tasks that are network or IO heavy. It's not good at crunching lots of
+inputs quickly or parallelizing fine grained / CPU intense pieces of work.
+
+GitHub uses divvy with [ModelIterator](https://github.com/technoweenie/model_iterator)
+to perform one-off and regular maintenance tasks on different types of records
+and their associated storage components.
+
+## example
+
+This is a simple and contrived example of a divvy job script. You must define a
+class that includes the `Divvy::Parallelizable` module and implement the
+`#dispatch` and `#perform` methods. There are also hooks available for tapping
+into the worker process lifecycle.
+
+``` ruby
+# This is a dance party. We're going to hand out tickets. We need to generate
+# codes for each available ticket. Thing is, the ticket codes have to be
+# generated by this external ticket code generator service (this part is
+# just pretend) and there's a lot of latency involved. We can generate multiple
+# ticket codes at the same time by making multiple connections.
+require 'divvy'
+require 'digest/sha1' # <-- your humble ticket code generator service
+
+class DanceParty
+  # The Parallelizable module provides default method implementations and marks
+  # the object as following the interface defined below.
+  include Divvy::Parallelizable
+
+  # This is the main loop responsible for generating work items for worker
+  # processes. It runs in the master process only. Each item yielded from this
+  # method is marshalled over a pipe and distributed to the next available
+  # worker process where it arrives at the #perform method (see below).
+  #
+  # In this example we're just going to generate a series of numbers to pass
+  # to the workers. The workers just write the number out with their pid and the
+  # SHA1 hex digest of the number given.
+  def dispatch
+    tickets_available = ARGV[0] ? ARGV[0].to_i : 10
+    puts "Generating #{tickets_available} ticket codes for the show..."
+    (0...tickets_available).each do |ticket_number|
+      yield ticket_number
+    end
+  end
+
+  # The individual work item processing method. Each item produced by the
+  # dispatch method is sent to this method in the worker processes. The
+  # arguments to this method must match the arity of the work item yielded
+  # from the #dispatch method.
+  #
+  # In this example we're given a Fixnum ticket number and asked to produce a
+  # code. Pretend this is a network intense operation where you're mostly
+  # sleeping waiting for a reply.
+  def perform(ticket_number)
+    ticket_sha1 = Digest::SHA1.hexdigest(ticket_number.to_s)
+    printf "%5d %6d %s\n" % [$$, ticket_number, ticket_sha1]
+    sleep 0.150 # fake some latency
+  end
+
+  # Hook called after a worker process is forked off from the master process.
+  # This runs in the worker process only. Typically used to re-establish
+  # connections to external services or open files (logs and such).
+  def after_fork(worker)
+    # warn "In after_fork for worker #{worker.number}"
+  end
+
+  # Hook called before a worker process is forked off from the master process.
+  # This runs in the master process only. This can be used to monitor the rate
+  # at which workers are being created or to set a starting process state for
+  # the newly forked process.
+  def before_fork(worker)
+    # warn "In before_fork for worker #{worker.number}"
+  end
+end
+```
+
+### divvy command
+
+You can run the example script above with the `divvy` command, which includes
+options for controlling concurrency and other cool stuff. Here we use five
+worker processes to generate 10 dance party ticket codes:
+
+```
+$ divvy -n 5 danceparty.rb
+51589        0 b6589fc6ab0dc82cf12099d1c2d40ab994e8410c
+51590        1 356a192b7913b04c54574d18c28d46e6395428ab
+51589        4 1b6453892473a467d07372d45eb05abc2031647a
+51590        5 ac3478d69a3c81fa62e60f5c3696165a4e5e6ac4
+51591        2 da4b9237bacccdf19c0760cab7aec4a8359010b0
+51589        6 c1dfd96eea8cc2b62785275bca38ac261256e278
+51592        3 77de68daecd823babbb58edb1c8e14d7106e83bb
+51590        8 fe5dbbcea5ce7e2988b8c69bcfdfde8904aabc1f
+51591        9 0ade7c2cf97f75d009975f4d720d1fa6c19f4897
+51593        7 902ba3cda1883801594b6e1b452790cc53948fda
+```
+
+The columns of output are the worker pid, the ticket number input, and the
+generated ticket code result. You can see items are distributed between
+available workers evenly-ish and may not be processed in order.
+
+### manual runner
+
+You can also turn the current ruby process into a divvy master by creating a
+`Divvy::Master` object, passing an instance of `Parallelizable` and the amount
+of desired concurrency:
+
+``` ruby
+require 'danceparty'
+task = DanceParty.new
+master = Divvy::Master.new(task, 10)
+master.run
+```

data/Rakefile

@@ -0,0 +1,6 @@
+task :default => :test
+
+desc "Run tests"
+task :test do
+  sh "script/cibuild"
+end

data/bin/divvy

@@ -0,0 +1,31 @@
+#!/usr/bin/env ruby
+#/ Usage: divvy [-n <workers>] <script>.rb
+#/ Run a divvy script with the given number of workers.
+require 'optparse'
+$stderr.sync = true
+
+# Number of work processes to spawn.
+worker_count = 1
+
+# Whether to write verbose output to stderr
+verbose = false
+
+# The divvy script used to setup
+script_file = nil
+
+# parse arguments
+file = __FILE__
+ARGV.options do |opts|
+  opts.on("-n", "--workers=val", Integer)  { |val| worker_count = val }
+  opts.on("-v", "--verbose")               { |val| verbose = val }
+  opts.on_tail("-h", "--help")             { exec "grep ^#/<'#{file}'|cut -c4-" }
+  opts.parse!
+end
+script_file = ARGV.shift
+
+require 'divvy'
+task = Divvy.load(script_file)
+warn "divvy: booting #{worker_count} workers for #{task.class}" if verbose
+
+master = Divvy::Master.new(task, worker_count, verbose)
+master.run

data/divvy.gemspec

@@ -0,0 +1,20 @@
+# -*- encoding: utf-8 -*-
+
+Gem::Specification.new do |s|
+  s.name        = "divvy"
+  s.version     = "1.0"
+  s.platform    = Gem::Platform::RUBY
+  s.authors     = %w[@rtomayko]
+  s.email       = ["rtomayko@gmail.com"]
+  s.homepage    = "https://github.com/rtomayko/divvy"
+  s.summary     = "little ruby parallel script runner"
+  s.description = "..."
+
+  s.add_development_dependency "rake"
+  s.add_development_dependency "minitest"
+
+  s.files         = `git ls-files`.split("\n")
+  s.test_files    = `git ls-files -- test`.split("\n").select { |f| f =~ /_test.rb$/ }
+  s.executables   = `git ls-files -- bin`.split("\n").map { |f| File.basename(f) }
+  s.require_paths = %w[lib]
+end

data/example.rb

@@ -0,0 +1,58 @@
+# This is a dance party. We're going to hand out tickets. We need to generate
+# codes for each available ticket. Thing is, the ticket codes have to be
+# generated by this external ticket code generator service (this part is
+# just pretend) and there's a lot of latency involved. We can generate multiple
+# ticket codes at the same time by making multiple connections.
+require 'divvy'
+require 'digest/sha1' # <-- your humble ticket code generator service
+
+class DanceParty
+  # The Parallelizable module provides default method implementations and marks
+  # the object as following the interface defined below.
+  include Divvy::Parallelizable
+
+  # This is the main loop responsible for generating work items for worker
+  # processes. It runs in the master process only. Each item yielded from this
+  # method is marshalled over a pipe and distributed to the next available
+  # worker process where it arrives at the #perform method (see below).
+  #
+  # In this example we're just going to generate a series of numbers to pass
+  # to the workers. The workers just write the number out with their pid and the
+  # SHA1 hex digest of the number given.
+  def dispatch
+    tickets_available = ARGV[0] ? ARGV[0].to_i : 10
+    puts "Generating #{tickets_available} ticket codes for the show..."
+    (0...tickets_available).each do |ticket_number|
+      yield ticket_number
+    end
+  end
+
+  # The individual work item processing method. Each item produced by the
+  # dispatch method is sent to this method in the worker processes. The
+  # arguments to this method must match the arity of the work item yielded
+  # from the #dispatch method.
+  #
+  # In this example we're given a Fixnum ticket number and asked to produce a
+  # code. Pretend this is a network intense operation where you're mostly
+  # sleeping waiting for a reply.
+  def perform(ticket_number)
+    ticket_sha1 = Digest::SHA1.hexdigest(ticket_number.to_s)
+    printf "%5d %6d %s\n" % [$$, ticket_number, ticket_sha1]
+    sleep 0.150 # fake some latency
+  end
+
+  # Hook called after a worker process is forked off from the master process.
+  # This runs in the worker process only. Typically used to re-establish
+  # connections to external services or open files (logs and such).
+  def after_fork(worker)
+    # warn "In after_fork for worker #{worker.number}"
+  end
+
+  # Hook called before a worker process is forked off from the master process.
+  # This runs in the master process only. This can be used to monitor the rate
+  # at which workers are being created or to set a starting process state for
+  # the newly forked process.
+  def before_fork(worker)
+    # warn "In before_fork for worker #{worker.number}"
+  end
+end

data/lib/divvy.rb

@@ -0,0 +1,22 @@
+require 'divvy/parallelizable'
+require 'divvy/master'
+require 'divvy/worker'
+
+module Divvy
+  # Load a script that defines a Divvy::Parallelizable object. A class that
+  # includes the Parallelizable module must be defined in order for this to work.
+  #
+  # file - Script file to load.
+  #
+  # Returns an object that implements the Parallelizable interface.
+  # Raises a RuntimeError when no parallelizable object was defined.
+  def self.load(file)
+    Kernel::load(file)
+
+    if subclass = Parallelizable.parallelizable.last
+      @receiver = subclass.new
+    else
+      fail "#{file} does not define a Divvy::Parallelizable object"
+    end
+  end
+end

data/lib/divvy/master.rb

@@ -0,0 +1,323 @@
+require 'socket'
+
+module Divvy
+  # The master process used to generate and distribute task items to the
+  # worker processes.
+  class Master
+    # The number of worker processes to boot.
+    attr_reader :worker_count
+
+    # The array of Worker objects this master is managing.
+    attr_reader :workers
+
+    # The string filename of the unix domain socket used to distribute work.
+    attr_reader :socket
+
+    # Enable verbose logging to stderr.
+    attr_accessor :verbose
+
+    # Number of tasks that have been distributed to worker processes.
+    attr_reader :tasks_distributed
+
+    # Number of worker processes that have exited with a failure status since
+    # the master started processing work.
+    attr_reader :failures
+
+    # Number of worker processes that have been spawned since the master
+    # started processing work.
+    attr_reader :spawn_count
+
+    # Raised from a signal handler when a forceful shutdown is requested.
+    class Shutdown < Exception
+    end
+
+    # Raised from the run loop when worker processes never fully booted and
+    # started making connections to the master.
+    class BootFailure < StandardError
+    end
+
+    # Create the master process object.
+    #
+    # task         - Object that implements the Parallelizable interface.
+    # worker_count - Number of worker processes.
+    # verbose      - Enable verbose error logging.
+    # socket       - The unix domain socket filename.
+    #
+    # The workers array is initialized with Worker objects for the number of
+    # worker processes requested. The processes are not actually started at this
+    # time though.
+    def initialize(task, worker_count, verbose = false, socket = nil)
+      @task = task
+      @worker_count = worker_count
+      @verbose = verbose
+      @socket = socket || "/tmp/divvy-#{$$}-#{object_id}.sock"
+
+      # stats
+      @tasks_distributed = 0
+      @failures = 0
+      @spawn_count = 0
+
+      # shutdown state
+      @shutdown = false
+      @graceful = true
+      @reap = false
+
+      # worker objects
+      @workers = []
+      (1..@worker_count).each do |worker_num|
+        worker = Divvy::Worker.new(@task, worker_num, @socket, @verbose)
+        workers << worker
+      end
+    end
+
+    # Public: Start the main run loop. This installs signal handlers into the
+    # current process, binds to the unix domain socket, boots workers, and begins
+    # dispatching work.
+    #
+    # The run method does not return until all task items generated have been
+    # processed unless a shutdown signal is received or the #shutdown method is
+    # called within the loop.
+    #
+    # Returns nothing.
+    # Raises BootFailure when the workers fail to start.
+    # Raises Shutdown when a forceful shutdown is triggered (SIGTERM).
+    def run
+      fail "Already running!!!" if @server
+      fail "Attempt to run master in a worker process" if worker_process?
+      install_signal_traps
+      start_server
+
+      @task.dispatch do |*task_item|
+        # boot workers that haven't started yet or have been reaped
+        boot_workers
+
+        # check for shutdown or worker reap flag until a connection is pending
+        # in the domain socket queue. bail out if workers exited before even
+        # requesting a task item.
+        while IO.select([@server], nil, nil, 0.010).nil?
+          break if @shutdown
+          if @reap
+            reap_workers
+            if !workers_running? && @tasks_distributed == 0
+              raise BootFailure, "Worker processes failed to boot."
+            else
+              boot_workers
+            end
+          end
+        end
+        break if @shutdown
+
+        # at this point there should be at least one connection pending.
+        begin
+          data = Marshal.dump(task_item)
+          sock = @server.accept
+          sock.write(data)
+        ensure
+          sock.close if sock
+        end
+        @tasks_distributed += 1
+
+        break if @shutdown
+        reap_workers if @reap
+      end
+
+      nil
+    rescue Shutdown
+      @graceful = false
+      @shutdown = true
+    ensure
+      shutdown! if master_process?
+    end
+
+    # Public: Check if the current process is the master process.
+    #
+    # Returns true in the master process, false in the worker process.
+    def master_process?
+      @workers
+    end
+
+    # Public: Check if the current process is a worker process.
+    # This relies on the @workers array being set to a nil value.
+    #
+    # Returns true in the worker process, false in master processes.
+    def worker_process?
+      !master_process?
+    end
+
+    # Public: Are any worker processes currently running or have yet to be
+    # reaped by the master process?
+    def workers_running?
+      @workers.any? { |worker| worker.running? }
+    end
+
+    # Public: Initiate shutdown of the run loop. The loop will not be stopped when
+    # this method returns. The original run loop will return after the current
+    # iteration of task item.
+    def shutdown
+      @shutdown ||= Time.now
+    end
+
+    # Internal: Really shutdown the unix socket and reap all worker processes.
+    # This doesn't signal the workers. Instead, the socket shutdown is relied
+    # upon to trigger the workers to exit normally.
+    #
+    # TODO Send SIGKILL when workers stay running for configurable period.
+    def shutdown!
+      fail "Master#shutdown! called in worker process" if worker_process?
+      stop_server
+      while workers_running?
+        kill_workers("KILL") if !@graceful
+        reaped = reap_workers
+        sleep 0.010 if reaped.empty?
+      end
+      reset_signal_traps
+      raise Shutdown if !@graceful
+    end
+
+    # Internal: create and bind to the unix domain socket. Note that the
+    # requested backlog matches the number of workers. Otherwise workers will
+    # get ECONNREFUSED when attempting to connect to the master and exit.
+    def start_server
+      fail "Master#start_server called in worker process" if worker_process?
+      File.unlink(@socket) if File.exist?(@socket)
+      @server = UNIXServer.new(@socket)
+      @server.listen(worker_count)
+    end
+
+    # Internal: Close and remove the unix domain socket.
+    def stop_server
+      fail "Master#stop_server called in worker process" if worker_process?
+      File.unlink(@socket) if File.exist?(@socket)
+      @server.close if @server
+      @server = nil
+    end
+
+    # Internal: Boot any workers that are not currently running. This is a no-op
+    # if all workers are though to be running. No attempt is made to verify
+    # worker processes are running here. Only workers that have not yet been
+    # booted and those previously marked as reaped are started.
+    def boot_workers
+      workers.each do |worker|
+        next if worker.running?
+        boot_worker(worker)
+      end
+    end
+
+    # Internal: Boot and individual worker process. Don't call this if the
+    # worker is thought to be running.
+    #
+    # worker - The Worker object to boot.
+    #
+    # Returns the Worker object provided.
+    def boot_worker(worker)
+      fail "worker #{worker.number} already running" if worker.running?
+      fail "attempt to boot worker without server" if !@server
+
+      @task.before_fork(worker)
+
+      worker.spawn do
+        reset_signal_traps
+        @workers = nil
+
+        @server.close
+        @server = nil
+
+        $stdin.close
+      end
+      @spawn_count += 1
+
+      worker
+    end
+
+    # Internal: Send a signal to all running workers.
+    #
+    # signal - The string signal name.
+    #
+    # Returns nothing.
+    def kill_workers(signal = 'TERM')
+      workers.each do |worker|
+        next if !worker.running?
+        worker.kill(signal)
+      end
+    end
+
+    # Internal: Attempt to reap all worker processes via Process::waitpid. This
+    # method does not block waiting for processes to exit. Running processes are
+    # ignored.
+    #
+    # Returns an array of Worker objects whose process's were reaped. The
+    # Worker#status attribute can be used to access the Process::Status result.
+    def reap_workers
+      @reap = false
+      workers.select do |worker|
+        if status = worker.reap
+          @failures += 1 if !status.success?
+          worker
+        end
+      end
+    end
+
+    # Internal: Install traps for shutdown signals. Most signals deal with
+    # shutting down the master loop and socket.
+    #
+    # INFO      - Dump stack for all processes to stderr.
+    # TERM      - Initiate immediate forceful shutdown of all worker processes
+    #             along with the master process, aborting any existing jobs in
+    #             progress.
+    # INT, QUIT - Initiate graceful shutdown, allowing existing worker processes
+    #             to finish their current task and exit on their own. If this
+    #             signal is received again after 10s, instead initiate an
+    #             immediate forceful shutdown as with TERM. This is mostly so you
+    #             can interrupt sanely with Ctrl+C with the master foregrounded.
+    # CHLD      - Set the worker reap flag. An attempt is made to reap workers
+    #             immediately after the current dispatch iteration.
+    #
+    # Returns nothing.
+    def install_signal_traps
+      @traps =
+        %w[INT QUIT].map do |signal|
+          Signal.trap signal do
+            if @shutdown
+              raise Shutdown, "SIG#{signal}" if (Time.now - @shutdown) > 10 # seconds
+              next
+            else
+              shutdown
+              log "#{signal} received. initiating graceful shutdown..."
+            end
+          end
+        end
+      @traps << Signal.trap("CHLD") { @reap = true }
+      @traps << Signal.trap("TERM") { raise Shutdown, "SIGTERM" }
+
+      Signal.trap "INFO" do
+        message = "==> info: process #$$ dumping stack\n"
+        message << caller.join("\n").gsub(/^/, "    ").gsub("#{Dir.pwd}/", "")
+        $stderr.puts(message)
+      end
+
+      @traps
+    end
+
+    # Internal: Uninstall signal traps set up by the install_signal_traps
+    # method. This is called immediately after forking worker processes to reset
+    # traps to their default implementations and also when the master process
+    # shuts down.
+    def reset_signal_traps
+      return if @traps.nil? || @traps.empty?
+      %w[INT QUIT CHLD TERM].each do |signal|
+        handler = @traps.shift || "DEFAULT"
+        if handler.is_a?(String)
+          Signal.trap(signal, handler)
+        else
+          Signal.trap(signal, &handler)
+        end
+      end
+    end
+
+    # Internal: Write a verbose log message to stderr.
+    def log(message)
+      return if !verbose
+      $stderr.printf("master: %s\n", message)
+    end
+  end
+end

data/lib/divvy/parallelizable.rb

@@ -0,0 +1,61 @@
+module Divvy
+  # Module defining the main task interface. Parallelizable classes must respond
+  # to #dispatch and #perform and may override hook methods to tap into the
+  # worker process lifecycle.
+  module Parallelizable
+    # The main loop responsible for generating task items to process in workers.
+    # Runs in the master process only. Each item this method yields is distributed
+    # to one of a pool of worker processes where #perform (see below) is invoked
+    # to process the task item.
+    #
+    # The arguments yielded to the block are passed with same arity to
+    # the #perform method. Only marshallable types may be included.
+    #
+    # The dispatch method takes no arguments. It's expected that the receiving
+    # object is setup with all necessary state to generate task items or can
+    # retrieve the information it needs from elsewhere.
+    #
+    # When the dispatch method returns the master process exits.
+    # If an exception is raised, the program exits non-zero.
+    def dispatch
+      raise NotImplementedError, "#{self.class} must implement #dispatch method"
+    end
+
+    # Process an individual task item. Each item produced by #dispatch is sent here
+    # in one of a pool of the worker processes. The arguments to this method must
+    # match the arity of the task item yielded from #dispatch.
+    def perform(*args)
+      raise NotImplementedError, "#{self.class} must implement perform method"
+    end
+
+    # Hook called after a worker process is forked off from the master process.
+    # This runs in the worker process only. Typically used to re-establish
+    # connections to external services or open files (logs and such).
+    #
+    # worker - A Divvy::Worker object describing the process that was just
+    #          created. Always the current process ($$).
+    #
+    # Return value is ignored.
+    def after_fork(worker)
+    end
+
+    # Hook called before a worker process is forked off from the master process.
+    # This runs in the master process only.
+    #
+    # worker - Divvy::Worker object descibing the process that's about to fork.
+    #          Worker#pid will be nil but Worker#number (1..worker_count) is
+    #          always available.
+    #
+    # Return value is ignored.
+    def before_fork(worker)
+    end
+
+    # Track classes and modules that include this module.
+    @parallelizable = []
+    def self.included(mod)
+      @parallelizable << mod if self == Divvy::Parallelizable
+      super
+    end
+    def self.parallelizable; @parallelizable; end
+  end
+end

data/lib/divvy/worker.rb

@@ -0,0 +1,153 @@
+module Divvy
+  # Models an individual divvy worker process. These objects are used in both
+  # the master and the forked off workers to perform common tasks and for basic
+  # tracking.
+  class Worker
+    # The worker number. These are sequential starting from 1 and ending in the
+    # configured worker concurrency count.
+    attr_reader :number
+
+    # The Unix domain socket file used to communicate with the master process.
+    attr_reader :socket
+
+    # Whether verbose log info should be written to stderr.
+    attr_accessor :verbose
+
+    # Process::Status object result of reaping the worker.
+    attr_reader :status
+
+    # The worker processes's pid. This is $$ when inside the worker process.
+    attr_accessor :pid
+
+    # Create a Worker object. The Master object typically handles this.
+    def initialize(task, number, socket, verbose = false)
+      @task = task
+      @number = number
+      @socket = socket
+      @verbose = verbose
+      @pid = nil
+      @status = nil
+    end
+
+    # Public: Check whether the worker process is thought to be running. This
+    # does not attempt to verify the real state of the process with the system.
+    def running?
+      @pid && @status.nil?
+    end
+
+    # Public: Send a signal to a running worker process.
+    #
+    # signal - String signal name.
+    #
+    # Returns true when the process was signaled, false if the process is no
+    # longer running.
+    # Raises when the worker process is not thought to be running.
+    def kill(signal)
+      fail "worker not running" if @pid.nil?
+      log "sending signal #{signal}"
+      Process.kill(signal, @pid)
+      true
+    rescue Errno::ESRCH
+      false
+    end
+
+    # Public: Check whether the current process is this worker process.
+    #
+    # Returns true when we're in this worker, false in the master.
+    def worker_process?
+      @pid == $$
+    end
+
+    # Public: Fork off a new process for this worker and yield to the block
+    # immediately in the new child process.
+    #
+    # Returns the pid of the new process in the master process. Never returns in
+    # the child process.
+    # Raises when the worker process is already thought to be running or has not
+    # yet been reaped.
+    def spawn
+      fail "worker already running" if running?
+      @status = nil
+
+      if (@pid = fork).nil?
+        @pid = $$
+        yield
+        install_signal_traps
+        main
+        exit 0
+      end
+
+      @pid
+    end
+
+    # Public: Attempt to reap this worker's process using waitpid. This is a
+    # no-op if the process is not thought to be running or is marked as already
+    # being reaped. This should only be called in the master process.
+    #
+    # Returns the Process::Status object if the process was reaped, nil if the
+    # process was not reaped because it's still running or is already reaped.
+    def reap
+      if @status.nil? && @pid && Process::waitpid(@pid, Process::WNOHANG)
+        @status = $?
+        log "exited, reaped pid #{@pid} (status: #{@status.exitstatus})"
+        @status
+      end
+    end
+
+    # Internal: The main worker loop. This is called after a new worker process
+    # has been setup with signal traps and whatnot and connects to the master in
+    # a loop to retrieve task items. The worker process exits if this method
+    # returns or raises an exception.
+    def main
+      fail "Worker#main called in master process" if !worker_process?
+
+      log "booted with pid #{@pid}"
+      @task.after_fork(self)
+
+      while arguments = dequeue
+        @task.perform(*arguments)
+        break if @shutdown
+      end
+
+      # worker should exit on return
+    rescue Exception => boom
+      warn "error: worker [#{number}]: #{boom.class} #{boom.to_s}"
+      exit 1
+    end
+
+    # Internal: Retrieve an individual task item from the master process. Opens
+    # a new socket, reads and unmarshals a single task item.
+    #
+    # Returns an Array containing the arguments yielded by the dispatcher.
+    def dequeue
+      client = UNIXSocket.new(@socket)
+      r, w, e = IO.select([client], nil, [client], nil)
+      return if !e.empty?
+
+      if data = client.read(16384)
+        Marshal.load(data)
+      end
+    rescue Errno::ENOENT => boom
+      # socket file went away, bail out
+    ensure
+      client.close if client
+    end
+
+    def install_signal_traps
+      fail "attempt to install worker signal handling in master" if !worker_process?
+
+      %w[INT TERM QUIT].each do |signal|
+        trap signal do
+          next if @shutdown
+          @shutdown = true
+          log "#{signal} received. initiating graceful shutdown..."
+        end
+      end
+    end
+
+    def log(message)
+      return if !verbose
+      $stderr.printf("worker [%d]: %s\n", number, message)
+    end
+  end
+end

data/script/cibuild

@@ -0,0 +1,4 @@
+#!/bin/sh
+set -e
+cd "$(dirname "$0")/.."
+testrb test/*_test.rb

data/test/example_test.rb

@@ -0,0 +1,12 @@
+require File.expand_path("../setup", __FILE__)
+require "divvy"
+
+# Tests that make sure the example.rb file runs and generates tickets.
+class ExampleTest < MiniTest::Unit::TestCase
+  def test_running_the_example_program
+    example_script = File.expand_path("../../example.rb", __FILE__)
+    output = `divvy -n 2 '#{example_script}'`
+    assert $?.success?, "example program should exit successfully"
+    assert_equal 11, output.split("\n").size
+  end
+end

data/test/master_test.rb

@@ -0,0 +1,146 @@
+require File.expand_path("../setup", __FILE__)
+require "divvy"
+
+# Tests that make sure the example.rb file runs and generates tickets.
+class MasterTest < MiniTest::Unit::TestCase
+  class SimpleTask
+    include Divvy::Parallelizable
+
+    def dispatch
+      10.times(&block)
+    end
+
+    def perform(num)
+    end
+  end
+
+  def setup
+    @task = SimpleTask.new
+    @master = Divvy::Master.new(@task, 2)
+  end
+
+  def teardown
+    @master.shutdown!
+  end
+
+  def test_worker_object_instantiation
+    assert_equal 2, @master.workers.size
+
+    assert_equal 1, @master.workers[0].number
+    assert_equal 2, @master.workers[1].number
+
+    @master.workers.each { |worker| assert_nil worker.pid }
+    @master.workers.each { |worker| assert_nil worker.status }
+  end
+
+  def test_workers_running_check
+    assert !@master.workers_running?
+  end
+
+  def test_master_process_check
+    assert @master.master_process?
+    assert !@master.worker_process?
+  end
+
+  def test_start_server
+    @master.start_server
+    assert File.exist?(@master.socket)
+  end
+
+  def test_boot_workers
+    @master.start_server
+    @master.boot_workers
+    assert @master.workers_running?
+    assert @master.workers.all? { |w| w.running? }
+  end
+
+  def test_reaping_and_killing_workers
+    @master.start_server
+    @master.boot_workers
+    reaped = @master.reap_workers
+    assert_equal 0, reaped.size
+
+    @master.kill_workers("KILL")
+    sleep 0.100
+    reaped = @master.reap_workers
+    assert_equal 2, reaped.size
+  end
+
+  def test_installing_and_uninstalling_signal_traps
+    traps = @master.install_signal_traps
+    assert traps.is_a?(Array)
+    assert traps.size >= 4
+
+    @master.reset_signal_traps
+    assert_equal 0, traps.size
+  end
+
+  class SuccessfulTask
+    include Divvy::Parallelizable
+
+    def dispatch
+      yield 'just one thing'
+    end
+
+    def perform(arg)
+      if arg != 'just one thing'
+        fail "expected arg to be 'just one thing'"
+      end
+    end
+  end
+
+  def test_successful_run
+    task = SuccessfulTask.new
+    master = Divvy::Master.new(task, 1)
+    master.run
+  end
+
+  class StatsTask
+    include Divvy::Parallelizable
+
+    def dispatch(&block)
+      10.times(&block)
+    end
+
+    def perform(num)
+      if num % 2 == 0
+        fail "simulated failure"
+      else
+        true
+      end
+    end
+
+    def after_fork(worker)
+      $stderr.reopen("/dev/null")
+    end
+  end
+
+  def test_stats
+    task = StatsTask.new
+    master = Divvy::Master.new(task, 5)
+    master.run
+    assert_equal 5, master.failures
+    assert_equal 10, master.tasks_distributed
+  end
+
+  class FlappingWorkerTask
+    include Divvy::Parallelizable
+
+    def dispatch
+      yield 'never makes it'
+    end
+
+    def after_fork(worker)
+      exit! 1
+    end
+  end
+
+  def test_flapping_worker_detection
+    task = FlappingWorkerTask.new
+    master = Divvy::Master.new(task, 1)
+    assert_raises(Divvy::Master::BootFailure) do
+      master.run
+    end
+    assert_equal 1, master.failures
+  end
+end

data/test/setup.rb

@@ -0,0 +1,19 @@
+# Basic test environment.
+#
+# This should set up the load path for testing only. Don't require any support libs
+# or gitrpc stuff in here.
+
+# bring in minitest
+require 'minitest/autorun'
+
+# add bin dir to path for testing command
+ENV['PATH'] = [
+  File.expand_path("../../bin", __FILE__),
+  ENV['PATH']
+].join(":")
+
+# put lib dir directly to load path
+$LOAD_PATH.unshift File.expand_path("../../lib", __FILE__)
+
+# child processes inherit our load path
+ENV['RUBYLIB'] = $LOAD_PATH.join(":")

metadata

@@ -0,0 +1,94 @@
+--- !ruby/object:Gem::Specification
+name: divvy
+version: !ruby/object:Gem::Version
+  version: '1.0'
+  prerelease: 
+platform: ruby
+authors:
+- ! '@rtomayko'
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2013-04-26 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: minitest
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+description: ! '...'
+email:
+- rtomayko@gmail.com
+executables:
+- divvy
+extensions: []
+extra_rdoc_files: []
+files:
+- COPYING
+- README.md
+- Rakefile
+- bin/divvy
+- divvy.gemspec
+- example.rb
+- lib/divvy.rb
+- lib/divvy/master.rb
+- lib/divvy/parallelizable.rb
+- lib/divvy/worker.rb
+- script/cibuild
+- test/example_test.rb
+- test/master_test.rb
+- test/setup.rb
+homepage: https://github.com/rtomayko/divvy
+licenses: []
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.23
+signing_key: 
+specification_version: 3
+summary: little ruby parallel script runner
+test_files:
+- test/example_test.rb
+- test/master_test.rb