resque-pool 0.0.2 → 0.0.4

data/README.md

@@ -0,0 +1,132 @@
+Resque Pool
+===========
+
+Resque pool is a simple library for managing a pool of resque workers.  Given a
+configuration hash or a config file (resque-pool.yml or
+config/resque-pool.yml), it will manage your workers for you, starting up the
+appropriate number of workers for each.
+
+Benefits
+---------
+
+* Less memory consumption - If you are using Ruby Enterprise Edition, or any
+  ruby with copy-on-write safe garbage collection, this will save you a lot of
+  memory when you managing many workers.
+* Simpler (less) config - If you are using monit or god or an init script to
+  start up your workers, you can simply start up one pool, and it will manage
+  your workers for you.
+* Faster startup - if you are starting many workers at once, you would normally
+  have them competing for CPU as they load their environments.  Resque-pool can
+  load the environment once, and almost immediately fork all of your workers.
+
+How to use
+-----------
+
+To configure resque-pool, you can either set `Resque::Pool.config` to a hash in
+your `resque:setup` or you can set the same config in either `resque-pool.yml`
+or `config/resque-pool.yml`.  To use resque-pool, require its rake tasks in
+your rake file, and call the resque:pool task.
+
+For example, to use resque-pool with rails, in `config/resque-pool.yml`:
+
+    foo: 1
+    bar: 2
+    "foo,bar,baz": 4
+
+and in `lib/tasks/resque.rake`:
+
+    require 'resque/pool/tasks'
+    namespace :resque do
+
+      # preload the rails environment in the pool master
+      task :setup => :environment
+        # it's better to use a config file, but you can also config here:
+        # Resque::Pool.config = {"foo" => 1, "bar" => 1}
+
+        # close any sockets or files in pool master
+        ActiveRecord::Base.connection.disconnect!
+
+        # and re-open them in the resque worker parent
+        Resque::Pool.after_prefork do |job|
+          ActiveRecord::Base.establish_connection
+        end
+
+        # you could also re-open them in the resque worker child, using
+        # Resque.after_fork, but that probably isn't necessary, and
+        # Resque::Pool.after_prefork should be faster, since it won't run
+        # for every single job.
+      end
+    end
+
+Then you can start the queues via:
+
+    rake resque:pool RAILS_ENV=production
+
+This will start up seven worker processes, one each looking exclusively at each
+of the foo, bar, and baz queues, and four workers looking at all queues in
+priority.  This is similar to if you ran the following:
+
+    rake resque:worker RAILS_ENV=production QUEUES=foo
+    rake resque:worker RAILS_ENV=production QUEUES=bar
+    rake resque:worker RAILS_ENV=production QUEUES=bar
+    rake resque:worker RAILS_ENV=production QUEUES=foo,bar,baz
+    rake resque:worker RAILS_ENV=production QUEUES=foo,bar,baz
+    rake resque:worker RAILS_ENV=production QUEUES=foo,bar,baz
+    rake resque:worker RAILS_ENV=production QUEUES=foo,bar,baz
+
+Resque already forks for its own child processes, giving two levels.  The pool
+master will stay around monitoring the resque worker parents, giving three
+levels:
+
+* a single pool master
+* many worker parents
+* a worker child per worker (when the actual job is being processed)
+
+For example, `ps -ef f | grep [r]esque` might return something like the
+following:
+
+    rails    13858     1  0 13:44 ?        S      0:02 resque-pool-master: managing [13867, 13875, 13871, 13872, 13868, 13870, 13876]
+    rails    13867 13858  0 13:44 ?        S      0:00  \_ resque-1.9.9: Waiting for foo
+    rails    13868 13858  0 13:44 ?        S      0:00  \_ resque-1.9.9: Waiting for bar
+    rails    13870 13858  0 13:44 ?        S      0:00  \_ resque-1.9.9: Waiting for bar
+    rails    13871 13858  0 13:44 ?        S      0:00  \_ resque-1.9.9: Waiting for foo,bar,baz
+    rails    13872 13858  0 13:44 ?        S      0:00  \_ resque-1.9.9: Forked 7481 at 1280343254
+    rails     7481 13872  0 14:54 ?        S      0:00      \_ resque-1.9.9: Processing foo since 1280343254
+    rails    13875 13858  0 13:44 ?        S      0:00  \_ resque-1.9.9: Waiting for foo,bar,baz
+    rails    13876 13858  0 13:44 ?        S      0:00  \_ resque-1.9.9: Forked 7485 at 1280343255
+    rails     7485 13876  0 14:54 ?        S      0:00      \_ resque-1.9.9: Processing bar since 1280343254
+
+SIGNALS
+-------
+
+The pool master responds to the following signals:
+
+* `HUP`   - reload the config file, e.g. to change the number of workers per queue list
+* `QUIT`  - send `QUIT` to each worker parent and shutdown the master after all workers are done.
+* `INT`   - send `QUIT` to each worker parent and immediately shutdown master
+* `TERM`  - send `TERM` to each worker parent and immediately shutdown master
+* `WINCH` - send `QUIT` to each worker, but keep master running (send `HUP` to reload config and restart workers)
+* `USR1`/`USR2`/`CONT` - send the signal on to all worker parents (see Resque docs).
+
+`HUP` will no-op if you use a hash for configuration instead of a config file.
+So you should probably use a config file.  After a `HUP`, workers that are no
+longer needed will be gracefully shutdown via `QUIT`.
+
+Other Features
+--------------
+
+Workers will watch the pool master, and gracefully shutdown if the master
+process dies (for whatever reason) before them.
+
+TODO
+-----
+
+* do appropriate logging (e.g. all to one logfile, each queue to its own
+  logfile, or each worker to its own logfile).  Logfile location must be
+  configurable.
+* (optionally) daemonize, setting a PID file somewhere
+* recover gracefully from a malformed config file (on startup and HUP)
+* figure out a good way to test this (preferably via cucumber or rspec)
+* clean up the code (I stole most of it from unicorn, and it's still a bit
+  bastardized)
+* web interface for adding and removing workers (etc)

data/lib/resque/pool.rb

@@ -1,23 +1,30 @@
 # -*- encoding: utf-8 -*-
 require 'resque'
+require 'resque/pool/logging'
+require 'resque/pool/pooled_worker'
 require 'fcntl'
+require 'yaml'
 
 module Resque
   class Pool
-    attr_reader :pool_config
+    include Logging
+    attr_reader :config
     attr_reader :workers
 
-    SELF_PIPE = []
-    SIG_QUEUE = []
-    QUEUE_SIGS = [ :QUIT, :INT, :TERM, :USR1, :USR2, :HUP, ]
+    # CONSTANTS {{{
+    SIG_QUEUE_MAX_SIZE = 5
+    DEFAULT_WORKER_INTERVAL = 5
+    QUEUE_SIGS = [ :QUIT, :INT, :TERM, :USR1, :USR2, :CONT, :HUP, :WINCH, ]
     CHUNK_SIZE=(16 * 1024)
+    # }}}
 
     def initialize(config)
-      @pool_config = config.dup
-      @workers = pool_config.keys.inject({}) {|h,k| h[k] = {}; h}
+      init_config(config)
+      @workers = {}
       procline "(initialized)"
     end
 
+    # Config: after_prefork {{{
 
     # The `after_prefork` hook will be run in workers if you are using the
     # preforking master worker to save memory. Use this hook to reload
@@ -39,48 +46,131 @@ module Resque
       self.class.after_prefork && self.class.after_prefork.call
     end
 
-    # Given a string, sets the procline ($0)
-    # Procline is always in the format of:
-    #   resque-pool-master: STRING
-    def procline(string)
-      $0 = "resque-pool-master: #{string}"
+    # }}}
+    # Config: class methods to start up the pool using the default config {{{
+
+    @config_files = ["resque-pool.yml", "config/resque-pool.yml"]
+    class << self; attr_accessor :config, :config_files; end
+    def self.load_default_config
+      if @config
+        @config
+      else
+        @config_files.detect { |f| File.exist?(f) }
+      end
+    end
+
+    def self.run
+      if GC.respond_to?(:copy_on_write_friendly=)
+        GC.copy_on_write_friendly = true
+      end
+      Resque::Pool.new(load_default_config).start.join
+    end
+
+    # }}}
+    # Config: load config and config file {{{
+
+    def load_config_from_file
+      return unless @pool_config_file
+      log "**** loading config from #{@pool_config_file}"
+      @config = YAML.load_file(@pool_config_file)
     end
 
-    # TODO: make this use an actual logger
-    def log(message)
-      puts message
+    def init_config(config)
+      unless config
+        raise ArgumentError,
+          "No configuration found. Please setup config/resque-pool.yml"
+      end
+      if config.kind_of? String
+        @pool_config_file = config.to_s
+        load_config_from_file
+      else
+        @config = config.dup
+      end
+      log "**** config: #{@config.inspect}"
     end
 
+    # }}}
+
+    # Sig handlers and self pipe management {{{
+
+    def self_pipe; @self_pipe ||= [] end
+    def sig_queue; @sig_queue ||= [] end
+
     def init_self_pipe!
-      SELF_PIPE.each { |io| io.close rescue nil }
-      SELF_PIPE.replace(IO.pipe)
-      SELF_PIPE.each { |io| io.fcntl(Fcntl::F_SETFD, Fcntl::FD_CLOEXEC) }
+      self_pipe.each { |io| io.close rescue nil }
+      self_pipe.replace(IO.pipe)
+      self_pipe.each { |io| io.fcntl(Fcntl::F_SETFD, Fcntl::FD_CLOEXEC) }
     end
 
     def init_sig_handlers!
       QUEUE_SIGS.each { |sig| trap_deferred(sig) }
-      trap(:CHLD) { |_| awaken_master }
+      trap(:CHLD)     { |_| awaken_master }
+    end
+
+    def awaken_master
+      begin
+        self_pipe.last.write_nonblock('.') # wakeup master process from select
+      rescue Errno::EAGAIN, Errno::EINTR
+        # pipe is full, master should wake up anyways
+        retry
+      end
     end
 
     # defer a signal for later processing in #join (master process)
     def trap_deferred(signal)
       trap(signal) do |sig_nr|
-        if SIG_QUEUE.size < 5
-          SIG_QUEUE << signal
+        if sig_queue.size < SIG_QUEUE_MAX_SIZE
+          sig_queue << signal
           awaken_master
         else
-          log "ignoring SIG#{signal}, queue=#{SIG_QUEUE.inspect}"
+          log "ignoring SIG#{signal}, queue=#{sig_queue.inspect}"
         end
       end
     end
 
+    def reset_sig_handlers!
+      QUEUE_SIGS.each {|sig| trap(sig, "DEFAULT") }
+    end
+
+    def handle_sig_queue!
+      case signal = sig_queue.shift
+      when :USR1, :USR2, :CONT
+        log "#{signal}: sending to all workers"
+        signal_all_workers(signal)
+      when :HUP
+        log "HUP: reload config file"
+        load_config_from_file
+        maintain_worker_count
+      when :WINCH
+        log "WINCH: gracefully stopping all workers"
+        @config = {}
+        maintain_worker_count
+      when :QUIT
+        log "QUIT: graceful shutdown, waiting for children"
+        signal_all_workers(:QUIT)
+        reap_all_workers(0) # will hang until all workers are shutdown
+        :break
+      when :INT
+        log "INT: immediate shutdown (graceful worker shutdown)"
+        signal_all_workers(:QUIT)
+        :break
+      when :TERM
+        log "TERM: immediate shutdown (and immediate worker shutdown)"
+        signal_all_workers(:TERM)
+        :break
+      end
+    end
+
+    # }}}
+    # start, join, and master sleep {{{
+
     def start
       procline("(starting)")
       init_self_pipe!
       init_sig_handlers!
       maintain_worker_count
       procline("(started)")
-      log "**** done in pool master #initialize"
+      log "**** started master at PID: #{Process.pid}"
       log "**** Pool contains PIDs: #{all_pids.inspect}"
       self
     end
@@ -89,84 +179,78 @@ module Resque
       loop do
         reap_all_workers
         break if handle_sig_queue! == :break
-        maintain_worker_count if SIG_QUEUE.empty?
+        if sig_queue.empty?
+          master_sleep
+          maintain_worker_count
+        end
         procline("managing #{all_pids.inspect}")
       end
       procline("(shutting down)")
       #stop # gracefully shutdown all workers on our way out
+      log "**** master complete"
       #unlink_pid_safe(pid) if pid
     end
 
-    def handle_sig_queue!
-      case SIG_QUEUE.shift
-      when nil
-        master_sleep
-      when :QUIT # graceful shutdown
-        :break
-      when :TERM, :INT # immediate shutdown
-        #stop(false)
-        :break
+    def master_sleep
+      begin
+        ready = IO.select([self_pipe.first], nil, nil, 1) or return
+        ready.first && ready.first.first or return
+        loop { self_pipe.first.read_nonblock(CHUNK_SIZE) }
+      rescue Errno::EAGAIN, Errno::EINTR
       end
     end
 
-    def reap_all_workers
+    # }}}
+    # worker process management {{{
+
+    def reap_all_workers(waitpid_flags=Process::WNOHANG)
       begin
         loop do
-          wpid, status = Process.waitpid2(-1, Process::WNOHANG)
+          wpid, status = Process.waitpid2(-1, waitpid_flags)
           wpid or break
-          #if reexec_pid == wpid
-            #log "reaped #{status.inspect} exec()-ed"
-            #self.reexec_pid = 0
-            #self.pid = pid.chomp('.oldbin') if pid
-            #proc_name 'master'
-          #else
-            worker = delete_worker(wpid) #and worker.tmp.close rescue nil
-            log "reaped #{status.inspect} " \
-                        "worker=#{worker.nr rescue 'unknown'}"
-          #end
+          worker = delete_worker(wpid)
+          # TODO: close any file descriptors connected to worker, if any
+          log "** reaped #{status.inspect}, worker=#{worker.queues.join(",")}"
         end
       rescue Errno::ECHILD
       end
     end
 
     def delete_worker(pid)
-      workers.each do |queues, pid_to_worker|
-        pid_to_worker.delete(pid)
+      worker = nil
+      workers.detect do |queues, pid_to_worker|
+        worker = pid_to_worker.delete(pid)
       end
+      worker
     end
 
-    def master_sleep
-      begin
-        ready = IO.select([SELF_PIPE.first], nil, nil, 1) or return
-        ready.first && ready.first.first or return
-        loop { SELF_PIPE.first.read_nonblock(CHUNK_SIZE) }
-      rescue Errno::EAGAIN, Errno::EINTR
-      end
+    def all_pids
+      workers.map {|q,workers| workers.keys }.flatten
     end
 
-    def awaken_master
-      begin
-        SELF_PIPE.last.write_nonblock('.') # wakeup master process from select
-      rescue Errno::EAGAIN, Errno::EINTR
-        # pipe is full, master should wake up anyways
-        retry
+    def signal_all_workers(signal)
+      all_pids.each do |pid|
+        Process.kill signal, pid
       end
     end
 
+    # }}}
+    # ???: maintain_worker_count, all_known_queues {{{
+
     def maintain_worker_count
-      pool_config.each do |queues, count|
-        next if (delta = worker_delta_for(queues)) == 0
+      all_known_queues.each do |queues|
+        delta = worker_delta_for(queues)
         spawn_missing_workers_for(queues) if delta > 0
-        #TODO: quit_excess_workers_for(queues)
+        quit_excess_workers_for(queues)   if delta < 0
       end
     end
 
-    def all_pids
-      workers.map {|q,workers| workers.keys }.flatten
+    def all_known_queues
+      config.keys | workers.keys
     end
 
-    ##
-    # all methods below operate on a single grouping of queues
+    # }}}
+    # methods that operate on a single grouping of queues {{{
     # perhaps this means a class is waiting to be extracted
 
     def spawn_missing_workers_for(queues)
@@ -175,8 +259,19 @@ module Resque
       end
     end
 
+    def quit_excess_workers_for(queues)
+      delta = -worker_delta_for(queues)
+      pids_for(queues)[0...delta].each do |pid|
+        Process.kill("QUIT", pid)
+      end
+    end
+
     def worker_delta_for(queues)
-      pool_config[queues] - workers[queues].size
+      config.fetch(queues, 0) - workers.fetch(queues, []).size
+    end
+
+    def pids_for(queues)
+      workers[queues].keys
     end
 
     def spawn_worker!(queues)
@@ -184,20 +279,23 @@ module Resque
       pid = fork do
         log "*** Starting worker #{worker}"
         call_after_prefork!
-        worker.work(ENV['INTERVAL'] || 5) # interval, will block
+        reset_sig_handlers!
+        #self_pipe.each {|io| io.close }
+        worker.work(ENV['INTERVAL'] || DEFAULT_WORKER_INTERVAL) # interval, will block
       end
+      workers[queues] ||= {}
       workers[queues][pid] = worker
     end
 
     def create_worker(queues)
       queues = queues.to_s.split(',')
-      worker = Resque::Worker.new(*queues)
+      worker = PooledWorker.new(*queues)
       worker.verbose = ENV['LOGGING'] || ENV['VERBOSE']
       worker.very_verbose = ENV['VVERBOSE']
       worker
-    rescue Resque::NoQueueError
-      abort "set QUEUE env var, e.g. $ QUEUE=critical,high rake resque:work"
     end
 
+    # }}}
+
   end
 end

data/lib/resque/pool/logging.rb

@@ -0,0 +1,19 @@
+module Resque
+  class Pool
+    module Logging
+
+      # Given a string, sets the procline ($0)
+      # Procline is always in the format of:
+      #   resque-pool-master: STRING
+      def procline(string)
+        $0 = "resque-pool-master: #{string}"
+      end
+
+      # TODO: make this use an actual logger
+      def log(message)
+        puts message
+      end
+
+    end
+  end
+end

data/lib/resque/pool/pooled_worker.rb

@@ -0,0 +1,65 @@
+class Resque::Pool
+
+  class PooledWorker < ::Resque::Worker
+
+    def initialize(*args)
+      @pool_master_pid = Process.pid
+      super
+    end
+
+    def pool_master_has_gone_away?
+      @pool_master_pid && @pool_master_pid != Process.ppid
+    end
+
+    # this allows us to shutdown
+    def shutdown
+      @shutdown || pool_master_has_gone_away?
+    end
+
+    # this entire method (except for one line) is copied and pasted from
+    # resque-1.9.9.  If shutdown were used as a method (attr_reader) rather
+    # than an instance variable, I wouldn't need to reduplicate this. :-(
+    #
+    # hopefully I can get defunkt to accept my patch for this.
+    # Until it is, the resque-pool gem will depend on an exact version of
+    # resque.
+    def work(interval = 5, &block)
+      $0 = "resque: Starting"
+      startup
+
+      loop do
+        #### THIS IS THE ONLY LINE THAT IS CHANGED
+        break if shutdown
+        #### THAT WAS THE ONLY LINE THAT WAS CHANGED
+
+        if not @paused and job = reserve
+          log "got: #{job.inspect}"
+          run_hook :before_fork
+          working_on job
+
+          if @child = fork
+            rand # Reseeding
+            procline "Forked #{@child} at #{Time.now.to_i}"
+            Process.wait
+          else
+            procline "Processing #{job.queue} since #{Time.now.to_i}"
+            perform(job, &block)
+            exit! unless @cant_fork
+          end
+
+          done_working
+          @child = nil
+        else
+          break if interval.to_i == 0
+          log! "Sleeping for #{interval.to_i}"
+          procline @paused ? "Paused" : "Waiting for #{@queues.join(',')}"
+          sleep interval.to_i
+        end
+      end
+
+    ensure
+      unregister_worker
+    end
+  end
+
+end

data/lib/resque/pool/tasks.rb

@@ -1,17 +1,12 @@
 # -*- encoding: utf-8 -*-
 
-# require 'resque/pool/tasks'
-# and configure "resque:setup" task to start up the environment, initialize
-# RESQUE_POOL_CONFIG, and setup other resque hooks
-
 namespace :resque do
   task :setup
 
-  desc "Launch a pool of resque workers (set RESQUE_POOL_CONFIG)"
+  desc "Launch a pool of resque workers"
   task :pool => :setup do
-    GC.respond_to?(:copy_on_write_friendly=) && GC.copy_on_write_friendly = true
     require 'resque/pool'
-    Resque::Pool.new(RESQUE_POOL_CONFIG).start.join
+    Resque::Pool.run
   end
 
 end

metadata

@@ -1,13 +1,13 @@
 --- !ruby/object:Gem::Specification 
 name: resque-pool
 version: !ruby/object:Gem::Version 
-  hash: 27
+  hash: 23
   prerelease: false
   segments: 
   - 0
   - 0
-  - 2
-  version: 0.0.2
+  - 4
+  version: 0.0.4
 platform: ruby
 authors: 
 - nicholas a. evans
@@ -16,7 +16,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2010-06-15 00:00:00 -04:00
+date: 2010-07-29 00:00:00 -04:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency 
@@ -39,14 +39,14 @@ dependencies:
   requirement: &id002 !ruby/object:Gem::Requirement 
     none: false
     requirements: 
-    - - ~>
+    - - "="
       - !ruby/object:Gem::Version 
-        hash: 49
+        hash: 33
         segments: 
         - 1
         - 9
-        - 1
-        version: 1.9.1
+        - 9
+        version: 1.9.9
   type: :runtime
   version_requirements: *id002
 description: "    quickly and easily fork a pool of resque workers,\n    saving memory (w/REE) and monitoring their uptime\n"
@@ -61,6 +61,9 @@ extra_rdoc_files: []
 files: 
 - lib/resque/pool.rb
 - lib/resque/pool/tasks.rb
+- lib/resque/pool/pooled_worker.rb
+- lib/resque/pool/logging.rb
+- README.md
 has_rdoc: true
 homepage: http://github.com/nevans/resque-pool
 licenses: []