resque-pool 0.2.0 → 0.3.0.beta.1

data/Changelog.md

@@ -1,5 +1,19 @@
 ## unreleased
 
+* enhancement: new callbacks for configuration
+  * `Resque::Pool.configure do |pool| ... end`
+  * `pool.after_manager_wakeup`
+  * `pool.to_calculate_worker_offset`
+  * `pool.after_prefork` (instance callback prefered over class callback)
+* experimental: memory management
+* experimental: check orphaned workers
+* development: a good bit of code cleanup and rearrangement
+
+See ExperimentalFeatures.md for more info.  Thanks to Jason Haruska for these
+features!
+
+## 0.2.0 (2011-03-15)
+
 * new feature: sending `HUP` to pool manager will reload the logfiles and
   gracefully restart all workers.
 * enhancement: logging now includes timestamp, process "name" (worker or

data/ExperimentalFeatures.md

@@ -0,0 +1,48 @@
+Experimental Features
+---------------------
+
+Features listed here should not cause you problems if you don't use them... and
+probably won't cause you problems if you *do* use them.  Maybe.  We hope.  ;-)
+Once these features have stood the test of time, can be trusted to work cross
+platform, are well documented and tested, and have settled on a stable API,
+then we'll stop calling them experimental and probably give them command line
+options.  Until then, be forewarned that it might not work for you and the API
+might change between minor releases.
+
+### Memory management
+
+A memory manager is provided which can check once a minute to see if any of
+your workers are using too much memory.  If a worker is over the soft limit it
+will be sent a QUIT signal.  If a worker is over the hard signal it will be
+sent a TERM signal, and if it is still running a minute later it will be sent a
+KILL signal.  To use the memory manager, add something like the following to
+your Rakefile config:
+
+    task "resque:pool:setup" do
+      Resque::Pool.configure do |pool|
+        # memory limits are in MB
+        hard_limit = 250
+        soft_limit = 200
+        mm = Resque::Pool::MemoryManager.new(pool, hard_limit, soft_limit)
+        pool.after_manager_wakeup { mm.monitor_memory_usage }
+      end
+    end
+
+### Wait for orphaned workers to quit.
+
+When restarting resque-pool, some orphaned workers may be left over from the
+previous manager, even after the new manager has started up and forked its
+workers.  If you set the `RESQUE_WAIT_FOR_ORPHANS` environment variable, then
+the new manager will not start up all of its configured workers until all of
+the orphaned workers have finished.  This might be useful for long running jobs
+on memory constrained servers.
+
+To use this, add something like the following to your Rakefile config:
+
+    task "resque:pool:setup" do
+      Resque::Pool.configure do |pool|
+        orphan_watcher = Resque::Pool::OrphanWatcher.new(pool)
+        pool.to_calculate_worker_offset { orphan_watcher.worker_offset }
+      end
+    end
+

data/README.md

@@ -130,6 +130,11 @@ their current job) if the manager process disappears before them.
 You can specify an alternate config file by setting the `RESQUE_POOL_CONFIG` or
 with the `--config` command line option.
 
+Experimental Features
+---------------------
+
+See `ExperimentalFeatures.md` for some potentially useful works in progress.
+
 TODO
 -----
 
@@ -138,8 +143,8 @@ See [the TODO list](https://github.com/nevans/resque-pool/issues) at github issu
 Contributors
 -------------
 
+* Jason Haruska from Backupify (pidfile management, memory management, wait for orphaned workers)
 * John Schult (config file can be split by environment)
 * Stephen Celis (increased gemspec sanity)
 * Vincent Agnello, Robert Kamunyori, Paul Kauders; for pairing with me at
-  B'more on Rails Open Source Hack Nights. :)
-
+  [B'more on Rails](http://twitter.com/bmoreonrails) Open Source Hack Nights. :)

data/lib/resque/pool.rb

@@ -2,7 +2,12 @@
 require 'resque'
 require 'resque/pool/version'
 require 'resque/pool/logging'
-require 'resque/pool/pooled_worker'
+require 'resque/pool/manager'
+require 'resque/pool/worker_type_manager'
+# experimental features!
+require 'resque/pool/memory_manager'
+require 'resque/pool/orphan_watcher'
+
 require 'fcntl'
 require 'yaml'
 
@@ -23,26 +28,70 @@ module Resque
       procline "(initialized)"
     end
 
-    # Config: after_prefork {{{
+    # Config: hooks {{{
+
+    # The +configure+ block will be run once during pool startup, after the
+    # internal initialization is complete, but before any workers are started
+    # up.  This is for configuring any runtime callbacks that you want to
+    # customize (e.g.  +after_wakup+, +worker_offset_handler+)
+    #
+    # Call with a block to set.
+    # Call with no arguments to return the hook.
+    def self.configure(&block)
+      block ? (@configure = block) : @configure
+    end
+
+    def call_configure!
+      self.class.configure && self.class.configure.call(self)
+    end
+
+    # The +after_manager_wakeup+ hook will be run in the pool manager every
+    # time the pool manager wakes up from IO.select (normally once a second).
+    # It will run immediately before the pool manager performs its normal
+    # worker maintenance (starting and stopping workers as necessary).  If you
+    # have some special logic for killing workers that are taking too long or
+    # using too much memory, this is the place to put it.
+    #
+    # Call with a block to set the hook.
+    # Call with no arguments to return the hook.
+    def after_manager_wakeup(&block)
+      block ? (@after_manager_wakeup = block) : @after_manager_wakeup
+    end
+
+    def call_after_manager_wakeup!
+      after_manager_wakeup && after_manager_wakeup.call(self)
+    end
+
+    def to_calculate_worker_offset(&block)
+      block ? (@to_calculate_worker_offset = block) : @to_calculate_worker_offset
+    end
+
+    # deprecated by instance level +after_prefork+ hook
+    def self.after_prefork(&block)
+      #log '"Resque::Pool.after_prefork(&blk)" is deprecated.'
+      #log 'Please use "Resque::Pool.configure {|p| p.after_prefork(&blk) }" instead.'
+      block ? (@after_prefork = block) : @after_prefork
+    end
 
-    # The `after_prefork` hook will be run in workers if you are using the
+    # The +after_prefork+ hook will be run in workers if you are using the
     # preforking master worker to save memory. Use this hook to reload
     # database connections and so forth to ensure that they're not shared
     # among workers.
     #
     # Call with a block to set the hook.
     # Call with no arguments to return the hook.
-    def self.after_prefork(&block)
+    def after_prefork(&block)
       block ? (@after_prefork = block) : @after_prefork
     end
 
     # Set the after_prefork proc.
-    def self.after_prefork=(after_prefork)
+    def after_prefork=(after_prefork)
       @after_prefork = after_prefork
     end
 
     def call_after_prefork!
-      self.class.after_prefork && self.class.after_prefork.call
+      (after_prefork && after_prefork.call) ||
+        (self.class.after_prefork && self.class.after_prefork.call)
     end
 
     # }}}
@@ -93,7 +142,9 @@ module Resque
     end
 
     def environment
-      if defined? RAILS_ENV
+      if defined? Rails
+        Rails.env
+      elsif defined? RAILS_ENV # keep compatibility with older versions of rails
         RAILS_ENV
       else
         ENV['RACK_ENV'] || ENV['RAILS_ENV'] || ENV['RESQUE_ENV']
@@ -144,10 +195,6 @@ module Resque
       end
     end
 
-    def reset_sig_handlers!
-      QUEUE_SIGS.each {|sig| trap(sig, "DEFAULT") }
-    end
-
     def handle_sig_queue!
       case signal = sig_queue.shift
       when :USR1, :USR2, :CONT
@@ -185,6 +232,8 @@ module Resque
     # start, join, and master sleep {{{
 
     def start
+      procline("(configuring)")
+      call_configure!
       procline("(starting)")
       init_self_pipe!
       init_sig_handlers!
@@ -209,6 +258,7 @@ module Resque
         break if handle_sig_queue! == :break
         if sig_queue.empty?
           master_sleep
+          call_after_manager_wakeup!
           maintain_worker_count
         end
         procline("managing #{all_pids.inspect}")
@@ -242,6 +292,8 @@ module Resque
           # TODO: close any file descriptors connected to worker, if any
           log "Reaped resque worker[#{status.pid}] (status: #{status.exitstatus}) queues: #{worker.queues.join(",")}"
         end
+      rescue Errno::EINTR
+        retry
       rescue Errno::ECHILD, QuitNowException
       end
     end
@@ -265,64 +317,20 @@ module Resque
     end
 
     # }}}
-    # ???: maintain_worker_count, all_known_queues {{{
+    # maintain_worker_count, all_known_worker_types, worker_offset {{{
 
     def maintain_worker_count
-      all_known_queues.each do |queues|
-        delta = worker_delta_for(queues)
-        spawn_missing_workers_for(queues) if delta > 0
-        quit_excess_workers_for(queues)   if delta < 0
+      all_known_worker_types.each do |queues|
+        WorkerTypeManager.new(self, queues).maintain_worker_count(worker_offset)
       end
     end
 
-    def all_known_queues
+    def all_known_worker_types
       config.keys | workers.keys
     end
 
-    # }}}
-    # methods that operate on a single grouping of queues {{{
-    # perhaps this means a class is waiting to be extracted
-
-    def spawn_missing_workers_for(queues)
-      worker_delta_for(queues).times do |nr|
-        spawn_worker!(queues)
-      end
-    end
-
-    def quit_excess_workers_for(queues)
-      delta = -worker_delta_for(queues)
-      pids_for(queues)[0...delta].each do |pid|
-        Process.kill("QUIT", pid)
-      end
-    end
-
-    def worker_delta_for(queues)
-      config.fetch(queues, 0) - workers.fetch(queues, []).size
-    end
-
-    def pids_for(queues)
-      workers[queues].keys
-    end
-
-    def spawn_worker!(queues)
-      worker = create_worker(queues)
-      pid = fork do
-        log_worker "Starting worker #{worker}"
-        call_after_prefork!
-        reset_sig_handlers!
-        #self_pipe.each {|io| io.close }
-        worker.work(ENV['INTERVAL'] || DEFAULT_WORKER_INTERVAL) # interval, will block
-      end
-      workers[queues] ||= {}
-      workers[queues][pid] = worker
-    end
-
-    def create_worker(queues)
-      queues = queues.to_s.split(',')
-      worker = PooledWorker.new(*queues)
-      worker.verbose = ENV['LOGGING'] || ENV['VERBOSE']
-      worker.very_verbose = ENV['VVERBOSE']
-      worker
+    def worker_offset
+      to_calculate_worker_offset && to_calculate_worker_offset.call || 0
     end
 
     # }}}

data/lib/resque/pool/cli.rb

@@ -99,7 +99,7 @@ where [options] are:
       def setup_environment(opts)
         ENV["RACK_ENV"] = ENV["RAILS_ENV"] = ENV["RESQUE_ENV"] = opts[:environment] if opts[:environment]
         log "Resque Pool running in #{ENV["RAILS_ENV"] || "development"} environment"
-        ENV["RESQUE_POOL_CONFIG"] = opts[:config] if opts[:config]
+        ENV["RESQUE_POOL_CONFIG"] = opts[:config].to_s if opts[:config]
       end
 
       def start_pool

data/lib/resque/pool/manager.rb

@@ -0,0 +1,11 @@
+# TODO: reorganize code that is currently in resque/pool.rb into this file
+require 'resque/pool/logging'
+
+module Resque
+  class Pool
+    class Manager
+      include Logging
+
+    end
+  end
+end

data/lib/resque/pool/memory_manager.rb

@@ -0,0 +1,118 @@
+require 'resque/pool/logging'
+
+module Resque
+  class Pool
+    class MemoryManager
+      include Logging
+
+      attr_reader :hard_limit, :soft_limit
+
+      def initialize(hard_limit=250, soft_limit=200)
+        @hard_limit = hard_limit
+        @soft_limit = soft_limit
+      end
+
+      def monitor_memory_usage(pool_manager)
+        #only check every minute
+        if @last_mem_check.nil? || @last_mem_check < Time.now - 60
+          hard_kill_workers
+          pool_manager.all_pids.each do |pid|
+            total_usage  = memory_usage(pid)
+            child_pid    = find_child_pid(pid)
+            total_usage += memory_usage(child_pid) if child_pid
+
+            if total_usage > hard_limit
+              log "Terminating worker #{pid} for using #{total_usage}MB memory"
+              stop_worker(pid)
+            elsif total_usage > soft_limit
+              log "Gracefully shutting down worker #{pid} for using #{total_usage}MB memory"
+              stop_worker(pid, :QUIT)
+            end
+          end
+          @last_mem_check = Time.now
+        end
+      end
+
+      private
+
+      def add_killed_worker(pid)
+        @term_workers ||= []
+        @term_workers << pid if pid
+      end
+
+      def find_child_pid(parent_pid)
+        begin
+          p = `ps --ppid #{parent_pid} -o pid --no-header`.to_i
+          p == 0 ? nil : p
+        rescue Errno::EINTR
+          retry
+        end
+      end
+
+      def hard_kill_workers
+        @term_workers ||= []
+        #look for workers that didn't terminate
+        @term_workers.delete_if {|pid| !process_exists?(pid)}
+        #send the rest a -9
+        @term_workers.each {|pid| `kill -9 #{pid}`}
+      end
+
+      def hostname
+        begin
+          @hostname ||= `hostname`.strip
+        rescue Errno::EINTR
+          retry
+        end
+      end
+
+      def memory_usage(pid)
+        smaps_filename = "/proc/#{pid}/smaps"
+        #Grab actual memory usage from proc in MB
+        begin
+          mem_usage = `
+            if [ -f #{smaps_filename} ];
+              then
+                grep Private_Dirty #{smaps_filename} | awk '{s+=$2} END {printf("%d", s/1000)}'
+              else echo "0"
+            fi
+          `.to_i
+        rescue Errno::EINTR
+          retry
+        end
+      end
+
+      # TODO: DRY up this and the pidfile checker in Resque::Pool::CLI
+      def process_exists?(pid)
+        begin
+          ps_line = `ps -p #{pid} --no-header`
+        rescue Errno::EINTR
+          retry
+        end
+        !ps_line.nil? && ps_line.strip != ''
+      end
+
+      def stop_worker(pid, signal=:TERM)
+        begin
+          worker = Resque.working.find do |w|
+            host, worker_pid, queues = w.id.split(':')
+            w if worker_pid.to_i == pid.to_i && host == hostname
+          end
+          if worker
+            encoded_job = worker.job
+            verb = signal == :QUIT ? 'Graceful' : 'Forcing'
+            total_time = Time.now - Time.parse(encoded_job['run_at']) rescue 0
+            log "#{verb} shutdown while processing: #{encoded_job} -- ran for #{'%.2f' % total_time}s"
+          end
+          Process.kill signal, pid
+          if signal == :TERM
+            add_killed_worker(pid)
+            add_killed_worker(find_child_pid(pid))
+          end
+        rescue Errno::EINTR
+          retry
+        end
+      end
+
+    end
+  end
+end

data/lib/resque/pool/orphan_watcher.rb

@@ -0,0 +1,49 @@
+require 'resque/pool/logging'
+
+module Resque
+  class Pool
+    class OrphanWatcher
+      include Logging
+
+      attr_reader :pool_manager
+
+      def initialize(pool_manager)
+        @pool_manager = pool_manager
+      end
+
+      def worker_offset
+          orphaned_worker_count / pool_manager.all_known_worker_types.size
+      end
+
+      def orphaned_worker_count
+        if @last_orphaned_check.nil? || @last_orphaned_check < Time.now - 60
+          if @orphaned_pids.nil?
+            begin
+              pids_with_parents = `ps -Af | grep resque | grep -v grep | grep -v resque-web | grep -v master | awk '{printf("%d %d\\n", $2, $3)}'`.split("\n")
+            rescue Errno::EINTR
+              retry
+            end
+            pids = pids_with_parents.collect {|x| x.split[0].to_i}
+            parents = pids_with_parents.collect {|x| x.split[1].to_i}
+            pids.delete_if {|x| parents.include?(x)}
+            pids.delete_if {|x| all_pids.include?(x)}
+            @orphaned_pids = pids
+          elsif @orphaned_pids.size > 0
+            @orphaned_pids.delete_if do |pid|
+              begin
+                ps_out = `ps --no-heading p #{pid}`
+                ps_out.nil? || ps_out.strip == ''
+              rescue Errno::EINTR
+                retry
+              end
+            end
+          end
+          @last_orphaned_check = Time.now
+          log "Current orphaned pids: #{@orphaned_pids}" if @orphaned_pids.size > 0
+        end
+        @orphaned_pids.size
+      end
+
+    end
+  end
+end

data/lib/resque/pool/version.rb

@@ -1,5 +1,5 @@
 module Resque
   class Pool
-    VERSION = "0.2.0"
+    VERSION = "0.3.0.beta.1"
   end
 end

data/lib/resque/pool/worker_type_manager.rb

@@ -0,0 +1,102 @@
+require 'resque/pool/logging'
+require 'resque/pool/pooled_worker'
+
+module Resque
+  class Pool
+    class WorkerTypeManager
+      include Logging
+
+      attr_reader :pool_manager, :queues, :queue_array
+
+      def initialize(pool_manager, queues)
+        @pool_manager = pool_manager
+        @queues       = queues
+        @queue_array  = queues.to_s.split(',')
+      end
+
+      # TODO: Pool manager will hold onto WorkerTypeManagers, and will push the
+      # configuration directly into the WorkerTypeManagers
+      def configuration
+        { :count => pool_manager.config.fetch(queues, 0), }
+      end
+
+      def maintain_worker_count(offset)
+        delta = worker_delta - offset
+        spawn_missing_workers_for(delta) if delta > 0
+        quit_excess_workers_for(delta)   if delta < 0
+      end
+
+      def pids
+        running_workers.keys
+      end
+
+      # TODO: Pool manager will hold onto WorkerTypeManagers,
+      # and WorkerTypeManager will store the workers itself
+      def running_workers
+        pool_manager.workers.fetch(queues, {})
+      end
+
+      def worker_delta
+        configuration[:count] - running_workers.size
+      end
+
+      private
+
+      def create_worker
+        worker = PooledWorker.new(*queue_array)
+        worker.verbose = ENV['LOGGING'] || ENV['VERBOSE']
+        worker.very_verbose = ENV['VVERBOSE']
+        worker
+      end
+
+      def quit_excess_workers_for(delta)
+        if delta < 0
+          queue_pids = pids.clone
+          if queue_pids.size >= delta.abs
+            queue_pids[0...delta.abs].each {|pid| Process.kill("QUIT", pid)}
+          else
+            queue_pids.each {|pid| Process.kill("QUIT", pid)}
+          end
+        end
+      end
+
+      # TODO: Pool manager will hold onto WorkerTypeManagers,
+      # and WorkerTypeManager will store the pids itself
+      def register_new_worker_with_manager(worker, pid)
+        pool_manager.workers[queues] ||= {}
+        pool_manager.workers[queues][pid] = worker
+      end
+
+      def reset_sig_handlers!
+        QUEUE_SIGS.each {|sig| trap(sig, "DEFAULT") }
+      end
+
+      def spawn_missing_workers_for(delta)
+        delta.times { spawn_worker! } if delta > 0
+      end
+
+      def spawn_worker!
+        worker = create_worker
+        pid = fork do
+          start_forked_worker worker
+        end
+        register_new_worker_with_manager worker, pid
+      end
+
+      def start_forked_worker(worker)
+        reset_sig_handlers!
+        log_worker "Starting worker #{worker}"
+        pool_manager.call_after_prefork!
+        begin
+          worker.work(ENV['INTERVAL'] || DEFAULT_WORKER_INTERVAL) # interval, will block
+        rescue Errno::EINTR
+          log_worker "Caught interrupted system call Errno::EINTR. Retrying."
+          retry
+        end
+      end
+
+
+    end
+  end
+end
+

metadata

@@ -1,13 +1,15 @@
 --- !ruby/object:Gem::Specification 
 name: resque-pool
 version: !ruby/object:Gem::Version 
-  hash: 23
-  prerelease: false
+  hash: 62196369
+  prerelease: true
   segments: 
   - 0
-  - 2
+  - 3
   - 0
-  version: 0.2.0
+  - beta
+  - 1
+  version: 0.3.0.beta.1
 platform: ruby
 authors: 
 - nicholas a. evans
@@ -179,11 +181,16 @@ files:
 - lib/resque/pool/tasks.rb
 - lib/resque/pool/cli.rb
 - lib/resque/pool/version.rb
+- lib/resque/pool/manager.rb
 - lib/resque/pool/pooled_worker.rb
+- lib/resque/pool/memory_manager.rb
+- lib/resque/pool/orphan_watcher.rb
+- lib/resque/pool/worker_type_manager.rb
 - lib/resque/pool/logging.rb
 - config/cucumber.yml
 - config/alternate.yml
 - Gemfile
+- ExperimentalFeatures.md
 has_rdoc: true
 homepage: http://github.com/nevans/resque-pool
 licenses: []
@@ -205,12 +212,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
 required_rubygems_version: !ruby/object:Gem::Requirement 
   none: false
   requirements: 
-  - - ">="
+  - - ">"
     - !ruby/object:Gem::Version 
-      hash: 3
+      hash: 25
       segments: 
-      - 0
-      version: "0"
+      - 1
+      - 3
+      - 1
+      version: 1.3.1
 requirements: []
 
 rubyforge_project: