resque-pool 0.5.0 → 0.6.0.rc1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 3d0e06a4a17b962d8423595f52f762c2c648c323
-  data.tar.gz: e108e81c14763bc5f4603cbb31f8d68a63e0cde9
+  metadata.gz: eada222fbbc16be4ae78356bafd528aecaa318c7
+  data.tar.gz: d3794970570d267f8b3421a1547179e2171d6e73
 SHA512:
-  metadata.gz: d222c8f5475167a18b6a2115a7aed5b5513dda496d5e4eb3481f7735505ff28393e4b3f160d6b69a2e7eaf7843fac74f5912f35113f1f20eaf2a453bfc8e8505
-  data.tar.gz: 91e0de1439d031b0490fd752e78cd4d22398e142aa4642cff1c5415c2d051636263ef609d5ebe1c4494dd5e930f06972e40f72233966b909526311e8fbbcf3d5
+  metadata.gz: 4e1a25962a81a7de91830fe1c785fd97505ec26e1df211124f9c58b54961b6a439b50e2aa53cb6bcad40098c6253b11dada9c199f4fa9598983acaab840266fa
+  data.tar.gz: 41578e6985fa8f4581479c2626db8a57a3e1ba1417ad17903b3587845d3b6980871f38b62b257f1f624b352d9af782d79f0168b6f7415cfa7c04391008b54ec3

data/Changelog.md

@@ -1,6 +1,74 @@
-## 0.4.0.dev (unreleased)
-
- * ???
+## 0.6.0 _(unreleased)_
+[full changelog](https://github.com/nevans/resque-pool/compare/v0.5.0...master).
+
+One big new feature: [Custom Config
+Loader](https://github.com/nevans/resque-pool#custom-configuration-loader)
+thanks to [joshuaflanagan](https://github.com/joshuaflanagan)!
+
+Lots of cleanup in this release.  Thanks to the contributers:
+
+ * [joshuaflanagan](https://github.com/joshuaflanagan) Custom config loader
+ * [grosser](https://github/grosser)
+   * ship less files in the gem
+   * remove trollop dependency
+   * remove -n -t -r -n -i commandline options since they were added unintentionally
+ * no longer hijacks shutdown for normal resque worker processes.
+ * [PatrickTulskie](https://github.com/PatrickTulskie) Reopening log files now
+   reopens *all* logs in memory (append write only; code copied from Unicorn)
+ * [jonleighton](https://github.com/jonleighton) pass worker instance to
+   `after_prefork` hook
+
+## 0.5.0 (2015-03-24)
+
+Some more merges of long outstanding pull requests.
+
+ * _EVEN BETTER_ `TERM` support for Heroku than 0.4.0.  ;)
+ * _DOCKER SUPPORT_ (don't go crazy when master pid is 1).
+   _(example Dockerfile soon?)_
+ * `--spawn_delay` option in case workers respawn too quickly
+ * Support `RUN_AT_EXIT_HOOKS`.
+ * And [more](https://github.com/nevans/resque-pool/compare/v0.4.0...v0.5.0).
+
+Many thanks to the contributors! [JohnBat26](https://github.com/JohnBat26), Eric
+Chapweske, [werkshy](https://github.com/werkshy),
+[spajus](https://github.com/spajus), [greysteil](https://github.com/greysteil),
+[tjsousa](https://github.com/tjsousa), [jkrall](https://github.com/jkrall),
+[zmillman](https://github.com/zmillman), [nevans](http://github.com/nevans).
+
+## 0.4.0 (2015-01-28)
+
+Another _long_ overdue maintenance release.  Many users had been running the
+various release candidates in production for over 16 months.  0.4.0 was based
+on 0.4.0.rc2 and 0.4.0.rc3 was rolled up into 0.5.0 instead.
+
+Better Heroku/`TERM_CHILD` support, better `upstart` process group control, ERB
+in the config file, not-insane package size, and
+[more](https://github.com/nevans/resque-pool/compare/v0.3.0...v0.4.0).
+
+Many thanks to the contributors!
+
+ * Better `TERM_CHILD` support (useful for Heroku or anywhere else that only
+   sends `TERM` to quit) [@rayh](https://github.com/rayh) and
+   [@jjulian](https://github.com/jjulian)
+ * [@jjulian](https://github.com/jjulian):
+   * 0.3.0 accidentally packaged up 13MB of extra files!  OOOPS... SORRY!
+   * better MacOS X compatibility
+   * missing LICENCE in gemspec
+ * [@jasonrclark](https://github.com/jasonrclark): `after_prefork` hook manages
+   an array of hooks, rather than one single hook
+ * [@mlanett](https://github.com/mlanett): Parse ERB in the config file (_very_
+   useful for hostname/environment switched configuration)
+ * [@xjlu](https://github.com/xjlu): Match the task deps in resque:work
+ * [@darbyfrey](https://github.com/darbyfrey): Fixing deprecation warnings in
+   newer versions of resque
+ * [@ewoodh20](https://github.com/ewoodh20): Use `Rails.env` if available
+   (`RAILS_ENV` is deprecated)
+ * [@dlackty](https://github.com/dlackty): example `god` config file
+ * [@mattdbridges](https://github.com/mattdbridges): fix order dependent specs
+ * [@nevans](https://github.com/nevans):
+   * Ignore `WINCH` signal when running non-daemonized (often in the terminal)
+   * Do not run children in the same process group (solves problems with `upstart`
+     sending `TERM` to all processes at once)
 
 ## 0.3.0 (2012-05-22)
 

data/README.md

@@ -1,11 +1,11 @@
 Resque Pool
 ===========
 
-[![Build Status](https://secure.travis-ci.org/nevans/resque-pool.png)](http://travis-ci.org/nevans/resque-pool)
+[![Build Status](https://travis-ci.org/nevans/resque-pool.png)](https://travis-ci.org/nevans/resque-pool)
 [![Dependency Status](https://gemnasium.com/nevans/resque-pool.png)](https://gemnasium.com/nevans/resque-pool)
 
 Resque pool is a simple library for managing a pool of
-[resque](http://github.com/defunkt/resque) workers.  Given a a config file, it
+[resque](https://github.com/defunkt/resque) workers.  Given a a config file, it
 manages your workers for you, starting up the appropriate number of workers for
 each worker type.
 
@@ -40,12 +40,14 @@ environment specific overrides (`RACK_ENV`, `RAILS_ENV`, and `RESQUE_ENV`
 environment variables can be used to determine environment).  For example in
 `config/resque-pool.yml`:
 
-    foo: 1
-    bar: 2
-    "foo,bar,baz": 1
+```Yaml
+foo: 1
+bar: 2
+"foo,bar,baz": 1
 
-    production:
-      "foo,bar,baz": 4
+production:
+  "foo,bar,baz": 4
+```
 
 ### Rake task config
 
@@ -55,20 +57,24 @@ application environment, configure Resque as necessary, and configure
 manager and reconnect in the workers.  For example, with rails you should put
 the following into `lib/tasks/resque.rake`:
 
-    require 'resque/pool/tasks'
-    # this task will get called before resque:pool:setup
-    # and preload the rails environment in the pool manager
-    task "resque:setup" => :environment do
-      # generic worker setup, e.g. Hoptoad for failed jobs
-    end
-    task "resque:pool:setup" do
-      # close any sockets or files in pool manager
-      ActiveRecord::Base.connection.disconnect!
-      # and re-open them in the resque worker parent
-      Resque::Pool.after_prefork do |job|
-        ActiveRecord::Base.establish_connection
-      end
-    end
+```Ruby
+require 'resque/pool/tasks'
+
+# this task will get called before resque:pool:setup
+# and preload the rails environment in the pool manager
+task "resque:setup" => :environment do
+  # generic worker setup, e.g. Hoptoad for failed jobs
+end
+
+task "resque:pool:setup" do
+  # close any sockets or files in pool manager
+  ActiveRecord::Base.connection.disconnect!
+  # and re-open them in the resque worker parent
+  Resque::Pool.after_prefork do |job|
+    ActiveRecord::Base.establish_connection
+  end
+end
+```
 
 
 For normal work with fresh resque and resque-scheduler gems add next lines in lib/rake/resque.rake
@@ -124,7 +130,7 @@ SIGNALS
 
 The pool manager responds to the following signals:
 
-* `HUP`   - reload the config file, reload logfiles, restart all workers.
+* `HUP`   - reset config loader (reload the config file), reload logfiles, restart all workers.
 * `QUIT`  - gracefully shut down workers (via `QUIT`) and shutdown the manager
   after all workers are done.
 * `INT`   - gracefully shut down workers (via `QUIT`) and immediately shutdown manager
@@ -143,6 +149,37 @@ defined by Resque 1.22 and above. See http://hone.heroku.com/resque/2012/08/21/r
 for details, overriding any command-line configuration for `TERM`. Setting `TERM_CHILD` tells
 us you know what you're doing.
 
+Custom Configuration Loader
+---------------------------
+
+If the static YAML file configuration approach does not meet your needs, you can
+specify a custom configuration loader.
+
+Set the `config_loader` class variable on Resque::Pool to an object that
+responds to `#call` (which can simply be a lambda/Proc). The class attribute
+needs to be set before starting the pool. This is usually accomplished
+in the `resque:pool:setup` rake task, as described above.
+
+For example, if you wanted to vary the number of worker processes based on a
+value stored in Redis, you could do something like:
+
+```ruby
+task resque:pool:setup do
+  Resque::Pool.config_loader = lambda do |env|
+    worker_count = Redis.current.get("pool_workers_#{env}").to_i
+    {"queueA,queueB" => worker_count }
+  end
+end
+```
+
+The configuration loader's `#call` method will be invoked about once a second.
+This allows the configuration to constantly change, allowing you to scale the
+number of workers up or down based on different conditions.
+If the response is generally static, the loader may want to cache the value it
+returns. It can optionally implement a `#reset!` method, which will be invoked
+when the HUP signal is received, allowing the loader to flush its cache, or
+perform any other re-initialization.
+
 Other Features
 --------------
 

data/Rakefile

@@ -1,5 +1,5 @@
-require 'bundler'
-Bundler::GemHelper.install_tasks
+require 'bundler/setup'
+require 'bundler/gem_tasks'
 
 # for loading the example config file in config/resque-pool.yml
 require 'resque/pool/tasks'

data/lib/resque/pool.rb

@@ -4,6 +4,7 @@ require 'resque/worker'
 require 'resque/pool/version'
 require 'resque/pool/logging'
 require 'resque/pool/pooled_worker'
+require 'resque/pool/file_or_hash_loader'
 require 'erb'
 require 'fcntl'
 require 'yaml'
@@ -18,10 +19,11 @@ module Resque
     include Logging
     extend  Logging
     attr_reader :config
+    attr_reader :config_loader
     attr_reader :workers
 
-    def initialize(config)
-      init_config(config)
+    def initialize(config_loader=nil)
+      init_config(config_loader)
       @workers = Hash.new { |workers, queues| workers[queues] = {} }
       procline "(initialized)"
     end
@@ -31,7 +33,7 @@ module Resque
     # The `after_prefork` hooks will be run in workers if you are using the
     # preforking master worker to save memory. Use these hooks to reload
     # database connections and so forth to ensure that they're not shared
-    # among workers.
+    # among workers. The worker instance is passed as an argument to the block.
     #
     # Call with a block to set a hook.
     # Call with no arguments to return all registered hooks.
@@ -49,17 +51,16 @@ module Resque
       @after_prefork = [after_prefork]
     end
 
-    def call_after_prefork!
+    def call_after_prefork!(worker)
       self.class.after_prefork.each do |hook|
-        hook.call
+        hook.call(worker)
       end
     end
 
     # }}}
-    # Config: class methods to start up the pool using the default config {{{
+    # Config: class methods to start up the pool using the config loader {{{
 
-    @config_files = ["resque-pool.yml", "config/resque-pool.yml"]
-    class << self; attr_accessor :config_files, :app_name, :spawn_delay; end
+    class << self; attr_accessor :config_loader, :app_name, :spawn_delay; end
 
     def self.app_name
       @app_name ||= File.basename(Dir.pwd)
@@ -81,46 +82,37 @@ module Resque
       )
     end
 
-    def self.choose_config_file
-      if ENV["RESQUE_POOL_CONFIG"]
-        ENV["RESQUE_POOL_CONFIG"]
-      else
-        @config_files.detect { |f| File.exist?(f) }
-      end
-    end
-
     def self.run
       if GC.respond_to?(:copy_on_write_friendly=)
         GC.copy_on_write_friendly = true
       end
-      Resque::Pool.new(choose_config_file).start.join
+      create_configured.start.join
     end
 
-    # }}}
-    # Config: load config and config file {{{
-
-    def config_file
-      @config_file || (!@config && ::Resque::Pool.choose_config_file)
+    def self.create_configured
+      Resque::Pool.new(config_loader)
     end
 
-    def init_config(config)
-      case config
-      when String, nil
-        @config_file = config
+    # }}}
+    # Config: store loader and load config {{{
+
+    def init_config(loader)
+      case loader
+      when String, Hash, nil
+        @config_loader = FileOrHashLoader.new(loader)
       else
-        @config = config.dup
+        @config_loader = loader
       end
       load_config
     end
 
     def load_config
-      if config_file
-        @config = YAML.load(ERB.new(IO.read(config_file)).result)
-      else
-        @config ||= {}
-      end
-      environment and @config[environment] and config.merge!(@config[environment])
-      config.delete_if {|key, value| value.is_a? Hash }
+      @config = config_loader.call(environment)
+    end
+
+    def reset_config
+      config_loader.reset! if config_loader.respond_to?(:reset!)
+      load_config
     end
 
     def environment
@@ -189,8 +181,8 @@ module Resque
         log "#{signal}: sending to all workers"
         signal_all_workers(signal)
       when :HUP
-        log "HUP: reload config file and reload logfiles"
-        load_config
+        log "HUP: reset configuration and reload logfiles"
+        reset_config
         Logging.reopen_logs!
         log "HUP: gracefully shutdown old children (which have old logfiles open)"
         if term_child
@@ -289,6 +281,7 @@ module Resque
         break if handle_sig_queue! == :break
         if sig_queue.empty?
           master_sleep
+          load_config
           maintain_worker_count
         end
         procline("managing #{all_pids.inspect}")
@@ -397,7 +390,7 @@ module Resque
         Process.setpgrp unless Resque::Pool.single_process_group
         worker.worker_parent_pid = Process.pid
         log_worker "Starting worker #{worker}"
-        call_after_prefork!
+        call_after_prefork!(worker)
         reset_sig_handlers!
         #self_pipe.each {|io| io.close }
         worker.work(ENV['INTERVAL'] || DEFAULT_WORKER_INTERVAL) # interval, will block

data/lib/resque/pool/cli.rb

@@ -1,4 +1,4 @@
-require 'trollop'
+require 'optparse'
 require 'resque/pool'
 require 'resque/pool/logging'
 require 'fileutils'
@@ -21,32 +21,35 @@ module Resque
       end
 
       def parse_options
-        opts = Trollop::options do
-          version "resque-pool #{VERSION} (c) nicholas a. evans"
-          banner <<-EOS
-resque-pool is the best way to manage a group (pool) of resque workers
+        opts = {}
+        OptionParser.new do |opt|
+          opt.banner = <<-EOS.gsub(/^            /, '')
+            resque-pool is the best way to manage a group (pool) of resque workers
 
-When daemonized, stdout and stderr default to resque-pool.stdxxx.log files in
-the log directory and pidfile defaults to resque-pool.pid in the current dir.
+            When daemonized, stdout and stderr default to resque-pool.stdxxx.log files in
+            the log directory and pidfile defaults to resque-pool.pid in the current dir.
 
-Usage:
-   resque-pool [options]
-where [options] are:
+            Usage:
+               resque-pool [options]
+
+            where [options] are:
           EOS
-          opt :config, "Alternate path to config file", :type => String, :short => "-c"
-          opt :appname, "Alternate appname",         :type => String,    :short => "-a"
-          opt :daemon, "Run as a background daemon", :default => false,  :short => "-d"
-          opt :stdout, "Redirect stdout to logfile", :type => String,    :short => '-o'
-          opt :stderr, "Redirect stderr to logfile", :type => String,    :short => '-e'
-          opt :nosync, "Don't sync logfiles on every write"
-          opt :pidfile, "PID file location",         :type => String,    :short => "-p"
-          opt :environment, "Set RAILS_ENV/RACK_ENV/RESQUE_ENV", :type => String, :short => "-E"
-          opt :spawn_delay, "Delay in milliseconds between spawning missing workers", :type => Integer, :short => "-s"
-          opt :term_graceful_wait, "On TERM signal, wait for workers to shut down gracefully"
-          opt :term_graceful,      "On TERM signal, shut down workers gracefully"
-          opt :term_immediate,     "On TERM signal, shut down workers immediately (default)"
-          opt :single_process_group, "Workers remain in the same process group as the master", :default => false
-        end
+          opt.on('-c', '--config PATH', "Alternate path to config file") { |c| opts[:config] = c }
+          opt.on('-a', '--appname NAME', "Alternate appname") { |c| opts[:appname] = c }
+          opt.on("-d", '--daemon', "Run as a background daemon") { opts[:daemon] = true }
+          opt.on('-o', '--stdout FILE', "Redirect stdout to logfile") { |c| opts[:stdout] = c }
+          opt.on('-e', '--stderr FILE', "Redirect stderr to logfile") { |c| opts[:stderr] = c }
+          opt.on('--nosync', "Don't sync logfiles on every write") { opts[:nosync] = true }
+          opt.on("-p", '--pidfile FILE', "PID file location") { |c| opts[:pidfile] = c }
+          opt.on("-E", '--environment ENVIRONMENT', "Set RAILS_ENV/RACK_ENV/RESQUE_ENV") { |c| opts[:environment] = c }
+          opt.on("-s", '--spawn-delay MS', Integer, "Delay in milliseconds between spawning missing workers") { |c| opts[:spawn_delay] = c }
+          opt.on('--term-graceful-wait', "On TERM signal, wait for workers to shut down gracefully") { opts[:term_graceful_wait] = true }
+          opt.on('--term-graceful',      "On TERM signal, shut down workers gracefully") { opts[:term_graceful] = true }
+          opt.on('--term-immediate',     "On TERM signal, shut down workers immediately (default)") { opts[:term_immediate] = true }
+          opt.on('--single-process-group', "Workers remain in the same process group as the master") { opts[:single_process_group] = true }
+          opt.on("-h", "--help", "Show this.") { puts opt; exit }
+          opt.on("-v", "--version", "Show Version"){ puts "resque-pool #{VERSION} (c) nicholas a. evans"; exit}
+        end.parse!
         if opts[:daemon]
           opts[:stdout]  ||= "log/resque-pool.stdout.log"
           opts[:stderr]  ||= "log/resque-pool.stderr.log"

data/lib/resque/pool/file_or_hash_loader.rb

@@ -0,0 +1,55 @@
+class FileOrHashLoader
+  def initialize(filename_or_hash=nil)
+    case filename_or_hash
+    when String, nil
+      @filename = filename_or_hash
+    when Hash
+      @static_config = filename_or_hash.dup
+    else
+      raise "#{self.class} cannot be initialized with #{filename_or_hash.inspect}"
+    end
+  end
+
+  def call(environment)
+    @config ||= load_config_from_file(environment)
+  end
+
+  def reset!
+    @config = nil
+  end
+
+  private
+
+  def load_config_from_file(environment)
+    if @static_config
+      new_config = @static_config
+    else
+      filename = config_filename
+      new_config = load_config filename
+    end
+    apply_environment new_config, environment
+  end
+
+  def apply_environment(config, environment)
+    environment and config[environment] and config.merge!(config[environment])
+    config.delete_if {|key, value| value.is_a? Hash }
+  end
+
+  def config_filename
+    @filename || choose_config_file
+  end
+
+  def load_config(filename)
+    return {} unless filename
+    YAML.load(ERB.new(IO.read(filename)).result)
+  end
+
+  CONFIG_FILES = ["resque-pool.yml", "config/resque-pool.yml"]
+  def choose_config_file
+    if ENV["RESQUE_POOL_CONFIG"]
+      ENV["RESQUE_POOL_CONFIG"]
+    else
+      CONFIG_FILES.detect { |f| File.exist?(f) }
+    end
+  end
+end