resque-pool 0.0.12.1.alpha → 0.1.0

data/Changelog.md

@@ -0,0 +1,34 @@
+## 0.1.0 (unreleased)
+
+* new feature: `resque-pool` command line interface
+  * this replaces need for a special startup script.
+  * manages PID file, logfiles, daemonizing, etc.
+  * `resque-pool --help` for more info and options
+* updated example config, init.d script, including a chef recipe that should
+  work at EngineYard.
+
+## 0.0.10 (2010-08-31)
+
+* remove rubygems 1.3.6 dependency
+
+## 0.0.9 (2010-08-26)
+
+* new feature: `RESQUE_POOL_CONFIG` environment variable to set alt config file
+* upgraded to resque 1.10, removing `Resque::Worker` monkeypatch
+
+## 0.0.8 (2010-08-20)
+
+* bugfix: using (or not using) environments in config file
+
+## 0.0.7 (2010-08-16)
+
+* new feature: split by environments in config file
+* added example startup script, Rakefile, and monit config
+
+## 0.0.5 (2010-06-29)
+
+* bugfix: worker processes not shutting down after orphaned
+
+## 0.0.4 (2010-06-29)
+
+* first release used in production

data/README.md

@@ -2,33 +2,31 @@ Resque Pool
 ===========
 
 Resque pool is a simple library for managing a pool of resque workers.  Given a
-a config file (`resque-pool.yml` or `config/resque-pool.yml`), it will manage
-your workers for you, starting up the appropriate number of workers for each.
+a config file, it manages your workers for you, starting up the appropriate
+number of workers for each.
 
 Benefits
 ---------
 
-* Less memory consumption - If you are using Ruby Enterprise Edition, or any
-  ruby with copy-on-write safe garbage collection, this could save you a lot of
-  memory when you are managing many workers.
-* Simpler (less) config - If you are using monit or an init script to start up
-  your workers, you can start up one pool, and it will manage your workers for
-  you.
-* Faster startup - if you are starting many workers at once, you would normally
-  have them competing for CPU as they load their environments.  Resque-pool can
-  load the environment once and almost instantaneously fork all of the workers.
+* Less config - With a simple YAML file, you can start up a pool daemon, and it
+  will monitor your workers for you.  An example init.d script, monit config,
+  and chef cookbook are provided.
+* Less memory - If you are using Ruby Enterprise Edition, or any ruby with
+  copy-on-write safe garbage collection, this should save you a lot of memory
+  when you are managing many workers.
+* Faster startup - when you start many workers at once, they would normally
+  compete for CPU as they load their environments.  Resque-pool can load the
+  environment once and almost instantaneously fork all of the workers.
 
 How to use
 -----------
 
-To configure resque-pool, create a `config/resque-pool.yml` with your worker
-counts.  To use resque-pool, require its rake tasks (`resque/pool/tasks`) in
-your rake file, and run `resque-pool`
+### YAML file config
 
-The YAML file supports both using root level defaults as well as environment
-specific overrides (`RACK_ENV`, `RAILS_ENV`, and `RESQUE_ENV` environment
-variables will be used to determine environment).  For example, to use
-resque-pool with rails, in `config/resque-pool.yml`:
+Create a `config/resque-pool.yml` with your worker counts.  The YAML file
+supports both using root level defaults as well as environment specific
+overrides (`RACK_ENV`, `RAILS_ENV`, and `RESQUE_ENV` environment variables can
+be used to determine environment).  For example in `config/resque-pool.yml`:
 
     foo: 1
     bar: 2
@@ -37,16 +35,21 @@ resque-pool with rails, in `config/resque-pool.yml`:
     production:
       "foo,bar,baz": 4
 
-and in `lib/tasks/resque.rake`:
+### Rake task config
+
+Require the rake tasks (`resque/pool/tasks`) in your rake file, configure
+Resque as necessary, and configure `Resque::Pool` to disconnect all open
+sockets in the pool manager and reconnect in the workers.  For example, with
+rails you should put the following into `lib/tasks/resque.rake`:
 
     require 'resque/pool/tasks'
     # this task will get called before resque:pool:setup
-    # preload the rails environment in the pool master
+    # and preload the rails environment in the pool manager
     task "resque:setup" => :environment do
       # generic worker setup, e.g. Hoptoad for failed jobs
     end
     task "resque:pool:setup" do
-      # close any sockets or files in pool master
+      # close any sockets or files in pool manager
       ActiveRecord::Base.connection.disconnect!
       # and re-open them in the resque worker parent
       Resque::Pool.after_prefork do |job|
@@ -54,29 +57,30 @@ and in `lib/tasks/resque.rake`:
       end
     end
 
+### Start the pool manager
+
 Then you can start the queues via:
 
     resque-pool --environment production
 
 This will start up seven worker processes, one exclusively for the foo queue,
 two exclusively for the bar queue, and four workers looking at all queues in
-priority.  This is similar to if you ran the following:
-
-    rake resque:work RAILS_ENV=production VERBOSE=1 QUEUES=foo &
-    rake resque:work RAILS_ENV=production VERBOSE=1 QUEUES=bar &
-    rake resque:work RAILS_ENV=production VERBOSE=1 QUEUES=bar &
-    rake resque:work RAILS_ENV=production VERBOSE=1 QUEUES=foo,bar,baz &
-    rake resque:work RAILS_ENV=production VERBOSE=1 QUEUES=foo,bar,baz &
-    rake resque:work RAILS_ENV=production VERBOSE=1 QUEUES=foo,bar,baz &
-    rake resque:work RAILS_ENV=production VERBOSE=1 QUEUES=foo,bar,baz &
-
-Resque already forks for its own child processes.  The pool master will stay
-around monitoring the resque worker parents, giving three levels: a single pool
-master, many worker parents, and a worker child per worker (when the actual job
-is being processed).  For example, `ps -ef f | grep [r]esque` (in Linux) might
-return something like the following:
-
-    rails    13858     1  0 13:44 ?        S      0:02 resque-pool-master: managing [13867, 13875, 13871, 13872, 13868, 13870, 13876]
+priority.  With the config above, this is similar to if you ran the following:
+
+    rake resque:work RAILS_ENV=production QUEUES=foo &
+    rake resque:work RAILS_ENV=production QUEUES=bar &
+    rake resque:work RAILS_ENV=production QUEUES=bar &
+    rake resque:work RAILS_ENV=production QUEUES=foo,bar,baz &
+    rake resque:work RAILS_ENV=production QUEUES=foo,bar,baz &
+    rake resque:work RAILS_ENV=production QUEUES=foo,bar,baz &
+    rake resque:work RAILS_ENV=production QUEUES=foo,bar,baz &
+
+The pool manager will stay around monitoring the resque worker parents, giving
+three levels: a single pool manager, many worker parents, and one worker child
+per worker (when the actual job is being processed).  For example, `ps -ef f |
+grep [r]esque` (in Linux) might return something like the following:
+
+    rails    13858     1  0 13:44 ?        S      0:02 resque-pool-manager: managing [13867, 13875, 13871, 13872, 13868, 13870, 13876]
     rails    13867 13858  0 13:44 ?        S      0:00  \_ resque-1.9.9: Waiting for foo
     rails    13868 13858  0 13:44 ?        S      0:00  \_ resque-1.9.9: Waiting for bar
     rails    13870 13858  0 13:44 ?        S      0:00  \_ resque-1.9.9: Waiting for bar
@@ -87,16 +91,21 @@ return something like the following:
     rails    13876 13858  0 13:44 ?        S      0:00  \_ resque-1.9.9: Forked 7485 at 1280343255
     rails     7485 13876  0 14:54 ?        S      0:00      \_ resque-1.9.9: Processing bar since 1280343254
 
+You can also run resque-pool as a daemon via `--daemon`.  It will default to
+placing the pidfile and logfiles in the rails default locations, which you may
+want to configure.  The `RAILS_ENV` can be specified via `--environment`.  See
+`resque-pool --help` for more options.
+
 SIGNALS
 -------
 
-The pool master responds to the following signals:
+The pool manager responds to the following signals:
 
 * `HUP`   - reload the config file, e.g. to change the number of workers per queue list
-* `QUIT`  - send `QUIT` to each worker parent and shutdown the master after all workers are done.
-* `INT`   - send `QUIT` to each worker parent and immediately shutdown master
-* `TERM`  - send `TERM` to each worker parent and immediately shutdown master
-* `WINCH` - send `QUIT` to each worker, but keep master running (send `HUP` to reload config and restart workers)
+* `QUIT`  - send `QUIT` to each worker parent and shutdown the manager after all workers are done.
+* `INT`   - send `QUIT` to each worker parent and immediately shutdown manager
+* `TERM`  - send `TERM` to each worker parent and immediately shutdown manager
+* `WINCH` - send `QUIT` to each worker, but keep manager running (send `HUP` to reload config and restart workers)
 * `USR1`/`USR2`/`CONT` - send the signal on to all worker parents (see Resque docs).
 
 After a `HUP`, workers that are no longer needed will be gracefully shutdown
@@ -105,46 +114,43 @@ via `QUIT`.
 Other Features
 --------------
 
-You can run resque-pool as a daemon via `--daemon`.  It will default to placing
-the pidfile and logfiles in the rails default locations, and you may want to
-configure that.  See `resque-pool --help` for more options.
-
-An example init.d script and monitrc are provided in
-`examples/chef_cookbook/templates/default`, as erb templates.  Just copy them
-into place, filling in the erb variables as necessary.  An example chef recipe
-is also provided (it should work at Engine Yard as is; just provide a
-`/data/#{app_name}/shared/config/resque-pool.yml` on your utility servers).
-
-You can also run `resque-pool` via the `resque:pool` rake task or from a plain
-old ruby script by calling `Resque::Pool.run`.
+An example chef recipe is provided (it should work at Engine Yard as is; just
+provide a `/data/#{app_name}/shared/config/resque-pool.yml` on your utility
+servers).  Even if you don't use chef, you can still use the example init.d
+script and monitrc (erb templates) provided in
+`examples/chef_cookbook/templates/default`.
 
-Workers will watch the pool master, and gracefully shutdown (after completing
-their current job) if the master process dies (for whatever reason) before
-them.
+You can also start a pool manager via `rake resque:pool` or from a plain old
+ruby script by calling `Resque::Pool.run`.
 
-You can specify an alternate config file by setting the `RESQUE_POOL_CONFIG`
-environment variable like so:
+Workers will watch the pool manager, and gracefully shutdown (after completing
+their current job) if the manager process disappears before them.
 
-    resque-pool -c /path/to/my/config.yml
+You can specify an alternate config file by setting the `RESQUE_POOL_CONFIG` or
+with the `--config` command line option.
 
 TODO
 -----
 
-* cmd line option for non-rake loading
-* cmd line option for preload ruby file
-* provide Unix style log formatter
 * web interface for adding and removing workers (etc)
+* config file split by hostname
+* rename to `resque-squad`?
+* cmd line option for non-rake loading
+* cmd line option for preloading ruby file
+* provide Unix style log formatter (compatible with $stdout/$stderr)
 * recover gracefully from a malformed config file (on startup and HUP)
-* procline for malformed config file, graceful shutdown... and other states?
+* change procline for malformed config file, graceful shutdown... and other states?
 * figure out a good automated way to test this (cucumber or rspec?)
 * clean up the code (I stole most of it from unicorn, and it's still a bit
-  bastardized); excessive use of vim foldmarkers are a code smell.
-* rename to `resque-squad`?
+  bastardized); excessive use of vim foldmarkers are a code smell!
 * rdoc
-* incorporate resque-batchworker features? (v2.0)
+* integrate with or incorporate resque-batchworker features? (v2.0)
+* integrate with resque-scheduler? (v2.0)
 
 Contributors
 -------------
 
 * John Schult (config file can be split by environment)
 * Stephen Celis (increased gemspec sanity)
+* Vincent Agnello, Robert Kamunyori, Paul Kauders; for pairing with me at
+  B'more on Rails Open Source Hack Nights. :)

data/examples/chef_cookbook/recipes/default.rb

@@ -1,6 +1,9 @@
-if ['solo', 'util'].include?(node[:instance_role])
+roles = %w[solo util]
+if roles.include?(node[:instance_role])
   node[:applications].each do |app, data|
 
+    pidfile = "/data/#{app}/current/tmp/pids/#{app}_resque.pid"
+
     template "/etc/monit.d/#{app}_resque.monitrc" do
       owner 'root'
       group 'root'
@@ -8,6 +11,7 @@ if ['solo', 'util'].include?(node[:instance_role])
       source "monitrc.erb"
       variables({
         :app_name => app,
+        :pidfile  => pidfile,
         #:max_mem  => "400 MB",
       })
     end
@@ -16,9 +20,10 @@ if ['solo', 'util'].include?(node[:instance_role])
       owner 'root'
       group 'root'
       mode 0744
-      source "initd.rb"
+      source "initd.erb"
       variables({
         :app_name => app,
+        :pidfile  => pidfile,
       })
     end
 
@@ -30,6 +35,7 @@ if ['solo', 'util'].include?(node[:instance_role])
 
     execute "start-resque" do
       command %Q{/etc/init.d/#{app}_resque start}
+      creates pidfile
     end
 
     execute "ensure-resque-is-setup-with-monit" do

data/examples/chef_cookbook/templates/default/initd.erb

@@ -13,19 +13,24 @@
 ### END INIT INFO
 
 # configure these values here or in /etc/default/resque-pool
+# make sure your pidfile here matches your monitrc
 app_name="<%= @app_name %>"
+pidfile="<%= @pidfile %>"
 app_dir="/data/${app_name}/current"
 run_as_user="deploy"
-pidfile="/var/run/resque-pool-${app_name}.pid"
 sleep_time_during_restart=5
-bundler="bundle"
+stop_schedule="QUIT/30/INT/10/KILL/5"
+bundler="/usr/bin/bundle"
 environment="production"
 stdout_log="${app_dir}/log/resque-pool-${app_name}.stdout.log"
 stderr_log="${app_dir}/log/resque-pool-${app_name}.stderr.log"
 
 # override above values, and set any other env variables you need
-if [ -f /etc/default/resque-pool ] ; then
-  . /etc/default/resque-pool
+if [ -f /etc/default/resque ] ; then
+  . /etc/default/resque
+fi
+if [ -f /etc/default/${app_name}_resque ] ; then
+  . /etc/default/${app_name}_resque
 fi
 
 case "$1" in
@@ -38,6 +43,9 @@ case "$1" in
       --startas ${bundler} -- exec \
       resque-pool -d -p ${pidfile} -E ${environment} -o ${stdout_log} -e ${stderr_log}
     ;;
+  reload)
+    start-stop-daemon --stop --pidfile ${pidfile} --signal HUP
+    ;;
   graceful-stop)
     start-stop-daemon --stop --pidfile ${pidfile} --signal QUIT
     ;;
@@ -45,7 +53,7 @@ case "$1" in
     start-stop-daemon --stop --pidfile ${pidfile} --signal INT
     ;;
   stop)
-    start-stop-daemon --stop --pidfile ${pidfile} --signal QUIT/30/INT/10/KILL/5
+    start-stop-daemon --stop --pidfile ${pidfile} --retry=${stop_schedule}
     ;;
   restart)
     $0 stop
@@ -53,7 +61,7 @@ case "$1" in
     $0 start
     ;;
   *)
-    echo "Usage: $0 {start|stop|graceful-stop|quick-stop|restart}"
+    echo "Usage: $0 {start|stop|graceful-stop|quick-stop|restart|reload}"
     exit 1
     ;;
 esac

data/examples/chef_cookbook/templates/default/monitrc.erb

@@ -1,6 +1,6 @@
 check process <%= @app_name %>_resque
-  with pidfile /var/run/<%= @app_name %>_resque.pid
-  start program = "/etc/init.d/<%= @app_name %>_resque start" with timeout 90 seconds as uid deploy and gid deploy
+  with pidfile <%= @pidfile %>
+  start program = "/etc/init.d/<%= @app_name %>_resque start" with timeout 90 seconds
   stop program  = "/etc/init.d/<%= @app_name %>_resque stop"  with timeout 90 seconds
   #if totalmem is greater than <%= @max_mem || "300 MB" %> for 10 cycles then restart # eating up memory?
   group resque

data/lib/resque/pool/version.rb

@@ -1,5 +1,5 @@
 module Resque
   class Pool
-    VERSION = "0.0.12.1.alpha"
+    VERSION = "0.1.0"
   end
 end

metadata

@@ -1,15 +1,13 @@
 --- !ruby/object:Gem::Specification 
 name: resque-pool
 version: !ruby/object:Gem::Version 
-  hash: -1851332310
-  prerelease: true
+  hash: 27
+  prerelease: false
   segments: 
   - 0
-  - 0
-  - 12
   - 1
-  - alpha
-  version: 0.0.12.1.alpha
+  - 0
+  version: 0.1.0
 platform: ruby
 authors: 
 - nicholas a. evans
@@ -17,7 +15,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2011-01-11 00:00:00 -05:00
+date: 2011-01-18 00:00:00 -05:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency 
@@ -118,6 +116,7 @@ extra_rdoc_files: []
 
 files: 
 - .gitignore
+- Changelog.md
 - Gemfile
 - Gemfile.lock
 - LICENSE.txt
@@ -164,14 +163,12 @@ required_ruby_version: !ruby/object:Gem::Requirement
 required_rubygems_version: !ruby/object:Gem::Requirement 
   none: false
   requirements: 
-  - - ">"
+  - - ">="
     - !ruby/object:Gem::Version 
-      hash: 25
+      hash: 3
       segments: 
-      - 1
-      - 3
-      - 1
-      version: 1.3.1
+      - 0
+      version: "0"
 requirements: []
 
 rubyforge_project: