vitobotta-resque-pool 0.3.0.dev

data/Changelog.md

@@ -0,0 +1,49 @@
+## 0.2.0 (2011-03-15)
+
+* new feature: sending `HUP` to pool manager will reload the logfiles and
+  gracefully restart all workers.
+* enhancement: logging now includes timestamp, process "name" (worker or
+  manager), and PID.
+* enhancement: can be used with no config file or empty config file (not all
+  *that* useful, but it's better than unceromoniously dieing!)
+* bugfix: pidfile will be cleaned up on startup, e.g. if old process was
+  kill-9'd (Jason Haruska)
+* bugfix: TERM/INT are no longer ignored when HUP is waiting on children
+* bugfix: `resque-pool -c config.yml` command line option was broken
+* development: simple cucumber features for core functionality.
+* upstream: depends on resque ~> 1.13
+
+## 0.1.0 (2011-01-18)
+
+* new feature: `resque-pool` command line interface
+  * this replaces need for a special startup script.
+  * manages PID file, logfiles, daemonizing, etc.
+  * `resque-pool --help` for more info and options
+* updated example config, init.d script, including a chef recipe that should
+  work at EngineYard.
+
+## 0.0.10 (2010-08-31)
+
+* remove rubygems 1.3.6 dependency
+
+## 0.0.9 (2010-08-26)
+
+* new feature: `RESQUE_POOL_CONFIG` environment variable to set alt config file
+* upgraded to resque 1.10, removing `Resque::Worker` monkeypatch
+
+## 0.0.8 (2010-08-20)
+
+* bugfix: using (or not using) environments in config file
+
+## 0.0.7 (2010-08-16)
+
+* new feature: split by environments in config file
+* added example startup script, Rakefile, and monit config
+
+## 0.0.5 (2010-06-29)
+
+* bugfix: worker processes not shutting down after orphaned
+
+## 0.0.4 (2010-06-29)
+
+* first release used in production

data/Gemfile

@@ -0,0 +1,4 @@
+source "http://rubygems.org"
+
+# Specify your gem's dependencies in resque-pool.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,20 @@
+Copyright (C) 2010 by Nicholas Evans <nick@ekenosen.net>, et al.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+

data/README.md

@@ -0,0 +1,153 @@
+Resque Pool
+===========
+
+Resque pool is a simple library for managing a pool of
+[resque](http://github.com/defunkt/resque) workers.  Given a a config file, it
+manages your workers for you, starting up the appropriate number of workers for
+each worker type.
+
+Benefits
+---------
+
+* Less config - With a simple YAML file, you can start up a pool daemon, and it
+  will monitor your workers for you.  An example init.d script, monit config,
+  and chef cookbook are provided.
+* Less memory - If you are using Ruby Enterprise Edition, or any ruby with
+  copy-on-write safe garbage collection, this should save you a *lot* of memory
+  when you are managing many workers.
+* Faster startup - when you start many workers at once, they would normally
+  compete for CPU as they load their environments.  Resque-pool can load the
+  environment once and fork all of the workers almost instantly.
+
+Upgrading?
+-----------
+
+See
+[Changelog.md](https://github.com/nevans/resque-pool/blob/master/Changelog.md)
+in case there are important or helpful changes.
+
+How to use
+-----------
+
+### YAML file config
+
+Create a `config/resque-pool.yml` (or `resque-pool.yml`) with your worker
+counts.  The YAML file supports both using root level defaults as well as
+environment specific overrides (`RACK_ENV`, `RAILS_ENV`, and `RESQUE_ENV`
+environment variables can be used to determine environment).  For example in
+`config/resque-pool.yml`:
+
+    foo: 1
+    bar: 2
+    "foo,bar,baz": 1
+
+    production:
+      "foo,bar,baz": 4
+
+### Rake task config
+
+Require the rake tasks (`resque/pool/tasks`) in your `Rakefile`, load your
+application environment, configure Resque as necessary, and configure
+`resque:pool:setup` to disconnect all open files and sockets in the pool
+manager and reconnect in the workers.  For example, with rails you should put
+the following into `lib/tasks/resque.rake`:
+
+    require 'resque/pool/tasks'
+    # this task will get called before resque:pool:setup
+    # and preload the rails environment in the pool manager
+    task "resque:setup" => :environment do
+      # generic worker setup, e.g. Hoptoad for failed jobs
+    end
+    task "resque:pool:setup" do
+      # close any sockets or files in pool manager
+      ActiveRecord::Base.connection.disconnect!
+      # and re-open them in the resque worker parent
+      Resque::Pool.after_prefork do |job|
+        ActiveRecord::Base.establish_connection
+      end
+    end
+
+### Start the pool manager
+
+Then you can start the queues via:
+
+    resque-pool --daemon --environment production
+
+This will start up seven worker processes, one exclusively for the foo queue,
+two exclusively for the bar queue, and four workers looking at all queues in
+priority.  With the config above, this is similar to if you ran the following:
+
+    rake resque:work RAILS_ENV=production QUEUES=foo &
+    rake resque:work RAILS_ENV=production QUEUES=bar &
+    rake resque:work RAILS_ENV=production QUEUES=bar &
+    rake resque:work RAILS_ENV=production QUEUES=foo,bar,baz &
+    rake resque:work RAILS_ENV=production QUEUES=foo,bar,baz &
+    rake resque:work RAILS_ENV=production QUEUES=foo,bar,baz &
+    rake resque:work RAILS_ENV=production QUEUES=foo,bar,baz &
+
+The pool manager will stay around monitoring the resque worker parents, giving
+three levels: a single pool manager, many worker parents, and one worker child
+per worker (when the actual job is being processed).  For example, `ps -ef f |
+grep [r]esque` (in Linux) might return something like the following:
+
+    resque    13858     1  0 13:44 ?        S      0:02 resque-pool-manager: managing [13867, 13875, 13871, 13872, 13868, 13870, 13876]
+    resque    13867 13858  0 13:44 ?        S      0:00  \_ resque-1.9.9: Waiting for foo
+    resque    13868 13858  0 13:44 ?        S      0:00  \_ resque-1.9.9: Waiting for bar
+    resque    13870 13858  0 13:44 ?        S      0:00  \_ resque-1.9.9: Waiting for bar
+    resque    13871 13858  0 13:44 ?        S      0:00  \_ resque-1.9.9: Waiting for foo,bar,baz
+    resque    13872 13858  0 13:44 ?        S      0:00  \_ resque-1.9.9: Forked 7481 at 1280343254
+    resque     7481 13872  0 14:54 ?        S      0:00      \_ resque-1.9.9: Processing foo since 1280343254
+    resque    13875 13858  0 13:44 ?        S      0:00  \_ resque-1.9.9: Waiting for foo,bar,baz
+    resque    13876 13858  0 13:44 ?        S      0:00  \_ resque-1.9.9: Forked 7485 at 1280343255
+    resque     7485 13876  0 14:54 ?        S      0:00      \_ resque-1.9.9: Processing bar since 1280343254
+
+Running as a daemon will default to placing the pidfile and logfiles in the
+conventional rails locations, although you can configure that.  See
+`resque-pool --help` for more options.
+
+SIGNALS
+-------
+
+The pool manager responds to the following signals:
+
+* `HUP`   - reload the config file, reload logfiles, restart all workers.
+* `QUIT`  - send `QUIT` to each worker parent and shutdown the manager after all workers are done.
+* `INT`   - send `QUIT` to each worker parent and immediately shutdown manager
+* `TERM`  - send `TERM` to each worker parent and immediately shutdown manager
+* `WINCH` - send `QUIT` to each worker, but keep manager running (send `HUP` to reload config and restart workers)
+* `USR1`/`USR2`/`CONT` - pass the signal on to all worker parents (see Resque docs).
+
+Use `HUP` to help logrotate run smoothly and to change the number of workers
+per worker type.
+
+Other Features
+--------------
+
+An example chef cookbook is provided (and should Just Work at Engine Yard as
+is; just provide a `/data/#{app_name}/shared/config/resque-pool.yml` on your
+utility instances).  Even if you don't use chef you can use the example init.d
+and monitrc erb templates in `examples/chef_cookbook/templates/default`.
+
+You can also start a pool manager via `rake resque:pool:start` or from a plain
+old ruby script by calling `Resque::Pool.run`.
+
+You can stop a pool manager via `rake resque:pool:stop`.
+
+Workers will watch the pool manager, and gracefully shutdown (after completing
+their current job) if the manager process disappears before them.
+
+You can specify an alternate config file by setting the `RESQUE_POOL_CONFIG` or
+with the `--config` command line option.
+
+TODO
+-----
+
+See [the TODO list](https://github.com/nevans/resque-pool/issues) at github issues.
+
+Contributors
+-------------
+
+* John Schult (config file can be split by environment)
+* Stephen Celis (increased gemspec sanity)
+* Vincent Agnello, Robert Kamunyori, Paul Kauders; for pairing with me at
+  B'more on Rails Open Source Hack Nights. :)

data/Rakefile

@@ -0,0 +1,17 @@
+require 'bundler'
+Bundler::GemHelper.install_tasks
+
+# for loading the example config file in config/resque-pool.yml
+require 'resque/pool/tasks'
+
+require 'rspec/core/rake_task'
+RSpec::Core::RakeTask.new(:spec) do |t|
+  t.rspec_opts = ["-c", "-f progress"]
+end
+
+require 'cucumber/rake/task'
+Cucumber::Rake::Task.new(:features) do |c|
+  c.profile = "rake"
+end
+
+task :default => [:spec, :features]

data/bin/resque-pool

@@ -0,0 +1,5 @@
+#!/usr/bin/env ruby
+# -*- encoding: utf-8 -*-
+
+require 'resque/pool/cli'
+Resque::Pool::CLI.run

data/config/alternate.yml

@@ -0,0 +1,4 @@
+foo: 37
+bar: 42
+baz: 3
+quux: 7

data/config/cucumber.yml

@@ -0,0 +1,3 @@
+default: --format pretty   --strict --require features --tags ~@wip --tags ~@pending
+rake:    --format progress --strict --require features --tags ~@wip --tags ~@pending
+

data/examples/Gemfile

@@ -0,0 +1 @@
+gemspec :path => ".."

data/examples/Gemfile.lock

@@ -0,0 +1,67 @@
+PATH
+  remote: /home/nick/src/active/resque-pool
+  specs:
+    resque-pool (0.1.0.dev)
+      rake
+      resque (~> 1.13)
+      trollop (~> 1.16)
+
+GEM
+  specs:
+    SystemTimer (1.2.2)
+    aruba (0.3.2)
+      childprocess (~> 0.1.6)
+      cucumber (~> 0.10.0)
+      rspec (~> 2.3.0)
+    builder (3.0.0)
+    childprocess (0.1.7)
+      ffi (~> 0.6.3)
+    cucumber (0.10.0)
+      builder (>= 2.1.2)
+      diff-lcs (~> 1.1.2)
+      gherkin (~> 2.3.2)
+      json (~> 1.4.6)
+      term-ansicolor (~> 1.0.5)
+    diff-lcs (1.1.2)
+    ffi (0.6.3)
+      rake (>= 0.8.7)
+    gherkin (2.3.3)
+      json (~> 1.4.6)
+    json (1.4.6)
+    rack (1.2.1)
+    rake (0.8.7)
+    redis (2.1.1)
+    redis-namespace (0.10.0)
+      redis (< 3.0.0)
+    resque (1.13.0)
+      json (~> 1.4.6)
+      redis-namespace (>= 0.10.0)
+      sinatra (>= 0.9.2)
+      vegas (~> 0.1.2)
+    rspec (2.3.0)
+      rspec-core (~> 2.3.0)
+      rspec-expectations (~> 2.3.0)
+      rspec-mocks (~> 2.3.0)
+    rspec-core (2.3.1)
+    rspec-expectations (2.3.0)
+      diff-lcs (~> 1.1.2)
+    rspec-mocks (2.3.0)
+    sinatra (1.1.3)
+      rack (~> 1.1)
+      tilt (>= 1.2.2, < 2.0)
+    term-ansicolor (1.0.5)
+    tilt (1.2.2)
+    trollop (1.16.2)
+    vegas (0.1.8)
+      rack (>= 1.0.0)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  SystemTimer
+  aruba (~> 0.3.2)
+  bundler (~> 1.0)
+  cucumber (~> 0.10.0)
+  resque-pool!
+  rspec (~> 2.3.0)

data/examples/Rakefile

@@ -0,0 +1,8 @@
+# If you have redis running on the standard port on localhost, then you can
+# test out resque-pool in the examples directory by running:
+#
+#    rake --trace resque:pool
+#
+# And then you can poke and prod the various processes with signals and edit
+# the config file, etc.
+require 'resque/pool/tasks'

data/examples/chef_cookbook/recipes/default.rb

@@ -0,0 +1,46 @@
+roles = %w[solo util]
+if roles.include?(node[:instance_role])
+  node[:applications].each do |app, data|
+
+    pidfile = "/data/#{app}/current/tmp/pids/#{app}_resque.pid"
+
+    template "/etc/monit.d/#{app}_resque.monitrc" do
+      owner 'root'
+      group 'root'
+      mode 0644
+      source "monitrc.erb"
+      variables({
+        :app_name => app,
+        :pidfile  => pidfile,
+        #:max_mem  => "400 MB",
+      })
+    end
+
+    template "/etc/init.d/#{app}_resque" do
+      owner 'root'
+      group 'root'
+      mode 0744
+      source "initd.erb"
+      variables({
+        :app_name => app,
+        :pidfile  => pidfile,
+      })
+    end
+
+    execute "enable-resque" do
+      command "rc-update add #{app}_resque default"
+      action :run
+      not_if "rc-update show | grep -q '^ *#{app}_resque |.*default'"
+    end
+
+    execute "start-resque" do
+      command %Q{/etc/init.d/#{app}_resque start}
+      creates pidfile
+    end
+
+    execute "ensure-resque-is-setup-with-monit" do
+      command %Q{monit reload}
+    end
+
+  end
+end

data/examples/chef_cookbook/templates/default/initd.erb

@@ -0,0 +1,69 @@
+#!/bin/sh -e
+# Copyright (C) nicholas a. evans <nick@ekenosen.net>
+# See LICENSE.txt
+
+### BEGIN INIT INFO
+# Provides:          resque-pool-<%= @app_name %>
+# Required-Start:    $local_fs $remote_fs $network
+# Required-Stop:     $local_fs $remote_fs $network
+# Default-Start:     2 3 4 5
+# Default-Stop:      0 1 6
+# Short-Description: resque-pool init script for <%= @app_name %>
+# Description:       resque-pool manages the resque workers
+### END INIT INFO
+
+# configure these values here or in /etc/default/resque-pool
+# make sure your pidfile here matches your monitrc
+app_name="<%= @app_name %>"
+pidfile="<%= @pidfile %>"
+app_dir="/data/${app_name}/current"
+run_as_user="deploy"
+sleep_time_during_restart=5
+stop_schedule="QUIT/30/INT/10/KILL/5"
+bundler="/usr/bin/bundle"
+environment="production"
+stdout_log="${app_dir}/log/resque-pool-${app_name}.stdout.log"
+stderr_log="${app_dir}/log/resque-pool-${app_name}.stderr.log"
+
+# override above values, and set any other env variables you need
+if [ -f /etc/default/resque ] ; then
+  . /etc/default/resque
+fi
+if [ -f /etc/default/${app_name}_resque ] ; then
+  . /etc/default/${app_name}_resque
+fi
+
+case "$1" in
+  start)
+    # I'm assuming that you are using bundler.  If you are using rip or
+    # something else, you'll need to change this.  Remember to
+    # keep the double-dash; e.g.: --startas CMD -- ARGS
+    start-stop-daemon --start --pidfile ${pidfile} \
+      --chuid ${run_as_user} --chdir ${app_dir} \
+      --startas ${bundler} -- exec \
+      resque-pool -d -p ${pidfile} -E ${environment} -o ${stdout_log} -e ${stderr_log}
+    ;;
+  reload)
+    start-stop-daemon --stop --pidfile ${pidfile} --signal HUP
+    ;;
+  graceful-stop)
+    start-stop-daemon --stop --pidfile ${pidfile} --signal QUIT
+    ;;
+  quick-stop)
+    start-stop-daemon --stop --pidfile ${pidfile} --signal INT
+    ;;
+  stop)
+    start-stop-daemon --stop --pidfile ${pidfile} --retry=${stop_schedule}
+    ;;
+  restart)
+    $0 stop
+    sleep ${sleep_time_during_restart}
+    $0 start
+    ;;
+  *)
+    echo "Usage: $0 {start|stop|graceful-stop|quick-stop|restart|reload}"
+    exit 1
+    ;;
+esac
+
+# vim:ft=sh

data/examples/chef_cookbook/templates/default/monitrc.erb

@@ -0,0 +1,6 @@
+check process <%= @app_name %>_resque
+  with pidfile <%= @pidfile %>
+  start program = "/etc/init.d/<%= @app_name %>_resque start" with timeout 90 seconds
+  stop program  = "/etc/init.d/<%= @app_name %>_resque stop"  with timeout 90 seconds
+  #if totalmem is greater than <%= @max_mem || "300 MB" %> for 10 cycles then restart # eating up memory?
+  group resque

data/examples/rails-resque.rake

@@ -0,0 +1,22 @@
+require 'resque/pool/tasks'
+
+# this task will get called before resque:pool:setup
+# preload the rails environment in the pool master
+task "resque:setup" => :environment do
+  # generic worker setup, e.g. Hoptoad for failed jobs
+end
+
+task "resque:pool:setup" do
+  # close any sockets or files in pool master
+  ActiveRecord::Base.connection.disconnect!
+
+  # and re-open them in the resque worker parent
+  Resque::Pool.after_prefork do |job|
+    ActiveRecord::Base.establish_connection
+  end
+
+  # you could also re-open them in the resque worker child, using
+  # Resque.after_fork, but that probably isn't necessary, and
+  # Resque::Pool.after_prefork should be faster, since it won't run
+  # for every single job.
+end

data/examples/resque-pool.yml

@@ -0,0 +1,4 @@
+foo: 1
+bar: 2
+"foo,bar,baz": 4
+

data/features/basic_daemon_config.feature

@@ -0,0 +1,68 @@
+Feature: Basic resque-pool daemon configuration and operation
+  To easily manage a pool of resque workers, resque-pool provides a daemon with
+  simple configuration.  Static configuration is handled in the
+  config/config.yml file and dynamic configuration is handled in the Rakefile.
+
+  Background:
+    Given a file named "Rakefile" with:
+    """
+    require 'resque/pool/tasks'
+    """
+
+  Scenario: no config file
+    When I run the pool manager as "resque-pool"
+    Then the pool manager should report that it has started up
+    And the pool manager should report that the pool is empty
+    And the pool manager should have no child processes
+    When I send the pool manager the "QUIT" signal
+    Then the pool manager should finish
+    And the pool manager should report that it is finished
+
+  @slow_exit
+  Scenario: basic config file
+    Given a file named "config/resque-pool.yml" with:
+    """
+    foo: 1
+    bar: 2
+    "bar,baz": 3
+    """
+    When I run the pool manager as "resque-pool"
+    Then the pool manager should report that it has started up
+    And the pool manager should report that 6 workers are in the pool
+    And the pool manager should have 1 "foo" worker child processes
+    And the pool manager should have 2 "bar" worker child processes
+    And the pool manager should have 3 "bar,baz" worker child processes
+    When I send the pool manager the "QUIT" signal
+    Then the resque workers should all shutdown
+    And the pool manager should finish
+    And the pool manager should report that a "foo" worker has been reaped
+    And the pool manager should report that a "bar" worker has been reaped
+    And the pool manager should report that a "bar,baz" worker has been reaped
+    And the pool manager should report that it is finished
+
+  Scenario: daemonized
+    Given a directory named "log"
+    And a directory named "tmp/pids"
+    And a file named "config/resque-pool.yml" with:
+    """
+    foo: 2
+    bar: 4
+    "baz,quux": 4
+    """
+    When I run the pool manager as "resque-pool -d"
+    Then the pool manager should record its pid in "tmp/pids/resque-pool.pid"
+    And the pool manager should daemonize
+    And a file named "log/resque-pool.stdout.log" should exist
+    And a file named "log/resque-pool.stderr.log" should exist
+    And the pool manager should log that it has started up
+    And the pool manager should log that 10 workers are in the pool
+    And the pool manager should have 2 "foo" worker child processes
+    And the pool manager should have 4 "bar" worker child processes
+    And the pool manager should have 4 "baz,quux" worker child processes
+    When I send the pool manager the "QUIT" signal
+    Then the resque workers should all shutdown
+    And the pool manager daemon should finish
+    And the pool manager should log that a "foo" worker has been reaped
+    And the pool manager should log that a "bar" worker has been reaped
+    And the pool manager should log that a "baz,quux" worker has been reaped
+    And the pool manager should log that it is finished

data/features/step_definitions/daemon_steps.rb

@@ -0,0 +1,29 @@
+# syntactic sugar, and separate ivar.  daemons aren't interactive
+When /^I run "([^"]*)" in the background$/ do |cmd|
+  run_background(unescape(cmd))
+end
+
+Then /^the (output|logfiles) should contain the following lines \(with interpolated \$PID\):$/ do |output_logfiles, partial_output|
+  interpolate_background_pid(partial_output).split("\n").each do |line|
+    output_or_log(output_logfiles).should include(line)
+  end
+end
+
+When /^I send "([^"]*)" the "([^"]*)" signal$/ do |cmd, signal|
+  send_signal(cmd, signal)
+end
+
+Then /^the "([^"]*)" process should finish$/ do |cmd|
+  # doesn't actually stop... just polls for exit
+  processes[cmd].stop
+end
+
+Before("@slow_exit") do
+  @aruba_timeout_seconds = 10
+end
+
+After do
+  kill_all_processes!
+  # now kill the daemon!
+  `pkill -9 -f resque-pool`
+end