resque-pool-vinted 0.4.0.rc1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: d55af35f8c657528df7c50b44e89ccf3eca71f61
+  data.tar.gz: 4ef8c5944dba41bec27ff0261659b500e719cc0a
+SHA512:
+  metadata.gz: c4c664515d46859ba39ed51f31bf6c72e0d87399eb01e6091efac742ad7b6a148047d8053e787483b065967dd8ebe8a52af4ab1384a8fad34b38ffc160804d00
+  data.tar.gz: 9353984aadb57f4e865d82759a9ebc2296b4435066f3283f13438f1772abb4485396eb50054e482c992dd5d2bae998a32ce2ecb9868a55f7b0b90aea61421bd7

data/Changelog.md

@@ -0,0 +1,77 @@
+## 0.4.0.dev (unreleased)
+
+ * ???
+
+## 0.3.0 (2012-05-22)
+
+This is mostly just a long overdue maintenance release.  Many pull requests were
+merged.  A few non-pull-request branches were merged too.  This version supports
+ruby 1.9.3, 1.8.7, and even ancient 1.8.6, and all are checked by
+[travis-ci](http://travis-ci.org/nevans/resque-pool).  It also explicitly
+supports resque ~> 1.20.  And (if you have `gem-man` installed), it now has man
+pages for bin and yml config.
+
+Many thanks to the contributers!
+
+ * [@agnellvj](https://github.com/agnellvj): ruby 1.9 compatibility
+ * [@geoffgarside](https://github.com/geoffgarside): man pages!
+ * [@imajes](https://github.com/imajes) - bugfix: Handle when a pid no longer
+   exists by the time you try and kill it.
+ * [@jeremy](https://github.com/jeremy) & [@jamis](https://github.com/jamis) -
+   tasks require `resque/pool` lazily
+ * [@jhsu](https://github.com/jhsu) - bugfix: undefined variable 'e' for errors
+ * [@gaffneyc](https://github.com/gaffneyc) - compatibility fix:
+   Resque::Pool::PooledWorker as a module rather than class
+ * [@kcrayon](https://github.com/kcrayon) - bugfix: fix worker shutdown
+ * [@alexkwolfe](https://github.com/alexkwolfe) - added `app_name` for logging
+   (and maybe more in the future?)
+
+## 0.2.0 (2011-03-15)
+
+* new feature: sending `HUP` to pool manager will reload the logfiles and
+  gracefully restart all workers.
+* enhancement: logging now includes timestamp, process "name" (worker or
+  manager), and PID.
+* enhancement: can be used with no config file or empty config file (not all
+  *that* useful, but it's better than unceromoniously dieing!)
+* bugfix: pidfile will be cleaned up on startup, e.g. if old process was
+  kill-9'd (Jason Haruska)
+* bugfix: TERM/INT are no longer ignored when HUP is waiting on children
+* bugfix: `resque-pool -c config.yml` command line option was broken
+* development: simple cucumber features for core functionality.
+* upstream: depends on resque ~> 1.13
+
+## 0.1.0 (2011-01-18)
+
+* new feature: `resque-pool` command line interface
+  * this replaces need for a special startup script.
+  * manages PID file, logfiles, daemonizing, etc.
+  * `resque-pool --help` for more info and options
+* updated example config, init.d script, including a chef recipe that should
+  work at EngineYard.
+
+## 0.0.10 (2010-08-31)
+
+* remove rubygems 1.3.6 dependency
+
+## 0.0.9 (2010-08-26)
+
+* new feature: `RESQUE_POOL_CONFIG` environment variable to set alt config file
+* upgraded to resque 1.10, removing `Resque::Worker` monkeypatch
+
+## 0.0.8 (2010-08-20)
+
+* bugfix: using (or not using) environments in config file
+
+## 0.0.7 (2010-08-16)
+
+* new feature: split by environments in config file
+* added example startup script, Rakefile, and monit config
+
+## 0.0.5 (2010-06-29)
+
+* bugfix: worker processes not shutting down after orphaned
+
+## 0.0.4 (2010-06-29)
+
+* first release used in production

data/LICENSE.txt

@@ -0,0 +1,20 @@
+Copyright (C) 2010 by Nicholas Evans <nick@ekenosen.net>, et al.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+

data/README.md

@@ -0,0 +1,164 @@
+Resque Pool
+===========
+
+[![Build Status](https://secure.travis-ci.org/nevans/resque-pool.png)](http://travis-ci.org/nevans/resque-pool)
+[![Dependency Status](https://gemnasium.com/nevans/resque-pool.png)](https://gemnasium.com/nevans/resque-pool)
+
+Resque pool is a simple library for managing a pool of
+[resque](http://github.com/defunkt/resque) workers.  Given a a config file, it
+manages your workers for you, starting up the appropriate number of workers for
+each worker type.
+
+Benefits
+---------
+
+* Less config - With a simple YAML file, you can start up a pool daemon, and it
+  will monitor your workers for you.  An example init.d script, monit config,
+  and chef cookbook are provided.
+* Less memory - If you are using Ruby Enterprise Edition, or any ruby with
+  copy-on-write safe garbage collection, this should save you a *lot* of memory
+  when you are managing many workers.
+* Faster startup - when you start many workers at once, they would normally
+  compete for CPU as they load their environments.  Resque-pool can load the
+  environment once and fork all of the workers almost instantly.
+
+Upgrading?
+-----------
+
+See
+[Changelog.md](https://github.com/nevans/resque-pool/blob/master/Changelog.md)
+in case there are important or helpful changes.
+
+How to use
+-----------
+
+### YAML file config
+
+Create a `config/resque-pool.yml` (or `resque-pool.yml`) with your worker
+counts.  The YAML file supports both using root level defaults as well as
+environment specific overrides (`RACK_ENV`, `RAILS_ENV`, and `RESQUE_ENV`
+environment variables can be used to determine environment).  For example in
+`config/resque-pool.yml`:
+
+    foo: 1
+    bar: 2
+    "foo,bar,baz": 1
+
+    production:
+      "foo,bar,baz": 4
+
+### Rake task config
+
+Require the rake tasks (`resque/pool/tasks`) in your `Rakefile`, load your
+application environment, configure Resque as necessary, and configure
+`resque:pool:setup` to disconnect all open files and sockets in the pool
+manager and reconnect in the workers.  For example, with rails you should put
+the following into `lib/tasks/resque.rake`:
+
+    require 'resque/pool/tasks'
+    # this task will get called before resque:pool:setup
+    # and preload the rails environment in the pool manager
+    task "resque:setup" => :environment do
+      # generic worker setup, e.g. Hoptoad for failed jobs
+    end
+    task "resque:pool:setup" do
+      # close any sockets or files in pool manager
+      ActiveRecord::Base.connection.disconnect!
+      # and re-open them in the resque worker parent
+      Resque::Pool.after_prefork do |job|
+        ActiveRecord::Base.establish_connection
+      end
+    end
+
+### Start the pool manager
+
+Then you can start the queues via:
+
+    resque-pool --daemon --environment production
+
+This will start up seven worker processes, one exclusively for the foo queue,
+two exclusively for the bar queue, and four workers looking at all queues in
+priority.  With the config above, this is similar to if you ran the following:
+
+    rake resque:work RAILS_ENV=production QUEUES=foo &
+    rake resque:work RAILS_ENV=production QUEUES=bar &
+    rake resque:work RAILS_ENV=production QUEUES=bar &
+    rake resque:work RAILS_ENV=production QUEUES=foo,bar,baz &
+    rake resque:work RAILS_ENV=production QUEUES=foo,bar,baz &
+    rake resque:work RAILS_ENV=production QUEUES=foo,bar,baz &
+    rake resque:work RAILS_ENV=production QUEUES=foo,bar,baz &
+
+The pool manager will stay around monitoring the resque worker parents, giving
+three levels: a single pool manager, many worker parents, and one worker child
+per worker (when the actual job is being processed).  For example, `ps -ef f |
+grep [r]esque` (in Linux) might return something like the following:
+
+    resque    13858     1  0 13:44 ?        S      0:02 resque-pool-manager: managing [13867, 13875, 13871, 13872, 13868, 13870, 13876]
+    resque    13867 13858  0 13:44 ?        S      0:00  \_ resque-1.9.9: Waiting for foo
+    resque    13868 13858  0 13:44 ?        S      0:00  \_ resque-1.9.9: Waiting for bar
+    resque    13870 13858  0 13:44 ?        S      0:00  \_ resque-1.9.9: Waiting for bar
+    resque    13871 13858  0 13:44 ?        S      0:00  \_ resque-1.9.9: Waiting for foo,bar,baz
+    resque    13872 13858  0 13:44 ?        S      0:00  \_ resque-1.9.9: Forked 7481 at 1280343254
+    resque     7481 13872  0 14:54 ?        S      0:00      \_ resque-1.9.9: Processing foo since 1280343254
+    resque    13875 13858  0 13:44 ?        S      0:00  \_ resque-1.9.9: Waiting for foo,bar,baz
+    resque    13876 13858  0 13:44 ?        S      0:00  \_ resque-1.9.9: Forked 7485 at 1280343255
+    resque     7485 13876  0 14:54 ?        S      0:00      \_ resque-1.9.9: Processing bar since 1280343254
+
+Running as a daemon will default to placing the pidfile and logfiles in the
+conventional rails locations, although you can configure that.  See
+`resque-pool --help` for more options.
+
+SIGNALS
+-------
+
+The pool manager responds to the following signals:
+
+* `HUP`   - reload the config file, reload logfiles, restart all workers.
+* `QUIT`  - gracefully shut down workers (via `QUIT`) and shutdown the manager
+  after all workers are done.
+* `INT`   - gracefully shut down workers (via `QUIT`) and immediately shutdown manager
+* `TERM`  - immediately shut down workers (via `INT`) and immediately shutdown manager
+  _(configurable via command line options)_
+* `WINCH` - _(only when running as a daemon)_ send `QUIT` to each worker, but
+  keep manager running (send `HUP` to reload config and restart workers)
+* `USR1`/`USR2`/`CONT` - pass the signal on to all worker parents (see Resque docs).
+
+Use `HUP` to help logrotate run smoothly and to change the number of workers
+per worker type.  Signals can be sent via the `kill` command, e.g.
+`kill -HUP $master_pid`
+
+If the environment variable `TERM_CHILD` is set, `QUIT` and `TERM` will respond as
+defined by Resque 1.22 and above. See http://hone.heroku.com/resque/2012/08/21/resque-signals.html
+for details, overriding any command-line configuration for `TERM`. Setting `TERM_CHILD` tells
+us you know what you're doing.
+
+Other Features
+--------------
+
+An example chef cookbook is provided (and should Just Work at Engine Yard as
+is; just provide a `/data/#{app_name}/shared/config/resque-pool.yml` on your
+utility instances).  Even if you don't use chef you can use the example init.d
+and monitrc erb templates in `examples/chef_cookbook/templates/default`.
+
+You can also start a pool manager via `rake resque:pool` or from a plain old
+ruby script by calling `Resque::Pool.run`.
+
+Workers will watch the pool manager, and gracefully shutdown (after completing
+their current job) if the manager process disappears before them.
+
+You can specify an alternate config file by setting the `RESQUE_POOL_CONFIG` or
+with the `--config` command line option.
+
+TODO
+-----
+
+See [the TODO list](https://github.com/nevans/resque-pool/issues) at github issues.
+
+Contributors
+-------------
+
+* John Schult (config file can be split by environment)
+* Stephen Celis (increased gemspec sanity)
+* Vincent Agnello, Robert Kamunyori, Paul Kauders; for pairing with me at
+  B'more on Rails Open Source Hack Nights. :)
+

data/Rakefile

@@ -0,0 +1,30 @@
+require 'bundler'
+Bundler::GemHelper.install_tasks
+
+# for loading the example config file in config/resque-pool.yml
+require 'resque/pool/tasks'
+
+require 'rspec/core/rake_task'
+RSpec::Core::RakeTask.new(:spec) do |t|
+  t.rspec_opts = ["-c", "-f progress"]
+end
+
+require 'cucumber/rake/task'
+Cucumber::Rake::Task.new(:features) do |c|
+  c.profile = "rake"
+end
+
+task :default => [:spec, :features]
+
+rule(/\.[1-9]$/ => [proc { |tn| "#{tn}.ronn" }]) do |t|
+  name = Resque::Pool.name.sub('::','-').upcase
+  version = "%s %s" % [name, Resque::Pool::VERSION.upcase]
+
+  manual = '--manual "%s"' % name
+  organization = '--organization "%s"' % version
+  sh "ronn #{manual} #{organization} <#{t.source} >#{t.name}"
+end
+
+file 'man/resque-pool.1'
+file 'man/resque-pool.yml.5'
+task :manpages => ['man/resque-pool.1','man/resque-pool.yml.5']

data/bin/resque-pool

@@ -0,0 +1,7 @@
+#!/usr/bin/env ruby
+# -*- encoding: utf-8 -*-
+
+$LOAD_PATH.unshift File.expand_path('../../lib', __FILE__)
+
+require 'resque/pool/cli'
+Resque::Pool::CLI.run

data/features/basic_daemon_config.feature

@@ -0,0 +1,68 @@
+Feature: Basic resque-pool daemon configuration and operation
+  To easily manage a pool of resque workers, resque-pool provides a daemon with
+  simple configuration.  Static configuration is handled in the
+  config/config.yml file and dynamic configuration is handled in the Rakefile.
+
+  Background:
+    Given a file named "Rakefile" with:
+    """
+    require 'resque/pool/tasks'
+    """
+
+  Scenario: no config file
+    When I run the pool manager as "resque-pool"
+    Then the pool manager should report that it has started up
+    And the pool manager should report that the pool is empty
+    And the pool manager should have no child processes
+    When I send the pool manager the "QUIT" signal
+    Then the pool manager should finish
+    And the pool manager should report that it is finished
+
+  @slow_exit
+  Scenario: basic config file
+    Given a file named "config/resque-pool.yml" with:
+    """
+    foo: 1
+    bar: 2
+    "bar,baz": 3
+    """
+    When I run the pool manager as "resque-pool"
+    Then the pool manager should report that it has started up
+    And the pool manager should report that 6 workers are in the pool
+    And the pool manager should have 1 "foo" worker child processes
+    And the pool manager should have 2 "bar" worker child processes
+    And the pool manager should have 3 "bar,baz" worker child processes
+    When I send the pool manager the "QUIT" signal
+    Then the resque workers should all shutdown
+    And the pool manager should finish
+    And the pool manager should report that a "foo" worker has been reaped
+    And the pool manager should report that a "bar" worker has been reaped
+    And the pool manager should report that a "bar,baz" worker has been reaped
+    And the pool manager should report that it is finished
+
+  Scenario: daemonized
+    Given a directory named "log"
+    And a directory named "tmp/pids"
+    And a file named "config/resque-pool.yml" with:
+    """
+    foo: 2
+    bar: 4
+    "baz,quux": 4
+    """
+    When I run the pool manager as "resque-pool -d"
+    Then the pool manager should record its pid in "tmp/pids/resque-pool.pid"
+    And the pool manager should daemonize
+    And a file named "log/resque-pool.stdout.log" should exist
+    And a file named "log/resque-pool.stderr.log" should exist
+    And the pool manager should log that it has started up
+    And the pool manager should log that 10 workers are in the pool
+    And the pool manager should have 2 "foo" worker child processes
+    And the pool manager should have 4 "bar" worker child processes
+    And the pool manager should have 4 "baz,quux" worker child processes
+    When I send the pool manager the "QUIT" signal
+    Then the resque workers should all shutdown
+    And the pool manager daemon should finish
+    And the pool manager should log that a "foo" worker has been reaped
+    And the pool manager should log that a "bar" worker has been reaped
+    And the pool manager should log that a "baz,quux" worker has been reaped
+    And the pool manager should log that it is finished

data/features/step_definitions/daemon_steps.rb

@@ -0,0 +1,33 @@
+# syntactic sugar, and separate ivar.  daemons aren't interactive
+When /^I run "([^"]*)" in the background$/ do |cmd|
+  run_background(unescape(cmd))
+end
+
+Then /^the (output|logfiles) should contain the following lines \(with interpolated \$PID\):$/ do |output_logfiles, partial_output|
+  interpolate_background_pid(partial_output).split("\n").each do |line|
+    output_or_log(output_logfiles).should include(line)
+  end
+end
+
+When /^I send "([^"]*)" the "([^"]*)" signal$/ do |cmd, signal|
+  send_signal(cmd, signal)
+end
+
+Then /^the "([^"]*)" process should finish$/ do |cmd|
+  # doesn't actually stop... just polls for exit
+  processes[cmd].stop
+end
+
+Before("@slow_exit") do
+  @aruba_timeout_seconds = 10
+end
+
+After do
+  kill_all_processes!
+  # now kill the daemon!
+  begin
+    Process.kill(9, @pid_from_pidfile) if @pid_from_pidfile
+  rescue Errno::ESRCH
+  end
+  #`pkill -9 resque-pool`
+end

data/features/step_definitions/resque-pool_steps.rb

@@ -0,0 +1,159 @@
+def process_should_exist(pid)
+  lambda { Process.kill(0, pid) }.should_not raise_error(Errno::ESRCH)
+end
+
+def process_should_not_exist(pid)
+  lambda { Process.kill(0, pid) }.should raise_error(Errno::ESRCH)
+end
+
+def grab_worker_pids(count, str)
+  puts "TODO: check output_or_log for #{count} worker started messages"
+  pid_regex = (1..count).map { '(\d+)' }.join ', '
+  full_regex = /resque-pool-manager\[aruba\]\[\d+\]: Pool contains worker PIDs: \[#{pid_regex}\]/m
+  str.should =~ full_regex
+  @worker_pids = full_regex.match(str).captures.map {|pid| pid.to_i }
+end
+
+def output_or_logfiles_string(report_log)
+  case report_log
+  when "report", "output"
+    "output"
+  when "log", "logfiles"
+    "logfiles"
+  else
+    raise ArgumentError
+  end
+end
+
+def output_or_log(report_log)
+  case report_log
+  when "report", "output"
+    interactive_output
+  when "log", "logfiles"
+    in_current_dir do
+      File.read("log/resque-pool.stdout.log") << File.read("log/resque-pool.stderr.log")
+    end
+  else
+    raise ArgumentError
+  end
+end
+
+class NotFinishedStarting < StandardError; end
+def worker_processes_for(queues)
+  children_of(background_pid).select do |pid, cmd|
+    raise NotFinishedStarting if cmd =~ /Starting$/
+    cmd =~ /^resque-\d+.\d+.\d+: Waiting for #{queues}$/
+  end
+rescue NotFinishedStarting
+  retry
+end
+
+def children_of(ppid)
+  if RUBY_PLATFORM =~ /darwin/i
+    ps = `ps -eo ppid,pid,comm | grep '^ *#{ppid} '`
+  else
+    ps = `ps -eo ppid,pid,cmd | grep '^ *#{ppid} '`
+  end
+  ps.split(/\s*\n/).map do |line|
+    _, pid, cmd = line.strip.split(/\s+/, 3)
+    [pid, cmd]
+  end
+end
+
+When /^I run the pool manager as "([^"]*)"$/ do |cmd|
+  @pool_manager_process = run_background(unescape(cmd))
+end
+
+When /^I send the pool manager the "([^"]*)" signal$/ do |signal|
+  Process.kill signal, background_pid
+  output_logfiles = @pid_from_pidfile ? "logfiles" : "output"
+  case signal
+  when "QUIT"
+    keep_trying do
+      step "the #{output_logfiles} should contain the following lines (with interpolated $PID):", <<-EOF
+resque-pool-manager[aruba][$PID]: QUIT: graceful shutdown, waiting for children
+      EOF
+    end
+  else
+    raise ArgumentError
+  end
+end
+
+Then /^the pool manager should record its pid in "([^"]*)"$/ do |pidfile|
+  in_current_dir do
+    keep_trying do
+      File.should be_file(pidfile)
+      @pid_from_pidfile = File.read(pidfile).to_i
+      @pid_from_pidfile.should_not == 0
+      process_should_exist(@pid_from_pidfile)
+    end
+  end
+end
+
+Then /^the pool manager should daemonize$/ do
+  stop_processes!
+end
+
+Then /^the pool manager daemon should finish$/ do
+  keep_trying do
+    process_should_not_exist(@pid_from_pidfile)
+  end
+end
+
+# nomenclature: "report" => output to stdout/stderr
+#               "log"    => output to default logfile
+
+Then /^the pool manager should (report|log) that it has started up$/ do |report_log|
+  keep_trying do
+    step "the #{output_or_logfiles_string(report_log)} should contain the following lines (with interpolated $PID):", <<-EOF
+resque-pool-manager[aruba][$PID]: Resque Pool running in test environment
+resque-pool-manager[aruba][$PID]: started manager
+    EOF
+  end
+end
+
+Then /^the pool manager should (report|log) that the pool is empty$/ do |report_log|
+  step "the #{output_or_logfiles_string(report_log)} should contain the following lines (with interpolated $PID):", <<-EOF
+resque-pool-manager[aruba][$PID]: Pool is empty
+  EOF
+end
+
+Then /^the pool manager should (report|log) that (\d+) workers are in the pool$/ do |report_log, count|
+  grab_worker_pids Integer(count), output_or_log(report_log)
+end
+
+Then /^the resque workers should all shutdown$/ do
+  @worker_pids.each do |pid|
+    keep_trying do
+      process_should_not_exist(pid)
+    end
+  end
+end
+
+Then "the pool manager should have no child processes" do
+  children_of(background_pid).should have(:no).keys
+end
+
+Then /^the pool manager should have (\d+) "([^"]*)" worker child processes$/ do |count, queues|
+  worker_processes_for(queues).should have(Integer(count)).members
+end
+
+Then "the pool manager should finish" do
+  # assuming there will not be multiple processes running
+  stop_processes!
+end
+
+Then /^the pool manager should (report|log) that it is finished$/ do |report_log|
+  step "the #{output_or_logfiles_string(report_log)} should contain the following lines (with interpolated $PID):", <<-EOF
+resque-pool-manager[aruba][$PID]: manager finished
+  EOF
+end
+
+Then /^the pool manager should (report|log) that a "([^"]*)" worker has been reaped$/ do |report_log, worker_type|
+  step 'the '+ output_or_logfiles_string(report_log) +' should match /Reaped resque worker\[\d+\] \(status: 0\) queues: '+ worker_type + '/'
+end
+
+Then /^the logfiles should match \/([^\/]*)\/$/ do |partial_output|
+  output_or_log("log").should =~ /#{partial_output}/
+end
+