qless-pool 0.1.0

data/Changelog.md

@@ -0,0 +1,3 @@
+## 0.1.0 (2013-06-21)
+
+* initial version forked from [resque-pool](https://github.com/nevans/resque-pool) (specifically from a [backupify fork of it](https://github.com/backupify/resque-pool/tree/maintain_count)

data/LICENSE.txt

@@ -0,0 +1,20 @@
+Copyright (C) 2010 by Nicholas Evans <nick@ekenosen.net>, et al.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+

data/README.md

@@ -0,0 +1,159 @@
+Qless Pool
+===========
+
+[![Build Status](https://secure.travis-ci.org/wr0ngway/qless-pool.png)](http://travis-ci.org/wr0ngway/qless-pool)
+
+Qless pool is a simple library for managing a pool of
+[qless](http://github.com/seomoz/qless) workers.  Given a a config file, it
+manages your workers for you, starting up the appropriate number of workers for
+each worker type.  This is a fork of [resque-pool](https://github.com/nevans/resque-pool) with some commits from a [backupify fork of it](https://github.com/backupify/resque-pool/tree/maintain_count)
+
+Benefits
+---------
+
+* Less config - With a simple YAML file, you can start up a pool daemon, and it
+  will monitor your workers for you.  An example init.d script, monit config,
+  and chef cookbook are provided.
+* Less memory - If you are using Ruby Enterprise Edition, or any ruby with
+  copy-on-write safe garbage collection, this should save you a *lot* of memory
+  when you are managing many workers.
+* Faster startup - when you start many workers at once, they would normally
+  compete for CPU as they load their environments.  Qless-pool can load the
+  environment once and fork all of the workers almost instantly.
+
+Upgrading?
+-----------
+
+See
+[Changelog.md](https://github.com/wr0ngway/qless-pool/blob/master/Changelog.md)
+in case there are important or helpful changes.
+
+How to use
+-----------
+
+### YAML file config
+
+Create a `config/qless-pool.yml` (or `qless-pool.yml`) with your worker
+counts.  The YAML file supports both using root level defaults as well as
+environment specific overrides (`RACK_ENV`, `RAILS_ENV`, and `QLESS_ENV`
+environment variables can be used to determine environment).  For example in
+`config/qless-pool.yml`:
+
+    foo: 1
+    bar: 2
+    "foo,bar,baz": 1
+
+    production:
+      "foo,bar,baz": 4
+
+### Rake task config
+
+Require the rake tasks (`qless/pool/tasks`) in your `Rakefile`, load your
+application environment, configure Qless as necessary, and configure
+`qless:pool:setup` to disconnect all open files and sockets in the pool
+manager and reconnect in the workers.  For example, with rails you should put
+the following into `lib/tasks/qless.rake`:
+
+    require 'qless/pool/tasks'
+    # this task will get called before qless:pool:setup
+    # and preload the rails environment in the pool manager
+    task "qless:setup" => :environment do
+      # generic worker setup, e.g. Hoptoad for failed jobs
+    end
+    task "qless:pool:setup" do
+      # close any sockets or files in pool manager
+      ActiveRecord::Base.connection.disconnect!
+      # and re-open them in the qless worker parent
+      Qless::Pool.after_prefork do |job|
+        ActiveRecord::Base.establish_connection
+      end
+    end
+
+### Start the pool manager
+
+Then you can start the queues via:
+
+    qless-pool --daemon --environment production
+
+This will start up seven worker processes, one exclusively for the foo queue,
+two exclusively for the bar queue, and four workers looking at all queues in
+priority.  With the config above, this is similar to if you ran the following:
+
+    rake qless:work RAILS_ENV=production QUEUES=foo &
+    rake qless:work RAILS_ENV=production QUEUES=bar &
+    rake qless:work RAILS_ENV=production QUEUES=bar &
+    rake qless:work RAILS_ENV=production QUEUES=foo,bar,baz &
+    rake qless:work RAILS_ENV=production QUEUES=foo,bar,baz &
+    rake qless:work RAILS_ENV=production QUEUES=foo,bar,baz &
+    rake qless:work RAILS_ENV=production QUEUES=foo,bar,baz &
+
+The pool manager will stay around monitoring the qless worker parents, giving
+three levels: a single pool manager, many worker parents, and one worker child
+per worker (when the actual job is being processed).  For example, `ps -ef f |
+grep [r]esque` (in Linux) might return something like the following:
+
+    qless    13858     1  0 13:44 ?        S      0:02 qless-pool-manager: managing [13867, 13875, 13871, 13872, 13868, 13870, 13876]
+    qless    13867 13858  0 13:44 ?        S      0:00  \_ qless-1.9.9: Waiting for foo
+    qless    13868 13858  0 13:44 ?        S      0:00  \_ qless-1.9.9: Waiting for bar
+    qless    13870 13858  0 13:44 ?        S      0:00  \_ qless-1.9.9: Waiting for bar
+    qless    13871 13858  0 13:44 ?        S      0:00  \_ qless-1.9.9: Waiting for foo,bar,baz
+    qless    13872 13858  0 13:44 ?        S      0:00  \_ qless-1.9.9: Forked 7481 at 1280343254
+    qless     7481 13872  0 14:54 ?        S      0:00      \_ qless-1.9.9: Processing foo since 1280343254
+    qless    13875 13858  0 13:44 ?        S      0:00  \_ qless-1.9.9: Waiting for foo,bar,baz
+    qless    13876 13858  0 13:44 ?        S      0:00  \_ qless-1.9.9: Forked 7485 at 1280343255
+    qless     7485 13876  0 14:54 ?        S      0:00      \_ qless-1.9.9: Processing bar since 1280343254
+
+Running as a daemon will default to placing the pidfile and logfiles in the
+conventional rails locations, although you can configure that.  See
+`qless-pool --help` for more options.
+
+SIGNALS
+-------
+
+The pool manager responds to the following signals:
+
+* `HUP`   - reload the config file, reload logfiles, restart all workers.
+* `QUIT`  - gracefully shut down workers (via `QUIT`) and shutdown the manager
+  after all workers are done.
+* `INT`   - gracefully shut down workers (via `QUIT`) and immediately shutdown manager
+* `TERM`  - immediately shut down workers (via `INT`) and immediately shutdown manager
+  _(configurable via command line options)_
+* `WINCH` - _(only when running as a daemon)_ send `QUIT` to each worker, but
+  keep manager running (send `HUP` to reload config and restart workers)
+* `USR1`/`USR2`/`CONT` - pass the signal on to all worker parents (see Qless docs).
+
+Use `HUP` to help logrotate run smoothly and to change the number of workers
+per worker type.  Signals can be sent via the `kill` command, e.g.
+`kill -HUP $master_pid`
+
+Other Features
+--------------
+
+An example chef cookbook is provided (and should Just Work at Engine Yard as
+is; just provide a `/data/#{app_name}/shared/config/qless-pool.yml` on your
+utility instances).  Even if you don't use chef you can use the example init.d
+and monitrc erb templates in `examples/chef_cookbook/templates/default`.
+
+You can also start a pool manager via `rake qless:pool` or from a plain old
+ruby script by calling `Qless::Pool.run`.
+
+Workers will watch the pool manager, and gracefully shutdown (after completing
+their current job) if the manager process disappears before them.
+
+You can specify an alternate config file by setting the `QLESS_POOL_CONFIG` or
+with the `--config` command line option.
+
+TODO
+-----
+
+See [the TODO list](https://github.com/wr0ngway/qless-pool/issues) at github issues.
+
+Contributors
+-------------
+
+* Nicholas Evans (Author of resque-pool which this is a fork of)
+* Jason Haruska (from resque-pool)
+* John Schult (from resque-pool)
+* Stephen Celis (from resque-pool)
+* Vincent Agnello, Robert Kamunyori, Paul Kauders (from resque-pool)
+

data/Rakefile

@@ -0,0 +1,30 @@
+require 'bundler'
+Bundler::GemHelper.install_tasks
+
+# for loading the example config file in config/qless-pool.yml
+require 'qless/pool/tasks'
+
+require 'rspec/core/rake_task'
+RSpec::Core::RakeTask.new(:spec) do |t|
+  t.rspec_opts = ["-c", "-f progress"]
+end
+
+require 'cucumber/rake/task'
+Cucumber::Rake::Task.new(:features) do |c|
+  c.profile = "rake"
+end
+
+task :default => [:spec, :features]
+
+rule(/\.[1-9]$/ => [proc { |tn| "#{tn}.ronn" }]) do |t|
+  name = Qless::Pool.name.sub('::','-').upcase
+  version = "%s %s" % [name, Qless::Pool::VERSION.upcase]
+
+  manual = '--manual "%s"' % name
+  organization = '--organization "%s"' % version
+  sh "ronn #{manual} #{organization} <#{t.source} >#{t.name}"
+end
+
+file 'man/qless-pool.1'
+file 'man/qless-pool.yml.5'
+task :manpages => ['man/qless-pool.1','man/qless-pool.yml.5']

data/bin/qless-pool

@@ -0,0 +1,7 @@
+#!/usr/bin/env ruby
+# -*- encoding: utf-8 -*-
+
+$LOAD_PATH.unshift File.expand_path('../../lib', __FILE__)
+
+require 'qless/pool/cli'
+Qless::Pool::CLI.run

data/features/basic_daemon_config.feature

@@ -0,0 +1,68 @@
+Feature: Basic qless-pool daemon configuration and operation
+  To easily manage a pool of qless workers, qless-pool provides a daemon with
+  simple configuration.  Static configuration is handled in the
+  config/config.yml file and dynamic configuration is handled in the Rakefile.
+
+  Background:
+    Given a file named "Rakefile" with:
+    """
+    require 'qless/pool/tasks'
+    """
+
+  Scenario: no config file
+    When I run the pool manager as "qless-pool"
+    Then the pool manager should report that it has started up
+    And the pool manager should report that the pool is empty
+    And the pool manager should have no child processes
+    When I send the pool manager the "QUIT" signal
+    Then the pool manager should finish
+    And the pool manager should report that it is finished
+
+  @slow_exit
+  Scenario: basic config file
+    Given a file named "config/qless-pool.yml" with:
+    """
+    foo: 1
+    bar: 2
+    "bar,baz": 3
+    """
+    When I run the pool manager as "qless-pool"
+    Then the pool manager should report that it has started up
+    And the pool manager should report that 6 workers are in the pool
+    And the pool manager should have 1 "foo" worker child processes
+    And the pool manager should have 2 "bar" worker child processes
+    And the pool manager should have 3 "bar, baz" worker child processes
+    When I send the pool manager the "QUIT" signal
+    Then the qless workers should all shutdown
+    And the pool manager should finish
+    And the pool manager should report that a "foo" worker has been reaped
+    And the pool manager should report that a "bar" worker has been reaped
+    And the pool manager should report that a "bar,baz" worker has been reaped
+    And the pool manager should report that it is finished
+
+  Scenario: daemonized
+    Given a directory named "log"
+    And a directory named "tmp/pids"
+    And a file named "config/qless-pool.yml" with:
+    """
+    foo: 2
+    bar: 4
+    "baz,quux": 4
+    """
+    When I run the pool manager as "qless-pool -d"
+    Then the pool manager should record its pid in "tmp/pids/qless-pool.pid"
+    And the pool manager should daemonize
+    And a file named "log/qless-pool.stdout.log" should exist
+    And a file named "log/qless-pool.stderr.log" should exist
+    And the pool manager should log that it has started up
+    And the pool manager should log that 10 workers are in the pool
+    And the pool manager should have 2 "foo" worker child processes
+    And the pool manager should have 4 "bar" worker child processes
+    And the pool manager should have 4 "baz, quux" worker child processes
+    When I send the pool manager the "QUIT" signal
+    Then the qless workers should all shutdown
+    And the pool manager daemon should finish
+    And the pool manager should log that a "foo" worker has been reaped
+    And the pool manager should log that a "bar" worker has been reaped
+    And the pool manager should log that a "baz,quux" worker has been reaped
+    And the pool manager should log that it is finished

data/features/step_definitions/daemon_steps.rb

@@ -0,0 +1,33 @@
+# syntactic sugar, and separate ivar.  daemons aren't interactive
+When /^I run "([^"]*)" in the background$/ do |cmd|
+  run_background(unescape(cmd))
+end
+
+Then /^the (output|logfiles) should contain the following lines \(with interpolated \$PID\):$/ do |output_logfiles, partial_output|
+  interpolate_background_pid(partial_output).split("\n").each do |line|
+    output_or_log(output_logfiles).should include(line)
+  end
+end
+
+When /^I send "([^"]*)" the "([^"]*)" signal$/ do |cmd, signal|
+  send_signal(cmd, signal)
+end
+
+Then /^the "([^"]*)" process should finish$/ do |cmd|
+  # doesn't actually stop... just polls for exit
+  processes[cmd].stop
+end
+
+Before("@slow_exit") do
+  @aruba_timeout_seconds = 10
+end
+
+After do
+  kill_all_processes!
+  # now kill the daemon!
+  begin
+    Process.kill(9, @pid_from_pidfile) if @pid_from_pidfile
+  rescue Errno::ESRCH
+  end
+  #`pkill -9 qless-pool`
+end

data/features/step_definitions/qless-pool_steps.rb

@@ -0,0 +1,156 @@
+def process_should_exist(pid)
+  lambda { Process.kill(0, pid) }.should_not raise_error(Errno::ESRCH)
+end
+
+def process_should_not_exist(pid)
+  lambda { Process.kill(0, pid) }.should raise_error(Errno::ESRCH)
+end
+
+def grab_worker_pids(count, str)
+  puts "TODO: check output_or_log for #{count} worker started messages"
+  pid_regex = (1..count).map { '(\d+)' }.join ', '
+  full_regex = /qless-pool-manager\[aruba\]\[\d+\]: Pool contains worker PIDs: \[#{pid_regex}\]/m
+  str.should =~ full_regex
+  @worker_pids = full_regex.match(str).captures.map {|pid| pid.to_i }
+end
+
+def output_or_logfiles_string(report_log)
+  case report_log
+  when "report", "output"
+    "output"
+  when "log", "logfiles"
+    "logfiles"
+  else
+    raise ArgumentError
+  end
+end
+
+def output_or_log(report_log)
+  case report_log
+  when "report", "output"
+    interactive_output
+  when "log", "logfiles"
+    in_current_dir do
+      File.read("log/qless-pool.stdout.log") << File.read("log/qless-pool.stderr.log")
+    end
+  else
+    raise ArgumentError
+  end
+end
+
+class NotFinishedStarting < StandardError; end
+def worker_processes_for(queues)
+  children_of(background_pid).select do |pid, cmd|
+    raise NotFinishedStarting if cmd =~ /Starting$/
+    cmd =~ /^Qless-\d+.\d+.\d+: Waiting for #{queues} \(ordered\) at/
+  end
+rescue NotFinishedStarting
+  retry
+end
+
+def children_of(ppid)
+  osx = RUBY_PLATFORM =~ /darwin/i
+  ps = `ps -eo ppid,pid,#{osx ? 'args' : 'cmd'} | grep '^ *#{ppid} '`
+  ps.split(/\s*\n/).map do |line|
+    _, pid, cmd = line.strip.split(/\s+/, 3)
+    [pid, cmd]
+  end
+end
+
+When /^I run the pool manager as "([^"]*)"$/ do |cmd|
+  @pool_manager_process = run_background(unescape(cmd))
+end
+
+When /^I send the pool manager the "([^"]*)" signal$/ do |signal|
+  Process.kill signal, background_pid
+  output_logfiles = @pid_from_pidfile ? "logfiles" : "output"
+  case signal
+  when "QUIT"
+    keep_trying do
+      step "the #{output_logfiles} should contain the following lines (with interpolated $PID):", <<-EOF
+qless-pool-manager[aruba][$PID]: QUIT: graceful shutdown, waiting for children
+      EOF
+    end
+  else
+    raise ArgumentError
+  end
+end
+
+Then /^the pool manager should record its pid in "([^"]*)"$/ do |pidfile|
+  in_current_dir do
+    keep_trying do
+      File.should be_file(pidfile)
+      @pid_from_pidfile = File.read(pidfile).to_i
+      @pid_from_pidfile.should_not == 0
+      process_should_exist(@pid_from_pidfile)
+    end
+  end
+end
+
+Then /^the pool manager should daemonize$/ do
+  stop_processes!
+end
+
+Then /^the pool manager daemon should finish$/ do
+  keep_trying do
+    process_should_not_exist(@pid_from_pidfile)
+  end
+end
+
+# nomenclature: "report" => output to stdout/stderr
+#               "log"    => output to default logfile
+
+Then /^the pool manager should (report|log) that it has started up$/ do |report_log|
+  keep_trying do
+    step "the #{output_or_logfiles_string(report_log)} should contain the following lines (with interpolated $PID):", <<-EOF
+qless-pool-manager[aruba][$PID]: Qless Pool running in test environment
+qless-pool-manager[aruba][$PID]: started manager
+    EOF
+  end
+end
+
+Then /^the pool manager should (report|log) that the pool is empty$/ do |report_log|
+  step "the #{output_or_logfiles_string(report_log)} should contain the following lines (with interpolated $PID):", <<-EOF
+qless-pool-manager[aruba][$PID]: Pool is empty
+  EOF
+end
+
+Then /^the pool manager should (report|log) that (\d+) workers are in the pool$/ do |report_log, count|
+  grab_worker_pids Integer(count), output_or_log(report_log)
+end
+
+Then /^the qless workers should all shutdown$/ do
+  @worker_pids.each do |pid|
+    keep_trying do
+      process_should_not_exist(pid)
+    end
+  end
+end
+
+Then "the pool manager should have no child processes" do
+  children_of(background_pid).should have(:no).keys
+end
+
+Then /^the pool manager should have (\d+) "([^"]*)" worker child processes$/ do |count, queues|
+  worker_processes_for(queues).should have(Integer(count)).members
+end
+
+Then "the pool manager should finish" do
+  # assuming there will not be multiple processes running
+  stop_processes!
+end
+
+Then /^the pool manager should (report|log) that it is finished$/ do |report_log|
+  step "the #{output_or_logfiles_string(report_log)} should contain the following lines (with interpolated $PID):", <<-EOF
+qless-pool-manager[aruba][$PID]: manager finished
+  EOF
+end
+
+Then /^the pool manager should (report|log) that a "([^"]*)" worker has been reaped$/ do |report_log, worker_type|
+  step 'the '+ output_or_logfiles_string(report_log) +' should match /Reaped qless worker\[\d+\] \(status: 0\) queues: '+ worker_type + '/'
+end
+
+Then /^the logfiles should match \/([^\/]*)\/$/ do |partial_output|
+  output_or_log("log").should =~ /#{partial_output}/
+end
+