vitobotta-resque-pool 0.3.0.dev
Sign up to get free protection for your applications and to get access to all the features.
- data/Changelog.md +49 -0
- data/Gemfile +4 -0
- data/LICENSE.txt +20 -0
- data/README.md +153 -0
- data/Rakefile +17 -0
- data/bin/resque-pool +5 -0
- data/config/alternate.yml +4 -0
- data/config/cucumber.yml +3 -0
- data/examples/Gemfile +1 -0
- data/examples/Gemfile.lock +67 -0
- data/examples/Rakefile +8 -0
- data/examples/chef_cookbook/recipes/default.rb +46 -0
- data/examples/chef_cookbook/templates/default/initd.erb +69 -0
- data/examples/chef_cookbook/templates/default/monitrc.erb +6 -0
- data/examples/rails-resque.rake +22 -0
- data/examples/resque-pool.yml +4 -0
- data/features/basic_daemon_config.feature +68 -0
- data/features/step_definitions/daemon_steps.rb +29 -0
- data/features/step_definitions/resque-pool_steps.rb +147 -0
- data/features/support/aruba_daemon_support.rb +60 -0
- data/features/support/env.rb +1 -0
- data/lib/resque/pool/cli.rb +116 -0
- data/lib/resque/pool/logging.rb +58 -0
- data/lib/resque/pool/pooled_worker.rb +21 -0
- data/lib/resque/pool/tasks.rb +20 -0
- data/lib/resque/pool/version.rb +5 -0
- data/lib/resque/pool.rb +331 -0
- data/resque-pool.gemspec +34 -0
- data/spec/mock_config.rb +6 -0
- data/spec/resque-pool.yml +13 -0
- data/spec/resque_pool_spec.rb +134 -0
- data/spec/spec_helper.rb +3 -0
- data/tmp/aruba/Rakefile +1 -0
- metadata +175 -0
data/Changelog.md
ADDED
@@ -0,0 +1,49 @@
|
|
1
|
+
## 0.2.0 (2011-03-15)
|
2
|
+
|
3
|
+
* new feature: sending `HUP` to pool manager will reload the logfiles and
|
4
|
+
gracefully restart all workers.
|
5
|
+
* enhancement: logging now includes timestamp, process "name" (worker or
|
6
|
+
manager), and PID.
|
7
|
+
* enhancement: can be used with no config file or empty config file (not all
|
8
|
+
*that* useful, but it's better than unceromoniously dieing!)
|
9
|
+
* bugfix: pidfile will be cleaned up on startup, e.g. if old process was
|
10
|
+
kill-9'd (Jason Haruska)
|
11
|
+
* bugfix: TERM/INT are no longer ignored when HUP is waiting on children
|
12
|
+
* bugfix: `resque-pool -c config.yml` command line option was broken
|
13
|
+
* development: simple cucumber features for core functionality.
|
14
|
+
* upstream: depends on resque ~> 1.13
|
15
|
+
|
16
|
+
## 0.1.0 (2011-01-18)
|
17
|
+
|
18
|
+
* new feature: `resque-pool` command line interface
|
19
|
+
* this replaces need for a special startup script.
|
20
|
+
* manages PID file, logfiles, daemonizing, etc.
|
21
|
+
* `resque-pool --help` for more info and options
|
22
|
+
* updated example config, init.d script, including a chef recipe that should
|
23
|
+
work at EngineYard.
|
24
|
+
|
25
|
+
## 0.0.10 (2010-08-31)
|
26
|
+
|
27
|
+
* remove rubygems 1.3.6 dependency
|
28
|
+
|
29
|
+
## 0.0.9 (2010-08-26)
|
30
|
+
|
31
|
+
* new feature: `RESQUE_POOL_CONFIG` environment variable to set alt config file
|
32
|
+
* upgraded to resque 1.10, removing `Resque::Worker` monkeypatch
|
33
|
+
|
34
|
+
## 0.0.8 (2010-08-20)
|
35
|
+
|
36
|
+
* bugfix: using (or not using) environments in config file
|
37
|
+
|
38
|
+
## 0.0.7 (2010-08-16)
|
39
|
+
|
40
|
+
* new feature: split by environments in config file
|
41
|
+
* added example startup script, Rakefile, and monit config
|
42
|
+
|
43
|
+
## 0.0.5 (2010-06-29)
|
44
|
+
|
45
|
+
* bugfix: worker processes not shutting down after orphaned
|
46
|
+
|
47
|
+
## 0.0.4 (2010-06-29)
|
48
|
+
|
49
|
+
* first release used in production
|
data/Gemfile
ADDED
data/LICENSE.txt
ADDED
@@ -0,0 +1,20 @@
|
|
1
|
+
Copyright (C) 2010 by Nicholas Evans <nick@ekenosen.net>, et al.
|
2
|
+
|
3
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
4
|
+
of this software and associated documentation files (the "Software"), to deal
|
5
|
+
in the Software without restriction, including without limitation the rights
|
6
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
7
|
+
copies of the Software, and to permit persons to whom the Software is
|
8
|
+
furnished to do so, subject to the following conditions:
|
9
|
+
|
10
|
+
The above copyright notice and this permission notice shall be included in
|
11
|
+
all copies or substantial portions of the Software.
|
12
|
+
|
13
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
14
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
15
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
16
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
17
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
18
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
19
|
+
THE SOFTWARE.
|
20
|
+
|
data/README.md
ADDED
@@ -0,0 +1,153 @@
|
|
1
|
+
Resque Pool
|
2
|
+
===========
|
3
|
+
|
4
|
+
Resque pool is a simple library for managing a pool of
|
5
|
+
[resque](http://github.com/defunkt/resque) workers. Given a a config file, it
|
6
|
+
manages your workers for you, starting up the appropriate number of workers for
|
7
|
+
each worker type.
|
8
|
+
|
9
|
+
Benefits
|
10
|
+
---------
|
11
|
+
|
12
|
+
* Less config - With a simple YAML file, you can start up a pool daemon, and it
|
13
|
+
will monitor your workers for you. An example init.d script, monit config,
|
14
|
+
and chef cookbook are provided.
|
15
|
+
* Less memory - If you are using Ruby Enterprise Edition, or any ruby with
|
16
|
+
copy-on-write safe garbage collection, this should save you a *lot* of memory
|
17
|
+
when you are managing many workers.
|
18
|
+
* Faster startup - when you start many workers at once, they would normally
|
19
|
+
compete for CPU as they load their environments. Resque-pool can load the
|
20
|
+
environment once and fork all of the workers almost instantly.
|
21
|
+
|
22
|
+
Upgrading?
|
23
|
+
-----------
|
24
|
+
|
25
|
+
See
|
26
|
+
[Changelog.md](https://github.com/nevans/resque-pool/blob/master/Changelog.md)
|
27
|
+
in case there are important or helpful changes.
|
28
|
+
|
29
|
+
How to use
|
30
|
+
-----------
|
31
|
+
|
32
|
+
### YAML file config
|
33
|
+
|
34
|
+
Create a `config/resque-pool.yml` (or `resque-pool.yml`) with your worker
|
35
|
+
counts. The YAML file supports both using root level defaults as well as
|
36
|
+
environment specific overrides (`RACK_ENV`, `RAILS_ENV`, and `RESQUE_ENV`
|
37
|
+
environment variables can be used to determine environment). For example in
|
38
|
+
`config/resque-pool.yml`:
|
39
|
+
|
40
|
+
foo: 1
|
41
|
+
bar: 2
|
42
|
+
"foo,bar,baz": 1
|
43
|
+
|
44
|
+
production:
|
45
|
+
"foo,bar,baz": 4
|
46
|
+
|
47
|
+
### Rake task config
|
48
|
+
|
49
|
+
Require the rake tasks (`resque/pool/tasks`) in your `Rakefile`, load your
|
50
|
+
application environment, configure Resque as necessary, and configure
|
51
|
+
`resque:pool:setup` to disconnect all open files and sockets in the pool
|
52
|
+
manager and reconnect in the workers. For example, with rails you should put
|
53
|
+
the following into `lib/tasks/resque.rake`:
|
54
|
+
|
55
|
+
require 'resque/pool/tasks'
|
56
|
+
# this task will get called before resque:pool:setup
|
57
|
+
# and preload the rails environment in the pool manager
|
58
|
+
task "resque:setup" => :environment do
|
59
|
+
# generic worker setup, e.g. Hoptoad for failed jobs
|
60
|
+
end
|
61
|
+
task "resque:pool:setup" do
|
62
|
+
# close any sockets or files in pool manager
|
63
|
+
ActiveRecord::Base.connection.disconnect!
|
64
|
+
# and re-open them in the resque worker parent
|
65
|
+
Resque::Pool.after_prefork do |job|
|
66
|
+
ActiveRecord::Base.establish_connection
|
67
|
+
end
|
68
|
+
end
|
69
|
+
|
70
|
+
### Start the pool manager
|
71
|
+
|
72
|
+
Then you can start the queues via:
|
73
|
+
|
74
|
+
resque-pool --daemon --environment production
|
75
|
+
|
76
|
+
This will start up seven worker processes, one exclusively for the foo queue,
|
77
|
+
two exclusively for the bar queue, and four workers looking at all queues in
|
78
|
+
priority. With the config above, this is similar to if you ran the following:
|
79
|
+
|
80
|
+
rake resque:work RAILS_ENV=production QUEUES=foo &
|
81
|
+
rake resque:work RAILS_ENV=production QUEUES=bar &
|
82
|
+
rake resque:work RAILS_ENV=production QUEUES=bar &
|
83
|
+
rake resque:work RAILS_ENV=production QUEUES=foo,bar,baz &
|
84
|
+
rake resque:work RAILS_ENV=production QUEUES=foo,bar,baz &
|
85
|
+
rake resque:work RAILS_ENV=production QUEUES=foo,bar,baz &
|
86
|
+
rake resque:work RAILS_ENV=production QUEUES=foo,bar,baz &
|
87
|
+
|
88
|
+
The pool manager will stay around monitoring the resque worker parents, giving
|
89
|
+
three levels: a single pool manager, many worker parents, and one worker child
|
90
|
+
per worker (when the actual job is being processed). For example, `ps -ef f |
|
91
|
+
grep [r]esque` (in Linux) might return something like the following:
|
92
|
+
|
93
|
+
resque 13858 1 0 13:44 ? S 0:02 resque-pool-manager: managing [13867, 13875, 13871, 13872, 13868, 13870, 13876]
|
94
|
+
resque 13867 13858 0 13:44 ? S 0:00 \_ resque-1.9.9: Waiting for foo
|
95
|
+
resque 13868 13858 0 13:44 ? S 0:00 \_ resque-1.9.9: Waiting for bar
|
96
|
+
resque 13870 13858 0 13:44 ? S 0:00 \_ resque-1.9.9: Waiting for bar
|
97
|
+
resque 13871 13858 0 13:44 ? S 0:00 \_ resque-1.9.9: Waiting for foo,bar,baz
|
98
|
+
resque 13872 13858 0 13:44 ? S 0:00 \_ resque-1.9.9: Forked 7481 at 1280343254
|
99
|
+
resque 7481 13872 0 14:54 ? S 0:00 \_ resque-1.9.9: Processing foo since 1280343254
|
100
|
+
resque 13875 13858 0 13:44 ? S 0:00 \_ resque-1.9.9: Waiting for foo,bar,baz
|
101
|
+
resque 13876 13858 0 13:44 ? S 0:00 \_ resque-1.9.9: Forked 7485 at 1280343255
|
102
|
+
resque 7485 13876 0 14:54 ? S 0:00 \_ resque-1.9.9: Processing bar since 1280343254
|
103
|
+
|
104
|
+
Running as a daemon will default to placing the pidfile and logfiles in the
|
105
|
+
conventional rails locations, although you can configure that. See
|
106
|
+
`resque-pool --help` for more options.
|
107
|
+
|
108
|
+
SIGNALS
|
109
|
+
-------
|
110
|
+
|
111
|
+
The pool manager responds to the following signals:
|
112
|
+
|
113
|
+
* `HUP` - reload the config file, reload logfiles, restart all workers.
|
114
|
+
* `QUIT` - send `QUIT` to each worker parent and shutdown the manager after all workers are done.
|
115
|
+
* `INT` - send `QUIT` to each worker parent and immediately shutdown manager
|
116
|
+
* `TERM` - send `TERM` to each worker parent and immediately shutdown manager
|
117
|
+
* `WINCH` - send `QUIT` to each worker, but keep manager running (send `HUP` to reload config and restart workers)
|
118
|
+
* `USR1`/`USR2`/`CONT` - pass the signal on to all worker parents (see Resque docs).
|
119
|
+
|
120
|
+
Use `HUP` to help logrotate run smoothly and to change the number of workers
|
121
|
+
per worker type.
|
122
|
+
|
123
|
+
Other Features
|
124
|
+
--------------
|
125
|
+
|
126
|
+
An example chef cookbook is provided (and should Just Work at Engine Yard as
|
127
|
+
is; just provide a `/data/#{app_name}/shared/config/resque-pool.yml` on your
|
128
|
+
utility instances). Even if you don't use chef you can use the example init.d
|
129
|
+
and monitrc erb templates in `examples/chef_cookbook/templates/default`.
|
130
|
+
|
131
|
+
You can also start a pool manager via `rake resque:pool:start` or from a plain
|
132
|
+
old ruby script by calling `Resque::Pool.run`.
|
133
|
+
|
134
|
+
You can stop a pool manager via `rake resque:pool:stop`.
|
135
|
+
|
136
|
+
Workers will watch the pool manager, and gracefully shutdown (after completing
|
137
|
+
their current job) if the manager process disappears before them.
|
138
|
+
|
139
|
+
You can specify an alternate config file by setting the `RESQUE_POOL_CONFIG` or
|
140
|
+
with the `--config` command line option.
|
141
|
+
|
142
|
+
TODO
|
143
|
+
-----
|
144
|
+
|
145
|
+
See [the TODO list](https://github.com/nevans/resque-pool/issues) at github issues.
|
146
|
+
|
147
|
+
Contributors
|
148
|
+
-------------
|
149
|
+
|
150
|
+
* John Schult (config file can be split by environment)
|
151
|
+
* Stephen Celis (increased gemspec sanity)
|
152
|
+
* Vincent Agnello, Robert Kamunyori, Paul Kauders; for pairing with me at
|
153
|
+
B'more on Rails Open Source Hack Nights. :)
|
data/Rakefile
ADDED
@@ -0,0 +1,17 @@
|
|
1
|
+
require 'bundler'
|
2
|
+
Bundler::GemHelper.install_tasks
|
3
|
+
|
4
|
+
# for loading the example config file in config/resque-pool.yml
|
5
|
+
require 'resque/pool/tasks'
|
6
|
+
|
7
|
+
require 'rspec/core/rake_task'
|
8
|
+
RSpec::Core::RakeTask.new(:spec) do |t|
|
9
|
+
t.rspec_opts = ["-c", "-f progress"]
|
10
|
+
end
|
11
|
+
|
12
|
+
require 'cucumber/rake/task'
|
13
|
+
Cucumber::Rake::Task.new(:features) do |c|
|
14
|
+
c.profile = "rake"
|
15
|
+
end
|
16
|
+
|
17
|
+
task :default => [:spec, :features]
|
data/bin/resque-pool
ADDED
data/config/cucumber.yml
ADDED
data/examples/Gemfile
ADDED
@@ -0,0 +1 @@
|
|
1
|
+
gemspec :path => ".."
|
@@ -0,0 +1,67 @@
|
|
1
|
+
PATH
|
2
|
+
remote: /home/nick/src/active/resque-pool
|
3
|
+
specs:
|
4
|
+
resque-pool (0.1.0.dev)
|
5
|
+
rake
|
6
|
+
resque (~> 1.13)
|
7
|
+
trollop (~> 1.16)
|
8
|
+
|
9
|
+
GEM
|
10
|
+
specs:
|
11
|
+
SystemTimer (1.2.2)
|
12
|
+
aruba (0.3.2)
|
13
|
+
childprocess (~> 0.1.6)
|
14
|
+
cucumber (~> 0.10.0)
|
15
|
+
rspec (~> 2.3.0)
|
16
|
+
builder (3.0.0)
|
17
|
+
childprocess (0.1.7)
|
18
|
+
ffi (~> 0.6.3)
|
19
|
+
cucumber (0.10.0)
|
20
|
+
builder (>= 2.1.2)
|
21
|
+
diff-lcs (~> 1.1.2)
|
22
|
+
gherkin (~> 2.3.2)
|
23
|
+
json (~> 1.4.6)
|
24
|
+
term-ansicolor (~> 1.0.5)
|
25
|
+
diff-lcs (1.1.2)
|
26
|
+
ffi (0.6.3)
|
27
|
+
rake (>= 0.8.7)
|
28
|
+
gherkin (2.3.3)
|
29
|
+
json (~> 1.4.6)
|
30
|
+
json (1.4.6)
|
31
|
+
rack (1.2.1)
|
32
|
+
rake (0.8.7)
|
33
|
+
redis (2.1.1)
|
34
|
+
redis-namespace (0.10.0)
|
35
|
+
redis (< 3.0.0)
|
36
|
+
resque (1.13.0)
|
37
|
+
json (~> 1.4.6)
|
38
|
+
redis-namespace (>= 0.10.0)
|
39
|
+
sinatra (>= 0.9.2)
|
40
|
+
vegas (~> 0.1.2)
|
41
|
+
rspec (2.3.0)
|
42
|
+
rspec-core (~> 2.3.0)
|
43
|
+
rspec-expectations (~> 2.3.0)
|
44
|
+
rspec-mocks (~> 2.3.0)
|
45
|
+
rspec-core (2.3.1)
|
46
|
+
rspec-expectations (2.3.0)
|
47
|
+
diff-lcs (~> 1.1.2)
|
48
|
+
rspec-mocks (2.3.0)
|
49
|
+
sinatra (1.1.3)
|
50
|
+
rack (~> 1.1)
|
51
|
+
tilt (>= 1.2.2, < 2.0)
|
52
|
+
term-ansicolor (1.0.5)
|
53
|
+
tilt (1.2.2)
|
54
|
+
trollop (1.16.2)
|
55
|
+
vegas (0.1.8)
|
56
|
+
rack (>= 1.0.0)
|
57
|
+
|
58
|
+
PLATFORMS
|
59
|
+
ruby
|
60
|
+
|
61
|
+
DEPENDENCIES
|
62
|
+
SystemTimer
|
63
|
+
aruba (~> 0.3.2)
|
64
|
+
bundler (~> 1.0)
|
65
|
+
cucumber (~> 0.10.0)
|
66
|
+
resque-pool!
|
67
|
+
rspec (~> 2.3.0)
|
data/examples/Rakefile
ADDED
@@ -0,0 +1,8 @@
|
|
1
|
+
# If you have redis running on the standard port on localhost, then you can
|
2
|
+
# test out resque-pool in the examples directory by running:
|
3
|
+
#
|
4
|
+
# rake --trace resque:pool
|
5
|
+
#
|
6
|
+
# And then you can poke and prod the various processes with signals and edit
|
7
|
+
# the config file, etc.
|
8
|
+
require 'resque/pool/tasks'
|
@@ -0,0 +1,46 @@
|
|
1
|
+
roles = %w[solo util]
|
2
|
+
if roles.include?(node[:instance_role])
|
3
|
+
node[:applications].each do |app, data|
|
4
|
+
|
5
|
+
pidfile = "/data/#{app}/current/tmp/pids/#{app}_resque.pid"
|
6
|
+
|
7
|
+
template "/etc/monit.d/#{app}_resque.monitrc" do
|
8
|
+
owner 'root'
|
9
|
+
group 'root'
|
10
|
+
mode 0644
|
11
|
+
source "monitrc.erb"
|
12
|
+
variables({
|
13
|
+
:app_name => app,
|
14
|
+
:pidfile => pidfile,
|
15
|
+
#:max_mem => "400 MB",
|
16
|
+
})
|
17
|
+
end
|
18
|
+
|
19
|
+
template "/etc/init.d/#{app}_resque" do
|
20
|
+
owner 'root'
|
21
|
+
group 'root'
|
22
|
+
mode 0744
|
23
|
+
source "initd.erb"
|
24
|
+
variables({
|
25
|
+
:app_name => app,
|
26
|
+
:pidfile => pidfile,
|
27
|
+
})
|
28
|
+
end
|
29
|
+
|
30
|
+
execute "enable-resque" do
|
31
|
+
command "rc-update add #{app}_resque default"
|
32
|
+
action :run
|
33
|
+
not_if "rc-update show | grep -q '^ *#{app}_resque |.*default'"
|
34
|
+
end
|
35
|
+
|
36
|
+
execute "start-resque" do
|
37
|
+
command %Q{/etc/init.d/#{app}_resque start}
|
38
|
+
creates pidfile
|
39
|
+
end
|
40
|
+
|
41
|
+
execute "ensure-resque-is-setup-with-monit" do
|
42
|
+
command %Q{monit reload}
|
43
|
+
end
|
44
|
+
|
45
|
+
end
|
46
|
+
end
|
@@ -0,0 +1,69 @@
|
|
1
|
+
#!/bin/sh -e
|
2
|
+
# Copyright (C) nicholas a. evans <nick@ekenosen.net>
|
3
|
+
# See LICENSE.txt
|
4
|
+
|
5
|
+
### BEGIN INIT INFO
|
6
|
+
# Provides: resque-pool-<%= @app_name %>
|
7
|
+
# Required-Start: $local_fs $remote_fs $network
|
8
|
+
# Required-Stop: $local_fs $remote_fs $network
|
9
|
+
# Default-Start: 2 3 4 5
|
10
|
+
# Default-Stop: 0 1 6
|
11
|
+
# Short-Description: resque-pool init script for <%= @app_name %>
|
12
|
+
# Description: resque-pool manages the resque workers
|
13
|
+
### END INIT INFO
|
14
|
+
|
15
|
+
# configure these values here or in /etc/default/resque-pool
|
16
|
+
# make sure your pidfile here matches your monitrc
|
17
|
+
app_name="<%= @app_name %>"
|
18
|
+
pidfile="<%= @pidfile %>"
|
19
|
+
app_dir="/data/${app_name}/current"
|
20
|
+
run_as_user="deploy"
|
21
|
+
sleep_time_during_restart=5
|
22
|
+
stop_schedule="QUIT/30/INT/10/KILL/5"
|
23
|
+
bundler="/usr/bin/bundle"
|
24
|
+
environment="production"
|
25
|
+
stdout_log="${app_dir}/log/resque-pool-${app_name}.stdout.log"
|
26
|
+
stderr_log="${app_dir}/log/resque-pool-${app_name}.stderr.log"
|
27
|
+
|
28
|
+
# override above values, and set any other env variables you need
|
29
|
+
if [ -f /etc/default/resque ] ; then
|
30
|
+
. /etc/default/resque
|
31
|
+
fi
|
32
|
+
if [ -f /etc/default/${app_name}_resque ] ; then
|
33
|
+
. /etc/default/${app_name}_resque
|
34
|
+
fi
|
35
|
+
|
36
|
+
case "$1" in
|
37
|
+
start)
|
38
|
+
# I'm assuming that you are using bundler. If you are using rip or
|
39
|
+
# something else, you'll need to change this. Remember to
|
40
|
+
# keep the double-dash; e.g.: --startas CMD -- ARGS
|
41
|
+
start-stop-daemon --start --pidfile ${pidfile} \
|
42
|
+
--chuid ${run_as_user} --chdir ${app_dir} \
|
43
|
+
--startas ${bundler} -- exec \
|
44
|
+
resque-pool -d -p ${pidfile} -E ${environment} -o ${stdout_log} -e ${stderr_log}
|
45
|
+
;;
|
46
|
+
reload)
|
47
|
+
start-stop-daemon --stop --pidfile ${pidfile} --signal HUP
|
48
|
+
;;
|
49
|
+
graceful-stop)
|
50
|
+
start-stop-daemon --stop --pidfile ${pidfile} --signal QUIT
|
51
|
+
;;
|
52
|
+
quick-stop)
|
53
|
+
start-stop-daemon --stop --pidfile ${pidfile} --signal INT
|
54
|
+
;;
|
55
|
+
stop)
|
56
|
+
start-stop-daemon --stop --pidfile ${pidfile} --retry=${stop_schedule}
|
57
|
+
;;
|
58
|
+
restart)
|
59
|
+
$0 stop
|
60
|
+
sleep ${sleep_time_during_restart}
|
61
|
+
$0 start
|
62
|
+
;;
|
63
|
+
*)
|
64
|
+
echo "Usage: $0 {start|stop|graceful-stop|quick-stop|restart|reload}"
|
65
|
+
exit 1
|
66
|
+
;;
|
67
|
+
esac
|
68
|
+
|
69
|
+
# vim:ft=sh
|
@@ -0,0 +1,6 @@
|
|
1
|
+
check process <%= @app_name %>_resque
|
2
|
+
with pidfile <%= @pidfile %>
|
3
|
+
start program = "/etc/init.d/<%= @app_name %>_resque start" with timeout 90 seconds
|
4
|
+
stop program = "/etc/init.d/<%= @app_name %>_resque stop" with timeout 90 seconds
|
5
|
+
#if totalmem is greater than <%= @max_mem || "300 MB" %> for 10 cycles then restart # eating up memory?
|
6
|
+
group resque
|
@@ -0,0 +1,22 @@
|
|
1
|
+
require 'resque/pool/tasks'
|
2
|
+
|
3
|
+
# this task will get called before resque:pool:setup
|
4
|
+
# preload the rails environment in the pool master
|
5
|
+
task "resque:setup" => :environment do
|
6
|
+
# generic worker setup, e.g. Hoptoad for failed jobs
|
7
|
+
end
|
8
|
+
|
9
|
+
task "resque:pool:setup" do
|
10
|
+
# close any sockets or files in pool master
|
11
|
+
ActiveRecord::Base.connection.disconnect!
|
12
|
+
|
13
|
+
# and re-open them in the resque worker parent
|
14
|
+
Resque::Pool.after_prefork do |job|
|
15
|
+
ActiveRecord::Base.establish_connection
|
16
|
+
end
|
17
|
+
|
18
|
+
# you could also re-open them in the resque worker child, using
|
19
|
+
# Resque.after_fork, but that probably isn't necessary, and
|
20
|
+
# Resque::Pool.after_prefork should be faster, since it won't run
|
21
|
+
# for every single job.
|
22
|
+
end
|
@@ -0,0 +1,68 @@
|
|
1
|
+
Feature: Basic resque-pool daemon configuration and operation
|
2
|
+
To easily manage a pool of resque workers, resque-pool provides a daemon with
|
3
|
+
simple configuration. Static configuration is handled in the
|
4
|
+
config/config.yml file and dynamic configuration is handled in the Rakefile.
|
5
|
+
|
6
|
+
Background:
|
7
|
+
Given a file named "Rakefile" with:
|
8
|
+
"""
|
9
|
+
require 'resque/pool/tasks'
|
10
|
+
"""
|
11
|
+
|
12
|
+
Scenario: no config file
|
13
|
+
When I run the pool manager as "resque-pool"
|
14
|
+
Then the pool manager should report that it has started up
|
15
|
+
And the pool manager should report that the pool is empty
|
16
|
+
And the pool manager should have no child processes
|
17
|
+
When I send the pool manager the "QUIT" signal
|
18
|
+
Then the pool manager should finish
|
19
|
+
And the pool manager should report that it is finished
|
20
|
+
|
21
|
+
@slow_exit
|
22
|
+
Scenario: basic config file
|
23
|
+
Given a file named "config/resque-pool.yml" with:
|
24
|
+
"""
|
25
|
+
foo: 1
|
26
|
+
bar: 2
|
27
|
+
"bar,baz": 3
|
28
|
+
"""
|
29
|
+
When I run the pool manager as "resque-pool"
|
30
|
+
Then the pool manager should report that it has started up
|
31
|
+
And the pool manager should report that 6 workers are in the pool
|
32
|
+
And the pool manager should have 1 "foo" worker child processes
|
33
|
+
And the pool manager should have 2 "bar" worker child processes
|
34
|
+
And the pool manager should have 3 "bar,baz" worker child processes
|
35
|
+
When I send the pool manager the "QUIT" signal
|
36
|
+
Then the resque workers should all shutdown
|
37
|
+
And the pool manager should finish
|
38
|
+
And the pool manager should report that a "foo" worker has been reaped
|
39
|
+
And the pool manager should report that a "bar" worker has been reaped
|
40
|
+
And the pool manager should report that a "bar,baz" worker has been reaped
|
41
|
+
And the pool manager should report that it is finished
|
42
|
+
|
43
|
+
Scenario: daemonized
|
44
|
+
Given a directory named "log"
|
45
|
+
And a directory named "tmp/pids"
|
46
|
+
And a file named "config/resque-pool.yml" with:
|
47
|
+
"""
|
48
|
+
foo: 2
|
49
|
+
bar: 4
|
50
|
+
"baz,quux": 4
|
51
|
+
"""
|
52
|
+
When I run the pool manager as "resque-pool -d"
|
53
|
+
Then the pool manager should record its pid in "tmp/pids/resque-pool.pid"
|
54
|
+
And the pool manager should daemonize
|
55
|
+
And a file named "log/resque-pool.stdout.log" should exist
|
56
|
+
And a file named "log/resque-pool.stderr.log" should exist
|
57
|
+
And the pool manager should log that it has started up
|
58
|
+
And the pool manager should log that 10 workers are in the pool
|
59
|
+
And the pool manager should have 2 "foo" worker child processes
|
60
|
+
And the pool manager should have 4 "bar" worker child processes
|
61
|
+
And the pool manager should have 4 "baz,quux" worker child processes
|
62
|
+
When I send the pool manager the "QUIT" signal
|
63
|
+
Then the resque workers should all shutdown
|
64
|
+
And the pool manager daemon should finish
|
65
|
+
And the pool manager should log that a "foo" worker has been reaped
|
66
|
+
And the pool manager should log that a "bar" worker has been reaped
|
67
|
+
And the pool manager should log that a "baz,quux" worker has been reaped
|
68
|
+
And the pool manager should log that it is finished
|
@@ -0,0 +1,29 @@
|
|
1
|
+
# syntactic sugar, and separate ivar. daemons aren't interactive
|
2
|
+
When /^I run "([^"]*)" in the background$/ do |cmd|
|
3
|
+
run_background(unescape(cmd))
|
4
|
+
end
|
5
|
+
|
6
|
+
Then /^the (output|logfiles) should contain the following lines \(with interpolated \$PID\):$/ do |output_logfiles, partial_output|
|
7
|
+
interpolate_background_pid(partial_output).split("\n").each do |line|
|
8
|
+
output_or_log(output_logfiles).should include(line)
|
9
|
+
end
|
10
|
+
end
|
11
|
+
|
12
|
+
When /^I send "([^"]*)" the "([^"]*)" signal$/ do |cmd, signal|
|
13
|
+
send_signal(cmd, signal)
|
14
|
+
end
|
15
|
+
|
16
|
+
Then /^the "([^"]*)" process should finish$/ do |cmd|
|
17
|
+
# doesn't actually stop... just polls for exit
|
18
|
+
processes[cmd].stop
|
19
|
+
end
|
20
|
+
|
21
|
+
Before("@slow_exit") do
|
22
|
+
@aruba_timeout_seconds = 10
|
23
|
+
end
|
24
|
+
|
25
|
+
After do
|
26
|
+
kill_all_processes!
|
27
|
+
# now kill the daemon!
|
28
|
+
`pkill -9 -f resque-pool`
|
29
|
+
end
|