resque-pool 0.0.7 → 0.0.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- data/README.md +25 -38
- data/lib/resque/pool.rb +3 -7
- metadata +4 -4
data/README.md
CHANGED
|
@@ -2,9 +2,8 @@ Resque Pool
|
|
|
2
2
|
===========
|
|
3
3
|
|
|
4
4
|
Resque pool is a simple library for managing a pool of resque workers. Given a
|
|
5
|
-
a config file (`resque-pool.yml` or `config/resque-pool.yml`)
|
|
6
|
-
|
|
7
|
-
workers for each.
|
|
5
|
+
a config file (`resque-pool.yml` or `config/resque-pool.yml`), it will manage
|
|
6
|
+
your workers for you, starting up the appropriate number of workers for each.
|
|
8
7
|
|
|
9
8
|
Benefits
|
|
10
9
|
---------
|
|
@@ -23,8 +22,8 @@ How to use
|
|
|
23
22
|
-----------
|
|
24
23
|
|
|
25
24
|
To configure resque-pool, you can use either `resque-pool.yml` or
|
|
26
|
-
`config/resque-pool.yml`. To use resque-pool, require its rake tasks
|
|
27
|
-
rake file, and call the resque:pool task.
|
|
25
|
+
`config/resque-pool.yml`. To use resque-pool, require its rake tasks
|
|
26
|
+
(`resque/pool/tasks`) in your rake file, and call the `resque:pool` task.
|
|
28
27
|
|
|
29
28
|
The YAML file supports both using root level defaults as well as environment
|
|
30
29
|
specific overrides. For example, to use resque-pool with rails, in
|
|
@@ -32,63 +31,49 @@ specific overrides. For example, to use resque-pool with rails, in
|
|
|
32
31
|
|
|
33
32
|
foo: 1
|
|
34
33
|
bar: 2
|
|
35
|
-
"foo,bar,baz":
|
|
34
|
+
"foo,bar,baz": 1
|
|
36
35
|
|
|
37
36
|
production:
|
|
38
|
-
"foo,bar,baz":
|
|
37
|
+
"foo,bar,baz": 4
|
|
39
38
|
|
|
40
39
|
and in `lib/tasks/resque.rake`:
|
|
41
40
|
|
|
42
41
|
require 'resque/pool/tasks'
|
|
43
|
-
|
|
44
42
|
# this task will get called before resque:pool:setup
|
|
45
43
|
# preload the rails environment in the pool master
|
|
46
44
|
task "resque:setup" => :environment do
|
|
47
45
|
# generic worker setup, e.g. Hoptoad for failed jobs
|
|
48
46
|
end
|
|
49
|
-
|
|
50
|
-
# preload the rails environment in the pool master
|
|
51
47
|
task "resque:pool:setup" do
|
|
52
48
|
# close any sockets or files in pool master
|
|
53
49
|
ActiveRecord::Base.connection.disconnect!
|
|
54
|
-
|
|
55
50
|
# and re-open them in the resque worker parent
|
|
56
51
|
Resque::Pool.after_prefork do |job|
|
|
57
52
|
ActiveRecord::Base.establish_connection
|
|
58
53
|
end
|
|
59
|
-
|
|
60
|
-
# you could also re-open them in the resque worker child, using
|
|
61
|
-
# Resque.after_fork, but that probably isn't necessary, and
|
|
62
|
-
# Resque::Pool.after_prefork should be faster, since it won't run
|
|
63
|
-
# for every single job.
|
|
64
54
|
end
|
|
65
55
|
|
|
66
56
|
Then you can start the queues via:
|
|
67
57
|
|
|
68
58
|
rake resque:pool RAILS_ENV=production VERBOSE=1
|
|
69
59
|
|
|
70
|
-
This will start up seven worker processes, one
|
|
71
|
-
|
|
60
|
+
This will start up seven worker processes, one exclusively for the foo queue,
|
|
61
|
+
two exclusively for the bar queue, and four workers looking at all queues in
|
|
72
62
|
priority. This is similar to if you ran the following:
|
|
73
63
|
|
|
74
|
-
rake resque:work RAILS_ENV=production VERBOSE=1 QUEUES=foo
|
|
75
|
-
rake resque:work RAILS_ENV=production VERBOSE=1 QUEUES=bar
|
|
76
|
-
rake resque:work RAILS_ENV=production VERBOSE=1 QUEUES=bar
|
|
77
|
-
rake resque:work RAILS_ENV=production VERBOSE=1 QUEUES=foo,bar,baz
|
|
78
|
-
rake resque:work RAILS_ENV=production VERBOSE=1 QUEUES=foo,bar,baz
|
|
79
|
-
rake resque:work RAILS_ENV=production VERBOSE=1 QUEUES=foo,bar,baz
|
|
80
|
-
rake resque:work RAILS_ENV=production VERBOSE=1 QUEUES=foo,bar,baz
|
|
81
|
-
|
|
82
|
-
Resque already forks for its own child processes, giving two levels. The pool
|
|
83
|
-
master will stay around monitoring the resque worker parents, giving three
|
|
84
|
-
levels:
|
|
64
|
+
rake resque:work RAILS_ENV=production VERBOSE=1 QUEUES=foo &
|
|
65
|
+
rake resque:work RAILS_ENV=production VERBOSE=1 QUEUES=bar &
|
|
66
|
+
rake resque:work RAILS_ENV=production VERBOSE=1 QUEUES=bar &
|
|
67
|
+
rake resque:work RAILS_ENV=production VERBOSE=1 QUEUES=foo,bar,baz &
|
|
68
|
+
rake resque:work RAILS_ENV=production VERBOSE=1 QUEUES=foo,bar,baz &
|
|
69
|
+
rake resque:work RAILS_ENV=production VERBOSE=1 QUEUES=foo,bar,baz &
|
|
70
|
+
rake resque:work RAILS_ENV=production VERBOSE=1 QUEUES=foo,bar,baz &
|
|
85
71
|
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
following:
|
|
72
|
+
Resque already forks for its own child processes. The pool master will stay
|
|
73
|
+
around monitoring the resque worker parents, giving three levels: a single pool
|
|
74
|
+
master, many worker parents, and a worker child per worker (when the actual job
|
|
75
|
+
is being processed). For example, `ps -ef f | grep [r]esque` (in Linux) might
|
|
76
|
+
return something like the following:
|
|
92
77
|
|
|
93
78
|
rails 13858 1 0 13:44 ? S 0:02 resque-pool-master: managing [13867, 13875, 13871, 13872, 13868, 13870, 13876]
|
|
94
79
|
rails 13867 13858 0 13:44 ? S 0:00 \_ resque-1.9.9: Waiting for foo
|
|
@@ -101,6 +86,9 @@ following:
|
|
|
101
86
|
rails 13876 13858 0 13:44 ? S 0:00 \_ resque-1.9.9: Forked 7485 at 1280343255
|
|
102
87
|
rails 7485 13876 0 14:54 ? S 0:00 \_ resque-1.9.9: Processing bar since 1280343254
|
|
103
88
|
|
|
89
|
+
An example startup script, which redirects STDOUT and STDERR and creates a pid
|
|
90
|
+
file, is given in the examples directory.
|
|
91
|
+
|
|
104
92
|
SIGNALS
|
|
105
93
|
-------
|
|
106
94
|
|
|
@@ -113,9 +101,8 @@ The pool master responds to the following signals:
|
|
|
113
101
|
* `WINCH` - send `QUIT` to each worker, but keep master running (send `HUP` to reload config and restart workers)
|
|
114
102
|
* `USR1`/`USR2`/`CONT` - send the signal on to all worker parents (see Resque docs).
|
|
115
103
|
|
|
116
|
-
`HUP
|
|
117
|
-
|
|
118
|
-
longer needed will be gracefully shutdown via `QUIT`.
|
|
104
|
+
After a `HUP`, workers that are no longer needed will be gracefully shutdown
|
|
105
|
+
via `QUIT`.
|
|
119
106
|
|
|
120
107
|
Other Features
|
|
121
108
|
--------------
|
data/lib/resque/pool.rb
CHANGED
|
@@ -79,16 +79,12 @@ module Resque
|
|
|
79
79
|
end
|
|
80
80
|
|
|
81
81
|
def load_config
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
end
|
|
85
|
-
if config_environment
|
|
86
|
-
config.merge!(@config[config_environment])
|
|
87
|
-
end
|
|
82
|
+
@config_file and @config = YAML.load_file(@config_file)
|
|
83
|
+
@config[environment] and config.merge!(@config[environment])
|
|
88
84
|
config.delete_if {|key, value| value.is_a? Hash }
|
|
89
85
|
end
|
|
90
86
|
|
|
91
|
-
def
|
|
87
|
+
def environment
|
|
92
88
|
if defined? RAILS_ENV
|
|
93
89
|
RAILS_ENV
|
|
94
90
|
else
|
metadata
CHANGED
|
@@ -1,13 +1,13 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: resque-pool
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
hash:
|
|
4
|
+
hash: 15
|
|
5
5
|
prerelease: false
|
|
6
6
|
segments:
|
|
7
7
|
- 0
|
|
8
8
|
- 0
|
|
9
|
-
-
|
|
10
|
-
version: 0.0.
|
|
9
|
+
- 8
|
|
10
|
+
version: 0.0.8
|
|
11
11
|
platform: ruby
|
|
12
12
|
authors:
|
|
13
13
|
- nicholas a. evans
|
|
@@ -15,7 +15,7 @@ autorequire:
|
|
|
15
15
|
bindir: bin
|
|
16
16
|
cert_chain: []
|
|
17
17
|
|
|
18
|
-
date: 2010-08-
|
|
18
|
+
date: 2010-08-20 00:00:00 -04:00
|
|
19
19
|
default_executable:
|
|
20
20
|
dependencies:
|
|
21
21
|
- !ruby/object:Gem::Dependency
|