resque-pool 0.0.7 → 0.0.8
Sign up to get free protection for your applications and to get access to all the features.
- data/README.md +25 -38
- data/lib/resque/pool.rb +3 -7
- metadata +4 -4
data/README.md
CHANGED
@@ -2,9 +2,8 @@ Resque Pool
|
|
2
2
|
===========
|
3
3
|
|
4
4
|
Resque pool is a simple library for managing a pool of resque workers. Given a
|
5
|
-
a config file (`resque-pool.yml` or `config/resque-pool.yml`)
|
6
|
-
|
7
|
-
workers for each.
|
5
|
+
a config file (`resque-pool.yml` or `config/resque-pool.yml`), it will manage
|
6
|
+
your workers for you, starting up the appropriate number of workers for each.
|
8
7
|
|
9
8
|
Benefits
|
10
9
|
---------
|
@@ -23,8 +22,8 @@ How to use
|
|
23
22
|
-----------
|
24
23
|
|
25
24
|
To configure resque-pool, you can use either `resque-pool.yml` or
|
26
|
-
`config/resque-pool.yml`. To use resque-pool, require its rake tasks
|
27
|
-
rake file, and call the resque:pool task.
|
25
|
+
`config/resque-pool.yml`. To use resque-pool, require its rake tasks
|
26
|
+
(`resque/pool/tasks`) in your rake file, and call the `resque:pool` task.
|
28
27
|
|
29
28
|
The YAML file supports both using root level defaults as well as environment
|
30
29
|
specific overrides. For example, to use resque-pool with rails, in
|
@@ -32,63 +31,49 @@ specific overrides. For example, to use resque-pool with rails, in
|
|
32
31
|
|
33
32
|
foo: 1
|
34
33
|
bar: 2
|
35
|
-
"foo,bar,baz":
|
34
|
+
"foo,bar,baz": 1
|
36
35
|
|
37
36
|
production:
|
38
|
-
"foo,bar,baz":
|
37
|
+
"foo,bar,baz": 4
|
39
38
|
|
40
39
|
and in `lib/tasks/resque.rake`:
|
41
40
|
|
42
41
|
require 'resque/pool/tasks'
|
43
|
-
|
44
42
|
# this task will get called before resque:pool:setup
|
45
43
|
# preload the rails environment in the pool master
|
46
44
|
task "resque:setup" => :environment do
|
47
45
|
# generic worker setup, e.g. Hoptoad for failed jobs
|
48
46
|
end
|
49
|
-
|
50
|
-
# preload the rails environment in the pool master
|
51
47
|
task "resque:pool:setup" do
|
52
48
|
# close any sockets or files in pool master
|
53
49
|
ActiveRecord::Base.connection.disconnect!
|
54
|
-
|
55
50
|
# and re-open them in the resque worker parent
|
56
51
|
Resque::Pool.after_prefork do |job|
|
57
52
|
ActiveRecord::Base.establish_connection
|
58
53
|
end
|
59
|
-
|
60
|
-
# you could also re-open them in the resque worker child, using
|
61
|
-
# Resque.after_fork, but that probably isn't necessary, and
|
62
|
-
# Resque::Pool.after_prefork should be faster, since it won't run
|
63
|
-
# for every single job.
|
64
54
|
end
|
65
55
|
|
66
56
|
Then you can start the queues via:
|
67
57
|
|
68
58
|
rake resque:pool RAILS_ENV=production VERBOSE=1
|
69
59
|
|
70
|
-
This will start up seven worker processes, one
|
71
|
-
|
60
|
+
This will start up seven worker processes, one exclusively for the foo queue,
|
61
|
+
two exclusively for the bar queue, and four workers looking at all queues in
|
72
62
|
priority. This is similar to if you ran the following:
|
73
63
|
|
74
|
-
rake resque:work RAILS_ENV=production VERBOSE=1 QUEUES=foo
|
75
|
-
rake resque:work RAILS_ENV=production VERBOSE=1 QUEUES=bar
|
76
|
-
rake resque:work RAILS_ENV=production VERBOSE=1 QUEUES=bar
|
77
|
-
rake resque:work RAILS_ENV=production VERBOSE=1 QUEUES=foo,bar,baz
|
78
|
-
rake resque:work RAILS_ENV=production VERBOSE=1 QUEUES=foo,bar,baz
|
79
|
-
rake resque:work RAILS_ENV=production VERBOSE=1 QUEUES=foo,bar,baz
|
80
|
-
rake resque:work RAILS_ENV=production VERBOSE=1 QUEUES=foo,bar,baz
|
81
|
-
|
82
|
-
Resque already forks for its own child processes, giving two levels. The pool
|
83
|
-
master will stay around monitoring the resque worker parents, giving three
|
84
|
-
levels:
|
64
|
+
rake resque:work RAILS_ENV=production VERBOSE=1 QUEUES=foo &
|
65
|
+
rake resque:work RAILS_ENV=production VERBOSE=1 QUEUES=bar &
|
66
|
+
rake resque:work RAILS_ENV=production VERBOSE=1 QUEUES=bar &
|
67
|
+
rake resque:work RAILS_ENV=production VERBOSE=1 QUEUES=foo,bar,baz &
|
68
|
+
rake resque:work RAILS_ENV=production VERBOSE=1 QUEUES=foo,bar,baz &
|
69
|
+
rake resque:work RAILS_ENV=production VERBOSE=1 QUEUES=foo,bar,baz &
|
70
|
+
rake resque:work RAILS_ENV=production VERBOSE=1 QUEUES=foo,bar,baz &
|
85
71
|
|
86
|
-
|
87
|
-
|
88
|
-
|
89
|
-
|
90
|
-
|
91
|
-
following:
|
72
|
+
Resque already forks for its own child processes. The pool master will stay
|
73
|
+
around monitoring the resque worker parents, giving three levels: a single pool
|
74
|
+
master, many worker parents, and a worker child per worker (when the actual job
|
75
|
+
is being processed). For example, `ps -ef f | grep [r]esque` (in Linux) might
|
76
|
+
return something like the following:
|
92
77
|
|
93
78
|
rails 13858 1 0 13:44 ? S 0:02 resque-pool-master: managing [13867, 13875, 13871, 13872, 13868, 13870, 13876]
|
94
79
|
rails 13867 13858 0 13:44 ? S 0:00 \_ resque-1.9.9: Waiting for foo
|
@@ -101,6 +86,9 @@ following:
|
|
101
86
|
rails 13876 13858 0 13:44 ? S 0:00 \_ resque-1.9.9: Forked 7485 at 1280343255
|
102
87
|
rails 7485 13876 0 14:54 ? S 0:00 \_ resque-1.9.9: Processing bar since 1280343254
|
103
88
|
|
89
|
+
An example startup script, which redirects STDOUT and STDERR and creates a pid
|
90
|
+
file, is given in the examples directory.
|
91
|
+
|
104
92
|
SIGNALS
|
105
93
|
-------
|
106
94
|
|
@@ -113,9 +101,8 @@ The pool master responds to the following signals:
|
|
113
101
|
* `WINCH` - send `QUIT` to each worker, but keep master running (send `HUP` to reload config and restart workers)
|
114
102
|
* `USR1`/`USR2`/`CONT` - send the signal on to all worker parents (see Resque docs).
|
115
103
|
|
116
|
-
`HUP
|
117
|
-
|
118
|
-
longer needed will be gracefully shutdown via `QUIT`.
|
104
|
+
After a `HUP`, workers that are no longer needed will be gracefully shutdown
|
105
|
+
via `QUIT`.
|
119
106
|
|
120
107
|
Other Features
|
121
108
|
--------------
|
data/lib/resque/pool.rb
CHANGED
@@ -79,16 +79,12 @@ module Resque
|
|
79
79
|
end
|
80
80
|
|
81
81
|
def load_config
|
82
|
-
|
83
|
-
|
84
|
-
end
|
85
|
-
if config_environment
|
86
|
-
config.merge!(@config[config_environment])
|
87
|
-
end
|
82
|
+
@config_file and @config = YAML.load_file(@config_file)
|
83
|
+
@config[environment] and config.merge!(@config[environment])
|
88
84
|
config.delete_if {|key, value| value.is_a? Hash }
|
89
85
|
end
|
90
86
|
|
91
|
-
def
|
87
|
+
def environment
|
92
88
|
if defined? RAILS_ENV
|
93
89
|
RAILS_ENV
|
94
90
|
else
|
metadata
CHANGED
@@ -1,13 +1,13 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: resque-pool
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
hash:
|
4
|
+
hash: 15
|
5
5
|
prerelease: false
|
6
6
|
segments:
|
7
7
|
- 0
|
8
8
|
- 0
|
9
|
-
-
|
10
|
-
version: 0.0.
|
9
|
+
- 8
|
10
|
+
version: 0.0.8
|
11
11
|
platform: ruby
|
12
12
|
authors:
|
13
13
|
- nicholas a. evans
|
@@ -15,7 +15,7 @@ autorequire:
|
|
15
15
|
bindir: bin
|
16
16
|
cert_chain: []
|
17
17
|
|
18
|
-
date: 2010-08-
|
18
|
+
date: 2010-08-20 00:00:00 -04:00
|
19
19
|
default_executable:
|
20
20
|
dependencies:
|
21
21
|
- !ruby/object:Gem::Dependency
|