tr_resque 1.20.1
Sign up to get free protection for your applications and to get access to all the features.
- data/HISTORY.md +354 -0
- data/LICENSE +20 -0
- data/README.markdown +908 -0
- data/Rakefile +70 -0
- data/bin/resque +81 -0
- data/bin/resque-web +27 -0
- data/lib/resque.rb +369 -0
- data/lib/resque/errors.rb +10 -0
- data/lib/resque/failure.rb +96 -0
- data/lib/resque/failure/airbrake.rb +17 -0
- data/lib/resque/failure/base.rb +64 -0
- data/lib/resque/failure/hoptoad.rb +33 -0
- data/lib/resque/failure/multiple.rb +54 -0
- data/lib/resque/failure/redis.rb +51 -0
- data/lib/resque/failure/thoughtbot.rb +33 -0
- data/lib/resque/helpers.rb +94 -0
- data/lib/resque/job.rb +227 -0
- data/lib/resque/plugin.rb +66 -0
- data/lib/resque/server.rb +248 -0
- data/lib/resque/server/public/favicon.ico +0 -0
- data/lib/resque/server/public/idle.png +0 -0
- data/lib/resque/server/public/jquery-1.3.2.min.js +19 -0
- data/lib/resque/server/public/jquery.relatize_date.js +95 -0
- data/lib/resque/server/public/poll.png +0 -0
- data/lib/resque/server/public/ranger.js +73 -0
- data/lib/resque/server/public/reset.css +44 -0
- data/lib/resque/server/public/style.css +86 -0
- data/lib/resque/server/public/working.png +0 -0
- data/lib/resque/server/test_helper.rb +19 -0
- data/lib/resque/server/views/error.erb +1 -0
- data/lib/resque/server/views/failed.erb +67 -0
- data/lib/resque/server/views/key_sets.erb +19 -0
- data/lib/resque/server/views/key_string.erb +11 -0
- data/lib/resque/server/views/layout.erb +44 -0
- data/lib/resque/server/views/next_more.erb +10 -0
- data/lib/resque/server/views/overview.erb +4 -0
- data/lib/resque/server/views/queues.erb +49 -0
- data/lib/resque/server/views/stats.erb +62 -0
- data/lib/resque/server/views/workers.erb +109 -0
- data/lib/resque/server/views/working.erb +72 -0
- data/lib/resque/stat.rb +53 -0
- data/lib/resque/tasks.rb +61 -0
- data/lib/resque/version.rb +3 -0
- data/lib/resque/worker.rb +546 -0
- data/lib/tasks/redis.rake +161 -0
- data/lib/tasks/resque.rake +2 -0
- data/test/airbrake_test.rb +27 -0
- data/test/hoptoad_test.rb +26 -0
- data/test/job_hooks_test.rb +423 -0
- data/test/job_plugins_test.rb +230 -0
- data/test/plugin_test.rb +116 -0
- data/test/redis-test-cluster.conf +115 -0
- data/test/redis-test.conf +115 -0
- data/test/resque-web_test.rb +59 -0
- data/test/resque_test.rb +278 -0
- data/test/test_helper.rb +160 -0
- data/test/worker_test.rb +434 -0
- metadata +186 -0
data/HISTORY.md
ADDED
@@ -0,0 +1,354 @@
|
|
1
|
+
## 1.20.0 (2012-02-17)
|
2
|
+
|
3
|
+
* Fixed demos for ruby 1.9 (@BMorearty, #445)
|
4
|
+
* Fixed `#requeue` tests (@hone, #500)
|
5
|
+
* Web UI: optional trailing slashes of URLs (@elisehuard, #449)
|
6
|
+
* Allow * to appear anywhere in queue list (@tapajos, #405, #407)
|
7
|
+
* Wait for child with specific PID (@jacobkg)
|
8
|
+
* #decode raise takes a string when re-raising as a different exception class (Trevor Hart)
|
9
|
+
* Use Sinatra's `pubilc_folder` if it exists (@defunkt, #420, #421)
|
10
|
+
* Assign the job's worker before calling `before_fork` (@quirkey)
|
11
|
+
* Fix Resque::Helpers#constantize to work correctly on 1.9.2 (@rtlong)
|
12
|
+
* Added before & after hooks for dequeue (@humancopy, #398)
|
13
|
+
* daemonize support using `ENV["BACKGROUND"]` (@chrisleishman)
|
14
|
+
* requeue and remove failed jobs by queue name (@evanwhalen)
|
15
|
+
* `-r` flag for resque-web for redis connection (@gjastrab)
|
16
|
+
* Added `Resque.enqueue_to`: allows you to specif the queue and still run hooks (@dan-g)
|
17
|
+
* Web UI: Set the default encoding to UTF-8 (@elubow)
|
18
|
+
* fix finding worker pids on JRuby (John Andrews + Andrew Grieser)
|
19
|
+
* Added distributed redis support (@stipple)
|
20
|
+
* Added better failure hooks (@raykrueger)
|
21
|
+
* Added before & after dequeue hooks (@humancopy)
|
22
|
+
|
23
|
+
## 1.19.0 (2011-09-01)
|
24
|
+
|
25
|
+
* Added Airbrake (formerly Hoptoad) support.
|
26
|
+
* Web UI: Added retry all button to failed jobs page
|
27
|
+
* Web UI: Show focus outline
|
28
|
+
|
29
|
+
## 1.18.6 (2011-08-30)
|
30
|
+
|
31
|
+
* Bugfix: Use Rails 3 eager loading for resque:preload
|
32
|
+
|
33
|
+
## 1.18.5 (2011-08-24)
|
34
|
+
|
35
|
+
* Added support for Travis CI
|
36
|
+
* Bugfix: preload only happens in production Rails environment
|
37
|
+
|
38
|
+
## 1.18.4 (2011-08-23)
|
39
|
+
|
40
|
+
* Bugfix: preload task depends on setup
|
41
|
+
|
42
|
+
## 1.18.3 (2011-08-23)
|
43
|
+
|
44
|
+
* Bugfix: Fix preloading on Rails 3.x.
|
45
|
+
|
46
|
+
## 1.18.2 (2011-08-19)
|
47
|
+
|
48
|
+
* Fix RAILS_ROOT deprecation warning
|
49
|
+
|
50
|
+
## 1.18.1 (2011-08-19)
|
51
|
+
|
52
|
+
* Bugfix: Use RAILS_ROOT in preload task
|
53
|
+
|
54
|
+
## 1.18.0 (2011-08-18)
|
55
|
+
|
56
|
+
* Added before_enqueue hook.
|
57
|
+
* Resque workers now preload files under app/ in Rails
|
58
|
+
* Switch to MultiJSON
|
59
|
+
* Bugfix: Finding worker pids on Solaris
|
60
|
+
* Web UI: Fix NaN days ago for worker screens
|
61
|
+
* Web UI: Add Cache-Control header to prevent proxy caching
|
62
|
+
* Web UI: Update Resque.redis_id so it can be used in a distributed ring.
|
63
|
+
|
64
|
+
## 1.17.1 (2011-05-27)
|
65
|
+
|
66
|
+
* Reverted `exit` change. Back to `exit!`.
|
67
|
+
|
68
|
+
## 1.17.0 (2011-05-26)
|
69
|
+
|
70
|
+
* Workers exit with `exit` instead of `exit!`. This means you
|
71
|
+
can now use `at_exit` hooks inside workers.
|
72
|
+
* More monit typo fixes.
|
73
|
+
* Fixed bug in Hoptoad backend.
|
74
|
+
* Web UI: Wrap preformatted arguments.
|
75
|
+
|
76
|
+
## 1.16.1 (2011-05-17)
|
77
|
+
|
78
|
+
* Bugfix: Resque::Failure::Hoptoad.configure works again
|
79
|
+
* Bugfix: Loading rake tasks
|
80
|
+
|
81
|
+
## 1.16.0 (2011-05-16)
|
82
|
+
|
83
|
+
* Optional Hoptoad backend extracted into hoptoad_notifier. Install the gem to use it.
|
84
|
+
* Added `Worker#paused?` method
|
85
|
+
* Bugfix: Properly reseed random number generator after forking.
|
86
|
+
* Bugfix: Resque.redis=(<a Redis::Namespace>)
|
87
|
+
* Bugfix: Monit example stdout/stderr redirection
|
88
|
+
* Bugfix: Removing single failure now works with multiple failure backends
|
89
|
+
* Web: 'Remove Queue' now requires confirmation
|
90
|
+
* Web: Favicon!
|
91
|
+
* Web Bugfix: Dates display in Safari
|
92
|
+
* Web Bugfix: Dates display timezone
|
93
|
+
* Web Bugfix: Race condition querying working workers
|
94
|
+
* Web Bugfix: Fix polling /workers/all in resque-web
|
95
|
+
|
96
|
+
## 1.15.0 (2011-03-18)
|
97
|
+
|
98
|
+
* Fallback to Redis.connect. Makes ENV variables and whatnot work.
|
99
|
+
* Fixed Sinatra 1.2 compatibility
|
100
|
+
|
101
|
+
## 1.14.0 (2011-03-17)
|
102
|
+
|
103
|
+
* Sleep interval can now be a float
|
104
|
+
* Added Resque.inline to allow in-process performing of jobs (for testing)
|
105
|
+
* Fixed tests for Ruby 1.9.2
|
106
|
+
* Added Resque.validate(klass) to validate a Job
|
107
|
+
* Decode errors are no longer ignored to help debugging
|
108
|
+
* Web: Sinatra 1.2 compatibility
|
109
|
+
* Fixed after_enqueue hook to actually run in `Resque.enqueue`
|
110
|
+
* Fixed very_verbose timestamps to use 24 hour time (AM/PM wasn't included)
|
111
|
+
* Fixed monit example
|
112
|
+
* Fixed Worker#pid
|
113
|
+
|
114
|
+
## 1.13.0 (2011-02-07)
|
115
|
+
|
116
|
+
* Depend on redis-namespace >= 0.10
|
117
|
+
* README tweaks
|
118
|
+
* Use thread_safe option when setting redis url
|
119
|
+
* Bugfix: worker pruning
|
120
|
+
|
121
|
+
## 1.12.0 (2011-02-03)
|
122
|
+
|
123
|
+
* Added pidfile writing from `rake resque:work`
|
124
|
+
* Added Worker#pid method
|
125
|
+
* Added configurable location for `rake install`
|
126
|
+
* Bugfix: Errors in failure backend are rescue'd
|
127
|
+
* Bugfix: Non-working workers no longer counted in "working" count
|
128
|
+
* Bugfix: Don't think resque-web is a worker
|
129
|
+
|
130
|
+
## 1.11.0 (2010-08-23)
|
131
|
+
|
132
|
+
* Web UI: Group /workers page by hostnames
|
133
|
+
|
134
|
+
## 1.10.0 (2010-08-23)
|
135
|
+
|
136
|
+
* Support redis:// string format in `Resque.redis=`
|
137
|
+
* Using new cross-platform JSON gem.
|
138
|
+
* Added `after_enqueue` plugin hook.
|
139
|
+
* Added `shutdown?` method which can be overridden.
|
140
|
+
* Added support for the "leftright" gem when running tests.
|
141
|
+
* Grammarfix: In the README
|
142
|
+
|
143
|
+
## 1.9.10 (2010-08-06)
|
144
|
+
|
145
|
+
* Bugfix: before_fork should get passed the job
|
146
|
+
|
147
|
+
## 1.9.9 (2010-07-26)
|
148
|
+
|
149
|
+
* Depend on redis-namespace 0.8.0
|
150
|
+
* Depend on json_pure instead of json (for JRuby compat)
|
151
|
+
* Bugfix: rails_env display in stats view
|
152
|
+
|
153
|
+
## 1.9.8 (2010-07-20)
|
154
|
+
|
155
|
+
* Bugfix: Worker.all should never return nil
|
156
|
+
* monit example: Fixed Syntax Error and adding environment to the rake task
|
157
|
+
* redis rake task: Fixed typo in copy command
|
158
|
+
|
159
|
+
## 1.9.7 (2010-07-09)
|
160
|
+
|
161
|
+
* Improved memory usage in Job.destroy
|
162
|
+
* redis-namespace 0.7.0 now required
|
163
|
+
* Bugfix: Reverted $0 changes
|
164
|
+
* Web Bugfix: Payload-less failures in the web ui work
|
165
|
+
|
166
|
+
## 1.9.6 (2010-06-22)
|
167
|
+
|
168
|
+
* Bugfix: Rakefile logging works the same as all the other logging
|
169
|
+
|
170
|
+
## 1.9.5 (2010-06-16)
|
171
|
+
|
172
|
+
* Web Bugfix: Display the configured namespace on the stats page
|
173
|
+
* Revert Bugfix: Make ps -o more cross platform friendly
|
174
|
+
|
175
|
+
## 1.9.4 (2010-06-14)
|
176
|
+
|
177
|
+
* Bugfix: Multiple failure backend gets exception information when created
|
178
|
+
|
179
|
+
## 1.9.3 (2010-06-14)
|
180
|
+
|
181
|
+
* Bugfix: Resque#queues always returns an array
|
182
|
+
|
183
|
+
## 1.9.2 (2010-06-13)
|
184
|
+
|
185
|
+
* Bugfix: Worker.all returning nil fix
|
186
|
+
* Bugfix: Make ps -o more cross platform friendly
|
187
|
+
|
188
|
+
## 1.9.1 (2010-06-04)
|
189
|
+
|
190
|
+
* Less strict JSON dependency
|
191
|
+
* Included HISTORY.md in gem
|
192
|
+
|
193
|
+
## 1.9.0 (2010-06-04)
|
194
|
+
|
195
|
+
* Redis 2 support
|
196
|
+
* Depend on redis-namespace 0.5.0
|
197
|
+
* Added Resque::VERSION constant (alias of Resque::Version)
|
198
|
+
* Bugfix: Specify JSON dependency
|
199
|
+
* Bugfix: Hoptoad plugin now works on 1.9
|
200
|
+
|
201
|
+
## 1.8.5 (2010-05-18)
|
202
|
+
|
203
|
+
* Bugfix: Be more liberal in which Redis clients we accept.
|
204
|
+
|
205
|
+
## 1.8.4 (2010-05-18)
|
206
|
+
|
207
|
+
* Try to resolve redis-namespace dependency issue
|
208
|
+
|
209
|
+
## 1.8.3 (2010-05-17)
|
210
|
+
|
211
|
+
* Depend on redis-rb ~> 1.0.7
|
212
|
+
|
213
|
+
## 1.8.2 (2010-05-03)
|
214
|
+
|
215
|
+
* Bugfix: Include "tasks/" dir in RubyGem
|
216
|
+
|
217
|
+
## 1.8.1 (2010-04-29)
|
218
|
+
|
219
|
+
* Bugfix: Multiple failure backend did not support requeue-ing failed jobs
|
220
|
+
* Bugfix: Fix /failed when error has no backtrace
|
221
|
+
* Bugfix: Add `Redis::DistRedis` as a valid client
|
222
|
+
|
223
|
+
## 1.8.0 (2010-04-07)
|
224
|
+
|
225
|
+
* Jobs that never complete due to killed worker are now failed.
|
226
|
+
* Worker "working" state is now maintained by the parent, not the child.
|
227
|
+
* Stopped using deprecated redis.rb methods
|
228
|
+
* `Worker.working` race condition fixed
|
229
|
+
* `Worker#process` has been deprecated.
|
230
|
+
* Monit example fixed
|
231
|
+
* Redis::Client and Redis::Namespace can be passed to `Resque.redis=`
|
232
|
+
|
233
|
+
## 1.7.1 (2010-04-02)
|
234
|
+
|
235
|
+
* Bugfix: Make job hook execution order consistent
|
236
|
+
* Bugfix: stdout buffering in child process
|
237
|
+
|
238
|
+
## 1.7.0 (2010-03-31)
|
239
|
+
|
240
|
+
* Job hooks API. See docs/HOOKS.md.
|
241
|
+
* web: Hovering over dates shows a timestamp
|
242
|
+
* web: AJAXify retry action for failed jobs
|
243
|
+
* web bugfix: Fix pagination bug
|
244
|
+
|
245
|
+
## 1.6.1 (2010-03-25)
|
246
|
+
|
247
|
+
* Bugfix: Workers may not be clearing their state correctly on
|
248
|
+
shutdown
|
249
|
+
* Added example monit config.
|
250
|
+
* Exception class is now recorded when an error is raised in a
|
251
|
+
worker.
|
252
|
+
* web: Unit tests
|
253
|
+
* web: Show namespace in header and footer
|
254
|
+
* web: Remove a queue
|
255
|
+
* web: Retry failed jobs
|
256
|
+
|
257
|
+
## 1.6.0 (2010-03-09)
|
258
|
+
|
259
|
+
* Added `before_first_fork`, `before_fork`, and `after_fork` hooks.
|
260
|
+
* Hoptoad: Added server_environment config setting
|
261
|
+
* Hoptoad bugfix: Don't depend on RAILS_ROOT
|
262
|
+
* 1.8.6 compat fixes
|
263
|
+
|
264
|
+
## 1.5.2 (2010-03-03)
|
265
|
+
|
266
|
+
* Bugfix: JSON check was crazy.
|
267
|
+
|
268
|
+
## 1.5.1 (2010-03-03)
|
269
|
+
|
270
|
+
* `Job.destroy` and `Resque.dequeue` return the # of destroyed jobs.
|
271
|
+
* Hoptoad notifier improvements
|
272
|
+
* Specify the namespace with `resque-web` by passing `-N namespace`
|
273
|
+
* Bugfix: Don't crash when trying to parse invalid JSON.
|
274
|
+
* Bugfix: Non-standard namespace support
|
275
|
+
* Web: Red backgound for queue "failed" only shown if there are failed jobs.
|
276
|
+
* Web bugfix: Tabs highlight properly now
|
277
|
+
* Web bugfix: ZSET partial support in stats
|
278
|
+
* Web bugfix: Deleting failed jobs works again
|
279
|
+
* Web bugfix: Sets (or zsets, lists, etc) now paginate.
|
280
|
+
|
281
|
+
## 1.5.0 (2010-02-17)
|
282
|
+
|
283
|
+
* Version now included in procline, e.g. `resque-1.5.0: Message`
|
284
|
+
* Web bugfix: Ignore idle works in the "working" page
|
285
|
+
* Added `Resque::Job.destroy(queue, klass, *args)`
|
286
|
+
* Added `Resque.dequeue(klass, *args)`
|
287
|
+
|
288
|
+
## 1.4.0 (2010-02-11)
|
289
|
+
|
290
|
+
* Fallback when unable to bind QUIT and USR1 for Windows and JRuby.
|
291
|
+
* Fallback when no `Kernel.fork` is provided (for IronRuby).
|
292
|
+
* Web: Rounded corners in Firefox
|
293
|
+
* Cut down system calls in `Worker#prune_dead_workers`
|
294
|
+
* Enable switching DB in a Redis server from config
|
295
|
+
* Support USR2 and CONT to stop and start job processing.
|
296
|
+
* Web: Add example failing job
|
297
|
+
* Bugfix: `Worker#unregister_worker` shouldn't call `done_working`
|
298
|
+
* Bugfix: Example god config now restarts Resque properly.
|
299
|
+
* Multiple failure backends now permitted.
|
300
|
+
* Hoptoad failure backend updated to new API
|
301
|
+
|
302
|
+
## 1.3.1 (2010-01-11)
|
303
|
+
|
304
|
+
* Vegas bugfix: Don't error without a config
|
305
|
+
|
306
|
+
## 1.3.0 (2010-01-11)
|
307
|
+
|
308
|
+
* Use Vegas for resque-web
|
309
|
+
* Web Bugfix: Show proper date/time value for failed_at on Failures
|
310
|
+
* Web Bugfix: Make the / route more flexible
|
311
|
+
* Add Resque::Server.tabs array (so plugins can add their own tabs)
|
312
|
+
* Start using [Semantic Versioning](http://semver.org/)
|
313
|
+
|
314
|
+
## 1.2.4 (2009-12-15)
|
315
|
+
|
316
|
+
* Web Bugfix: fix key links on stat page
|
317
|
+
|
318
|
+
## 1.2.3 (2009-12-15)
|
319
|
+
|
320
|
+
* Bugfix: Fixed `rand` seeding in child processes.
|
321
|
+
* Bugfix: Better JSON encoding/decoding without Yajl.
|
322
|
+
* Bugfix: Avoid `ps` flag error on Linux
|
323
|
+
* Add `PREFIX` observance to `rake` install tasks.
|
324
|
+
|
325
|
+
## 1.2.2 (2009-12-08)
|
326
|
+
|
327
|
+
* Bugfix: Job equality was not properly implemented.
|
328
|
+
|
329
|
+
## 1.2.1 (2009-12-07)
|
330
|
+
|
331
|
+
* Added `rake resque:workers` task for starting multiple workers.
|
332
|
+
* 1.9.x compatibility
|
333
|
+
* Bugfix: Yajl decoder doesn't care about valid UTF-8
|
334
|
+
* config.ru loads RESQUECONFIG if the ENV variable is set.
|
335
|
+
* `resque-web` now sets RESQUECONFIG
|
336
|
+
* Job objects know if they are equal.
|
337
|
+
* Jobs can be re-queued using `Job#recreate`
|
338
|
+
|
339
|
+
## 1.2.0 (2009-11-25)
|
340
|
+
|
341
|
+
* If USR1 is sent and no child is found, shutdown.
|
342
|
+
* Raise when a job class does not respond to `perform`.
|
343
|
+
* Added `Resque.remove_queue` for deleting a queue
|
344
|
+
|
345
|
+
## 1.1.0 (2009-11-04)
|
346
|
+
|
347
|
+
* Bugfix: Broken ERB tag in failure UI
|
348
|
+
* Bugfix: Save the worker's ID, not the worker itself, in the failure module
|
349
|
+
* Redesigned the sinatra web interface
|
350
|
+
* Added option to clear failed jobs
|
351
|
+
|
352
|
+
## 1.0.0 (2009-11-03)
|
353
|
+
|
354
|
+
* First release.
|
data/LICENSE
ADDED
@@ -0,0 +1,20 @@
|
|
1
|
+
Copyright (c) Chris Wanstrath
|
2
|
+
|
3
|
+
Permission is hereby granted, free of charge, to any person obtaining
|
4
|
+
a copy of this software and associated documentation files (the
|
5
|
+
"Software"), to deal in the Software without restriction, including
|
6
|
+
without limitation the rights to use, copy, modify, merge, publish,
|
7
|
+
distribute, sublicense, and/or sell copies of the Software, and to
|
8
|
+
permit persons to whom the Software is furnished to do so, subject to
|
9
|
+
the following conditions:
|
10
|
+
|
11
|
+
The above copyright notice and this permission notice shall be
|
12
|
+
included in all copies or substantial portions of the Software.
|
13
|
+
|
14
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
15
|
+
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
16
|
+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
17
|
+
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
18
|
+
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
19
|
+
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
20
|
+
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
data/README.markdown
ADDED
@@ -0,0 +1,908 @@
|
|
1
|
+
Resque
|
2
|
+
======
|
3
|
+
|
4
|
+
Resque (pronounced like "rescue") is a Redis-backed library for creating
|
5
|
+
background jobs, placing those jobs on multiple queues, and processing
|
6
|
+
them later.
|
7
|
+
|
8
|
+
Background jobs can be any Ruby class or module that responds to
|
9
|
+
`perform`. Your existing classes can easily be converted to background
|
10
|
+
jobs or you can create new classes specifically to do work. Or, you
|
11
|
+
can do both.
|
12
|
+
|
13
|
+
Resque is heavily inspired by DelayedJob (which rocks) and comprises
|
14
|
+
three parts:
|
15
|
+
|
16
|
+
1. A Ruby library for creating, querying, and processing jobs
|
17
|
+
2. A Rake task for starting a worker which processes jobs
|
18
|
+
3. A Sinatra app for monitoring queues, jobs, and workers.
|
19
|
+
|
20
|
+
Resque workers can be distributed between multiple machines,
|
21
|
+
support priorities, are resilient to memory bloat / "leaks," are
|
22
|
+
optimized for REE (but work on MRI and JRuby), tell you what they're
|
23
|
+
doing, and expect failure.
|
24
|
+
|
25
|
+
Resque queues are persistent; support constant time, atomic push and
|
26
|
+
pop (thanks to Redis); provide visibility into their contents; and
|
27
|
+
store jobs as simple JSON packages.
|
28
|
+
|
29
|
+
The Resque frontend tells you what workers are doing, what workers are
|
30
|
+
not doing, what queues you're using, what's in those queues, provides
|
31
|
+
general usage stats, and helps you track failures.
|
32
|
+
|
33
|
+
|
34
|
+
The Blog Post
|
35
|
+
-------------
|
36
|
+
|
37
|
+
For the backstory, philosophy, and history of Resque's beginnings,
|
38
|
+
please see [the blog post][0].
|
39
|
+
|
40
|
+
|
41
|
+
Overview
|
42
|
+
--------
|
43
|
+
|
44
|
+
Resque allows you to create jobs and place them on a queue, then,
|
45
|
+
later, pull those jobs off the queue and process them.
|
46
|
+
|
47
|
+
Resque jobs are Ruby classes (or modules) which respond to the
|
48
|
+
`perform` method. Here's an example:
|
49
|
+
|
50
|
+
|
51
|
+
``` ruby
|
52
|
+
class Archive
|
53
|
+
@queue = :file_serve
|
54
|
+
|
55
|
+
def self.perform(repo_id, branch = 'master')
|
56
|
+
repo = Repository.find(repo_id)
|
57
|
+
repo.create_archive(branch)
|
58
|
+
end
|
59
|
+
end
|
60
|
+
```
|
61
|
+
|
62
|
+
The `@queue` class instance variable determines which queue `Archive`
|
63
|
+
jobs will be placed in. Queues are arbitrary and created on the fly -
|
64
|
+
you can name them whatever you want and have as many as you want.
|
65
|
+
|
66
|
+
To place an `Archive` job on the `file_serve` queue, we might add this
|
67
|
+
to our application's pre-existing `Repository` class:
|
68
|
+
|
69
|
+
``` ruby
|
70
|
+
class Repository
|
71
|
+
def async_create_archive(branch)
|
72
|
+
Resque.enqueue(Archive, self.id, branch)
|
73
|
+
end
|
74
|
+
end
|
75
|
+
```
|
76
|
+
|
77
|
+
Now when we call `repo.async_create_archive('masterbrew')` in our
|
78
|
+
application, a job will be created and placed on the `file_serve`
|
79
|
+
queue.
|
80
|
+
|
81
|
+
Later, a worker will run something like this code to process the job:
|
82
|
+
|
83
|
+
``` ruby
|
84
|
+
klass, args = Resque.reserve(:file_serve)
|
85
|
+
klass.perform(*args) if klass.respond_to? :perform
|
86
|
+
```
|
87
|
+
|
88
|
+
Which translates to:
|
89
|
+
|
90
|
+
``` ruby
|
91
|
+
Archive.perform(44, 'masterbrew')
|
92
|
+
```
|
93
|
+
|
94
|
+
Let's start a worker to run `file_serve` jobs:
|
95
|
+
|
96
|
+
$ cd app_root
|
97
|
+
$ QUEUE=file_serve rake resque:work
|
98
|
+
|
99
|
+
This starts one Resque worker and tells it to work off the
|
100
|
+
`file_serve` queue. As soon as it's ready it'll try to run the
|
101
|
+
`Resque.reserve` code snippet above and process jobs until it can't
|
102
|
+
find any more, at which point it will sleep for a small period and
|
103
|
+
repeatedly poll the queue for more jobs.
|
104
|
+
|
105
|
+
Workers can be given multiple queues (a "queue list") and run on
|
106
|
+
multiple machines. In fact they can be run anywhere with network
|
107
|
+
access to the Redis server.
|
108
|
+
|
109
|
+
|
110
|
+
Jobs
|
111
|
+
----
|
112
|
+
|
113
|
+
What should you run in the background? Anything that takes any time at
|
114
|
+
all. Slow INSERT statements, disk manipulating, data processing, etc.
|
115
|
+
|
116
|
+
At GitHub we use Resque to process the following types of jobs:
|
117
|
+
|
118
|
+
* Warming caches
|
119
|
+
* Counting disk usage
|
120
|
+
* Building tarballs
|
121
|
+
* Building Rubygems
|
122
|
+
* Firing off web hooks
|
123
|
+
* Creating events in the db and pre-caching them
|
124
|
+
* Building graphs
|
125
|
+
* Deleting users
|
126
|
+
* Updating our search index
|
127
|
+
|
128
|
+
As of writing we have about 35 different types of background jobs.
|
129
|
+
|
130
|
+
Keep in mind that you don't need a web app to use Resque - we just
|
131
|
+
mention "foreground" and "background" because they make conceptual
|
132
|
+
sense. You could easily be spidering sites and sticking data which
|
133
|
+
needs to be crunched later into a queue.
|
134
|
+
|
135
|
+
|
136
|
+
### Persistence
|
137
|
+
|
138
|
+
Jobs are persisted to queues as JSON objects. Let's take our `Archive`
|
139
|
+
example from above. We'll run the following code to create a job:
|
140
|
+
|
141
|
+
``` ruby
|
142
|
+
repo = Repository.find(44)
|
143
|
+
repo.async_create_archive('masterbrew')
|
144
|
+
```
|
145
|
+
|
146
|
+
The following JSON will be stored in the `file_serve` queue:
|
147
|
+
|
148
|
+
``` javascript
|
149
|
+
{
|
150
|
+
'class': 'Archive',
|
151
|
+
'args': [ 44, 'masterbrew' ]
|
152
|
+
}
|
153
|
+
```
|
154
|
+
|
155
|
+
Because of this your jobs must only accept arguments that can be JSON encoded.
|
156
|
+
|
157
|
+
So instead of doing this:
|
158
|
+
|
159
|
+
``` ruby
|
160
|
+
Resque.enqueue(Archive, self, branch)
|
161
|
+
```
|
162
|
+
|
163
|
+
do this:
|
164
|
+
|
165
|
+
``` ruby
|
166
|
+
Resque.enqueue(Archive, self.id, branch)
|
167
|
+
```
|
168
|
+
|
169
|
+
This is why our above example (and all the examples in `examples/`)
|
170
|
+
uses object IDs instead of passing around the objects.
|
171
|
+
|
172
|
+
While this is less convenient than just sticking a marshaled object
|
173
|
+
in the database, it gives you a slight advantage: your jobs will be
|
174
|
+
run against the most recent version of an object because they need to
|
175
|
+
pull from the DB or cache.
|
176
|
+
|
177
|
+
If your jobs were run against marshaled objects, they could
|
178
|
+
potentially be operating on a stale record with out-of-date information.
|
179
|
+
|
180
|
+
|
181
|
+
### send_later / async
|
182
|
+
|
183
|
+
Want something like DelayedJob's `send_later` or the ability to use
|
184
|
+
instance methods instead of just methods for jobs? See the `examples/`
|
185
|
+
directory for goodies.
|
186
|
+
|
187
|
+
We plan to provide first class `async` support in a future release.
|
188
|
+
|
189
|
+
|
190
|
+
### Failure
|
191
|
+
|
192
|
+
If a job raises an exception, it is logged and handed off to the
|
193
|
+
`Resque::Failure` module. Failures are logged either locally in Redis
|
194
|
+
or using some different backend.
|
195
|
+
|
196
|
+
For example, Resque ships with Hoptoad support.
|
197
|
+
|
198
|
+
Keep this in mind when writing your jobs: you may want to throw
|
199
|
+
exceptions you would not normally throw in order to assist debugging.
|
200
|
+
|
201
|
+
|
202
|
+
Workers
|
203
|
+
-------
|
204
|
+
|
205
|
+
Resque workers are rake tasks that run forever. They basically do this:
|
206
|
+
|
207
|
+
``` ruby
|
208
|
+
start
|
209
|
+
loop do
|
210
|
+
if job = reserve
|
211
|
+
job.process
|
212
|
+
else
|
213
|
+
sleep 5 # Polling frequency = 5
|
214
|
+
end
|
215
|
+
end
|
216
|
+
shutdown
|
217
|
+
```
|
218
|
+
|
219
|
+
Starting a worker is simple. Here's our example from earlier:
|
220
|
+
|
221
|
+
$ QUEUE=file_serve rake resque:work
|
222
|
+
|
223
|
+
By default Resque won't know about your application's
|
224
|
+
environment. That is, it won't be able to find and run your jobs - it
|
225
|
+
needs to load your application into memory.
|
226
|
+
|
227
|
+
If we've installed Resque as a Rails plugin, we might run this command
|
228
|
+
from our RAILS_ROOT:
|
229
|
+
|
230
|
+
$ QUEUE=file_serve rake environment resque:work
|
231
|
+
|
232
|
+
This will load the environment before starting a worker. Alternately
|
233
|
+
we can define a `resque:setup` task with a dependency on the
|
234
|
+
`environment` rake task:
|
235
|
+
|
236
|
+
``` ruby
|
237
|
+
task "resque:setup" => :environment
|
238
|
+
```
|
239
|
+
|
240
|
+
GitHub's setup task looks like this:
|
241
|
+
|
242
|
+
``` ruby
|
243
|
+
task "resque:setup" => :environment do
|
244
|
+
Grit::Git.git_timeout = 10.minutes
|
245
|
+
end
|
246
|
+
```
|
247
|
+
|
248
|
+
We don't want the `git_timeout` as high as 10 minutes in our web app,
|
249
|
+
but in the Resque workers it's fine.
|
250
|
+
|
251
|
+
|
252
|
+
### Logging
|
253
|
+
|
254
|
+
Workers support basic logging to STDOUT. If you start them with the
|
255
|
+
`VERBOSE` env variable set, they will print basic debugging
|
256
|
+
information. You can also set the `VVERBOSE` (very verbose) env
|
257
|
+
variable.
|
258
|
+
|
259
|
+
$ VVERBOSE=1 QUEUE=file_serve rake environment resque:work
|
260
|
+
|
261
|
+
### Process IDs (PIDs)
|
262
|
+
|
263
|
+
There are scenarios where it's helpful to record the PID of a resque
|
264
|
+
worker process. Use the PIDFILE option for easy access to the PID:
|
265
|
+
|
266
|
+
$ PIDFILE=./resque.pid QUEUE=file_serve rake environment resque:work
|
267
|
+
|
268
|
+
### Running in the background
|
269
|
+
|
270
|
+
(Only supported with ruby >= 1.9). There are scenarios where it's helpful for
|
271
|
+
the resque worker to run itself in the background (usually in combination with
|
272
|
+
PIDFILE). Use the BACKGROUND option so that rake will return as soon as the
|
273
|
+
worker is started.
|
274
|
+
|
275
|
+
$ PIDFILE=./resque.pid BACKGROUND=yes QUEUE=file_serve \
|
276
|
+
rake environment resque:work
|
277
|
+
|
278
|
+
### Polling frequency
|
279
|
+
|
280
|
+
You can pass an INTERVAL option which is a float representing the polling frequency.
|
281
|
+
The default is 5 seconds, but for a semi-active app you may want to use a smaller value.
|
282
|
+
|
283
|
+
$ INTERVAL=0.1 QUEUE=file_serve rake environment resque:work
|
284
|
+
|
285
|
+
### Priorities and Queue Lists
|
286
|
+
|
287
|
+
Resque doesn't support numeric priorities but instead uses the order
|
288
|
+
of queues you give it. We call this list of queues the "queue list."
|
289
|
+
|
290
|
+
Let's say we add a `warm_cache` queue in addition to our `file_serve`
|
291
|
+
queue. We'd now start a worker like so:
|
292
|
+
|
293
|
+
$ QUEUES=file_serve,warm_cache rake resque:work
|
294
|
+
|
295
|
+
When the worker looks for new jobs, it will first check
|
296
|
+
`file_serve`. If it finds a job, it'll process it then check
|
297
|
+
`file_serve` again. It will keep checking `file_serve` until no more
|
298
|
+
jobs are available. At that point, it will check `warm_cache`. If it
|
299
|
+
finds a job it'll process it then check `file_serve` (repeating the
|
300
|
+
whole process).
|
301
|
+
|
302
|
+
In this way you can prioritize certain queues. At GitHub we start our
|
303
|
+
workers with something like this:
|
304
|
+
|
305
|
+
$ QUEUES=critical,archive,high,low rake resque:work
|
306
|
+
|
307
|
+
Notice the `archive` queue - it is specialized and in our future
|
308
|
+
architecture will only be run from a single machine.
|
309
|
+
|
310
|
+
At that point we'll start workers on our generalized background
|
311
|
+
machines with this command:
|
312
|
+
|
313
|
+
$ QUEUES=critical,high,low rake resque:work
|
314
|
+
|
315
|
+
And workers on our specialized archive machine with this command:
|
316
|
+
|
317
|
+
$ QUEUE=archive rake resque:work
|
318
|
+
|
319
|
+
|
320
|
+
### Running All Queues
|
321
|
+
|
322
|
+
If you want your workers to work off of every queue, including new
|
323
|
+
queues created on the fly, you can use a splat:
|
324
|
+
|
325
|
+
$ QUEUE=* rake resque:work
|
326
|
+
|
327
|
+
Queues will be processed in alphabetical order.
|
328
|
+
|
329
|
+
|
330
|
+
### Running Multiple Workers
|
331
|
+
|
332
|
+
At GitHub we use god to start and stop multiple workers. A sample god
|
333
|
+
configuration file is included under `examples/god`. We recommend this
|
334
|
+
method.
|
335
|
+
|
336
|
+
If you'd like to run multiple workers in development mode, you can do
|
337
|
+
so using the `resque:workers` rake task:
|
338
|
+
|
339
|
+
$ COUNT=5 QUEUE=* rake resque:workers
|
340
|
+
|
341
|
+
This will spawn five Resque workers, each in its own thread. Hitting
|
342
|
+
ctrl-c should be sufficient to stop them all.
|
343
|
+
|
344
|
+
|
345
|
+
### Forking
|
346
|
+
|
347
|
+
On certain platforms, when a Resque worker reserves a job it
|
348
|
+
immediately forks a child process. The child processes the job then
|
349
|
+
exits. When the child has exited successfully, the worker reserves
|
350
|
+
another job and repeats the process.
|
351
|
+
|
352
|
+
Why?
|
353
|
+
|
354
|
+
Because Resque assumes chaos.
|
355
|
+
|
356
|
+
Resque assumes your background workers will lock up, run too long, or
|
357
|
+
have unwanted memory growth.
|
358
|
+
|
359
|
+
If Resque workers processed jobs themselves, it'd be hard to whip them
|
360
|
+
into shape. Let's say one is using too much memory: you send it a
|
361
|
+
signal that says "shutdown after you finish processing the current
|
362
|
+
job," and it does so. It then starts up again - loading your entire
|
363
|
+
application environment. This adds useless CPU cycles and causes a
|
364
|
+
delay in queue processing.
|
365
|
+
|
366
|
+
Plus, what if it's using too much memory and has stopped responding to
|
367
|
+
signals?
|
368
|
+
|
369
|
+
Thanks to Resque's parent / child architecture, jobs that use too much memory
|
370
|
+
release that memory upon completion. No unwanted growth.
|
371
|
+
|
372
|
+
And what if a job is running too long? You'd need to `kill -9` it then
|
373
|
+
start the worker again. With Resque's parent / child architecture you
|
374
|
+
can tell the parent to forcefully kill the child then immediately
|
375
|
+
start processing more jobs. No startup delay or wasted cycles.
|
376
|
+
|
377
|
+
The parent / child architecture helps us keep tabs on what workers are
|
378
|
+
doing, too. By eliminating the need to `kill -9` workers we can have
|
379
|
+
parents remove themselves from the global listing of workers. If we
|
380
|
+
just ruthlessly killed workers, we'd need a separate watchdog process
|
381
|
+
to add and remove them to the global listing - which becomes
|
382
|
+
complicated.
|
383
|
+
|
384
|
+
Workers instead handle their own state.
|
385
|
+
|
386
|
+
|
387
|
+
### Parents and Children
|
388
|
+
|
389
|
+
Here's a parent / child pair doing some work:
|
390
|
+
|
391
|
+
$ ps -e -o pid,command | grep [r]esque
|
392
|
+
92099 resque: Forked 92102 at 1253142769
|
393
|
+
92102 resque: Processing file_serve since 1253142769
|
394
|
+
|
395
|
+
You can clearly see that process 92099 forked 92102, which has been
|
396
|
+
working since 1253142769.
|
397
|
+
|
398
|
+
(By advertising the time they began processing you can easily use monit
|
399
|
+
or god to kill stale workers.)
|
400
|
+
|
401
|
+
When a parent process is idle, it lets you know what queues it is
|
402
|
+
waiting for work on:
|
403
|
+
|
404
|
+
$ ps -e -o pid,command | grep [r]esque
|
405
|
+
92099 resque: Waiting for file_serve,warm_cache
|
406
|
+
|
407
|
+
|
408
|
+
### Signals
|
409
|
+
|
410
|
+
Resque workers respond to a few different signals:
|
411
|
+
|
412
|
+
* `QUIT` - Wait for child to finish processing then exit
|
413
|
+
* `TERM` / `INT` - Immediately kill child then exit
|
414
|
+
* `USR1` - Immediately kill child but don't exit
|
415
|
+
* `USR2` - Don't start to process any new jobs
|
416
|
+
* `CONT` - Start to process new jobs again after a USR2
|
417
|
+
|
418
|
+
If you want to gracefully shutdown a Resque worker, use `QUIT`.
|
419
|
+
|
420
|
+
If you want to kill a stale or stuck child, use `USR1`. Processing
|
421
|
+
will continue as normal unless the child was not found. In that case
|
422
|
+
Resque assumes the parent process is in a bad state and shuts down.
|
423
|
+
|
424
|
+
If you want to kill a stale or stuck child and shutdown, use `TERM`
|
425
|
+
|
426
|
+
If you want to stop processing jobs, but want to leave the worker running
|
427
|
+
(for example, to temporarily alleviate load), use `USR2` to stop processing,
|
428
|
+
then `CONT` to start it again.
|
429
|
+
|
430
|
+
### Mysql::Error: MySQL server has gone away
|
431
|
+
|
432
|
+
If your workers remain idle for too long they may lose their MySQL
|
433
|
+
connection. If that happens we recommend using [this
|
434
|
+
Gist](http://gist.github.com/238999).
|
435
|
+
|
436
|
+
|
437
|
+
The Front End
|
438
|
+
-------------
|
439
|
+
|
440
|
+
Resque comes with a Sinatra-based front end for seeing what's up with
|
441
|
+
your queue.
|
442
|
+
|
443
|
+
![The Front End](https://img.skitch.com/20110528-pc67a8qsfapgjxf5gagxd92fcu.png)
|
444
|
+
|
445
|
+
### Standalone
|
446
|
+
|
447
|
+
If you've installed Resque as a gem running the front end standalone is easy:
|
448
|
+
|
449
|
+
$ resque-web
|
450
|
+
|
451
|
+
It's a thin layer around `rackup` so it's configurable as well:
|
452
|
+
|
453
|
+
$ resque-web -p 8282
|
454
|
+
|
455
|
+
If you have a Resque config file you want evaluated just pass it to
|
456
|
+
the script as the final argument:
|
457
|
+
|
458
|
+
$ resque-web -p 8282 rails_root/config/initializers/resque.rb
|
459
|
+
|
460
|
+
You can also set the namespace directly using `resque-web`:
|
461
|
+
|
462
|
+
$ resque-web -p 8282 -N myapp
|
463
|
+
|
464
|
+
or set the Redis connection string if you need to do something like select a different database:
|
465
|
+
|
466
|
+
$ resque-web -p 8282 -r localhost:6379:2
|
467
|
+
|
468
|
+
### Passenger
|
469
|
+
|
470
|
+
Using Passenger? Resque ships with a `config.ru` you can use. See
|
471
|
+
Phusion's guide:
|
472
|
+
|
473
|
+
Apache: <http://www.modrails.com/documentation/Users%20guide%20Apache.html#_deploying_a_rack_based_ruby_application>
|
474
|
+
Nginx: <http://www.modrails.com/documentation/Users%20guide%20Nginx.html#deploying_a_rack_app>
|
475
|
+
|
476
|
+
### Rack::URLMap
|
477
|
+
|
478
|
+
If you want to load Resque on a subpath, possibly alongside other
|
479
|
+
apps, it's easy to do with Rack's `URLMap`:
|
480
|
+
|
481
|
+
``` ruby
|
482
|
+
require 'resque/server'
|
483
|
+
|
484
|
+
run Rack::URLMap.new \
|
485
|
+
"/" => Your::App.new,
|
486
|
+
"/resque" => Resque::Server.new
|
487
|
+
```
|
488
|
+
|
489
|
+
Check `examples/demo/config.ru` for a functional example (including
|
490
|
+
HTTP basic auth).
|
491
|
+
|
492
|
+
### Rails 3
|
493
|
+
|
494
|
+
You can also mount Resque on a subpath in your existing Rails 3 app by adding `require 'resque/server'` to the top of your routes file or in an initializer then adding this to `routes.rb`:
|
495
|
+
|
496
|
+
``` ruby
|
497
|
+
mount Resque::Server.new, :at => "/resque"
|
498
|
+
```
|
499
|
+
|
500
|
+
|
501
|
+
Resque vs DelayedJob
|
502
|
+
--------------------
|
503
|
+
|
504
|
+
How does Resque compare to DelayedJob, and why would you choose one
|
505
|
+
over the other?
|
506
|
+
|
507
|
+
* Resque supports multiple queues
|
508
|
+
* DelayedJob supports finer grained priorities
|
509
|
+
* Resque workers are resilient to memory leaks / bloat
|
510
|
+
* DelayedJob workers are extremely simple and easy to modify
|
511
|
+
* Resque requires Redis
|
512
|
+
* DelayedJob requires ActiveRecord
|
513
|
+
* Resque can only place JSONable Ruby objects on a queue as arguments
|
514
|
+
* DelayedJob can place _any_ Ruby object on its queue as arguments
|
515
|
+
* Resque includes a Sinatra app for monitoring what's going on
|
516
|
+
* DelayedJob can be queried from within your Rails app if you want to
|
517
|
+
add an interface
|
518
|
+
|
519
|
+
If you're doing Rails development, you already have a database and
|
520
|
+
ActiveRecord. DelayedJob is super easy to setup and works great.
|
521
|
+
GitHub used it for many months to process almost 200 million jobs.
|
522
|
+
|
523
|
+
Choose Resque if:
|
524
|
+
|
525
|
+
* You need multiple queues
|
526
|
+
* You don't care / dislike numeric priorities
|
527
|
+
* You don't need to persist every Ruby object ever
|
528
|
+
* You have potentially huge queues
|
529
|
+
* You want to see what's going on
|
530
|
+
* You expect a lot of failure / chaos
|
531
|
+
* You can setup Redis
|
532
|
+
* You're not running short on RAM
|
533
|
+
|
534
|
+
Choose DelayedJob if:
|
535
|
+
|
536
|
+
* You like numeric priorities
|
537
|
+
* You're not doing a gigantic amount of jobs each day
|
538
|
+
* Your queue stays small and nimble
|
539
|
+
* There is not a lot failure / chaos
|
540
|
+
* You want to easily throw anything on the queue
|
541
|
+
* You don't want to setup Redis
|
542
|
+
|
543
|
+
In no way is Resque a "better" DelayedJob, so make sure you pick the
|
544
|
+
tool that's best for your app.
|
545
|
+
|
546
|
+
|
547
|
+
Installing Redis
|
548
|
+
----------------
|
549
|
+
|
550
|
+
Resque requires Redis 0.900 or higher.
|
551
|
+
|
552
|
+
Resque uses Redis' lists for its queues. It also stores worker state
|
553
|
+
data in Redis.
|
554
|
+
|
555
|
+
#### Homebrew
|
556
|
+
|
557
|
+
If you're on OS X, Homebrew is the simplest way to install Redis:
|
558
|
+
|
559
|
+
$ brew install redis
|
560
|
+
$ redis-server /usr/local/etc/redis.conf
|
561
|
+
|
562
|
+
You now have a Redis daemon running on 6379.
|
563
|
+
|
564
|
+
#### Via Resque
|
565
|
+
|
566
|
+
Resque includes Rake tasks (thanks to Ezra's redis-rb) that will
|
567
|
+
install and run Redis for you:
|
568
|
+
|
569
|
+
$ git clone git://github.com/defunkt/resque.git
|
570
|
+
$ cd resque
|
571
|
+
$ rake redis:install dtach:install
|
572
|
+
$ rake redis:start
|
573
|
+
|
574
|
+
Or, if you don't have admin access on your machine:
|
575
|
+
|
576
|
+
$ git clone git://github.com/defunkt/resque.git
|
577
|
+
$ cd resque
|
578
|
+
$ PREFIX=<your_prefix> rake redis:install dtach:install
|
579
|
+
$ rake redis:start
|
580
|
+
|
581
|
+
You now have Redis running on 6379. Wait a second then hit ctrl-\ to
|
582
|
+
detach and keep it running in the background.
|
583
|
+
|
584
|
+
The demo is probably the best way to figure out how to put the parts
|
585
|
+
together. But, it's not that hard.
|
586
|
+
|
587
|
+
|
588
|
+
Resque Dependencies
|
589
|
+
-------------------
|
590
|
+
|
591
|
+
$ gem install bundler
|
592
|
+
$ bundle install
|
593
|
+
|
594
|
+
|
595
|
+
Installing Resque
|
596
|
+
-----------------
|
597
|
+
|
598
|
+
### In a Rack app, as a gem
|
599
|
+
|
600
|
+
First install the gem.
|
601
|
+
|
602
|
+
$ gem install resque
|
603
|
+
|
604
|
+
Next include it in your application.
|
605
|
+
|
606
|
+
``` ruby
|
607
|
+
require 'resque'
|
608
|
+
```
|
609
|
+
|
610
|
+
Now start your application:
|
611
|
+
|
612
|
+
rackup config.ru
|
613
|
+
|
614
|
+
That's it! You can now create Resque jobs from within your app.
|
615
|
+
|
616
|
+
To start a worker, create a Rakefile in your app's root (or add this
|
617
|
+
to an existing Rakefile):
|
618
|
+
|
619
|
+
``` ruby
|
620
|
+
require 'your/app'
|
621
|
+
require 'resque/tasks'
|
622
|
+
```
|
623
|
+
|
624
|
+
Now:
|
625
|
+
|
626
|
+
$ QUEUE=* rake resque:work
|
627
|
+
|
628
|
+
Alternately you can define a `resque:setup` hook in your Rakefile if you
|
629
|
+
don't want to load your app every time rake runs.
|
630
|
+
|
631
|
+
|
632
|
+
### In a Rails 2.x app, as a gem
|
633
|
+
|
634
|
+
First install the gem.
|
635
|
+
|
636
|
+
$ gem install resque
|
637
|
+
|
638
|
+
Next include it in your application.
|
639
|
+
|
640
|
+
$ cat config/initializers/load_resque.rb
|
641
|
+
require 'resque'
|
642
|
+
|
643
|
+
Now start your application:
|
644
|
+
|
645
|
+
$ ./script/server
|
646
|
+
|
647
|
+
That's it! You can now create Resque jobs from within your app.
|
648
|
+
|
649
|
+
To start a worker, add this to your Rakefile in `RAILS_ROOT`:
|
650
|
+
|
651
|
+
``` ruby
|
652
|
+
require 'resque/tasks'
|
653
|
+
```
|
654
|
+
|
655
|
+
Now:
|
656
|
+
|
657
|
+
$ QUEUE=* rake environment resque:work
|
658
|
+
|
659
|
+
Don't forget you can define a `resque:setup` hook in
|
660
|
+
`lib/tasks/whatever.rake` that loads the `environment` task every time.
|
661
|
+
|
662
|
+
|
663
|
+
### In a Rails 2.x app, as a plugin
|
664
|
+
|
665
|
+
$ ./script/plugin install git://github.com/defunkt/resque
|
666
|
+
|
667
|
+
That's it! Resque will automatically be available when your Rails app
|
668
|
+
loads.
|
669
|
+
|
670
|
+
To start a worker:
|
671
|
+
|
672
|
+
$ QUEUE=* rake environment resque:work
|
673
|
+
|
674
|
+
Don't forget you can define a `resque:setup` hook in
|
675
|
+
`lib/tasks/whatever.rake` that loads the `environment` task every time.
|
676
|
+
|
677
|
+
|
678
|
+
### In a Rails 3 app, as a gem
|
679
|
+
|
680
|
+
First include it in your Gemfile.
|
681
|
+
|
682
|
+
$ cat Gemfile
|
683
|
+
...
|
684
|
+
gem 'resque'
|
685
|
+
...
|
686
|
+
|
687
|
+
Next install it with Bundler.
|
688
|
+
|
689
|
+
$ bundle install
|
690
|
+
|
691
|
+
Now start your application:
|
692
|
+
|
693
|
+
$ rails server
|
694
|
+
|
695
|
+
That's it! You can now create Resque jobs from within your app.
|
696
|
+
|
697
|
+
To start a worker, add this to a file in `lib/tasks` (ex:
|
698
|
+
`lib/tasks/resque.rake`):
|
699
|
+
|
700
|
+
``` ruby
|
701
|
+
require 'resque/tasks'
|
702
|
+
```
|
703
|
+
|
704
|
+
Now:
|
705
|
+
|
706
|
+
$ QUEUE=* rake environment resque:work
|
707
|
+
|
708
|
+
Don't forget you can define a `resque:setup` hook in
|
709
|
+
`lib/tasks/whatever.rake` that loads the `environment` task every time.
|
710
|
+
|
711
|
+
|
712
|
+
Configuration
|
713
|
+
-------------
|
714
|
+
|
715
|
+
You may want to change the Redis host and port Resque connects to, or
|
716
|
+
set various other options at startup.
|
717
|
+
|
718
|
+
Resque has a `redis` setter which can be given a string or a Redis
|
719
|
+
object. This means if you're already using Redis in your app, Resque
|
720
|
+
can re-use the existing connection.
|
721
|
+
|
722
|
+
String: `Resque.redis = 'localhost:6379'`
|
723
|
+
|
724
|
+
Redis: `Resque.redis = $redis`
|
725
|
+
|
726
|
+
For our rails app we have a `config/initializers/resque.rb` file where
|
727
|
+
we load `config/resque.yml` by hand and set the Redis information
|
728
|
+
appropriately.
|
729
|
+
|
730
|
+
Here's our `config/resque.yml`:
|
731
|
+
|
732
|
+
development: localhost:6379
|
733
|
+
test: localhost:6379
|
734
|
+
staging: redis1.se.github.com:6379
|
735
|
+
fi: localhost:6379
|
736
|
+
production: redis1.ae.github.com:6379
|
737
|
+
|
738
|
+
And our initializer:
|
739
|
+
|
740
|
+
``` ruby
|
741
|
+
rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..'
|
742
|
+
rails_env = ENV['RAILS_ENV'] || 'development'
|
743
|
+
|
744
|
+
resque_config = YAML.load_file(rails_root + '/config/resque.yml')
|
745
|
+
Resque.redis = resque_config[rails_env]
|
746
|
+
```
|
747
|
+
|
748
|
+
Easy peasy! Why not just use `RAILS_ROOT` and `RAILS_ENV`? Because
|
749
|
+
this way we can tell our Sinatra app about the config file:
|
750
|
+
|
751
|
+
$ RAILS_ENV=production resque-web rails_root/config/initializers/resque.rb
|
752
|
+
|
753
|
+
Now everyone is on the same page.
|
754
|
+
|
755
|
+
Also, you could disable jobs queueing by setting 'inline' attribute.
|
756
|
+
For example, if you want to run all jobs in the same process for cucumber, try:
|
757
|
+
|
758
|
+
``` ruby
|
759
|
+
Resque.inline = ENV['RAILS_ENV'] == "cucumber"
|
760
|
+
```
|
761
|
+
|
762
|
+
|
763
|
+
Plugins and Hooks
|
764
|
+
-----------------
|
765
|
+
|
766
|
+
For a list of available plugins see
|
767
|
+
<http://wiki.github.com/defunkt/resque/plugins>.
|
768
|
+
|
769
|
+
If you'd like to write your own plugin, or want to customize Resque
|
770
|
+
using hooks (such as `Resque.after_fork`), see
|
771
|
+
[docs/HOOKS.md](http://github.com/defunkt/resque/blob/master/docs/HOOKS.md).
|
772
|
+
|
773
|
+
|
774
|
+
Namespaces
|
775
|
+
----------
|
776
|
+
|
777
|
+
If you're running multiple, separate instances of Resque you may want
|
778
|
+
to namespace the keyspaces so they do not overlap. This is not unlike
|
779
|
+
the approach taken by many memcached clients.
|
780
|
+
|
781
|
+
This feature is provided by the [redis-namespace][rs] library, which
|
782
|
+
Resque uses by default to separate the keys it manages from other keys
|
783
|
+
in your Redis server.
|
784
|
+
|
785
|
+
Simply use the `Resque.redis.namespace` accessor:
|
786
|
+
|
787
|
+
``` ruby
|
788
|
+
Resque.redis.namespace = "resque:GitHub"
|
789
|
+
```
|
790
|
+
|
791
|
+
We recommend sticking this in your initializer somewhere after Redis
|
792
|
+
is configured.
|
793
|
+
|
794
|
+
|
795
|
+
Demo
|
796
|
+
----
|
797
|
+
|
798
|
+
Resque ships with a demo Sinatra app for creating jobs that are later
|
799
|
+
processed in the background.
|
800
|
+
|
801
|
+
Try it out by looking at the README, found at `examples/demo/README.markdown`.
|
802
|
+
|
803
|
+
|
804
|
+
Monitoring
|
805
|
+
----------
|
806
|
+
|
807
|
+
### god
|
808
|
+
|
809
|
+
If you're using god to monitor Resque, we have provided example
|
810
|
+
configs in `examples/god/`. One is for starting / stopping workers,
|
811
|
+
the other is for killing workers that have been running too long.
|
812
|
+
|
813
|
+
### monit
|
814
|
+
|
815
|
+
If you're using monit, `examples/monit/resque.monit` is provided free
|
816
|
+
of charge. This is **not** used by GitHub in production, so please
|
817
|
+
send patches for any tweaks or improvements you can make to it.
|
818
|
+
|
819
|
+
|
820
|
+
Questions
|
821
|
+
---------
|
822
|
+
|
823
|
+
Please add them to the [FAQ](https://github.com/defunkt/resque/wiki/FAQ) or
|
824
|
+
ask on the Mailing List. The Mailing List is explained further below
|
825
|
+
|
826
|
+
|
827
|
+
Development
|
828
|
+
-----------
|
829
|
+
|
830
|
+
Want to hack on Resque?
|
831
|
+
|
832
|
+
First clone the repo and run the tests:
|
833
|
+
|
834
|
+
git clone git://github.com/defunkt/resque.git
|
835
|
+
cd resque
|
836
|
+
rake test
|
837
|
+
|
838
|
+
If the tests do not pass make sure you have Redis installed
|
839
|
+
correctly (though we make an effort to tell you if we feel this is the
|
840
|
+
case). The tests attempt to start an isolated instance of Redis to
|
841
|
+
run against.
|
842
|
+
|
843
|
+
Also make sure you've installed all the dependencies correctly. For
|
844
|
+
example, try loading the `redis-namespace` gem after you've installed
|
845
|
+
it:
|
846
|
+
|
847
|
+
$ irb
|
848
|
+
>> require 'rubygems'
|
849
|
+
=> true
|
850
|
+
>> require 'redis/namespace'
|
851
|
+
=> true
|
852
|
+
|
853
|
+
If you get an error requiring any of the dependencies, you may have
|
854
|
+
failed to install them or be seeing load path issues.
|
855
|
+
|
856
|
+
Feel free to ping the mailing list with your problem and we'll try to
|
857
|
+
sort it out.
|
858
|
+
|
859
|
+
|
860
|
+
Contributing
|
861
|
+
------------
|
862
|
+
|
863
|
+
Read the [Contributing][cb] wiki page first.
|
864
|
+
|
865
|
+
Once you've made your great commits:
|
866
|
+
|
867
|
+
1. [Fork][1] Resque
|
868
|
+
2. Create a topic branch - `git checkout -b my_branch`
|
869
|
+
3. Push to your branch - `git push origin my_branch`
|
870
|
+
4. Create a [Pull Request](http://help.github.com/pull-requests/) from your branch
|
871
|
+
5. That's it!
|
872
|
+
|
873
|
+
|
874
|
+
Mailing List
|
875
|
+
------------
|
876
|
+
|
877
|
+
To join the list simply send an email to <resque@librelist.com>. This
|
878
|
+
will subscribe you and send you information about your subscription,
|
879
|
+
including unsubscribe information.
|
880
|
+
|
881
|
+
The archive can be found at <http://librelist.com/browser/resque/>.
|
882
|
+
|
883
|
+
|
884
|
+
Meta
|
885
|
+
----
|
886
|
+
|
887
|
+
* Code: `git clone git://github.com/defunkt/resque.git`
|
888
|
+
* Home: <http://github.com/defunkt/resque>
|
889
|
+
* Docs: <http://defunkt.github.com/resque/>
|
890
|
+
* Bugs: <http://github.com/defunkt/resque/issues>
|
891
|
+
* List: <resque@librelist.com>
|
892
|
+
* Chat: <irc://irc.freenode.net/resque>
|
893
|
+
* Gems: <http://gemcutter.org/gems/resque>
|
894
|
+
|
895
|
+
This project uses [Semantic Versioning][sv].
|
896
|
+
|
897
|
+
|
898
|
+
Author
|
899
|
+
------
|
900
|
+
|
901
|
+
Chris Wanstrath :: chris@ozmm.org :: @defunkt
|
902
|
+
|
903
|
+
[0]: http://github.com/blog/542-introducing-resque
|
904
|
+
[1]: http://help.github.com/forking/
|
905
|
+
[2]: http://github.com/defunkt/resque/issues
|
906
|
+
[sv]: http://semver.org/
|
907
|
+
[rs]: http://github.com/defunkt/redis-namespace
|
908
|
+
[cb]: http://wiki.github.com/defunkt/resque/contributing
|