opengotham_resque 1.8.2

Sign up to get free protection for your applications and to get access to all the features.
Files changed (70) hide show
  1. data/.gitignore +2 -0
  2. data/.kick +26 -0
  3. data/HISTORY.md +142 -0
  4. data/LICENSE +20 -0
  5. data/README.markdown +794 -0
  6. data/Rakefile +112 -0
  7. data/bin/resque +57 -0
  8. data/bin/resque-web +23 -0
  9. data/config.ru +14 -0
  10. data/deps.rip +7 -0
  11. data/docs/HOOKS.md +121 -0
  12. data/docs/PLUGINS.md +93 -0
  13. data/examples/async_helper.rb +31 -0
  14. data/examples/demo/README.markdown +71 -0
  15. data/examples/demo/Rakefile +8 -0
  16. data/examples/demo/app.rb +38 -0
  17. data/examples/demo/config.ru +19 -0
  18. data/examples/demo/job.rb +22 -0
  19. data/examples/god/resque.god +53 -0
  20. data/examples/god/stale.god +26 -0
  21. data/examples/instance.rb +11 -0
  22. data/examples/monit/resque.monit +6 -0
  23. data/examples/simple.rb +30 -0
  24. data/init.rb +1 -0
  25. data/lib/resque.rb +287 -0
  26. data/lib/resque/errors.rb +10 -0
  27. data/lib/resque/failure.rb +66 -0
  28. data/lib/resque/failure/base.rb +61 -0
  29. data/lib/resque/failure/hoptoad.rb +132 -0
  30. data/lib/resque/failure/multiple.rb +48 -0
  31. data/lib/resque/failure/redis.rb +40 -0
  32. data/lib/resque/helpers.rb +63 -0
  33. data/lib/resque/job.rb +207 -0
  34. data/lib/resque/plugin.rb +46 -0
  35. data/lib/resque/server.rb +201 -0
  36. data/lib/resque/server/public/idle.png +0 -0
  37. data/lib/resque/server/public/jquery-1.3.2.min.js +19 -0
  38. data/lib/resque/server/public/jquery.relatize_date.js +95 -0
  39. data/lib/resque/server/public/poll.png +0 -0
  40. data/lib/resque/server/public/ranger.js +67 -0
  41. data/lib/resque/server/public/reset.css +48 -0
  42. data/lib/resque/server/public/style.css +81 -0
  43. data/lib/resque/server/public/working.png +0 -0
  44. data/lib/resque/server/test_helper.rb +19 -0
  45. data/lib/resque/server/views/error.erb +1 -0
  46. data/lib/resque/server/views/failed.erb +53 -0
  47. data/lib/resque/server/views/key_sets.erb +20 -0
  48. data/lib/resque/server/views/key_string.erb +11 -0
  49. data/lib/resque/server/views/layout.erb +44 -0
  50. data/lib/resque/server/views/next_more.erb +10 -0
  51. data/lib/resque/server/views/overview.erb +4 -0
  52. data/lib/resque/server/views/queues.erb +49 -0
  53. data/lib/resque/server/views/stats.erb +62 -0
  54. data/lib/resque/server/views/workers.erb +78 -0
  55. data/lib/resque/server/views/working.erb +69 -0
  56. data/lib/resque/stat.rb +53 -0
  57. data/lib/resque/tasks.rb +39 -0
  58. data/lib/resque/version.rb +3 -0
  59. data/lib/resque/worker.rb +478 -0
  60. data/tasks/redis.rake +159 -0
  61. data/tasks/resque.rake +2 -0
  62. data/test/job_hooks_test.rb +302 -0
  63. data/test/job_plugins_test.rb +209 -0
  64. data/test/plugin_test.rb +116 -0
  65. data/test/redis-test.conf +132 -0
  66. data/test/resque-web_test.rb +54 -0
  67. data/test/resque_test.rb +225 -0
  68. data/test/test_helper.rb +111 -0
  69. data/test/worker_test.rb +302 -0
  70. metadata +199 -0
@@ -0,0 +1,2 @@
1
+ pkg
2
+ nbproject
data/.kick ADDED
@@ -0,0 +1,26 @@
1
+ # take control of the growl notifications
2
+ module GrowlHacks
3
+ def growl(type, subject, body, *args, &block)
4
+ case type
5
+ when Kicker::GROWL_NOTIFICATIONS[:succeeded]
6
+ puts subject = "Success"
7
+ body = body.split("\n").last
8
+ when Kicker::GROWL_NOTIFICATIONS[:failed]
9
+ subject = "Failure"
10
+ puts body
11
+ body = body.split("\n").last
12
+ else
13
+ return nil
14
+ end
15
+ super(type, subject, body, *args, &block)
16
+ end
17
+ end
18
+
19
+ Kicker.send :extend, GrowlHacks
20
+
21
+ # no logging
22
+ Kicker::Utils.module_eval do
23
+ def log(message)
24
+ nil
25
+ end
26
+ end
@@ -0,0 +1,142 @@
1
+ ## 1.8.2 (2010-05-03)
2
+
3
+ * Bugfix: Include "tasks/" dir in RubyGem
4
+
5
+ ## 1.8.1 (2010-04-29)
6
+
7
+ * Bugfix: Multiple failure backend did not support requeue-ing failed jobs
8
+ * Bugfix: Fix /failed when error has no backtrace
9
+ * Bugfix: Add `Redis::DistRedis` as a valid client
10
+
11
+ ## 1.8.0 (2010-04-07)
12
+
13
+ * Jobs that never complete due to killed worker are now failed.
14
+ * Worker "working" state is now maintained by the parent, not the child.
15
+ * Stopped using deprecated redis.rb methods
16
+ * `Worker.working` race condition fixed
17
+ * `Worker#process` has been deprecated.
18
+ * Monit example fixed
19
+ * Redis::Client and Redis::Namespace can be passed to `Resque.redis=`
20
+
21
+ ## 1.7.1 (2010-04-02)
22
+
23
+ * Bugfix: Make job hook execution order consistent
24
+ * Bugfix: stdout buffering in child process
25
+
26
+ ## 1.7.0 (2010-03-31)
27
+
28
+ * Job hooks API. See docs/HOOKS.md.
29
+ * web: Hovering over dates shows a timestamp
30
+ * web: AJAXify retry action for failed jobs
31
+ * web bugfix: Fix pagination bug
32
+
33
+ ## 1.6.1 (2010-03-25)
34
+
35
+ * Bugfix: Workers may not be clearing their state correctly on
36
+ shutdown
37
+ * Added example monit config.
38
+ * Exception class is now recorded when an error is raised in a
39
+ worker.
40
+ * web: Unit tests
41
+ * web: Show namespace in header and footer
42
+ * web: Remove a queue
43
+ * web: Retry failed jobs
44
+
45
+ ## 1.6.0 (2010-03-09)
46
+
47
+ * Added `before_first_fork`, `before_fork`, and `after_fork` hooks.
48
+ * Hoptoad: Added server_environment config setting
49
+ * Hoptoad bugfix: Don't depend on RAILS_ROOT
50
+ * 1.8.6 compat fixes
51
+
52
+ ## 1.5.2 (2010-03-03)
53
+
54
+ * Bugfix: JSON check was crazy.
55
+
56
+ ## 1.5.1 (2010-03-03)
57
+
58
+ * `Job.destroy` and `Resque.dequeue` return the # of destroyed jobs.
59
+ * Hoptoad notifier improvements
60
+ * Specify the namespace with `resque-web` by passing `-N namespace`
61
+ * Bugfix: Don't crash when trying to parse invalid JSON.
62
+ * Bugfix: Non-standard namespace support
63
+ * Web: Red backgound for queue "failed" only shown if there are failed jobs.
64
+ * Web bugfix: Tabs highlight properly now
65
+ * Web bugfix: ZSET partial support in stats
66
+ * Web bugfix: Deleting failed jobs works again
67
+ * Web bugfix: Sets (or zsets, lists, etc) now paginate.
68
+
69
+ ## 1.5.0 (2010-02-17)
70
+
71
+ * Version now included in procline, e.g. `resque-1.5.0: Message`
72
+ * Web bugfix: Ignore idle works in the "working" page
73
+ * Added `Resque::Job.destroy(queue, klass, *args)`
74
+ * Added `Resque.dequeue(klass, *args)`
75
+
76
+ ## 1.4.0 (2010-02-11)
77
+
78
+ * Fallback when unable to bind QUIT and USR1 for Windows and JRuby.
79
+ * Fallback when no `Kernel.fork` is provided (for IronRuby).
80
+ * Web: Rounded corners in Firefox
81
+ * Cut down system calls in `Worker#prune_dead_workers`
82
+ * Enable switching DB in a Redis server from config
83
+ * Support USR2 and CONT to stop and start job processing.
84
+ * Web: Add example failing job
85
+ * Bugfix: `Worker#unregister_worker` shouldn't call `done_working`
86
+ * Bugfix: Example god config now restarts Resque properly.
87
+ * Multiple failure backends now permitted.
88
+ * Hoptoad failure backend updated to new API
89
+
90
+ ## 1.3.1 (2010-01-11)
91
+
92
+ * Vegas bugfix: Don't error without a config
93
+
94
+ ## 1.3.0 (2010-01-11)
95
+
96
+ * Use Vegas for resque-web
97
+ * Web Bugfix: Show proper date/time value for failed_at on Failures
98
+ * Web Bugfix: Make the / route more flexible
99
+ * Add Resque::Server.tabs array (so plugins can add their own tabs)
100
+ * Start using [Semantic Versioning](http://semver.org/)
101
+
102
+ ## 1.2.4 (2009-12-15)
103
+
104
+ * Web Bugfix: fix key links on stat page
105
+
106
+ ## 1.2.3 (2009-12-15)
107
+
108
+ * Bugfix: Fixed `rand` seeding in child processes.
109
+ * Bugfix: Better JSON encoding/decoding without Yajl.
110
+ * Bugfix: Avoid `ps` flag error on Linux
111
+ * Add `PREFIX` observance to `rake` install tasks.
112
+
113
+ ## 1.2.2 (2009-12-08)
114
+
115
+ * Bugfix: Job equality was not properly implemented.
116
+
117
+ ## 1.2.1 (2009-12-07)
118
+
119
+ * Added `rake resque:workers` task for starting multiple workers.
120
+ * 1.9.x compatibility
121
+ * Bugfix: Yajl decoder doesn't care about valid UTF-8
122
+ * config.ru loads RESQUECONFIG if the ENV variable is set.
123
+ * `resque-web` now sets RESQUECONFIG
124
+ * Job objects know if they are equal.
125
+ * Jobs can be re-queued using `Job#recreate`
126
+
127
+ ## 1.2.0 (2009-11-25)
128
+
129
+ * If USR1 is sent and no child is found, shutdown.
130
+ * Raise when a job class does not respond to `perform`.
131
+ * Added `Resque.remove_queue` for deleting a queue
132
+
133
+ ## 1.1.0 (2009-11-04)
134
+
135
+ * Bugfix: Broken ERB tag in failure UI
136
+ * Bugfix: Save the worker's ID, not the worker itself, in the failure module
137
+ * Redesigned the sinatra web interface
138
+ * Added option to clear failed jobs
139
+
140
+ ## 1.0.0 (2009-11-03)
141
+
142
+ * First release.
data/LICENSE ADDED
@@ -0,0 +1,20 @@
1
+ Copyright (c) 2009 Chris Wanstrath
2
+
3
+ Permission is hereby granted, free of charge, to any person obtaining
4
+ a copy of this software and associated documentation files (the
5
+ "Software"), to deal in the Software without restriction, including
6
+ without limitation the rights to use, copy, modify, merge, publish,
7
+ distribute, sublicense, and/or sell copies of the Software, and to
8
+ permit persons to whom the Software is furnished to do so, subject to
9
+ the following conditions:
10
+
11
+ The above copyright notice and this permission notice shall be
12
+ included in all copies or substantial portions of the Software.
13
+
14
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
15
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
16
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
17
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
18
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
19
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
20
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@@ -0,0 +1,794 @@
1
+ Resque
2
+ ======
3
+
4
+ Resque is a Redis-backed library for creating background jobs, placing
5
+ those jobs on multiple queues, and processing them later.
6
+
7
+ Background jobs can be any Ruby class or module that responds to
8
+ `perform`. Your existing classes can easily be converted to background
9
+ jobs or you can create new classes specifically to do work. Or, you
10
+ can do both.
11
+
12
+ Resque is heavily inspired by DelayedJob (which rocks) and is
13
+ comprised of three parts:
14
+
15
+ 1. A Ruby library for creating, querying, and processing jobs
16
+ 2. A Rake task for starting a worker which processes jobs
17
+ 3. A Sinatra app for monitoring queues, jobs, and workers.
18
+
19
+ Resque workers can be distributed between multiple machines,
20
+ support priorities, are resilient to memory bloat / "leaks," are
21
+ optimized for REE (but work on MRI and JRuby), tell you what they're
22
+ doing, and expect failure.
23
+
24
+ Resque queues are persistent; support constant time, atomic push and
25
+ pop (thanks to Redis); provide visibility into their contents; and
26
+ store jobs as simple JSON packages.
27
+
28
+ The Resque frontend tells you what workers are doing, what workers are
29
+ not doing, what queues you're using, what's in those queues, provides
30
+ general usage stats, and helps you track failures.
31
+
32
+
33
+ The Blog Post
34
+ -------------
35
+
36
+ For the backstory, philosophy, and history of Resque's beginnings,
37
+ please see [the blog post][0].
38
+
39
+
40
+ Overview
41
+ --------
42
+
43
+ Resque allows you to create jobs and place them on a queue, then,
44
+ later, pull those jobs off the queue and process them.
45
+
46
+ Resque jobs are Ruby classes (or modules) which respond to the
47
+ `perform` method. Here's an example:
48
+
49
+ class Archive
50
+ @queue = :file_serve
51
+
52
+ def self.perform(repo_id, branch = 'master')
53
+ repo = Repository.find(repo_id)
54
+ repo.create_archive(branch)
55
+ end
56
+ end
57
+
58
+ The `@queue` class instance variable determines which queue `Archive`
59
+ jobs will be placed in. Queues are arbitrary and created on the fly -
60
+ you can name them whatever you want and have as many as you want.
61
+
62
+ To place an `Archive` job on the `file_serve` queue, we might add this
63
+ to our application's pre-existing `Repository` class:
64
+
65
+ class Repository
66
+ def async_create_archive(branch)
67
+ Resque.enqueue(Archive, self.id, branch)
68
+ end
69
+ end
70
+
71
+ Now when we call `repo.async_create_archive('masterbrew')` in our
72
+ application, a job will be created and placed on the `file_serve`
73
+ queue.
74
+
75
+ Later, a worker will run something like this code to process the job:
76
+
77
+ klass, args = Resque.reserve(:file_serve)
78
+ klass.perform(*args) if klass.respond_to? :perform
79
+
80
+ Which translates to:
81
+
82
+ Archive.perform(44, 'masterbrew')
83
+
84
+ Let's start a worker to run `file_serve` jobs:
85
+
86
+ $ cd app_root
87
+ $ QUEUE=file_serve rake resque:work
88
+
89
+ This starts one Resque worker and tells it to work off the
90
+ `file_serve` queue. As soon as it's ready it'll try to run the
91
+ `Resque.reserve` code snippet above and process jobs until it can't
92
+ find any more, at which point it will sleep for a small period and
93
+ repeatedly poll the queue for more jobs.
94
+
95
+ Workers can be given multiple queues (a "queue list") and run on
96
+ multiple machines. In fact they can be run anywhere with network
97
+ access to the Redis server.
98
+
99
+
100
+ Jobs
101
+ ----
102
+
103
+ What should you run in the background? Anything that takes any time at
104
+ all. Slow INSERT statements, disk manipulating, data processing, etc.
105
+
106
+ At GitHub we use Resque to process the following types of jobs:
107
+
108
+ * Warming caches
109
+ * Counting disk usage
110
+ * Building tarballs
111
+ * Building Rubygems
112
+ * Firing off web hooks
113
+ * Creating events in the db and pre-caching them
114
+ * Building graphs
115
+ * Deleting users
116
+ * Updating our search index
117
+
118
+ As of writing we have about 35 different types of background jobs.
119
+
120
+ Keep in mind that you don't need a web app to use Resque - we just
121
+ mention "foreground" and "background" because they make conceptual
122
+ sense. You could easily be spidering sites and sticking data which
123
+ needs to be crunched later into a queue.
124
+
125
+
126
+ ### Persistence
127
+
128
+ Jobs are persisted to queues as JSON objects. Let's take our `Archive`
129
+ example from above. We'll run the following code to create a job:
130
+
131
+ repo = Repository.find(44)
132
+ repo.async_create_archive('masterbrew')
133
+
134
+ The following JSON will be stored in the `file_serve` queue:
135
+
136
+ {
137
+ 'class': 'Archive',
138
+ 'args': [ 44, 'masterbrew' ]
139
+ }
140
+
141
+ Because of this your jobs must only accept arguments that can be JSON encoded.
142
+
143
+ So instead of doing this:
144
+
145
+ Resque.enqueue(Archive, self, branch)
146
+
147
+ do this:
148
+
149
+ Resque.enqueue(Archive, self.id, branch)
150
+
151
+ This is why our above example (and all the examples in `examples/`)
152
+ uses object IDs instead of passing around the objects.
153
+
154
+ While this is less convenient than just sticking a marshaled object
155
+ in the database, it gives you a slight advantage: your jobs will be
156
+ run against the most recent version of an object because they need to
157
+ pull from the DB or cache.
158
+
159
+ If your jobs were run against marshaled objects, they could
160
+ potentially be operating on a stale record with out-of-date information.
161
+
162
+
163
+ ### send_later / async
164
+
165
+ Want something like DelayedJob's `send_later` or the ability to use
166
+ instance methods instead of just methods for jobs? See the `examples/`
167
+ directory for goodies.
168
+
169
+ We plan to provide first class `async` support in a future release.
170
+
171
+
172
+ ### Failure
173
+
174
+ If a job raises an exception, it is logged and handed off to the
175
+ `Resque::Failure` module. Failures are logged either locally in Redis
176
+ or using some different backend.
177
+
178
+ For example, Resque ships with Hoptoad support.
179
+
180
+ Keep this in mind when writing your jobs: you may want to throw
181
+ exceptions you would not normally throw in order to assist debugging.
182
+
183
+
184
+ Workers
185
+ -------
186
+
187
+ Resque workers are rake tasks that run forever. They basically do this:
188
+
189
+ start
190
+ loop do
191
+ if job = reserve
192
+ job.process
193
+ else
194
+ sleep 5
195
+ end
196
+ end
197
+ shutdown
198
+
199
+ Starting a worker is simple. Here's our example from earlier:
200
+
201
+ $ QUEUE=file_serve rake resque:work
202
+
203
+ By default Resque won't know about your application's
204
+ environment. That is, it won't be able to find and run your jobs - it
205
+ needs to load your application into memory.
206
+
207
+ If we've installed Resque as a Rails plugin, we might run this command
208
+ from our RAILS_ROOT:
209
+
210
+ $ QUEUE=file_serve rake environment resque:work
211
+
212
+ This will load the environment before starting a worker. Alternately
213
+ we can define a `resque:setup` task with a dependency on the
214
+ `environment` rake task:
215
+
216
+ task "resque:setup" => :environment
217
+
218
+ GitHub's setup task looks like this:
219
+
220
+ task "resque:setup" => :environment do
221
+ Grit::Git.git_timeout = 10.minutes
222
+ end
223
+
224
+ We don't want the `git_timeout` as high as 10 minutes in our web app,
225
+ but in the Resque workers it's fine.
226
+
227
+
228
+ ### Logging
229
+
230
+ Workers support basic logging to STDOUT. If you start them with the
231
+ `VERBOSE` env variable set, they will print basic debugging
232
+ information. You can also set the `VVERBOSE` (very verbose) env
233
+ variable.
234
+
235
+ $ VVERBOSE=1 QUEUE=file_serve rake environment resque:work
236
+
237
+
238
+ ### Priorities and Queue Lists
239
+
240
+ Resque doesn't support numeric priorities but instead uses the order
241
+ of queues you give it. We call this list of queues the "queue list."
242
+
243
+ Let's say we add a `warm_cache` queue in addition to our `file_serve`
244
+ queue. We'd now start a worker like so:
245
+
246
+ $ QUEUES=file_serve,warm_cache rake resque:work
247
+
248
+ When the worker looks for new jobs, it will first check
249
+ `file_serve`. If it finds a job, it'll process it then check
250
+ `file_serve` again. It will keep checking `file_serve` until no more
251
+ jobs are available. At that point, it will check `warm_cache`. If it
252
+ finds a job it'll process it then check `file_serve` (repeating the
253
+ whole process).
254
+
255
+ In this way you can prioritize certain queues. At GitHub we start our
256
+ workers with something like this:
257
+
258
+ $ QUEUES=critical,archive,high,low rake resque:work
259
+
260
+ Notice the `archive` queue - it is specialized and in our future
261
+ architecture will only be run from a single machine.
262
+
263
+ At that point we'll start workers on our generalized background
264
+ machines with this command:
265
+
266
+ $ QUEUES=critical,high,low rake resque:work
267
+
268
+ And workers on our specialized archive machine with this command:
269
+
270
+ $ QUEUE=archive rake resque:work
271
+
272
+
273
+ ### Running All Queues
274
+
275
+ If you want your workers to work off of every queue, including new
276
+ queues created on the fly, you can use a splat:
277
+
278
+ $ QUEUE=* rake resque:work
279
+
280
+ Queues will be processed in alphabetical order.
281
+
282
+
283
+ ### Running Multiple Workers
284
+
285
+ At GitHub we use god to start and stop multiple workers. A sample god
286
+ configuration file is included under `examples/god`. We recommend this
287
+ method.
288
+
289
+ If you'd like to run multiple workers in development mode, you can do
290
+ so using the `resque:workers` rake task:
291
+
292
+ $ COUNT=5 QUEUE=* rake resque:workers
293
+
294
+ This will spawn five Resque workers, each in its own thread. Hitting
295
+ ctrl-c should be sufficient to stop them all.
296
+
297
+
298
+ ### Forking
299
+
300
+ On certain platforms, when a Resque worker reserves a job it
301
+ immediately forks a child process. The child processes the job then
302
+ exits. When the child has exited successfully, the worker reserves
303
+ another job and repeats the process.
304
+
305
+ Why?
306
+
307
+ Because Resque assumes chaos.
308
+
309
+ Resque assumes your background workers will lock up, run too long, or
310
+ have unwanted memory growth.
311
+
312
+ If Resque workers processed jobs themselves, it'd be hard to whip them
313
+ into shape. Let's say one is using too much memory: you send it a
314
+ signal that says "shutdown after you finish processing the current
315
+ job," and it does so. It then starts up again - loading your entire
316
+ application environment. This adds useless CPU cycles and causes a
317
+ delay in queue processing.
318
+
319
+ Plus, what if it's using too much memory and has stopped responding to
320
+ signals?
321
+
322
+ Thanks to Resque's parent / child architecture, jobs that use too much memory
323
+ release that memory upon completion. No unwanted growth.
324
+
325
+ And what if a job is running too long? You'd need to `kill -9` it then
326
+ start the worker again. With Resque's parent / child architecture you
327
+ can tell the parent to forcefully kill the child then immediately
328
+ start processing more jobs. No startup delay or wasted cycles.
329
+
330
+ The parent / child architecture helps us keep tabs on what workers are
331
+ doing, too. By eliminating the need to `kill -9` workers we can have
332
+ parents remove themselves from the global listing of workers. If we
333
+ just ruthlessly killed workers, we'd need a separate watchdog process
334
+ to add and remove them to the global listing - which becomes
335
+ complicated.
336
+
337
+ Workers instead handle their own state.
338
+
339
+
340
+ ### Parents and Children
341
+
342
+ Here's a parent / child pair doing some work:
343
+
344
+ $ ps -e -o pid,command | grep [r]esque
345
+ 92099 resque: Forked 92102 at 1253142769
346
+ 92102 resque: Processing file_serve since 1253142769
347
+
348
+ You can clearly see that process 92099 forked 92102, which has been
349
+ working since 1253142769.
350
+
351
+ (By advertising the time they began processing you can easily use monit
352
+ or god to kill stale workers.)
353
+
354
+ When a parent process is idle, it lets you know what queues it is
355
+ waiting for work on:
356
+
357
+ $ ps -e -o pid,command | grep [r]esque
358
+ 92099 resque: Waiting for file_serve,warm_cache
359
+
360
+
361
+ ### Signals
362
+
363
+ Resque workers respond to a few different signals:
364
+
365
+ * `QUIT` - Wait for child to finish processing then exit
366
+ * `TERM` / `INT` - Immediately kill child then exit
367
+ * `USR1` - Immediately kill child but don't exit
368
+ * `USR2` - Don't start to process any new jobs
369
+ * `CONT` - Start to process new jobs again after a USR2
370
+
371
+ If you want to gracefully shutdown a Resque worker, use `QUIT`.
372
+
373
+ If you want to kill a stale or stuck child, use `USR1`. Processing
374
+ will continue as normal unless the child was not found. In that case
375
+ Resque assumes the parent process is in a bad state and shuts down.
376
+
377
+ If you want to kill a stale or stuck child and shutdown, use `TERM`
378
+
379
+ If you want to stop processing jobs, but want to leave the worker running
380
+ (for example, to temporarily alleviate load), use `USR2` to stop processing,
381
+ then `CONT` to start it again.
382
+
383
+ ### Mysql::Error: MySQL server has gone away
384
+
385
+ If your workers remain idle for too long they may lose their MySQL
386
+ connection. If that happens we recommend using [this
387
+ Gist](http://gist.github.com/238999).
388
+
389
+
390
+ The Front End
391
+ -------------
392
+
393
+ Resque comes with a Sinatra-based front end for seeing what's up with
394
+ your queue.
395
+
396
+ ![The Front End](http://img.skitch.com/20091104-tqh5pgkwgbskjbk7qbtmpesnyw.jpg)
397
+
398
+ ### Standalone
399
+
400
+ If you've installed Resque as a gem running the front end standalone is easy:
401
+
402
+ $ resque-web
403
+
404
+ It's a thin layer around `rackup` so it's configurable as well:
405
+
406
+ $ resque-web -p 8282
407
+
408
+ If you have a Resque config file you want evaluated just pass it to
409
+ the script as the final argument:
410
+
411
+ $ resque-web -p 8282 rails_root/config/initializers/resque.rb
412
+
413
+ You can also set the namespace directly using `resque-web`:
414
+
415
+ $ resque-web -p 8282 -N myapp
416
+
417
+ ### Passenger
418
+
419
+ Using Passenger? Resque ships with a `config.ru` you can use. See
420
+ Phusion's guide:
421
+
422
+ <http://www.modrails.com/documentation/Users%20guide.html#_deploying_a_rack_based_ruby_application>
423
+
424
+ ### Rack::URLMap
425
+
426
+ If you want to load Resque on a subpath, possibly alongside other
427
+ apps, it's easy to do with Rack's `URLMap`:
428
+
429
+ require 'resque/server'
430
+
431
+ run Rack::URLMap.new \
432
+ "/" => Your::App.new,
433
+ "/resque" => Resque::Server.new
434
+
435
+ Check `examples/demo/config.ru` for a functional example (including
436
+ HTTP basic auth).
437
+
438
+
439
+ Resque vs DelayedJob
440
+ --------------------
441
+
442
+ How does Resque compare to DelayedJob, and why would you choose one
443
+ over the other?
444
+
445
+ * Resque supports multiple queues
446
+ * DelayedJob supports finer grained priorities
447
+ * Resque workers are resilient to memory leaks / bloat
448
+ * DelayedJob workers are extremely simple and easy to modify
449
+ * Resque requires Redis
450
+ * DelayedJob requires ActiveRecord
451
+ * Resque can only place JSONable Ruby objects on a queue as arguments
452
+ * DelayedJob can place _any_ Ruby object on its queue as arguments
453
+ * Resque includes a Sinatra app for monitoring what's going on
454
+ * DelayedJob can be queried from within your Rails app if you want to
455
+ add an interface
456
+
457
+ If you're doing Rails development, you already have a database and
458
+ ActiveRecord. DelayedJob is super easy to setup and works great.
459
+ GitHub used it for many months to process almost 200 million jobs.
460
+
461
+ Choose Resque if:
462
+
463
+ * You need multiple queues
464
+ * You don't care / dislike numeric priorities
465
+ * You don't need to persist every Ruby object ever
466
+ * You have potentially huge queues
467
+ * You want to see what's going on
468
+ * You expect a lot of failure / chaos
469
+ * You can setup Redis
470
+ * You're not running short on RAM
471
+
472
+ Choose DelayedJob if:
473
+
474
+ * You like numeric priorities
475
+ * You're not doing a gigantic amount of jobs each day
476
+ * Your queue stays small and nimble
477
+ * There is not a lot failure / chaos
478
+ * You want to easily throw anything on the queue
479
+ * You don't want to setup Redis
480
+
481
+ In no way is Resque a "better" DelayedJob, so make sure you pick the
482
+ tool that's best for your app.
483
+
484
+
485
+ Installing Redis
486
+ ----------------
487
+
488
+ Resque requires Redis 0.900 or higher.
489
+
490
+ Resque uses Redis' lists for its queues. It also stores worker state
491
+ data in Redis.
492
+
493
+ #### Homebrew
494
+
495
+ If you're on OS X, Homebrew is the simplest way to install Redis:
496
+
497
+ $ brew install redis
498
+ $ redis-server /usr/local/etc/redis.conf
499
+
500
+ You now have a Redis daemon running on 6379.
501
+
502
+ #### Via Resque
503
+
504
+ Resque includes Rake tasks (thanks to Ezra's redis-rb) that will
505
+ install and run Redis for you:
506
+
507
+ $ git clone git://github.com/defunkt/resque.git
508
+ $ cd resque
509
+ $ rake redis:install dtach:install
510
+ $ rake redis:start
511
+
512
+ Or, if you don't have admin access on your machine:
513
+
514
+ $ git clone git://github.com/defunkt/resque.git
515
+ $ cd resque
516
+ $ PREFIX=<your_prefix> rake redis:install dtach:install
517
+ $ rake redis:start
518
+
519
+ You now have Redis running on 6379. Wait a second then hit ctrl-\ to
520
+ detach and keep it running in the background.
521
+
522
+ The demo is probably the best way to figure out how to put the parts
523
+ together. But, it's not that hard.
524
+
525
+
526
+ Resque Dependencies
527
+ -------------------
528
+
529
+ gem install redis redis-namespace yajl-ruby
530
+
531
+ If you cannot install `yajl-ruby` (JRuby?), you can install the `json`
532
+ gem and Resque will use it instead.
533
+
534
+ When problems arise, make sure you have the newest versions of the
535
+ `redis` and `redis-namespace` gems.
536
+
537
+
538
+ Installing Resque
539
+ -----------------
540
+
541
+ ### In a Rack app, as a gem
542
+
543
+ First install the gem.
544
+
545
+ $ gem install resque
546
+
547
+ Next include it in your application.
548
+
549
+ require 'resque'
550
+
551
+ Now start your application:
552
+
553
+ rackup config.ru
554
+
555
+ That's it! You can now create Resque jobs from within your app.
556
+
557
+ To start a worker, create a Rakefile in your app's root (or add this
558
+ to an existing Rakefile):
559
+
560
+ require 'your/app'
561
+ require 'resque/tasks'
562
+
563
+ Now:
564
+
565
+ $ QUEUE=* rake resque:work
566
+
567
+ Alternately you can define a `resque:setup` hook in your Rakefile if you
568
+ don't want to load your app every time rake runs.
569
+
570
+
571
+ ### In a Rails app, as a gem
572
+
573
+ First install the gem.
574
+
575
+ $ gem install resque
576
+
577
+ Next include it in your application.
578
+
579
+ $ cat config/initializers/load_resque.rb
580
+ require 'resque'
581
+
582
+ Now start your application:
583
+
584
+ $ ./script/server
585
+
586
+ That's it! You can now create Resque jobs from within your app.
587
+
588
+ To start a worker, add this to your Rakefile in `RAILS_ROOT`:
589
+
590
+ require 'resque/tasks'
591
+
592
+ Now:
593
+
594
+ $ QUEUE=* rake environment resque:work
595
+
596
+ Don't forget you can define a `resque:setup` hook in
597
+ `lib/tasks/whatever.rake` that loads the `environment` task every time.
598
+
599
+
600
+ ### In a Rails app, as a plugin
601
+
602
+ $ ./script/plugin install git://github.com/defunkt/resque
603
+
604
+ That's it! Resque will automatically be available when your Rails app
605
+ loads.
606
+
607
+ To start a worker:
608
+
609
+ $ QUEUE=* rake environment resque:work
610
+
611
+ Don't forget you can define a `resque:setup` hook in
612
+ `lib/tasks/whatever.rake` that loads the `environment` task every time.
613
+
614
+
615
+ Configuration
616
+ -------------
617
+
618
+ You may want to change the Redis host and port Resque connects to, or
619
+ set various other options at startup.
620
+
621
+ Resque has a `redis` setter which can be given a string or a Redis
622
+ object. This means if you're already using Redis in your app, Resque
623
+ can re-use the existing connection.
624
+
625
+ String: `Resque.redis = 'localhost:6379'`
626
+
627
+ Redis: `Resque.redis = $redis`
628
+
629
+ For our rails app we have a `config/initializers/resque.rb` file where
630
+ we load `config/resque.yml` by hand and set the Redis information
631
+ appropriately.
632
+
633
+ Here's our `config/resque.yml`:
634
+
635
+ development: localhost:6379
636
+ test: localhost:6379
637
+ staging: redis1.se.github.com:6379
638
+ fi: localhost:6379
639
+ production: redis1.ae.github.com:6379
640
+
641
+ And our initializer:
642
+
643
+ rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..'
644
+ rails_env = ENV['RAILS_ENV'] || 'development'
645
+
646
+ resque_config = YAML.load_file(rails_root + '/config/resque.yml')
647
+ Resque.redis = resque_config[rails_env]
648
+
649
+ Easy peasy! Why not just use `RAILS_ROOT` and `RAILS_ENV`? Because
650
+ this way we can tell our Sinatra app about the config file:
651
+
652
+ $ RAILS_ENV=production resque-web rails_root/config/initializers/resque.rb
653
+
654
+ Now everyone is on the same page.
655
+
656
+
657
+ Plugins and Hooks
658
+ -----------------
659
+
660
+ For a list of available plugins see
661
+ <http://wiki.github.com/defunkt/resque/plugins>.
662
+
663
+ If you'd like to write your own plugin, or want to customize Resque
664
+ using hooks (such as `Resque.after_fork`), see
665
+ [docs/HOOKS.md](http://github.com/defunkt/resque/blob/master/HOOKS.md).
666
+
667
+
668
+ Namespaces
669
+ ----------
670
+
671
+ If you're running multiple, separate instances of Resque you may want
672
+ to namespace the keyspaces so they do not overlap. This is not unlike
673
+ the approach taken by many memcached clients.
674
+
675
+ This feature is provided by the [redis-namespace][rs] library, which
676
+ Resque uses by default to separate the keys it manages from other keys
677
+ in your Redis server.
678
+
679
+ Simply use the `Resque.redis.namespace` accessor:
680
+
681
+ Resque.redis.namespace = "resque:GitHub"
682
+
683
+ We recommend sticking this in your initializer somewhere after Redis
684
+ is configured.
685
+
686
+
687
+ Demo
688
+ ----
689
+
690
+ Resque ships with a demo Sinatra app for creating jobs that are later
691
+ processed in the background.
692
+
693
+ Try it out by looking at the README, found at `examples/demo/README.markdown`.
694
+
695
+
696
+ Monitoring
697
+ ----------
698
+
699
+ ### god
700
+
701
+ If you're using god to monitor Resque, we have provided example
702
+ configs in `examples/god/`. One is for starting / stopping workers,
703
+ the other is for killing workers that have been running too long.
704
+
705
+ ### monit
706
+
707
+ If you're using monit, `examples/monit/resque.monit` is provided free
708
+ of charge. This is **not** used by GitHub in production, so please
709
+ send patches for any tweaks or improvements you can make to it.
710
+
711
+
712
+ Development
713
+ -----------
714
+
715
+ Want to hack on Resque?
716
+
717
+ First clone the repo and run the tests:
718
+
719
+ git clone git://github.com/defunkt/resque.git
720
+ cd resque
721
+ rake test
722
+
723
+ If the tests do not pass make sure you have Redis installed
724
+ correctly (though we make an effort to tell you if we feel this is the
725
+ case). The tests attempt to start an isolated instance of Redis to
726
+ run against.
727
+
728
+ Also make sure you've installed all the dependencies correctly. For
729
+ example, try loading the `redis-namespace` gem after you've installed
730
+ it:
731
+
732
+ $ irb
733
+ >> require 'rubygems'
734
+ => true
735
+ >> require 'redis/namespace'
736
+ => true
737
+
738
+ If you get an error requiring any of the dependencies, you may have
739
+ failed to install them or be seeing load path issues.
740
+
741
+ Feel free to ping the mailing list with your problem and we'll try to
742
+ sort it out.
743
+
744
+
745
+ Contributing
746
+ ------------
747
+
748
+ Once you've made your great commits:
749
+
750
+ 1. [Fork][1] Resque
751
+ 2. Create a topic branch - `git checkout -b my_branch`
752
+ 3. Push to your branch - `git push origin my_branch`
753
+ 4. Create an [Issue][2] with a link to your branch
754
+ 5. That's it!
755
+
756
+ You might want to checkout our [Contributing][cb] wiki page for information
757
+ on coding standards, new features, etc.
758
+
759
+
760
+ Mailing List
761
+ ------------
762
+
763
+ To join the list simply send an email to <resque@librelist.com>. This
764
+ will subscribe you and send you information about your subscription,
765
+ including unsubscribe information.
766
+
767
+ The archive can be found at <http://librelist.com/browser/>.
768
+
769
+
770
+ Meta
771
+ ----
772
+
773
+ * Code: `git clone git://github.com/defunkt/resque.git`
774
+ * Home: <http://github.com/defunkt/resque>
775
+ * Docs: <http://defunkt.github.com/resque/>
776
+ * Bugs: <http://github.com/defunkt/resque/issues>
777
+ * List: <resque@librelist.com>
778
+ * Chat: <irc://irc.freenode.net/resque>
779
+ * Gems: <http://gemcutter.org/gems/resque>
780
+
781
+ This project uses [Semantic Versioning][sv].
782
+
783
+
784
+ Author
785
+ ------
786
+
787
+ Chris Wanstrath :: chris@ozmm.org :: @defunkt
788
+
789
+ [0]: http://github.com/blog/542-introducing-resque
790
+ [1]: http://help.github.com/forking/
791
+ [2]: http://github.com/defunkt/resque/issues
792
+ [sv]: http://semver.org/
793
+ [rs]: http://github.com/defunkt/redis-namespace
794
+ [cb]: http://wiki.github.com/defunkt/resque/contributing