opengotham_resque 1.8.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (70) hide show
  1. data/.gitignore +2 -0
  2. data/.kick +26 -0
  3. data/HISTORY.md +142 -0
  4. data/LICENSE +20 -0
  5. data/README.markdown +794 -0
  6. data/Rakefile +112 -0
  7. data/bin/resque +57 -0
  8. data/bin/resque-web +23 -0
  9. data/config.ru +14 -0
  10. data/deps.rip +7 -0
  11. data/docs/HOOKS.md +121 -0
  12. data/docs/PLUGINS.md +93 -0
  13. data/examples/async_helper.rb +31 -0
  14. data/examples/demo/README.markdown +71 -0
  15. data/examples/demo/Rakefile +8 -0
  16. data/examples/demo/app.rb +38 -0
  17. data/examples/demo/config.ru +19 -0
  18. data/examples/demo/job.rb +22 -0
  19. data/examples/god/resque.god +53 -0
  20. data/examples/god/stale.god +26 -0
  21. data/examples/instance.rb +11 -0
  22. data/examples/monit/resque.monit +6 -0
  23. data/examples/simple.rb +30 -0
  24. data/init.rb +1 -0
  25. data/lib/resque.rb +287 -0
  26. data/lib/resque/errors.rb +10 -0
  27. data/lib/resque/failure.rb +66 -0
  28. data/lib/resque/failure/base.rb +61 -0
  29. data/lib/resque/failure/hoptoad.rb +132 -0
  30. data/lib/resque/failure/multiple.rb +48 -0
  31. data/lib/resque/failure/redis.rb +40 -0
  32. data/lib/resque/helpers.rb +63 -0
  33. data/lib/resque/job.rb +207 -0
  34. data/lib/resque/plugin.rb +46 -0
  35. data/lib/resque/server.rb +201 -0
  36. data/lib/resque/server/public/idle.png +0 -0
  37. data/lib/resque/server/public/jquery-1.3.2.min.js +19 -0
  38. data/lib/resque/server/public/jquery.relatize_date.js +95 -0
  39. data/lib/resque/server/public/poll.png +0 -0
  40. data/lib/resque/server/public/ranger.js +67 -0
  41. data/lib/resque/server/public/reset.css +48 -0
  42. data/lib/resque/server/public/style.css +81 -0
  43. data/lib/resque/server/public/working.png +0 -0
  44. data/lib/resque/server/test_helper.rb +19 -0
  45. data/lib/resque/server/views/error.erb +1 -0
  46. data/lib/resque/server/views/failed.erb +53 -0
  47. data/lib/resque/server/views/key_sets.erb +20 -0
  48. data/lib/resque/server/views/key_string.erb +11 -0
  49. data/lib/resque/server/views/layout.erb +44 -0
  50. data/lib/resque/server/views/next_more.erb +10 -0
  51. data/lib/resque/server/views/overview.erb +4 -0
  52. data/lib/resque/server/views/queues.erb +49 -0
  53. data/lib/resque/server/views/stats.erb +62 -0
  54. data/lib/resque/server/views/workers.erb +78 -0
  55. data/lib/resque/server/views/working.erb +69 -0
  56. data/lib/resque/stat.rb +53 -0
  57. data/lib/resque/tasks.rb +39 -0
  58. data/lib/resque/version.rb +3 -0
  59. data/lib/resque/worker.rb +478 -0
  60. data/tasks/redis.rake +159 -0
  61. data/tasks/resque.rake +2 -0
  62. data/test/job_hooks_test.rb +302 -0
  63. data/test/job_plugins_test.rb +209 -0
  64. data/test/plugin_test.rb +116 -0
  65. data/test/redis-test.conf +132 -0
  66. data/test/resque-web_test.rb +54 -0
  67. data/test/resque_test.rb +225 -0
  68. data/test/test_helper.rb +111 -0
  69. data/test/worker_test.rb +302 -0
  70. metadata +199 -0
@@ -0,0 +1,2 @@
1
+ pkg
2
+ nbproject
data/.kick ADDED
@@ -0,0 +1,26 @@
1
+ # take control of the growl notifications
2
+ module GrowlHacks
3
+ def growl(type, subject, body, *args, &block)
4
+ case type
5
+ when Kicker::GROWL_NOTIFICATIONS[:succeeded]
6
+ puts subject = "Success"
7
+ body = body.split("\n").last
8
+ when Kicker::GROWL_NOTIFICATIONS[:failed]
9
+ subject = "Failure"
10
+ puts body
11
+ body = body.split("\n").last
12
+ else
13
+ return nil
14
+ end
15
+ super(type, subject, body, *args, &block)
16
+ end
17
+ end
18
+
19
+ Kicker.send :extend, GrowlHacks
20
+
21
+ # no logging
22
+ Kicker::Utils.module_eval do
23
+ def log(message)
24
+ nil
25
+ end
26
+ end
@@ -0,0 +1,142 @@
1
+ ## 1.8.2 (2010-05-03)
2
+
3
+ * Bugfix: Include "tasks/" dir in RubyGem
4
+
5
+ ## 1.8.1 (2010-04-29)
6
+
7
+ * Bugfix: Multiple failure backend did not support requeue-ing failed jobs
8
+ * Bugfix: Fix /failed when error has no backtrace
9
+ * Bugfix: Add `Redis::DistRedis` as a valid client
10
+
11
+ ## 1.8.0 (2010-04-07)
12
+
13
+ * Jobs that never complete due to killed worker are now failed.
14
+ * Worker "working" state is now maintained by the parent, not the child.
15
+ * Stopped using deprecated redis.rb methods
16
+ * `Worker.working` race condition fixed
17
+ * `Worker#process` has been deprecated.
18
+ * Monit example fixed
19
+ * Redis::Client and Redis::Namespace can be passed to `Resque.redis=`
20
+
21
+ ## 1.7.1 (2010-04-02)
22
+
23
+ * Bugfix: Make job hook execution order consistent
24
+ * Bugfix: stdout buffering in child process
25
+
26
+ ## 1.7.0 (2010-03-31)
27
+
28
+ * Job hooks API. See docs/HOOKS.md.
29
+ * web: Hovering over dates shows a timestamp
30
+ * web: AJAXify retry action for failed jobs
31
+ * web bugfix: Fix pagination bug
32
+
33
+ ## 1.6.1 (2010-03-25)
34
+
35
+ * Bugfix: Workers may not be clearing their state correctly on
36
+ shutdown
37
+ * Added example monit config.
38
+ * Exception class is now recorded when an error is raised in a
39
+ worker.
40
+ * web: Unit tests
41
+ * web: Show namespace in header and footer
42
+ * web: Remove a queue
43
+ * web: Retry failed jobs
44
+
45
+ ## 1.6.0 (2010-03-09)
46
+
47
+ * Added `before_first_fork`, `before_fork`, and `after_fork` hooks.
48
+ * Hoptoad: Added server_environment config setting
49
+ * Hoptoad bugfix: Don't depend on RAILS_ROOT
50
+ * 1.8.6 compat fixes
51
+
52
+ ## 1.5.2 (2010-03-03)
53
+
54
+ * Bugfix: JSON check was crazy.
55
+
56
+ ## 1.5.1 (2010-03-03)
57
+
58
+ * `Job.destroy` and `Resque.dequeue` return the # of destroyed jobs.
59
+ * Hoptoad notifier improvements
60
+ * Specify the namespace with `resque-web` by passing `-N namespace`
61
+ * Bugfix: Don't crash when trying to parse invalid JSON.
62
+ * Bugfix: Non-standard namespace support
63
+ * Web: Red backgound for queue "failed" only shown if there are failed jobs.
64
+ * Web bugfix: Tabs highlight properly now
65
+ * Web bugfix: ZSET partial support in stats
66
+ * Web bugfix: Deleting failed jobs works again
67
+ * Web bugfix: Sets (or zsets, lists, etc) now paginate.
68
+
69
+ ## 1.5.0 (2010-02-17)
70
+
71
+ * Version now included in procline, e.g. `resque-1.5.0: Message`
72
+ * Web bugfix: Ignore idle works in the "working" page
73
+ * Added `Resque::Job.destroy(queue, klass, *args)`
74
+ * Added `Resque.dequeue(klass, *args)`
75
+
76
+ ## 1.4.0 (2010-02-11)
77
+
78
+ * Fallback when unable to bind QUIT and USR1 for Windows and JRuby.
79
+ * Fallback when no `Kernel.fork` is provided (for IronRuby).
80
+ * Web: Rounded corners in Firefox
81
+ * Cut down system calls in `Worker#prune_dead_workers`
82
+ * Enable switching DB in a Redis server from config
83
+ * Support USR2 and CONT to stop and start job processing.
84
+ * Web: Add example failing job
85
+ * Bugfix: `Worker#unregister_worker` shouldn't call `done_working`
86
+ * Bugfix: Example god config now restarts Resque properly.
87
+ * Multiple failure backends now permitted.
88
+ * Hoptoad failure backend updated to new API
89
+
90
+ ## 1.3.1 (2010-01-11)
91
+
92
+ * Vegas bugfix: Don't error without a config
93
+
94
+ ## 1.3.0 (2010-01-11)
95
+
96
+ * Use Vegas for resque-web
97
+ * Web Bugfix: Show proper date/time value for failed_at on Failures
98
+ * Web Bugfix: Make the / route more flexible
99
+ * Add Resque::Server.tabs array (so plugins can add their own tabs)
100
+ * Start using [Semantic Versioning](http://semver.org/)
101
+
102
+ ## 1.2.4 (2009-12-15)
103
+
104
+ * Web Bugfix: fix key links on stat page
105
+
106
+ ## 1.2.3 (2009-12-15)
107
+
108
+ * Bugfix: Fixed `rand` seeding in child processes.
109
+ * Bugfix: Better JSON encoding/decoding without Yajl.
110
+ * Bugfix: Avoid `ps` flag error on Linux
111
+ * Add `PREFIX` observance to `rake` install tasks.
112
+
113
+ ## 1.2.2 (2009-12-08)
114
+
115
+ * Bugfix: Job equality was not properly implemented.
116
+
117
+ ## 1.2.1 (2009-12-07)
118
+
119
+ * Added `rake resque:workers` task for starting multiple workers.
120
+ * 1.9.x compatibility
121
+ * Bugfix: Yajl decoder doesn't care about valid UTF-8
122
+ * config.ru loads RESQUECONFIG if the ENV variable is set.
123
+ * `resque-web` now sets RESQUECONFIG
124
+ * Job objects know if they are equal.
125
+ * Jobs can be re-queued using `Job#recreate`
126
+
127
+ ## 1.2.0 (2009-11-25)
128
+
129
+ * If USR1 is sent and no child is found, shutdown.
130
+ * Raise when a job class does not respond to `perform`.
131
+ * Added `Resque.remove_queue` for deleting a queue
132
+
133
+ ## 1.1.0 (2009-11-04)
134
+
135
+ * Bugfix: Broken ERB tag in failure UI
136
+ * Bugfix: Save the worker's ID, not the worker itself, in the failure module
137
+ * Redesigned the sinatra web interface
138
+ * Added option to clear failed jobs
139
+
140
+ ## 1.0.0 (2009-11-03)
141
+
142
+ * First release.
data/LICENSE ADDED
@@ -0,0 +1,20 @@
1
+ Copyright (c) 2009 Chris Wanstrath
2
+
3
+ Permission is hereby granted, free of charge, to any person obtaining
4
+ a copy of this software and associated documentation files (the
5
+ "Software"), to deal in the Software without restriction, including
6
+ without limitation the rights to use, copy, modify, merge, publish,
7
+ distribute, sublicense, and/or sell copies of the Software, and to
8
+ permit persons to whom the Software is furnished to do so, subject to
9
+ the following conditions:
10
+
11
+ The above copyright notice and this permission notice shall be
12
+ included in all copies or substantial portions of the Software.
13
+
14
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
15
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
16
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
17
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
18
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
19
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
20
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@@ -0,0 +1,794 @@
1
+ Resque
2
+ ======
3
+
4
+ Resque is a Redis-backed library for creating background jobs, placing
5
+ those jobs on multiple queues, and processing them later.
6
+
7
+ Background jobs can be any Ruby class or module that responds to
8
+ `perform`. Your existing classes can easily be converted to background
9
+ jobs or you can create new classes specifically to do work. Or, you
10
+ can do both.
11
+
12
+ Resque is heavily inspired by DelayedJob (which rocks) and is
13
+ comprised of three parts:
14
+
15
+ 1. A Ruby library for creating, querying, and processing jobs
16
+ 2. A Rake task for starting a worker which processes jobs
17
+ 3. A Sinatra app for monitoring queues, jobs, and workers.
18
+
19
+ Resque workers can be distributed between multiple machines,
20
+ support priorities, are resilient to memory bloat / "leaks," are
21
+ optimized for REE (but work on MRI and JRuby), tell you what they're
22
+ doing, and expect failure.
23
+
24
+ Resque queues are persistent; support constant time, atomic push and
25
+ pop (thanks to Redis); provide visibility into their contents; and
26
+ store jobs as simple JSON packages.
27
+
28
+ The Resque frontend tells you what workers are doing, what workers are
29
+ not doing, what queues you're using, what's in those queues, provides
30
+ general usage stats, and helps you track failures.
31
+
32
+
33
+ The Blog Post
34
+ -------------
35
+
36
+ For the backstory, philosophy, and history of Resque's beginnings,
37
+ please see [the blog post][0].
38
+
39
+
40
+ Overview
41
+ --------
42
+
43
+ Resque allows you to create jobs and place them on a queue, then,
44
+ later, pull those jobs off the queue and process them.
45
+
46
+ Resque jobs are Ruby classes (or modules) which respond to the
47
+ `perform` method. Here's an example:
48
+
49
+ class Archive
50
+ @queue = :file_serve
51
+
52
+ def self.perform(repo_id, branch = 'master')
53
+ repo = Repository.find(repo_id)
54
+ repo.create_archive(branch)
55
+ end
56
+ end
57
+
58
+ The `@queue` class instance variable determines which queue `Archive`
59
+ jobs will be placed in. Queues are arbitrary and created on the fly -
60
+ you can name them whatever you want and have as many as you want.
61
+
62
+ To place an `Archive` job on the `file_serve` queue, we might add this
63
+ to our application's pre-existing `Repository` class:
64
+
65
+ class Repository
66
+ def async_create_archive(branch)
67
+ Resque.enqueue(Archive, self.id, branch)
68
+ end
69
+ end
70
+
71
+ Now when we call `repo.async_create_archive('masterbrew')` in our
72
+ application, a job will be created and placed on the `file_serve`
73
+ queue.
74
+
75
+ Later, a worker will run something like this code to process the job:
76
+
77
+ klass, args = Resque.reserve(:file_serve)
78
+ klass.perform(*args) if klass.respond_to? :perform
79
+
80
+ Which translates to:
81
+
82
+ Archive.perform(44, 'masterbrew')
83
+
84
+ Let's start a worker to run `file_serve` jobs:
85
+
86
+ $ cd app_root
87
+ $ QUEUE=file_serve rake resque:work
88
+
89
+ This starts one Resque worker and tells it to work off the
90
+ `file_serve` queue. As soon as it's ready it'll try to run the
91
+ `Resque.reserve` code snippet above and process jobs until it can't
92
+ find any more, at which point it will sleep for a small period and
93
+ repeatedly poll the queue for more jobs.
94
+
95
+ Workers can be given multiple queues (a "queue list") and run on
96
+ multiple machines. In fact they can be run anywhere with network
97
+ access to the Redis server.
98
+
99
+
100
+ Jobs
101
+ ----
102
+
103
+ What should you run in the background? Anything that takes any time at
104
+ all. Slow INSERT statements, disk manipulating, data processing, etc.
105
+
106
+ At GitHub we use Resque to process the following types of jobs:
107
+
108
+ * Warming caches
109
+ * Counting disk usage
110
+ * Building tarballs
111
+ * Building Rubygems
112
+ * Firing off web hooks
113
+ * Creating events in the db and pre-caching them
114
+ * Building graphs
115
+ * Deleting users
116
+ * Updating our search index
117
+
118
+ As of writing we have about 35 different types of background jobs.
119
+
120
+ Keep in mind that you don't need a web app to use Resque - we just
121
+ mention "foreground" and "background" because they make conceptual
122
+ sense. You could easily be spidering sites and sticking data which
123
+ needs to be crunched later into a queue.
124
+
125
+
126
+ ### Persistence
127
+
128
+ Jobs are persisted to queues as JSON objects. Let's take our `Archive`
129
+ example from above. We'll run the following code to create a job:
130
+
131
+ repo = Repository.find(44)
132
+ repo.async_create_archive('masterbrew')
133
+
134
+ The following JSON will be stored in the `file_serve` queue:
135
+
136
+ {
137
+ 'class': 'Archive',
138
+ 'args': [ 44, 'masterbrew' ]
139
+ }
140
+
141
+ Because of this your jobs must only accept arguments that can be JSON encoded.
142
+
143
+ So instead of doing this:
144
+
145
+ Resque.enqueue(Archive, self, branch)
146
+
147
+ do this:
148
+
149
+ Resque.enqueue(Archive, self.id, branch)
150
+
151
+ This is why our above example (and all the examples in `examples/`)
152
+ uses object IDs instead of passing around the objects.
153
+
154
+ While this is less convenient than just sticking a marshaled object
155
+ in the database, it gives you a slight advantage: your jobs will be
156
+ run against the most recent version of an object because they need to
157
+ pull from the DB or cache.
158
+
159
+ If your jobs were run against marshaled objects, they could
160
+ potentially be operating on a stale record with out-of-date information.
161
+
162
+
163
+ ### send_later / async
164
+
165
+ Want something like DelayedJob's `send_later` or the ability to use
166
+ instance methods instead of just methods for jobs? See the `examples/`
167
+ directory for goodies.
168
+
169
+ We plan to provide first class `async` support in a future release.
170
+
171
+
172
+ ### Failure
173
+
174
+ If a job raises an exception, it is logged and handed off to the
175
+ `Resque::Failure` module. Failures are logged either locally in Redis
176
+ or using some different backend.
177
+
178
+ For example, Resque ships with Hoptoad support.
179
+
180
+ Keep this in mind when writing your jobs: you may want to throw
181
+ exceptions you would not normally throw in order to assist debugging.
182
+
183
+
184
+ Workers
185
+ -------
186
+
187
+ Resque workers are rake tasks that run forever. They basically do this:
188
+
189
+ start
190
+ loop do
191
+ if job = reserve
192
+ job.process
193
+ else
194
+ sleep 5
195
+ end
196
+ end
197
+ shutdown
198
+
199
+ Starting a worker is simple. Here's our example from earlier:
200
+
201
+ $ QUEUE=file_serve rake resque:work
202
+
203
+ By default Resque won't know about your application's
204
+ environment. That is, it won't be able to find and run your jobs - it
205
+ needs to load your application into memory.
206
+
207
+ If we've installed Resque as a Rails plugin, we might run this command
208
+ from our RAILS_ROOT:
209
+
210
+ $ QUEUE=file_serve rake environment resque:work
211
+
212
+ This will load the environment before starting a worker. Alternately
213
+ we can define a `resque:setup` task with a dependency on the
214
+ `environment` rake task:
215
+
216
+ task "resque:setup" => :environment
217
+
218
+ GitHub's setup task looks like this:
219
+
220
+ task "resque:setup" => :environment do
221
+ Grit::Git.git_timeout = 10.minutes
222
+ end
223
+
224
+ We don't want the `git_timeout` as high as 10 minutes in our web app,
225
+ but in the Resque workers it's fine.
226
+
227
+
228
+ ### Logging
229
+
230
+ Workers support basic logging to STDOUT. If you start them with the
231
+ `VERBOSE` env variable set, they will print basic debugging
232
+ information. You can also set the `VVERBOSE` (very verbose) env
233
+ variable.
234
+
235
+ $ VVERBOSE=1 QUEUE=file_serve rake environment resque:work
236
+
237
+
238
+ ### Priorities and Queue Lists
239
+
240
+ Resque doesn't support numeric priorities but instead uses the order
241
+ of queues you give it. We call this list of queues the "queue list."
242
+
243
+ Let's say we add a `warm_cache` queue in addition to our `file_serve`
244
+ queue. We'd now start a worker like so:
245
+
246
+ $ QUEUES=file_serve,warm_cache rake resque:work
247
+
248
+ When the worker looks for new jobs, it will first check
249
+ `file_serve`. If it finds a job, it'll process it then check
250
+ `file_serve` again. It will keep checking `file_serve` until no more
251
+ jobs are available. At that point, it will check `warm_cache`. If it
252
+ finds a job it'll process it then check `file_serve` (repeating the
253
+ whole process).
254
+
255
+ In this way you can prioritize certain queues. At GitHub we start our
256
+ workers with something like this:
257
+
258
+ $ QUEUES=critical,archive,high,low rake resque:work
259
+
260
+ Notice the `archive` queue - it is specialized and in our future
261
+ architecture will only be run from a single machine.
262
+
263
+ At that point we'll start workers on our generalized background
264
+ machines with this command:
265
+
266
+ $ QUEUES=critical,high,low rake resque:work
267
+
268
+ And workers on our specialized archive machine with this command:
269
+
270
+ $ QUEUE=archive rake resque:work
271
+
272
+
273
+ ### Running All Queues
274
+
275
+ If you want your workers to work off of every queue, including new
276
+ queues created on the fly, you can use a splat:
277
+
278
+ $ QUEUE=* rake resque:work
279
+
280
+ Queues will be processed in alphabetical order.
281
+
282
+
283
+ ### Running Multiple Workers
284
+
285
+ At GitHub we use god to start and stop multiple workers. A sample god
286
+ configuration file is included under `examples/god`. We recommend this
287
+ method.
288
+
289
+ If you'd like to run multiple workers in development mode, you can do
290
+ so using the `resque:workers` rake task:
291
+
292
+ $ COUNT=5 QUEUE=* rake resque:workers
293
+
294
+ This will spawn five Resque workers, each in its own thread. Hitting
295
+ ctrl-c should be sufficient to stop them all.
296
+
297
+
298
+ ### Forking
299
+
300
+ On certain platforms, when a Resque worker reserves a job it
301
+ immediately forks a child process. The child processes the job then
302
+ exits. When the child has exited successfully, the worker reserves
303
+ another job and repeats the process.
304
+
305
+ Why?
306
+
307
+ Because Resque assumes chaos.
308
+
309
+ Resque assumes your background workers will lock up, run too long, or
310
+ have unwanted memory growth.
311
+
312
+ If Resque workers processed jobs themselves, it'd be hard to whip them
313
+ into shape. Let's say one is using too much memory: you send it a
314
+ signal that says "shutdown after you finish processing the current
315
+ job," and it does so. It then starts up again - loading your entire
316
+ application environment. This adds useless CPU cycles and causes a
317
+ delay in queue processing.
318
+
319
+ Plus, what if it's using too much memory and has stopped responding to
320
+ signals?
321
+
322
+ Thanks to Resque's parent / child architecture, jobs that use too much memory
323
+ release that memory upon completion. No unwanted growth.
324
+
325
+ And what if a job is running too long? You'd need to `kill -9` it then
326
+ start the worker again. With Resque's parent / child architecture you
327
+ can tell the parent to forcefully kill the child then immediately
328
+ start processing more jobs. No startup delay or wasted cycles.
329
+
330
+ The parent / child architecture helps us keep tabs on what workers are
331
+ doing, too. By eliminating the need to `kill -9` workers we can have
332
+ parents remove themselves from the global listing of workers. If we
333
+ just ruthlessly killed workers, we'd need a separate watchdog process
334
+ to add and remove them to the global listing - which becomes
335
+ complicated.
336
+
337
+ Workers instead handle their own state.
338
+
339
+
340
+ ### Parents and Children
341
+
342
+ Here's a parent / child pair doing some work:
343
+
344
+ $ ps -e -o pid,command | grep [r]esque
345
+ 92099 resque: Forked 92102 at 1253142769
346
+ 92102 resque: Processing file_serve since 1253142769
347
+
348
+ You can clearly see that process 92099 forked 92102, which has been
349
+ working since 1253142769.
350
+
351
+ (By advertising the time they began processing you can easily use monit
352
+ or god to kill stale workers.)
353
+
354
+ When a parent process is idle, it lets you know what queues it is
355
+ waiting for work on:
356
+
357
+ $ ps -e -o pid,command | grep [r]esque
358
+ 92099 resque: Waiting for file_serve,warm_cache
359
+
360
+
361
+ ### Signals
362
+
363
+ Resque workers respond to a few different signals:
364
+
365
+ * `QUIT` - Wait for child to finish processing then exit
366
+ * `TERM` / `INT` - Immediately kill child then exit
367
+ * `USR1` - Immediately kill child but don't exit
368
+ * `USR2` - Don't start to process any new jobs
369
+ * `CONT` - Start to process new jobs again after a USR2
370
+
371
+ If you want to gracefully shutdown a Resque worker, use `QUIT`.
372
+
373
+ If you want to kill a stale or stuck child, use `USR1`. Processing
374
+ will continue as normal unless the child was not found. In that case
375
+ Resque assumes the parent process is in a bad state and shuts down.
376
+
377
+ If you want to kill a stale or stuck child and shutdown, use `TERM`
378
+
379
+ If you want to stop processing jobs, but want to leave the worker running
380
+ (for example, to temporarily alleviate load), use `USR2` to stop processing,
381
+ then `CONT` to start it again.
382
+
383
+ ### Mysql::Error: MySQL server has gone away
384
+
385
+ If your workers remain idle for too long they may lose their MySQL
386
+ connection. If that happens we recommend using [this
387
+ Gist](http://gist.github.com/238999).
388
+
389
+
390
+ The Front End
391
+ -------------
392
+
393
+ Resque comes with a Sinatra-based front end for seeing what's up with
394
+ your queue.
395
+
396
+ ![The Front End](http://img.skitch.com/20091104-tqh5pgkwgbskjbk7qbtmpesnyw.jpg)
397
+
398
+ ### Standalone
399
+
400
+ If you've installed Resque as a gem running the front end standalone is easy:
401
+
402
+ $ resque-web
403
+
404
+ It's a thin layer around `rackup` so it's configurable as well:
405
+
406
+ $ resque-web -p 8282
407
+
408
+ If you have a Resque config file you want evaluated just pass it to
409
+ the script as the final argument:
410
+
411
+ $ resque-web -p 8282 rails_root/config/initializers/resque.rb
412
+
413
+ You can also set the namespace directly using `resque-web`:
414
+
415
+ $ resque-web -p 8282 -N myapp
416
+
417
+ ### Passenger
418
+
419
+ Using Passenger? Resque ships with a `config.ru` you can use. See
420
+ Phusion's guide:
421
+
422
+ <http://www.modrails.com/documentation/Users%20guide.html#_deploying_a_rack_based_ruby_application>
423
+
424
+ ### Rack::URLMap
425
+
426
+ If you want to load Resque on a subpath, possibly alongside other
427
+ apps, it's easy to do with Rack's `URLMap`:
428
+
429
+ require 'resque/server'
430
+
431
+ run Rack::URLMap.new \
432
+ "/" => Your::App.new,
433
+ "/resque" => Resque::Server.new
434
+
435
+ Check `examples/demo/config.ru` for a functional example (including
436
+ HTTP basic auth).
437
+
438
+
439
+ Resque vs DelayedJob
440
+ --------------------
441
+
442
+ How does Resque compare to DelayedJob, and why would you choose one
443
+ over the other?
444
+
445
+ * Resque supports multiple queues
446
+ * DelayedJob supports finer grained priorities
447
+ * Resque workers are resilient to memory leaks / bloat
448
+ * DelayedJob workers are extremely simple and easy to modify
449
+ * Resque requires Redis
450
+ * DelayedJob requires ActiveRecord
451
+ * Resque can only place JSONable Ruby objects on a queue as arguments
452
+ * DelayedJob can place _any_ Ruby object on its queue as arguments
453
+ * Resque includes a Sinatra app for monitoring what's going on
454
+ * DelayedJob can be queried from within your Rails app if you want to
455
+ add an interface
456
+
457
+ If you're doing Rails development, you already have a database and
458
+ ActiveRecord. DelayedJob is super easy to setup and works great.
459
+ GitHub used it for many months to process almost 200 million jobs.
460
+
461
+ Choose Resque if:
462
+
463
+ * You need multiple queues
464
+ * You don't care / dislike numeric priorities
465
+ * You don't need to persist every Ruby object ever
466
+ * You have potentially huge queues
467
+ * You want to see what's going on
468
+ * You expect a lot of failure / chaos
469
+ * You can setup Redis
470
+ * You're not running short on RAM
471
+
472
+ Choose DelayedJob if:
473
+
474
+ * You like numeric priorities
475
+ * You're not doing a gigantic amount of jobs each day
476
+ * Your queue stays small and nimble
477
+ * There is not a lot failure / chaos
478
+ * You want to easily throw anything on the queue
479
+ * You don't want to setup Redis
480
+
481
+ In no way is Resque a "better" DelayedJob, so make sure you pick the
482
+ tool that's best for your app.
483
+
484
+
485
+ Installing Redis
486
+ ----------------
487
+
488
+ Resque requires Redis 0.900 or higher.
489
+
490
+ Resque uses Redis' lists for its queues. It also stores worker state
491
+ data in Redis.
492
+
493
+ #### Homebrew
494
+
495
+ If you're on OS X, Homebrew is the simplest way to install Redis:
496
+
497
+ $ brew install redis
498
+ $ redis-server /usr/local/etc/redis.conf
499
+
500
+ You now have a Redis daemon running on 6379.
501
+
502
+ #### Via Resque
503
+
504
+ Resque includes Rake tasks (thanks to Ezra's redis-rb) that will
505
+ install and run Redis for you:
506
+
507
+ $ git clone git://github.com/defunkt/resque.git
508
+ $ cd resque
509
+ $ rake redis:install dtach:install
510
+ $ rake redis:start
511
+
512
+ Or, if you don't have admin access on your machine:
513
+
514
+ $ git clone git://github.com/defunkt/resque.git
515
+ $ cd resque
516
+ $ PREFIX=<your_prefix> rake redis:install dtach:install
517
+ $ rake redis:start
518
+
519
+ You now have Redis running on 6379. Wait a second then hit ctrl-\ to
520
+ detach and keep it running in the background.
521
+
522
+ The demo is probably the best way to figure out how to put the parts
523
+ together. But, it's not that hard.
524
+
525
+
526
+ Resque Dependencies
527
+ -------------------
528
+
529
+ gem install redis redis-namespace yajl-ruby
530
+
531
+ If you cannot install `yajl-ruby` (JRuby?), you can install the `json`
532
+ gem and Resque will use it instead.
533
+
534
+ When problems arise, make sure you have the newest versions of the
535
+ `redis` and `redis-namespace` gems.
536
+
537
+
538
+ Installing Resque
539
+ -----------------
540
+
541
+ ### In a Rack app, as a gem
542
+
543
+ First install the gem.
544
+
545
+ $ gem install resque
546
+
547
+ Next include it in your application.
548
+
549
+ require 'resque'
550
+
551
+ Now start your application:
552
+
553
+ rackup config.ru
554
+
555
+ That's it! You can now create Resque jobs from within your app.
556
+
557
+ To start a worker, create a Rakefile in your app's root (or add this
558
+ to an existing Rakefile):
559
+
560
+ require 'your/app'
561
+ require 'resque/tasks'
562
+
563
+ Now:
564
+
565
+ $ QUEUE=* rake resque:work
566
+
567
+ Alternately you can define a `resque:setup` hook in your Rakefile if you
568
+ don't want to load your app every time rake runs.
569
+
570
+
571
+ ### In a Rails app, as a gem
572
+
573
+ First install the gem.
574
+
575
+ $ gem install resque
576
+
577
+ Next include it in your application.
578
+
579
+ $ cat config/initializers/load_resque.rb
580
+ require 'resque'
581
+
582
+ Now start your application:
583
+
584
+ $ ./script/server
585
+
586
+ That's it! You can now create Resque jobs from within your app.
587
+
588
+ To start a worker, add this to your Rakefile in `RAILS_ROOT`:
589
+
590
+ require 'resque/tasks'
591
+
592
+ Now:
593
+
594
+ $ QUEUE=* rake environment resque:work
595
+
596
+ Don't forget you can define a `resque:setup` hook in
597
+ `lib/tasks/whatever.rake` that loads the `environment` task every time.
598
+
599
+
600
+ ### In a Rails app, as a plugin
601
+
602
+ $ ./script/plugin install git://github.com/defunkt/resque
603
+
604
+ That's it! Resque will automatically be available when your Rails app
605
+ loads.
606
+
607
+ To start a worker:
608
+
609
+ $ QUEUE=* rake environment resque:work
610
+
611
+ Don't forget you can define a `resque:setup` hook in
612
+ `lib/tasks/whatever.rake` that loads the `environment` task every time.
613
+
614
+
615
+ Configuration
616
+ -------------
617
+
618
+ You may want to change the Redis host and port Resque connects to, or
619
+ set various other options at startup.
620
+
621
+ Resque has a `redis` setter which can be given a string or a Redis
622
+ object. This means if you're already using Redis in your app, Resque
623
+ can re-use the existing connection.
624
+
625
+ String: `Resque.redis = 'localhost:6379'`
626
+
627
+ Redis: `Resque.redis = $redis`
628
+
629
+ For our rails app we have a `config/initializers/resque.rb` file where
630
+ we load `config/resque.yml` by hand and set the Redis information
631
+ appropriately.
632
+
633
+ Here's our `config/resque.yml`:
634
+
635
+ development: localhost:6379
636
+ test: localhost:6379
637
+ staging: redis1.se.github.com:6379
638
+ fi: localhost:6379
639
+ production: redis1.ae.github.com:6379
640
+
641
+ And our initializer:
642
+
643
+ rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..'
644
+ rails_env = ENV['RAILS_ENV'] || 'development'
645
+
646
+ resque_config = YAML.load_file(rails_root + '/config/resque.yml')
647
+ Resque.redis = resque_config[rails_env]
648
+
649
+ Easy peasy! Why not just use `RAILS_ROOT` and `RAILS_ENV`? Because
650
+ this way we can tell our Sinatra app about the config file:
651
+
652
+ $ RAILS_ENV=production resque-web rails_root/config/initializers/resque.rb
653
+
654
+ Now everyone is on the same page.
655
+
656
+
657
+ Plugins and Hooks
658
+ -----------------
659
+
660
+ For a list of available plugins see
661
+ <http://wiki.github.com/defunkt/resque/plugins>.
662
+
663
+ If you'd like to write your own plugin, or want to customize Resque
664
+ using hooks (such as `Resque.after_fork`), see
665
+ [docs/HOOKS.md](http://github.com/defunkt/resque/blob/master/HOOKS.md).
666
+
667
+
668
+ Namespaces
669
+ ----------
670
+
671
+ If you're running multiple, separate instances of Resque you may want
672
+ to namespace the keyspaces so they do not overlap. This is not unlike
673
+ the approach taken by many memcached clients.
674
+
675
+ This feature is provided by the [redis-namespace][rs] library, which
676
+ Resque uses by default to separate the keys it manages from other keys
677
+ in your Redis server.
678
+
679
+ Simply use the `Resque.redis.namespace` accessor:
680
+
681
+ Resque.redis.namespace = "resque:GitHub"
682
+
683
+ We recommend sticking this in your initializer somewhere after Redis
684
+ is configured.
685
+
686
+
687
+ Demo
688
+ ----
689
+
690
+ Resque ships with a demo Sinatra app for creating jobs that are later
691
+ processed in the background.
692
+
693
+ Try it out by looking at the README, found at `examples/demo/README.markdown`.
694
+
695
+
696
+ Monitoring
697
+ ----------
698
+
699
+ ### god
700
+
701
+ If you're using god to monitor Resque, we have provided example
702
+ configs in `examples/god/`. One is for starting / stopping workers,
703
+ the other is for killing workers that have been running too long.
704
+
705
+ ### monit
706
+
707
+ If you're using monit, `examples/monit/resque.monit` is provided free
708
+ of charge. This is **not** used by GitHub in production, so please
709
+ send patches for any tweaks or improvements you can make to it.
710
+
711
+
712
+ Development
713
+ -----------
714
+
715
+ Want to hack on Resque?
716
+
717
+ First clone the repo and run the tests:
718
+
719
+ git clone git://github.com/defunkt/resque.git
720
+ cd resque
721
+ rake test
722
+
723
+ If the tests do not pass make sure you have Redis installed
724
+ correctly (though we make an effort to tell you if we feel this is the
725
+ case). The tests attempt to start an isolated instance of Redis to
726
+ run against.
727
+
728
+ Also make sure you've installed all the dependencies correctly. For
729
+ example, try loading the `redis-namespace` gem after you've installed
730
+ it:
731
+
732
+ $ irb
733
+ >> require 'rubygems'
734
+ => true
735
+ >> require 'redis/namespace'
736
+ => true
737
+
738
+ If you get an error requiring any of the dependencies, you may have
739
+ failed to install them or be seeing load path issues.
740
+
741
+ Feel free to ping the mailing list with your problem and we'll try to
742
+ sort it out.
743
+
744
+
745
+ Contributing
746
+ ------------
747
+
748
+ Once you've made your great commits:
749
+
750
+ 1. [Fork][1] Resque
751
+ 2. Create a topic branch - `git checkout -b my_branch`
752
+ 3. Push to your branch - `git push origin my_branch`
753
+ 4. Create an [Issue][2] with a link to your branch
754
+ 5. That's it!
755
+
756
+ You might want to checkout our [Contributing][cb] wiki page for information
757
+ on coding standards, new features, etc.
758
+
759
+
760
+ Mailing List
761
+ ------------
762
+
763
+ To join the list simply send an email to <resque@librelist.com>. This
764
+ will subscribe you and send you information about your subscription,
765
+ including unsubscribe information.
766
+
767
+ The archive can be found at <http://librelist.com/browser/>.
768
+
769
+
770
+ Meta
771
+ ----
772
+
773
+ * Code: `git clone git://github.com/defunkt/resque.git`
774
+ * Home: <http://github.com/defunkt/resque>
775
+ * Docs: <http://defunkt.github.com/resque/>
776
+ * Bugs: <http://github.com/defunkt/resque/issues>
777
+ * List: <resque@librelist.com>
778
+ * Chat: <irc://irc.freenode.net/resque>
779
+ * Gems: <http://gemcutter.org/gems/resque>
780
+
781
+ This project uses [Semantic Versioning][sv].
782
+
783
+
784
+ Author
785
+ ------
786
+
787
+ Chris Wanstrath :: chris@ozmm.org :: @defunkt
788
+
789
+ [0]: http://github.com/blog/542-introducing-resque
790
+ [1]: http://help.github.com/forking/
791
+ [2]: http://github.com/defunkt/resque/issues
792
+ [sv]: http://semver.org/
793
+ [rs]: http://github.com/defunkt/redis-namespace
794
+ [cb]: http://wiki.github.com/defunkt/resque/contributing