grockit-resque 1.5.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (61) hide show
  1. data/.gitignore +2 -0
  2. data/.kick +26 -0
  3. data/CONTRIBUTORS +14 -0
  4. data/HISTORY.md +78 -0
  5. data/LICENSE +20 -0
  6. data/README.markdown +762 -0
  7. data/Rakefile +67 -0
  8. data/bin/resque +57 -0
  9. data/bin/resque-web +18 -0
  10. data/config.ru +14 -0
  11. data/deps.rip +6 -0
  12. data/examples/async_helper.rb +31 -0
  13. data/examples/demo/README.markdown +71 -0
  14. data/examples/demo/Rakefile +3 -0
  15. data/examples/demo/app.rb +38 -0
  16. data/examples/demo/config.ru +19 -0
  17. data/examples/demo/job.rb +22 -0
  18. data/examples/god/resque.god +53 -0
  19. data/examples/god/stale.god +26 -0
  20. data/examples/instance.rb +11 -0
  21. data/examples/simple.rb +30 -0
  22. data/init.rb +1 -0
  23. data/lib/resque.rb +247 -0
  24. data/lib/resque/errors.rb +7 -0
  25. data/lib/resque/failure.rb +63 -0
  26. data/lib/resque/failure/base.rb +58 -0
  27. data/lib/resque/failure/hoptoad.rb +122 -0
  28. data/lib/resque/failure/multiple.rb +44 -0
  29. data/lib/resque/failure/redis.rb +33 -0
  30. data/lib/resque/helpers.rb +57 -0
  31. data/lib/resque/job.rb +158 -0
  32. data/lib/resque/server.rb +182 -0
  33. data/lib/resque/server/public/idle.png +0 -0
  34. data/lib/resque/server/public/jquery-1.3.2.min.js +19 -0
  35. data/lib/resque/server/public/jquery.relatize_date.js +95 -0
  36. data/lib/resque/server/public/poll.png +0 -0
  37. data/lib/resque/server/public/ranger.js +24 -0
  38. data/lib/resque/server/public/reset.css +48 -0
  39. data/lib/resque/server/public/style.css +75 -0
  40. data/lib/resque/server/public/working.png +0 -0
  41. data/lib/resque/server/views/error.erb +1 -0
  42. data/lib/resque/server/views/failed.erb +35 -0
  43. data/lib/resque/server/views/key.erb +17 -0
  44. data/lib/resque/server/views/layout.erb +38 -0
  45. data/lib/resque/server/views/next_more.erb +10 -0
  46. data/lib/resque/server/views/overview.erb +4 -0
  47. data/lib/resque/server/views/queues.erb +46 -0
  48. data/lib/resque/server/views/stats.erb +62 -0
  49. data/lib/resque/server/views/workers.erb +78 -0
  50. data/lib/resque/server/views/working.erb +69 -0
  51. data/lib/resque/stat.rb +53 -0
  52. data/lib/resque/tasks.rb +39 -0
  53. data/lib/resque/version.rb +3 -0
  54. data/lib/resque/worker.rb +442 -0
  55. data/tasks/redis.rake +135 -0
  56. data/tasks/resque.rake +2 -0
  57. data/test/redis-test.conf +132 -0
  58. data/test/resque_test.rb +227 -0
  59. data/test/test_helper.rb +96 -0
  60. data/test/worker_test.rb +243 -0
  61. metadata +172 -0
@@ -0,0 +1,2 @@
1
+ pkg
2
+ nbproject
data/.kick ADDED
@@ -0,0 +1,26 @@
1
+ # take control of the growl notifications
2
+ module GrowlHacks
3
+ def growl(type, subject, body, *args, &block)
4
+ case type
5
+ when Kicker::GROWL_NOTIFICATIONS[:succeeded]
6
+ puts subject = "Success"
7
+ body = body.split("\n").last
8
+ when Kicker::GROWL_NOTIFICATIONS[:failed]
9
+ subject = "Failure"
10
+ puts body
11
+ body = body.split("\n").last
12
+ else
13
+ return nil
14
+ end
15
+ super(type, subject, body, *args, &block)
16
+ end
17
+ end
18
+
19
+ Kicker.send :extend, GrowlHacks
20
+
21
+ # no logging
22
+ Kicker::Utils.module_eval do
23
+ def log(message)
24
+ nil
25
+ end
26
+ end
@@ -0,0 +1,14 @@
1
+ * Chris Wanstrath
2
+ * John Barnette
3
+ * Adam Cooke
4
+ * Rob Hanlon
5
+ * Jason Amster
6
+ * jgeiger
7
+ * Aaron Quint
8
+ * Ben VandenBos
9
+ * PJ Hyett
10
+ * Arthur Zapparoli
11
+ * Simon Rozet
12
+ * Brian P O'Rourke
13
+ * Dave Hoover
14
+ * Michael Dwan
@@ -0,0 +1,78 @@
1
+ ## 1.5.1 (2010-??-??)
2
+
3
+ * `Job.destroy` and `Resque.dequeue` return the # of destroyed jobs.
4
+
5
+ ## 1.5.0 (2010-02-17)
6
+
7
+ * Version now included in procline, e.g. `resque-1.5.0: Message`
8
+ * Web bugfix: Ignore idle works in the "working" page
9
+ * Added `Resque::Job.destroy(queue, klass, *args)`
10
+ * Added `Resque.dequeue(klass, *args)`
11
+
12
+ ## 1.4.0 (2010-02-11)
13
+
14
+ * Fallback when unable to bind QUIT and USR1 for Windows and JRuby.
15
+ * Fallback when no `Kernel.fork` is provided (for IronRuby).
16
+ * Web: Rounded corners in Firefox
17
+ * Cut down system calls in `Worker#prune_dead_workers`
18
+ * Enable switching DB in a Redis server from config
19
+ * Support USR2 and CONT to stop and start job processing.
20
+ * Web: Add example failing job
21
+ * Bugfix: `Worker#unregister_worker` shouldn't call `done_working`
22
+ * Bugfix: Example god config now restarts Resque properly.
23
+ * Multiple failure backends now permitted.
24
+ * Hoptoad failure backend updated to new API
25
+
26
+ ## 1.3.1 (2010-01-11)
27
+
28
+ * Vegas bugfix: Don't error without a config
29
+
30
+ ## 1.3.0 (2010-01-11)
31
+
32
+ * Use Vegas for resque-web
33
+ * Web Bugfix: Show proper date/time value for failed_at on Failures
34
+ * Web Bugfix: Make the / route more flexible
35
+ * Add Resque::Server.tabs array (so plugins can add their own tabs)
36
+ * Start using [Semantic Versioning](http://semver.org/)
37
+
38
+ ## 1.2.4 (2009-12-15)
39
+
40
+ * Web Bugfix: fix key links on stat page
41
+
42
+ ## 1.2.3 (2009-12-15)
43
+
44
+ * Bugfix: Fixed `rand` seeding in child processes.
45
+ * Bugfix: Better JSON encoding/decoding without Yajl.
46
+ * Bugfix: Avoid `ps` flag error on Linux
47
+ * Add `PREFIX` observance to `rake` install tasks.
48
+
49
+ ## 1.2.2 (2009-12-08)
50
+
51
+ * Bugfix: Job equality was not properly implemented.
52
+
53
+ ## 1.2.1 (2009-12-07)
54
+
55
+ * Added `rake resque:workers` task for starting multiple workers.
56
+ * 1.9.x compatibility
57
+ * Bugfix: Yajl decoder doesn't care about valid UTF-8
58
+ * config.ru loads RESQUECONFIG if the ENV variable is set.
59
+ * `resque-web` now sets RESQUECONFIG
60
+ * Job objects know if they are equal.
61
+ * Jobs can be re-queued using `Job#recreate`
62
+
63
+ ## 1.2.0 (2009-11-25)
64
+
65
+ * If USR1 is sent and no child is found, shutdown.
66
+ * Raise when a job class does not respond to `perform`.
67
+ * Added `Resque.remove_queue` for deleting a queue
68
+
69
+ ## 1.1.0 (2009-11-04)
70
+
71
+ * Bugfix: Broken ERB tag in failure UI
72
+ * Bugfix: Save the worker's ID, not the worker itself, in the failure module
73
+ * Redesigned the sinatra web interface
74
+ * Added option to clear failed jobs
75
+
76
+ ## 1.0.0 (2009-11-03)
77
+
78
+ * First release.
data/LICENSE ADDED
@@ -0,0 +1,20 @@
1
+ Copyright (c) 2009 Chris Wanstrath
2
+
3
+ Permission is hereby granted, free of charge, to any person obtaining
4
+ a copy of this software and associated documentation files (the
5
+ "Software"), to deal in the Software without restriction, including
6
+ without limitation the rights to use, copy, modify, merge, publish,
7
+ distribute, sublicense, and/or sell copies of the Software, and to
8
+ permit persons to whom the Software is furnished to do so, subject to
9
+ the following conditions:
10
+
11
+ The above copyright notice and this permission notice shall be
12
+ included in all copies or substantial portions of the Software.
13
+
14
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
15
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
16
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
17
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
18
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
19
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
20
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@@ -0,0 +1,762 @@
1
+ Resque
2
+ ======
3
+
4
+ Resque is a Redis-backed library for creating background jobs, placing
5
+ those jobs on multiple queues, and processing them later.
6
+
7
+ Background jobs can be any Ruby class or module that responds to
8
+ `perform`. Your existing classes can easily be converted to background
9
+ jobs or you can create new classes specifically to do work. Or, you
10
+ can do both.
11
+
12
+ Resque is heavily inspired by DelayedJob (which rocks) and is
13
+ comprised of three parts:
14
+
15
+ 1. A Ruby library for creating, querying, and processing jobs
16
+ 2. A Rake task for starting a worker which processes jobs
17
+ 3. A Sinatra app for monitoring queues, jobs, and workers.
18
+
19
+ Resque workers can be distributed between multiple machines,
20
+ support priorities, are resilient to memory bloat / "leaks," are
21
+ optimized for REE (but work on MRI and JRuby), tell you what they're
22
+ doing, and expect failure.
23
+
24
+ Resque queues are persistent; support constant time, atomic push and
25
+ pop (thanks to Redis); provide visibility into their contents; and
26
+ store jobs as simple JSON packages.
27
+
28
+ The Resque frontend tells you what workers are doing, what workers are
29
+ not doing, what queues you're using, what's in those queues, provides
30
+ general usage stats, and helps you track failures.
31
+
32
+
33
+ The Blog Post
34
+ -------------
35
+
36
+ For the backstory, philosophy, and history of Resque's beginnings,
37
+ please see [the blog post][0].
38
+
39
+
40
+ Overview
41
+ --------
42
+
43
+ Resque allows you to create jobs and place them on a queue, then,
44
+ later, pull those jobs off the queue and process them.
45
+
46
+ Resque jobs are Ruby classes (or modules) which respond to the
47
+ `perform` method. Here's an example:
48
+
49
+ class Archive
50
+ @queue = :file_serve
51
+
52
+ def self.perform(repo_id, branch = 'master')
53
+ repo = Repository.find(repo_id)
54
+ repo.create_archive(branch)
55
+ end
56
+ end
57
+
58
+ The `@queue` class instance variable determines which queue `Archive`
59
+ jobs will be placed in. Queues are arbitrary and created on the fly -
60
+ you can name them whatever you want and have as many as you want.
61
+
62
+ To place an `Archive` job on the `file_serve` queue, we might add this
63
+ to our application's pre-existing `Repository` class:
64
+
65
+ class Repository
66
+ def async_create_archive(branch)
67
+ Resque.enqueue(Archive, self.id, branch)
68
+ end
69
+ end
70
+
71
+ Now when we call `repo.async_create_archive('masterbrew')` in our
72
+ application, a job will be created and placed on the `file_serve`
73
+ queue.
74
+
75
+ Later, a worker will run something like this code to process the job:
76
+
77
+ klass, args = Resque.reserve(:file_serve)
78
+ klass.perform(*args) if klass.respond_to? :perform
79
+
80
+ Which translates to:
81
+
82
+ Archive.perform(44, 'masterbrew')
83
+
84
+ Let's start a worker to run `file_serve` jobs:
85
+
86
+ $ cd app_root
87
+ $ QUEUE=file_serve rake resque:work
88
+
89
+ This starts one Resque worker and tells it to work off the
90
+ `file_serve` queue. As soon as it's ready it'll try to run the
91
+ `Resque.reserve` code snippet above and process jobs until it can't
92
+ find any more, at which point it will sleep for a small period and
93
+ repeatedly poll the queue for more jobs.
94
+
95
+ Workers can be given multiple queues (a "queue list") and run on
96
+ multiple machines. In fact they can be run anywhere with network
97
+ access to the Redis server.
98
+
99
+
100
+ Jobs
101
+ ----
102
+
103
+ What should you run in the background? Anything that takes any time at
104
+ all. Slow INSERT statements, disk manipulating, data processing, etc.
105
+
106
+ At GitHub we use Resque to process the following types of jobs:
107
+
108
+ * Warming caches
109
+ * Counting disk usage
110
+ * Building tarballs
111
+ * Building Rubygems
112
+ * Firing off web hooks
113
+ * Creating events in the db and pre-caching them
114
+ * Building graphs
115
+ * Deleting users
116
+ * Updating our search index
117
+
118
+ As of writing we have about 35 different types of background jobs.
119
+
120
+ Keep in mind that you don't need a web app to use Resque - we just
121
+ mention "foreground" and "background" because they make conceptual
122
+ sense. You could easily be spidering sites and sticking data which
123
+ needs to be crunched later into a queue.
124
+
125
+
126
+ ### Persistence
127
+
128
+ Jobs are persisted to queues as JSON objects. Let's take our `Archive`
129
+ example from above. We'll run the following code to create a job:
130
+
131
+ repo = Repository.find(44)
132
+ repo.async_create_archive('masterbrew')
133
+
134
+ The following JSON will be stored in the `file_serve` queue:
135
+
136
+ {
137
+ 'class': 'Archive',
138
+ 'args': [ 44, 'masterbrew' ]
139
+ }
140
+
141
+ Because of this your jobs must only accept arguments that can be JSON encoded.
142
+
143
+ So instead of doing this:
144
+
145
+ Resque.enqueue(Archive, self, branch)
146
+
147
+ do this:
148
+
149
+ Resque.enqueue(Archive, self.id, branch)
150
+
151
+ This is why our above example (and all the examples in `examples/`)
152
+ uses object IDs instead of passing around the objects.
153
+
154
+ While this is less convenient than just sticking a marshaled object
155
+ in the database, it gives you a slight advantage: your jobs will be
156
+ run against the most recent version of an object because they need to
157
+ pull from the DB or cache.
158
+
159
+ If your jobs were run against marshaled objects, they could
160
+ potentially be operating on a stale record with out-of-date information.
161
+
162
+
163
+ ### send_later / async
164
+
165
+ Want something like DelayedJob's `send_later` or the ability to use
166
+ instance methods instead of just methods for jobs? See the `examples/`
167
+ directory for goodies.
168
+
169
+ We plan to provide first class `async` support in a future release.
170
+
171
+
172
+ ### Failure
173
+
174
+ If a job raises an exception, it is logged and handed off to the
175
+ `Resque::Failure` module. Failures are logged either locally in Redis
176
+ or using some different backend.
177
+
178
+ For example, Resque ships with Hoptoad support.
179
+
180
+ Keep this in mind when writing your jobs: you may want to throw
181
+ exceptions you would not normally throw in order to assist debugging.
182
+
183
+
184
+ Workers
185
+ -------
186
+
187
+ Resque workers are rake tasks that run forever. They basically do this:
188
+
189
+ start
190
+ loop do
191
+ if job = reserve
192
+ job.process
193
+ else
194
+ sleep 5
195
+ end
196
+ end
197
+ shutdown
198
+
199
+ Starting a worker is simple. Here's our example from earlier:
200
+
201
+ $ QUEUE=file_serve rake resque:work
202
+
203
+ By default Resque won't know about your application's
204
+ environment. That is, it won't be able to find and run your jobs - it
205
+ needs to load your application into memory.
206
+
207
+ If we've installed Resque as a Rails plugin, we might run this command
208
+ from our RAILS_ROOT:
209
+
210
+ $ QUEUE=file_serve rake environment resque:work
211
+
212
+ This will load the environment before starting a worker. Alternately
213
+ we can define a `resque:setup` task with a dependency on the
214
+ `environment` rake task:
215
+
216
+ task "resque:setup" => :environment
217
+
218
+ GitHub's setup task looks like this:
219
+
220
+ task "resque:setup" => :environment do
221
+ Grit::Git.git_timeout = 10.minutes
222
+ end
223
+
224
+ We don't want the `git_timeout` as high as 10 minutes in our web app,
225
+ but in the Resque workers it's fine.
226
+
227
+
228
+ ### Logging
229
+
230
+ Workers support basic logging to STDOUT. If you start them with the
231
+ `VERBOSE` env variable set, they will print basic debugging
232
+ information. You can also set the `VVERBOSE` (very verbose) env
233
+ variable.
234
+
235
+ $ VVERBOSE=1 QUEUE=file_serve rake environment resque:work
236
+
237
+
238
+ ### Priorities and Queue Lists
239
+
240
+ Resque doesn't support numeric priorities but instead uses the order
241
+ of queues you give it. We call this list of queues the "queue list."
242
+
243
+ Let's say we add a `warm_cache` queue in addition to our `file_serve`
244
+ queue. We'd now start a worker like so:
245
+
246
+ $ QUEUES=file_serve,warm_cache rake resque:work
247
+
248
+ When the worker looks for new jobs, it will first check
249
+ `file_serve`. If it finds a job, it'll process it then check
250
+ `file_serve` again. It will keep checking `file_serve` until no more
251
+ jobs are available. At that point, it will check `warm_cache`. If it
252
+ finds a job it'll process it then check `file_serve` (repeating the
253
+ whole process).
254
+
255
+ In this way you can prioritize certain queues. At GitHub we start our
256
+ workers with something like this:
257
+
258
+ $ QUEUES=critical,archive,high,low rake resque:work
259
+
260
+ Notice the `archive` queue - it is specialized and in our future
261
+ architecture will only be run from a single machine.
262
+
263
+ At that point we'll start workers on our generalized background
264
+ machines with this command:
265
+
266
+ $ QUEUES=critical,high,low rake resque:work
267
+
268
+ And workers on our specialized archive machine with this command:
269
+
270
+ $ QUEUE=archive rake resque:work
271
+
272
+
273
+ ### Running All Queues
274
+
275
+ If you want your workers to work off of every queue, including new
276
+ queues created on the fly, you can use a splat:
277
+
278
+ $ QUEUE=* rake resque:work
279
+
280
+ Queues will be processed in alphabetical order.
281
+
282
+
283
+ ### Running Multiple Workers
284
+
285
+ At GitHub we use god to start and stop multiple workers. A sample god
286
+ configuration file is included under `examples/god`. We recommend this
287
+ method.
288
+
289
+ If you'd like to run multiple workers in development mode, you can do
290
+ so using the `resque:workers` rake task:
291
+
292
+ $ COUNT=5 QUEUE=* rake resque:workers
293
+
294
+ This will spawn five Resque workers, each in its own thread. Hitting
295
+ ctrl-c should be sufficient to stop them all.
296
+
297
+
298
+ ### Forking
299
+
300
+ On certain platforms, when a Resque worker reserves a job it
301
+ immediately forks a child process. The child processes the job then
302
+ exits. When the child has exited successfully, the worker reserves
303
+ another job and repeats the process.
304
+
305
+ Why?
306
+
307
+ Because Resque assumes chaos.
308
+
309
+ Resque assumes your background workers will lock up, run too long, or
310
+ have unwanted memory growth.
311
+
312
+ If Resque workers processed jobs themselves, it'd be hard to whip them
313
+ into shape. Let's say one is using too much memory: you send it a
314
+ signal that says "shutdown after you finish processing the current
315
+ job," and it does so. It then starts up again - loading your entire
316
+ application environment. This adds useless CPU cycles and causes a
317
+ delay in queue processing.
318
+
319
+ Plus, what if it's using too much memory and has stopped responding to
320
+ signals?
321
+
322
+ Thanks to Resque's parent / child architecture, jobs that use too much memory
323
+ release that memory upon completion. No unwanted growth.
324
+
325
+ And what if a job is running too long? You'd need to `kill -9` it then
326
+ start the worker again. With Resque's parent / child architecture you
327
+ can tell the parent to forcefully kill the child then immediately
328
+ start processing more jobs. No startup delay or wasted cycles.
329
+
330
+ The parent / child architecture helps us keep tabs on what workers are
331
+ doing, too. By eliminating the need to `kill -9` workers we can have
332
+ parents remove themselves from the global listing of workers. If we
333
+ just ruthlessly killed workers, we'd need a separate watchdog process
334
+ to add and remove them to the global listing - which becomes
335
+ complicated.
336
+
337
+ Workers instead handle their own state.
338
+
339
+
340
+ ### Parents and Children
341
+
342
+ Here's a parent / child pair doing some work:
343
+
344
+ $ ps -e -o pid,command | grep [r]esque
345
+ 92099 resque: Forked 92102 at 1253142769
346
+ 92102 resque: Processing file_serve since 1253142769
347
+
348
+ You can clearly see that process 92099 forked 92102, which has been
349
+ working since 1253142769.
350
+
351
+ (By advertising the time they began processing you can easily use monit
352
+ or god to kill stale workers.)
353
+
354
+ When a parent process is idle, it lets you know what queues it is
355
+ waiting for work on:
356
+
357
+ $ ps -e -o pid,command | grep [r]esque
358
+ 92099 resque: Waiting for file_serve,warm_cache
359
+
360
+
361
+ ### Signals
362
+
363
+ Resque workers respond to a few different signals:
364
+
365
+ * `QUIT` - Wait for child to finish processing then exit
366
+ * `TERM` / `INT` - Immediately kill child then exit
367
+ * `USR1` - Immediately kill child but don't exit
368
+ * `USR2` - Don't start to process any new jobs
369
+ * `CONT` - Start to process new jobs again after a USR2
370
+
371
+ If you want to gracefully shutdown a Resque worker, use `QUIT`.
372
+
373
+ If you want to kill a stale or stuck child, use `USR1`. Processing
374
+ will continue as normal unless the child was not found. In that case
375
+ Resque assumes the parent process is in a bad state and shuts down.
376
+
377
+ If you want to kill a stale or stuck child and shutdown, use `TERM`
378
+
379
+ If you want to stop processing jobs, but want to leave the worker running
380
+ (for example, to temporarily alleviate load), use `USR2` to stop processing,
381
+ then `CONT` to start it again.
382
+
383
+ ### Mysql::Error: MySQL server has gone away
384
+
385
+ If your workers remain idle for too long they may lose their MySQL
386
+ connection. If that happens we recommend using [this
387
+ Gist](http://gist.github.com/238999).
388
+
389
+
390
+ The Front End
391
+ -------------
392
+
393
+ Resque comes with a Sinatra-based front end for seeing what's up with
394
+ your queue.
395
+
396
+ ![The Front End](http://img.skitch.com/20091104-tqh5pgkwgbskjbk7qbtmpesnyw.jpg)
397
+
398
+ ### Standalone
399
+
400
+ If you've installed Resque as a gem running the front end standalone is easy:
401
+
402
+ $ resque-web
403
+
404
+ It's a thin layer around `rackup` so it's configurable as well:
405
+
406
+ $ resque-web -p 8282
407
+
408
+ If you have a Resque config file you want evaluated just pass it to
409
+ the script as the final argument:
410
+
411
+ $ resque-web -p 8282 rails_root/config/initializers/resque.rb
412
+
413
+ ### Passenger
414
+
415
+ Using Passenger? Resque ships with a `config.ru` you can use. See
416
+ Phusion's guide:
417
+
418
+ <http://www.modrails.com/documentation/Users%20guide.html#_deploying_a_rack_based_ruby_application>
419
+
420
+ ### Rack::URLMap
421
+
422
+ If you want to load Resque on a subpath, possibly alongside other
423
+ apps, it's easy to do with Rack's `URLMap`:
424
+
425
+ require 'resque/server'
426
+
427
+ run Rack::URLMap.new \
428
+ "/" => Your::App.new,
429
+ "/resque" => Resque::Server.new
430
+
431
+ Check `examples/demo/config.ru` for a functional example (including
432
+ HTTP basic auth).
433
+
434
+
435
+ Resque vs DelayedJob
436
+ --------------------
437
+
438
+ How does Resque compare to DelayedJob, and why would you choose one
439
+ over the other?
440
+
441
+ * Resque supports multiple queues
442
+ * DelayedJob supports finer grained priorities
443
+ * Resque workers are resilient to memory leaks / bloat
444
+ * DelayedJob workers are extremely simple and easy to modify
445
+ * Resque requires Redis
446
+ * DelayedJob requires ActiveRecord
447
+ * Resque can only place JSONable Ruby objects on a queue as arguments
448
+ * DelayedJob can place _any_ Ruby object on its queue as arguments
449
+ * Resque includes a Sinatra app for monitoring what's going on
450
+ * DelayedJob can be queried from within your Rails app if you want to
451
+ add an interface
452
+
453
+ If you're doing Rails development, you already have a database and
454
+ ActiveRecord. DelayedJob is super easy to setup and works great.
455
+ GitHub used it for many months to process almost 200 million jobs.
456
+
457
+ Choose Resque if:
458
+
459
+ * You need multiple queues
460
+ * You don't care / dislike numeric priorities
461
+ * You don't need to persist every Ruby object ever
462
+ * You have potentially huge queues
463
+ * You want to see what's going on
464
+ * You expect a lot of failure / chaos
465
+ * You can setup Redis
466
+ * You're not running short on RAM
467
+
468
+ Choose DelayedJob if:
469
+
470
+ * You like numeric priorities
471
+ * You're not doing a gigantic amount of jobs each day
472
+ * Your queue stays small and nimble
473
+ * There is not a lot failure / chaos
474
+ * You want to easily throw anything on the queue
475
+ * You don't want to setup Redis
476
+
477
+ In no way is Resque a "better" DelayedJob, so make sure you pick the
478
+ tool that's best for your app.
479
+
480
+
481
+ Installing Redis
482
+ ----------------
483
+
484
+ Resque uses Redis' lists for its queues. It also stores worker state
485
+ data in Redis.
486
+
487
+ #### Homebrew
488
+
489
+ If you're on OS X, Homebrew is the simplest way to install Redis:
490
+
491
+ $ brew install redis
492
+ $ redis-server /usr/local/etc/redis.conf
493
+
494
+ You now have a Redis daemon running on 6379.
495
+
496
+ #### Via Resque
497
+
498
+ Resque includes Rake tasks (thanks to Ezra's redis-rb) that will
499
+ install and run Redis for you:
500
+
501
+ $ git clone git://github.com/defunkt/resque.git
502
+ $ cd resque
503
+ $ rake redis:install dtach:install
504
+ $ rake redis:start
505
+
506
+ Or, if you don't have admin access on your machine:
507
+
508
+ $ git clone git://github.com/defunkt/resque.git
509
+ $ cd resque
510
+ $ PREFIX=<your_prefix> rake redis:install dtach:install
511
+ $ rake redis:start
512
+
513
+ You now have Redis running on 6379. Wait a second then hit ctrl-\ to
514
+ detach and keep it running in the background.
515
+
516
+ The demo is probably the best way to figure out how to put the parts
517
+ together. But, it's not that hard.
518
+
519
+
520
+ Resque Dependencies
521
+ -------------------
522
+
523
+ gem install redis redis-namespace yajl-ruby
524
+
525
+ If you cannot install `yajl-ruby` (JRuby?), you can install the `json`
526
+ gem and Resque will use it instead.
527
+
528
+
529
+ Installing Resque
530
+ -----------------
531
+
532
+ ### In a Rack app, as a gem
533
+
534
+ First install the gem.
535
+
536
+ $ gem install resque
537
+
538
+ Next include it in your application.
539
+
540
+ require 'resque'
541
+
542
+ Now start your application:
543
+
544
+ rackup config.ru
545
+
546
+ That's it! You can now create Resque jobs from within your app.
547
+
548
+ To start a worker, create a Rakefile in your app's root (or add this
549
+ to an existing Rakefile):
550
+
551
+ require 'your/app'
552
+ require 'resque/tasks'
553
+
554
+ Now:
555
+
556
+ $ QUEUE=* rake resque:work
557
+
558
+ Alternately you can define a `resque:setup` hook in your Rakefile if you
559
+ don't want to load your app every time rake runs.
560
+
561
+
562
+ ### In a Rails app, as a gem
563
+
564
+ First install the gem.
565
+
566
+ $ gem install resque
567
+
568
+ Next include it in your application.
569
+
570
+ $ cat config/initializers/load_resque.rb
571
+ require 'resque'
572
+
573
+ Now start your application:
574
+
575
+ $ ./script/server
576
+
577
+ That's it! You can now create Resque jobs from within your app.
578
+
579
+ To start a worker, add this to your Rakefile in `RAILS_ROOT`:
580
+
581
+ require 'resque/tasks'
582
+
583
+ Now:
584
+
585
+ $ QUEUE=* rake environment resque:work
586
+
587
+ Don't forget you can define a `resque:setup` hook in
588
+ `lib/tasks/whatever.rake` that loads the `environment` task every time.
589
+
590
+
591
+ ### In a Rails app, as a plugin
592
+
593
+ $ ./script/plugin install git://github.com/defunkt/resque
594
+
595
+ That's it! Resque will automatically be available when your Rails app
596
+ loads.
597
+
598
+ To start a worker:
599
+
600
+ $ QUEUE=* rake environment resque:work
601
+
602
+ Don't forget you can define a `resque:setup` hook in
603
+ `lib/tasks/whatever.rake` that loads the `environment` task every time.
604
+
605
+
606
+ Configuration
607
+ -------------
608
+
609
+ You may want to change the Redis host and port Resque connects to, or
610
+ set various other options at startup.
611
+
612
+ Resque has a `redis` setter which can be given a string or a Redis
613
+ object. This means if you're already using Redis in your app, Resque
614
+ can re-use the existing connection.
615
+
616
+ String: `Resque.redis = 'localhost:6379'`
617
+
618
+ Redis: `Resque.redis = $redis`
619
+
620
+ For our rails app we have a `config/initializers/resque.rb` file where
621
+ we load `config/resque.yml` by hand and set the Redis information
622
+ appropriately.
623
+
624
+ Here's our `config/resque.yml`:
625
+
626
+ development: localhost:6379
627
+ test: localhost:6379
628
+ staging: redis1.se.github.com:6379
629
+ fi: localhost:6379
630
+ production: redis1.ae.github.com:6379
631
+
632
+ And our initializer:
633
+
634
+ rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..'
635
+ rails_env = ENV['RAILS_ENV'] || 'development'
636
+
637
+ resque_config = YAML.load_file(rails_root + '/config/resque.yml')
638
+ Resque.redis = resque_config[rails_env]
639
+
640
+ Easy peasy! Why not just use `RAILS_ROOT` and `RAILS_ENV`? Because
641
+ this way we can tell our Sinatra app about the config file:
642
+
643
+ $ RAILS_ENV=production resque-web rails_root/config/initializers/resque.rb
644
+
645
+ Now everyone is on the same page.
646
+
647
+
648
+ Namespaces
649
+ ----------
650
+
651
+ If you're running multiple, separate instances of Resque you may want
652
+ to namespace the keyspaces so they do not overlap. This is not unlike
653
+ the approach taken by many memcached clients.
654
+
655
+ This feature is provided by the [redis-namespace][rs] library, which
656
+ Resque uses by default to separate the keys it manages from other keys
657
+ in your Redis server.
658
+
659
+ Simply use the `Resque.redis.namespace` accessor:
660
+
661
+ Resque.redis.namespace = "resque:GitHub"
662
+
663
+ We recommend sticking this in your initializer somewhere after Redis
664
+ is configured.
665
+
666
+
667
+ Demo
668
+ ----
669
+
670
+ Resque ships with a demo Sinatra app for creating jobs that are later
671
+ processed in the background.
672
+
673
+ Try it out by looking at the README, found at `examples/demo/README.markdown`.
674
+
675
+
676
+ Monitoring
677
+ ----------
678
+
679
+ If you're using god to monitor Resque, we have provided example
680
+ configs in `examples/god/`. One is for starting / stopping workers,
681
+ the other is for killing workers that have been running too long.
682
+
683
+
684
+ Development
685
+ -----------
686
+
687
+ Want to hack on Resque?
688
+
689
+ First clone the repo and run the tests:
690
+
691
+ git clone git://github.com/defunkt/resque.git
692
+ cd resque
693
+ rake test
694
+
695
+ If the tests do not pass make sure you have Redis installed
696
+ correctly (though we make an effort to tell you if we feel this is the
697
+ case). The tests attempt to start an isolated instance of Redis to
698
+ run against.
699
+
700
+ Also make sure you've installed all the dependencies correctly. For
701
+ example, try loading the `redis-namespace` gem after you've installed
702
+ it:
703
+
704
+ $ irb
705
+ >> require 'rubygems'
706
+ => true
707
+ >> require 'redis/namespace'
708
+ => true
709
+
710
+ If you get an error requiring any of the dependencies, you may have
711
+ failed to install them or be seeing load path issues.
712
+
713
+ Feel free to ping the mailing list with your problem and we'll try to
714
+ sort it out.
715
+
716
+
717
+ Contributing
718
+ ------------
719
+
720
+ Once you've made your great commits:
721
+
722
+ 1. [Fork][1] Resque
723
+ 2. Create a topic branch - `git checkout -b my_branch`
724
+ 3. Push to your branch - `git push origin my_branch`
725
+ 4. Create an [Issue][2] with a link to your branch
726
+ 5. That's it!
727
+
728
+
729
+ Mailing List
730
+ ------------
731
+
732
+ To join the list simply send an email to <resque@librelist.com>. This
733
+ will subscribe you and send you information about your subscription,
734
+ include unsubscribe information.
735
+
736
+ The archive can be found at <http://librelist.com/browser/>.
737
+
738
+
739
+ Meta
740
+ ----
741
+
742
+ * Code: `git clone git://github.com/defunkt/resque.git`
743
+ * Home: <http://github.com/defunkt/resque>
744
+ * Docs: <http://defunkt.github.com/resque/>
745
+ * Bugs: <http://github.com/defunkt/resque/issues>
746
+ * List: <resque@librelist.com>
747
+ * Chat: <irc://irc.freenode.net/resque>
748
+ * Gems: <http://gemcutter.org/gems/resque>
749
+
750
+ This project uses [Semantic Versioning][sv].
751
+
752
+
753
+ Author
754
+ ------
755
+
756
+ Chris Wanstrath :: chris@ozmm.org :: @defunkt
757
+
758
+ [0]: http://github.com/blog/542-introducing-resque
759
+ [1]: http://help.github.com/forking/
760
+ [2]: http://github.com/defunkt/resque/issues
761
+ [sv]: http://semver.org/
762
+ [rs]: http://github.com/defunkt/redis-namespace