resque-mongo 1.3.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (61) hide show
  1. data/.gitignore +3 -0
  2. data/.kick +26 -0
  3. data/CONTRIBUTORS +14 -0
  4. data/HISTORY.md +53 -0
  5. data/LICENSE +20 -0
  6. data/README.markdown +755 -0
  7. data/Rakefile +65 -0
  8. data/bin/resque +57 -0
  9. data/bin/resque-web +18 -0
  10. data/config.ru +14 -0
  11. data/deps.rip +8 -0
  12. data/examples/async_helper.rb +31 -0
  13. data/examples/demo/README.markdown +71 -0
  14. data/examples/demo/Rakefile +3 -0
  15. data/examples/demo/app.rb +27 -0
  16. data/examples/demo/config.ru +19 -0
  17. data/examples/demo/job.rb +12 -0
  18. data/examples/god/resque.god +52 -0
  19. data/examples/god/stale.god +26 -0
  20. data/examples/instance.rb +11 -0
  21. data/examples/simple.rb +30 -0
  22. data/init.rb +1 -0
  23. data/lib/resque.rb +224 -0
  24. data/lib/resque/errors.rb +7 -0
  25. data/lib/resque/failure.rb +63 -0
  26. data/lib/resque/failure/base.rb +58 -0
  27. data/lib/resque/failure/hoptoad.rb +88 -0
  28. data/lib/resque/failure/mongo.rb +32 -0
  29. data/lib/resque/helpers.rb +65 -0
  30. data/lib/resque/job.rb +103 -0
  31. data/lib/resque/server.rb +182 -0
  32. data/lib/resque/server/public/idle.png +0 -0
  33. data/lib/resque/server/public/jquery-1.3.2.min.js +19 -0
  34. data/lib/resque/server/public/jquery.relatize_date.js +95 -0
  35. data/lib/resque/server/public/poll.png +0 -0
  36. data/lib/resque/server/public/ranger.js +24 -0
  37. data/lib/resque/server/public/reset.css +48 -0
  38. data/lib/resque/server/public/style.css +75 -0
  39. data/lib/resque/server/public/working.png +0 -0
  40. data/lib/resque/server/views/error.erb +1 -0
  41. data/lib/resque/server/views/failed.erb +35 -0
  42. data/lib/resque/server/views/key.erb +17 -0
  43. data/lib/resque/server/views/layout.erb +38 -0
  44. data/lib/resque/server/views/next_more.erb +10 -0
  45. data/lib/resque/server/views/overview.erb +4 -0
  46. data/lib/resque/server/views/queues.erb +46 -0
  47. data/lib/resque/server/views/stats.erb +62 -0
  48. data/lib/resque/server/views/workers.erb +78 -0
  49. data/lib/resque/server/views/working.erb +67 -0
  50. data/lib/resque/stat.rb +55 -0
  51. data/lib/resque/tasks.rb +39 -0
  52. data/lib/resque/version.rb +3 -0
  53. data/lib/resque/worker.rb +415 -0
  54. data/tasks/redis.rake +133 -0
  55. data/tasks/resque.rake +2 -0
  56. data/test/dump.rdb +0 -0
  57. data/test/redis-test.conf +132 -0
  58. data/test/resque_test.rb +191 -0
  59. data/test/test_helper.rb +87 -0
  60. data/test/worker_test.rb +229 -0
  61. metadata +162 -0
@@ -0,0 +1,3 @@
1
+ *.gemspec
2
+ pkg
3
+ nbproject
data/.kick ADDED
@@ -0,0 +1,26 @@
1
+ # take control of the growl notifications
2
+ module GrowlHacks
3
+ def growl(type, subject, body, *args, &block)
4
+ case type
5
+ when Kicker::GROWL_NOTIFICATIONS[:succeeded]
6
+ puts subject = "Success"
7
+ body = body.split("\n").last
8
+ when Kicker::GROWL_NOTIFICATIONS[:failed]
9
+ subject = "Failure"
10
+ puts body
11
+ body = body.split("\n").last
12
+ else
13
+ return nil
14
+ end
15
+ super(type, subject, body, *args, &block)
16
+ end
17
+ end
18
+
19
+ Kicker.send :extend, GrowlHacks
20
+
21
+ # no logging
22
+ Kicker::Utils.module_eval do
23
+ def log(message)
24
+ nil
25
+ end
26
+ end
@@ -0,0 +1,14 @@
1
+ * Chris Wanstrath
2
+ * John Barnette
3
+ * Adam Cooke
4
+ * Rob Hanlon
5
+ * Jason Amster
6
+ * jgeiger
7
+ * Aaron Quint
8
+ * Ben VandenBos
9
+ * PJ Hyett
10
+ * Arthur Zapparoli
11
+ * Simon Rozet
12
+ * Brian P O'Rourke
13
+ * Dave Hoover
14
+ * Michael Dwan
@@ -0,0 +1,53 @@
1
+ ## 1.3.1 (2010-01-11)
2
+
3
+ * Vegas bugfix: Don't error without a config
4
+
5
+ ## 1.3.0 (2010-01-11)
6
+
7
+ * Use Vegas for resque-web
8
+ * Web Bugfix: Show proper date/time value for failed_at on Failures
9
+ * Web Bugfix: Make the / route more flexible
10
+ * Add Resque::Server.tabs array (so plugins can add their own tabs)
11
+ * Start using [Semantic Versioning](http://semver.org/)
12
+
13
+ ## 1.2.4 (2009-12-15)
14
+
15
+ * Web Bugfix: fix key links on stat page
16
+
17
+ ## 1.2.3 (2009-12-15)
18
+
19
+ * Bugfix: Fixed `rand` seeding in child processes.
20
+ * Bugfix: Better JSON encoding/decoding without Yajl.
21
+ * Bugfix: Avoid `ps` flag error on Linux
22
+ * Add `PREFIX` observance to `rake` install tasks.
23
+
24
+ ## 1.2.2 (2009-12-08)
25
+
26
+ * Bugfix: Job equality was not properly implemented.
27
+
28
+ ## 1.2.1 (2009-12-07)
29
+
30
+ * Added `rake resque:workers` task for starting multiple workers.
31
+ * 1.9.x compatibility
32
+ * Bugfix: Yajl decoder doesn't care about valid UTF-8
33
+ * config.ru loads RESQUECONFIG if the ENV variable is set.
34
+ * `resque-web` now sets RESQUECONFIG
35
+ * Job objects know if they are equal.
36
+ * Jobs can be re-queued using `Job#recreate`
37
+
38
+ ## 1.2.0 (2009-11-25)
39
+
40
+ * If USR1 is sent and no child is found, shutdown.
41
+ * Raise when a job class does not respond to `perform`.
42
+ * Added `Resque.remove_queue` for deleting a queue
43
+
44
+ ## 1.1.0 (2009-11-04)
45
+
46
+ * Bugfix: Broken ERB tag in failure UI
47
+ * Bugfix: Save the worker's ID, not the worker itself, in the failure module
48
+ * Redesigned the sinatra web interface
49
+ * Added option to clear failed jobs
50
+
51
+ ## 1.0.0 (2009-11-03)
52
+
53
+ * First release.
data/LICENSE ADDED
@@ -0,0 +1,20 @@
1
+ Copyright (c) 2009 Chris Wanstrath
2
+
3
+ Permission is hereby granted, free of charge, to any person obtaining
4
+ a copy of this software and associated documentation files (the
5
+ "Software"), to deal in the Software without restriction, including
6
+ without limitation the rights to use, copy, modify, merge, publish,
7
+ distribute, sublicense, and/or sell copies of the Software, and to
8
+ permit persons to whom the Software is furnished to do so, subject to
9
+ the following conditions:
10
+
11
+ The above copyright notice and this permission notice shall be
12
+ included in all copies or substantial portions of the Software.
13
+
14
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
15
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
16
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
17
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
18
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
19
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
20
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@@ -0,0 +1,755 @@
1
+ Resque-mongo
2
+ ============
3
+ Resque-mongo is a fork of [Resque][resque] that uses MongoDB as a
4
+ backend instead of Redis. This fork is a work in progress, all the
5
+ library functionality has been ported (all tests pass) and the
6
+ monitoring sinatra app works except the "stats" panel, although there are
7
+ a lot of details that need to been taken care of.
8
+
9
+ Resque-mongo uses a fairly new feature of mongo, [findAndModify()][fnr].
10
+ findAndModify is not yet supported by the ruby mongo driver because the
11
+ command's api might change a bit. You can use a [patched version
12
+ mongo-ruby-driver][fnr-ruby] that supports the current implementation.
13
+
14
+ Also, check your mongo version: 1.3.0 or higher
15
+
16
+ [fnr]: http://www.mongodb.org/display/DOCS/findandmodify+Command
17
+ [fnr-ruby]: http://github.com/ctrochalakis/mongo-ruby-driver/tree/find_replace
18
+
19
+ Resque
20
+ ======
21
+
22
+ Resque is a Redis-backed library for creating background jobs, placing
23
+ those jobs on multiple queues, and processing them later.
24
+
25
+ Background jobs can be any Ruby class or module that responds to
26
+ `perform`. Your existing classes can easily be converted to background
27
+ jobs or you can create new classes specifically to do work. Or, you
28
+ can do both.
29
+
30
+ Resque is heavily inspired by DelayedJob (which rocks) and is
31
+ comprised of three parts:
32
+
33
+ 1. A Ruby library for creating, querying, and processing jobs
34
+ 2. A Rake task for starting a worker which processes jobs
35
+ 3. A Sinatra app for monitoring queues, jobs, and workers.
36
+
37
+ Resque workers can be distributed between multiple machines,
38
+ support priorities, are resilient to memory bloat / "leaks," are
39
+ optimized for REE (but work on MRI and JRuby), tell you what they're
40
+ doing, and expect failure.
41
+
42
+ Resque queues are persistent; support constant time, atomic push and
43
+ pop (thanks to Redis); provide visibility into their contents; and
44
+ store jobs as simple JSON packages.
45
+
46
+ The Resque frontend tells you what workers are doing, what workers are
47
+ not doing, what queues you're using, what's in those queues, provides
48
+ general usage stats, and helps you track failures.
49
+
50
+
51
+ The Blog Post
52
+ -------------
53
+
54
+ For the backstory, philosophy, and history of Resque's beginnings,
55
+ please see [the blog post][0].
56
+
57
+
58
+ Overview
59
+ --------
60
+
61
+ Resque allows you to create jobs and place them on a queue, then,
62
+ later, pull those jobs off the queue and process them.
63
+
64
+ Resque jobs are Ruby classes (or modules) which respond to the
65
+ `perform` method. Here's an example:
66
+
67
+ class Archive
68
+ @queue = :file_serve
69
+
70
+ def self.perform(repo_id, branch = 'master')
71
+ repo = Repository.find(repo_id)
72
+ repo.create_archive(branch)
73
+ end
74
+ end
75
+
76
+ The `@queue` class instance variable determines which queue `Archive`
77
+ jobs will be placed in. Queues are arbitrary and created on the fly -
78
+ you can name them whatever you want and have as many as you want.
79
+
80
+ To place an `Archive` job on the `file_serve` queue, we might add this
81
+ to our application's pre-existing `Repository` class:
82
+
83
+ class Repository
84
+ def async_create_archive(branch)
85
+ Resque.enqueue(Archive, self.id, branch)
86
+ end
87
+ end
88
+
89
+ Now when we call `repo.async_create_archive('masterbrew')` in our
90
+ application, a job will be created and placed on the `file_serve`
91
+ queue.
92
+
93
+ Later, a worker will run something like this code to process the job:
94
+
95
+ klass, args = Resque.reserve(:file_serve)
96
+ klass.perform(*args) if klass.respond_to? :perform
97
+
98
+ Which translates to:
99
+
100
+ Archive.perform(44, 'masterbrew')
101
+
102
+ Let's start a worker to run `file_serve` jobs:
103
+
104
+ $ cd app_root
105
+ $ QUEUE=file_serve rake resque:work
106
+
107
+ This starts one Resque worker and tells it to work off the
108
+ `file_serve` queue. As soon as it's ready it'll try to run the
109
+ `Resque.reserve` code snippet above and process jobs until it can't
110
+ find any more, at which point it will sleep for a small period and
111
+ repeatedly poll the queue for more jobs.
112
+
113
+ Workers can be given multiple queues (a "queue list") and run on
114
+ multiple machines. In fact they can be run anywhere with network
115
+ access to the Redis server.
116
+
117
+
118
+ Jobs
119
+ ----
120
+
121
+ What should you run in the background? Anything that takes any time at
122
+ all. Slow INSERT statements, disk manipulating, data processing, etc.
123
+
124
+ At GitHub we use Resque to process the following types of jobs:
125
+
126
+ * Warming caches
127
+ * Counting disk usage
128
+ * Building tarballs
129
+ * Building Rubygems
130
+ * Firing off web hooks
131
+ * Creating events in the db and pre-caching them
132
+ * Building graphs
133
+ * Deleting users
134
+ * Updating our search index
135
+
136
+ As of writing we have about 35 different types of background jobs.
137
+
138
+ Keep in mind that you don't need a web app to use Resque - we just
139
+ mention "foreground" and "background" because they make conceptual
140
+ sense. You could easily be spidering sites and sticking data which
141
+ needs to be crunched later into a queue.
142
+
143
+
144
+ ### Persistence
145
+
146
+ Jobs are persisted to queues as JSON objects. Let's take our `Archive`
147
+ example from above. We'll run the following code to create a job:
148
+
149
+ repo = Repository.find(44)
150
+ repo.async_create_archive('masterbrew')
151
+
152
+ The following JSON will be stored in the `file_serve` queue:
153
+
154
+ {
155
+ 'class': 'Archive',
156
+ 'args': [ 44, 'masterbrew' ]
157
+ }
158
+
159
+ Because of this your jobs must only accept arguments that can be JSON encoded.
160
+
161
+ So instead of doing this:
162
+
163
+ Resque.enqueue(Archive, self, branch)
164
+
165
+ do this:
166
+
167
+ Resque.enqueue(Archive, self.id, branch)
168
+
169
+ This is why our above example (and all the examples in `examples/`)
170
+ uses object IDs instead of passing around the objects.
171
+
172
+ While this is less convenient than just sticking a marshaled object
173
+ in the database, it gives you a slight advantage: your jobs will be
174
+ run against the most recent version of an object because they need to
175
+ pull from the DB or cache.
176
+
177
+ If your jobs were run against marshaled objects, they could
178
+ potentially be operating on a stale record with out-of-date information.
179
+
180
+
181
+ ### send_later / async
182
+
183
+ Want something like DelayedJob's `send_later` or the ability to use
184
+ instance methods instead of just methods for jobs? See the `examples/`
185
+ directory for goodies.
186
+
187
+ We plan to provide first class `async` support in a future release.
188
+
189
+
190
+ ### Failure
191
+
192
+ If a job raises an exception, it is logged and handed off to the
193
+ `Resque::Failure` module. Failures are logged either locally in Redis
194
+ or using some different backend.
195
+
196
+ For example, Resque ships with Hoptoad support.
197
+
198
+ Keep this in mind when writing your jobs: you may want to throw
199
+ exceptions you would not normally throw in order to assist debugging.
200
+
201
+
202
+ Workers
203
+ -------
204
+
205
+ Resque workers are rake tasks that run forever. They basically do this:
206
+
207
+ start
208
+ loop do
209
+ if job = reserve
210
+ job.process
211
+ else
212
+ sleep 5
213
+ end
214
+ end
215
+ shutdown
216
+
217
+ Starting a worker is simple. Here's our example from earlier:
218
+
219
+ $ QUEUE=file_serve rake resque:work
220
+
221
+ By default Resque won't know about your application's
222
+ environment. That is, it won't be able to find and run your jobs - it
223
+ needs to load your application into memory.
224
+
225
+ If we've installed Resque as a Rails plugin, we might run this command
226
+ from our RAILS_ROOT:
227
+
228
+ $ QUEUE=file_serve rake environment resque:work
229
+
230
+ This will load the environment before starting a worker. Alternately
231
+ we can define a `resque:setup` task with a dependency on the
232
+ `environment` rake task:
233
+
234
+ task "resque:setup" => :environment
235
+
236
+ GitHub's setup task looks like this:
237
+
238
+ task "resque:setup" => :environment do
239
+ Grit::Git.git_timeout = 10.minutes
240
+ end
241
+
242
+ We don't want the `git_timeout` as high as 10 minutes in our web app,
243
+ but in the Resque workers it's fine.
244
+
245
+
246
+ ### Logging
247
+
248
+ Workers support basic logging to STDOUT. If you start them with the
249
+ `VERBOSE` env variable set, they will print basic debugging
250
+ information. You can also set the `VVERBOSE` (very verbose) env
251
+ variable.
252
+
253
+ $ VVERBOSE=1 QUEUE=file_serve rake environment resque:work
254
+
255
+
256
+ ### Priorities and Queue Lists
257
+
258
+ Resque doesn't support numeric priorities but instead uses the order
259
+ of queues you give it. We call this list of queues the "queue list."
260
+
261
+ Let's say we add a `warm_cache` queue in addition to our `file_serve`
262
+ queue. We'd now start a worker like so:
263
+
264
+ $ QUEUES=file_serve,warm_cache rake resque:work
265
+
266
+ When the worker looks for new jobs, it will first check
267
+ `file_serve`. If it finds a job, it'll process it then check
268
+ `file_serve` again. It will keep checking `file_serve` until no more
269
+ jobs are available. At that point, it will check `warm_cache`. If it
270
+ finds a job it'll process it then check `file_serve` (repeating the
271
+ whole process).
272
+
273
+ In this way you can prioritize certain queues. At GitHub we start our
274
+ workers with something like this:
275
+
276
+ $ QUEUES=critical,archive,high,low rake resque:work
277
+
278
+ Notice the `archive` queue - it is specialized and in our future
279
+ architecture will only be run from a single machine.
280
+
281
+ At that point we'll start workers on our generalized background
282
+ machines with this command:
283
+
284
+ $ QUEUES=critical,high,low rake resque:work
285
+
286
+ And workers on our specialized archive machine with this command:
287
+
288
+ $ QUEUE=archive rake resque:work
289
+
290
+
291
+ ### Running All Queues
292
+
293
+ If you want your workers to work off of every queue, including new
294
+ queues created on the fly, you can use a splat:
295
+
296
+ $ QUEUE=* rake resque:work
297
+
298
+ Queues will be processed in alphabetical order.
299
+
300
+
301
+ ### Running Multiple Workers
302
+
303
+ At GitHub we use god to start and stop multiple workers. A sample god
304
+ configuration file is included under `examples/god`. We recommend this
305
+ method.
306
+
307
+ If you'd like to run multiple workers in development mode, you can do
308
+ so using the `resque:workers` rake task:
309
+
310
+ $ COUNT=5 QUEUE=* rake resque:workers
311
+
312
+ This will spawn five Resque workers, each in its own thread. Hitting
313
+ ctrl-c should be sufficient to stop them all.
314
+
315
+
316
+ ### Forking
317
+
318
+ On certain platforms, when a Resque worker reserves a job it
319
+ immediately forks a child process. The child processes the job then
320
+ exits. When the child has exited successfully, the worker reserves
321
+ another job and repeats the process.
322
+
323
+ Why?
324
+
325
+ Because Resque assumes chaos.
326
+
327
+ Resque assumes your background workers will lock up, run too long, or
328
+ have unwanted memory growth.
329
+
330
+ If Resque workers processed jobs themselves, it'd be hard to whip them
331
+ into shape. Let's say one is using too much memory: you send it a
332
+ signal that says "shutdown after you finish processing the current
333
+ job," and it does so. It then starts up again - loading your entire
334
+ application environment. This adds useless CPU cycles and causes a
335
+ delay in queue processing.
336
+
337
+ Plus, what if it's using too much memory and has stopped responding to
338
+ signals?
339
+
340
+ Thanks to Resque's parent / child architecture, jobs that use too much memory
341
+ release that memory upon completion. No unwanted growth.
342
+
343
+ And what if a job is running too long? You'd need to `kill -9` it then
344
+ start the worker again. With Resque's parent / child architecture you
345
+ can tell the parent to forcefully kill the child then immediately
346
+ start processing more jobs. No startup delay or wasted cycles.
347
+
348
+ The parent / child architecture helps us keep tabs on what workers are
349
+ doing, too. By eliminating the need to `kill -9` workers we can have
350
+ parents remove themselves from the global listing of workers. If we
351
+ just ruthlessly killed workers, we'd need a separate watchdog process
352
+ to add and remove them to the global listing - which becomes
353
+ complicated.
354
+
355
+ Workers instead handle their own state.
356
+
357
+
358
+ ### Parents and Children
359
+
360
+ Here's a parent / child pair doing some work:
361
+
362
+ $ ps -e -o pid,command | grep [r]esque
363
+ 92099 resque: Forked 92102 at 1253142769
364
+ 92102 resque: Processing file_serve since 1253142769
365
+
366
+ You can clearly see that process 92099 forked 92102, which has been
367
+ working since 1253142769.
368
+
369
+ (By advertising the time they began processing you can easily use monit
370
+ or god to kill stale workers.)
371
+
372
+ When a parent process is idle, it lets you know what queues it is
373
+ waiting for work on:
374
+
375
+ $ ps -e -o pid,command | grep [r]esque
376
+ 92099 resque: Waiting for file_serve,warm_cache
377
+
378
+
379
+ ### Signals
380
+
381
+ Resque workers respond to a few different signals:
382
+
383
+ * `QUIT` - Wait for child to finish processing then exit
384
+ * `TERM` / `INT` - Immediately kill child then exit
385
+ * `USR1` - Immediately kill child but don't exit
386
+
387
+ If you want to gracefully shutdown a Resque worker, use `QUIT`.
388
+
389
+ If you want to kill a stale or stuck child, use `USR1`. Processing
390
+ will continue as normal unless the child was not found. In that case
391
+ Resque assumes the parent process is in a bad state and shuts down.
392
+
393
+ If you want to kill a stale or stuck child and shutdown, use `TERM`
394
+
395
+ ### Mysql::Error: MySQL server has gone away
396
+
397
+ If your workers remain idle for too long they may lose their MySQL
398
+ connection. If that happens we recommend using [this
399
+ Gist](http://gist.github.com/238999).
400
+
401
+
402
+ The Front End
403
+ -------------
404
+
405
+ Resque comes with a Sinatra-based front end for seeing what's up with
406
+ your queue.
407
+
408
+ ![The Front End](http://img.skitch.com/20091104-tqh5pgkwgbskjbk7qbtmpesnyw.jpg)
409
+
410
+ ### Standalone
411
+
412
+ If you've installed Resque as a gem running the front end standalone is easy:
413
+
414
+ $ resque-web
415
+
416
+ It's a thin layer around `rackup` so it's configurable as well:
417
+
418
+ $ resque-web -p 8282
419
+
420
+ If you have a Resque config file you want evaluated just pass it to
421
+ the script as the final argument:
422
+
423
+ $ resque-web -p 8282 rails_root/config/initializers/resque.rb
424
+
425
+ ### Passenger
426
+
427
+ Using Passenger? Resque ships with a `config.ru` you can use. See
428
+ Phusion's guide:
429
+
430
+ <http://www.modrails.com/documentation/Users%20guide.html#_deploying_a_rack_based_ruby_application>
431
+
432
+ ### Rack::URLMap
433
+
434
+ If you want to load Resque on a subpath, possibly alongside other
435
+ apps, it's easy to do with Rack's `URLMap`:
436
+
437
+ require 'resque/server'
438
+
439
+ run Rack::URLMap.new \
440
+ "/" => Your::App.new,
441
+ "/resque" => Resque::Server.new
442
+
443
+ Check `examples/demo/config.ru` for a functional example (including
444
+ HTTP basic auth).
445
+
446
+
447
+ Resque vs DelayedJob
448
+ --------------------
449
+
450
+ How does Resque compare to DelayedJob, and why would you choose one
451
+ over the other?
452
+
453
+ * Resque supports multiple queues
454
+ * DelayedJob supports finer grained priorities
455
+ * Resque workers are resilient to memory leaks / bloat
456
+ * DelayedJob workers are extremely simple and easy to modify
457
+ * Resque requires Redis
458
+ * DelayedJob requires ActiveRecord
459
+ * Resque can only place JSONable Ruby objects on a queue as arguments
460
+ * DelayedJob can place _any_ Ruby object on its queue as arguments
461
+ * Resque includes a Sinatra app for monitoring what's going on
462
+ * DelayedJob can be queried from within your Rails app if you want to
463
+ add an interface
464
+
465
+ If you're doing Rails development, you already have a database and
466
+ ActiveRecord. DelayedJob is super easy to setup and works great.
467
+ GitHub used it for many months to process almost 200 million jobs.
468
+
469
+ Choose Resque if:
470
+
471
+ * You need multiple queues
472
+ * You don't care / dislike numeric priorities
473
+ * You don't need to persist every Ruby object ever
474
+ * You have potentially huge queues
475
+ * You want to see what's going on
476
+ * You expect a lot of failure / chaos
477
+ * You can setup Redis
478
+ * You're not running short on RAM
479
+
480
+ Choose DelayedJob if:
481
+
482
+ * You like numeric priorities
483
+ * You're not doing a gigantic amount of jobs each day
484
+ * Your queue stays small and nimble
485
+ * There is not a lot failure / chaos
486
+ * You want to easily throw anything on the queue
487
+ * You don't want to setup Redis
488
+
489
+ In no way is Resque a "better" DelayedJob, so make sure you pick the
490
+ tool that's best for your app.
491
+
492
+
493
+ Installing Redis
494
+ ----------------
495
+
496
+ Resque uses Redis' lists for its queues. It also stores worker state
497
+ data in Redis.
498
+
499
+ #### Homebrew
500
+
501
+ If you're on OS X, Homebrew is the simplest way to install Redis:
502
+
503
+ $ brew install redis
504
+ $ redis-server /usr/local/etc/redis.conf
505
+
506
+ You now have a Redis daemon running on 6379.
507
+
508
+ #### Via Resque
509
+
510
+ Resque includes Rake tasks (thanks to Ezra's redis-rb) that will
511
+ install and run Redis for you:
512
+
513
+ $ git clone git://github.com/defunkt/resque.git
514
+ $ cd resque
515
+ $ rake redis:install dtach:install
516
+ $ rake redis:start
517
+
518
+ Or, if you don't have admin access on your machine:
519
+
520
+ $ git clone git://github.com/defunkt/resque.git
521
+ $ cd resque
522
+ $ PREFIX=<your_prefix> rake redis:install dtach:install
523
+ $ rake redis:start
524
+
525
+ You now have Redis running on 6379. Wait a second then hit ctrl-\ to
526
+ detach and keep it running in the background.
527
+
528
+ The demo is probably the best way to figure out how to put the parts
529
+ together. But, it's not that hard.
530
+
531
+
532
+ Resque Dependencies
533
+ -------------------
534
+
535
+ gem install redis redis-namespace yajl-ruby
536
+
537
+ If you cannot install `yajl-ruby` (JRuby?), you can install the `json`
538
+ gem and Resque will use it instead.
539
+
540
+
541
+ Installing Resque
542
+ -----------------
543
+
544
+ ### In a Rack app, as a gem
545
+
546
+ First install the gem.
547
+
548
+ $ gem install resque
549
+
550
+ Next include it in your application.
551
+
552
+ require 'resque'
553
+
554
+ Now start your application:
555
+
556
+ rackup config.ru
557
+
558
+ That's it! You can now create Resque jobs from within your app.
559
+
560
+ To start a worker, create a Rakefile in your app's root (or add this
561
+ to an existing Rakefile):
562
+
563
+ require 'your/app'
564
+ require 'resque/tasks'
565
+
566
+ Now:
567
+
568
+ $ QUEUE=* rake resque:work
569
+
570
+ Alternately you can define a `resque:setup` hook in your Rakefile if you
571
+ don't want to load your app every time rake runs.
572
+
573
+
574
+ ### In a Rails app, as a gem
575
+
576
+ First install the gem.
577
+
578
+ $ gem install resque
579
+
580
+ Next include it in your application.
581
+
582
+ $ cat config/initializers/load_resque.rb
583
+ require 'resque'
584
+
585
+ Now start your application:
586
+
587
+ $ ./script/server
588
+
589
+ That's it! You can now create Resque jobs from within your app.
590
+
591
+ To start a worker, add this to your Rakefile in `RAILS_ROOT`:
592
+
593
+ require 'resque/tasks'
594
+
595
+ Now:
596
+
597
+ $ QUEUE=* rake environment resque:work
598
+
599
+ Don't forget you can define a `resque:setup` hook in
600
+ `lib/tasks/whatever.rake` that loads the `environment` task every time.
601
+
602
+
603
+ ### In a Rails app, as a plugin
604
+
605
+ $ ./script/plugin install git://github.com/defunkt/resque
606
+
607
+ That's it! Resque will automatically be available when your Rails app
608
+ loads.
609
+
610
+ To start a worker:
611
+
612
+ $ QUEUE=* rake environment resque:work
613
+
614
+ Don't forget you can define a `resque:setup` hook in
615
+ `lib/tasks/whatever.rake` that loads the `environment` task every time.
616
+
617
+
618
+ Configuration
619
+ -------------
620
+
621
+ You may want to change the Redis host and port Resque connects to, or
622
+ set various other options at startup.
623
+
624
+ Resque has a `redis` setter which can be given a string or a Redis
625
+ object. This means if you're already using Redis in your app, Resque
626
+ can re-use the existing connection.
627
+
628
+ String: `Resque.redis = 'localhost:6379'`
629
+
630
+ Redis: `Redus.redis = $redis`
631
+
632
+ For our rails app we have a `config/initializers/resque.rb` file where
633
+ we load `config/resque.yml` by hand and set the Redis information
634
+ appropriately.
635
+
636
+ Here's our `config/resque.yml`:
637
+
638
+ development: localhost:6379
639
+ test: localhost:6379
640
+ staging: redis1.se.github.com:6379
641
+ fi: localhost:6379
642
+ production: redis1.ae.github.com:6379
643
+
644
+ And our initializer:
645
+
646
+ rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..'
647
+ rails_env = ENV['RAILS_ENV'] || 'development'
648
+
649
+ resque_config = YAML.load_file(rails_root + '/config/resque.yml')
650
+ Resque.redis = resque_config[rails_env]
651
+
652
+ Easy peasy! Why not just use `RAILS_ROOT` and `RAILS_ENV`? Because
653
+ this way we can tell our Sinatra app about the config file:
654
+
655
+ $ RAILS_ENV=production resque-web rails_root/config/initializers/resque.rb
656
+
657
+ Now everyone is on the same page.
658
+
659
+
660
+ Demo
661
+ ----
662
+
663
+ Resque ships with a demo Sinatra app for creating jobs that are later
664
+ processed in the background.
665
+
666
+ Try it out by looking at the README, found at `examples/demo/README.markdown`.
667
+
668
+
669
+ Monitoring
670
+ ----------
671
+
672
+ If you're using god to monitor Resque, we have provided example
673
+ configs in `examples/god/`. One is for starting / stopping workers,
674
+ the other is for killing workers that have been running too long.
675
+
676
+
677
+ Development
678
+ -----------
679
+
680
+ Want to hack on Resque?
681
+
682
+ First clone the repo and run the tests:
683
+
684
+ git clone git://github.com/defunkt/resque.git
685
+ cd resque
686
+ rake test
687
+
688
+ If the tests do not pass make sure you have Redis installed
689
+ correctly (though we make an effort to tell you if we feel this is the
690
+ case). The tests attempt to start an isolated instance of Redis to
691
+ run against.
692
+
693
+ Also make sure you've installed all the dependencies correctly. For
694
+ example, try loading the `redis-namespace` gem after you've installed
695
+ it:
696
+
697
+ $ irb
698
+ >> require 'rubygems'
699
+ => true
700
+ >> require 'redis/namespace'
701
+ => true
702
+
703
+ If you get an error requiring any of the dependencies, you may have
704
+ failed to install them or be seeing load path issues.
705
+
706
+ Feel free to ping the mailing list with your problem and we'll try to
707
+ sort it out.
708
+
709
+
710
+ Contributing
711
+ ------------
712
+
713
+ Once you've made your great commits:
714
+
715
+ 1. [Fork][1] Resque
716
+ 2. Create a topic branch - `git checkout -b my_branch`
717
+ 3. Push to your branch - `git push origin my_branch`
718
+ 4. Create an [Issue][2] with a link to your branch
719
+ 5. That's it!
720
+
721
+
722
+ Mailing List
723
+ ------------
724
+
725
+ To join the list simply send an email to <resque@librelist.com>. This
726
+ will subscribe you and send you information about your subscription,
727
+ include unsubscribe information.
728
+
729
+ The archive can be found at <http://librelist.com/browser/>.
730
+
731
+
732
+ Meta
733
+ ----
734
+
735
+ * Code: `git clone git://github.com/defunkt/resque.git`
736
+ * Home: <http://github.com/defunkt/resque>
737
+ * Docs: <http://defunkt.github.com/resque/>
738
+ * Bugs: <http://github.com/defunkt/resque/issues>
739
+ * List: <resque@librelist.com>
740
+ * Chat: <irc://irc.freenode.net/resque>
741
+ * Gems: <http://gemcutter.org/gems/resque>
742
+
743
+ This project uses [Semantic Versioning][sv].
744
+
745
+
746
+ Author
747
+ ------
748
+
749
+ Chris Wanstrath :: chris@ozmm.org :: @defunkt
750
+
751
+ [0]: http://github.com/blog/542-introducing-resque
752
+ [1]: http://help.github.com/forking/
753
+ [2]: http://github.com/defunkt/resque/issues
754
+ [sv]: http://semver.org/
755
+ [resque]: http://github.com/defunct/resque