resque-mongo 1.3.1

Sign up to get free protection for your applications and to get access to all the features.
Files changed (61) hide show
  1. data/.gitignore +3 -0
  2. data/.kick +26 -0
  3. data/CONTRIBUTORS +14 -0
  4. data/HISTORY.md +53 -0
  5. data/LICENSE +20 -0
  6. data/README.markdown +755 -0
  7. data/Rakefile +65 -0
  8. data/bin/resque +57 -0
  9. data/bin/resque-web +18 -0
  10. data/config.ru +14 -0
  11. data/deps.rip +8 -0
  12. data/examples/async_helper.rb +31 -0
  13. data/examples/demo/README.markdown +71 -0
  14. data/examples/demo/Rakefile +3 -0
  15. data/examples/demo/app.rb +27 -0
  16. data/examples/demo/config.ru +19 -0
  17. data/examples/demo/job.rb +12 -0
  18. data/examples/god/resque.god +52 -0
  19. data/examples/god/stale.god +26 -0
  20. data/examples/instance.rb +11 -0
  21. data/examples/simple.rb +30 -0
  22. data/init.rb +1 -0
  23. data/lib/resque.rb +224 -0
  24. data/lib/resque/errors.rb +7 -0
  25. data/lib/resque/failure.rb +63 -0
  26. data/lib/resque/failure/base.rb +58 -0
  27. data/lib/resque/failure/hoptoad.rb +88 -0
  28. data/lib/resque/failure/mongo.rb +32 -0
  29. data/lib/resque/helpers.rb +65 -0
  30. data/lib/resque/job.rb +103 -0
  31. data/lib/resque/server.rb +182 -0
  32. data/lib/resque/server/public/idle.png +0 -0
  33. data/lib/resque/server/public/jquery-1.3.2.min.js +19 -0
  34. data/lib/resque/server/public/jquery.relatize_date.js +95 -0
  35. data/lib/resque/server/public/poll.png +0 -0
  36. data/lib/resque/server/public/ranger.js +24 -0
  37. data/lib/resque/server/public/reset.css +48 -0
  38. data/lib/resque/server/public/style.css +75 -0
  39. data/lib/resque/server/public/working.png +0 -0
  40. data/lib/resque/server/views/error.erb +1 -0
  41. data/lib/resque/server/views/failed.erb +35 -0
  42. data/lib/resque/server/views/key.erb +17 -0
  43. data/lib/resque/server/views/layout.erb +38 -0
  44. data/lib/resque/server/views/next_more.erb +10 -0
  45. data/lib/resque/server/views/overview.erb +4 -0
  46. data/lib/resque/server/views/queues.erb +46 -0
  47. data/lib/resque/server/views/stats.erb +62 -0
  48. data/lib/resque/server/views/workers.erb +78 -0
  49. data/lib/resque/server/views/working.erb +67 -0
  50. data/lib/resque/stat.rb +55 -0
  51. data/lib/resque/tasks.rb +39 -0
  52. data/lib/resque/version.rb +3 -0
  53. data/lib/resque/worker.rb +415 -0
  54. data/tasks/redis.rake +133 -0
  55. data/tasks/resque.rake +2 -0
  56. data/test/dump.rdb +0 -0
  57. data/test/redis-test.conf +132 -0
  58. data/test/resque_test.rb +191 -0
  59. data/test/test_helper.rb +87 -0
  60. data/test/worker_test.rb +229 -0
  61. metadata +162 -0
@@ -0,0 +1,3 @@
1
+ *.gemspec
2
+ pkg
3
+ nbproject
data/.kick ADDED
@@ -0,0 +1,26 @@
1
+ # take control of the growl notifications
2
+ module GrowlHacks
3
+ def growl(type, subject, body, *args, &block)
4
+ case type
5
+ when Kicker::GROWL_NOTIFICATIONS[:succeeded]
6
+ puts subject = "Success"
7
+ body = body.split("\n").last
8
+ when Kicker::GROWL_NOTIFICATIONS[:failed]
9
+ subject = "Failure"
10
+ puts body
11
+ body = body.split("\n").last
12
+ else
13
+ return nil
14
+ end
15
+ super(type, subject, body, *args, &block)
16
+ end
17
+ end
18
+
19
+ Kicker.send :extend, GrowlHacks
20
+
21
+ # no logging
22
+ Kicker::Utils.module_eval do
23
+ def log(message)
24
+ nil
25
+ end
26
+ end
@@ -0,0 +1,14 @@
1
+ * Chris Wanstrath
2
+ * John Barnette
3
+ * Adam Cooke
4
+ * Rob Hanlon
5
+ * Jason Amster
6
+ * jgeiger
7
+ * Aaron Quint
8
+ * Ben VandenBos
9
+ * PJ Hyett
10
+ * Arthur Zapparoli
11
+ * Simon Rozet
12
+ * Brian P O'Rourke
13
+ * Dave Hoover
14
+ * Michael Dwan
@@ -0,0 +1,53 @@
1
+ ## 1.3.1 (2010-01-11)
2
+
3
+ * Vegas bugfix: Don't error without a config
4
+
5
+ ## 1.3.0 (2010-01-11)
6
+
7
+ * Use Vegas for resque-web
8
+ * Web Bugfix: Show proper date/time value for failed_at on Failures
9
+ * Web Bugfix: Make the / route more flexible
10
+ * Add Resque::Server.tabs array (so plugins can add their own tabs)
11
+ * Start using [Semantic Versioning](http://semver.org/)
12
+
13
+ ## 1.2.4 (2009-12-15)
14
+
15
+ * Web Bugfix: fix key links on stat page
16
+
17
+ ## 1.2.3 (2009-12-15)
18
+
19
+ * Bugfix: Fixed `rand` seeding in child processes.
20
+ * Bugfix: Better JSON encoding/decoding without Yajl.
21
+ * Bugfix: Avoid `ps` flag error on Linux
22
+ * Add `PREFIX` observance to `rake` install tasks.
23
+
24
+ ## 1.2.2 (2009-12-08)
25
+
26
+ * Bugfix: Job equality was not properly implemented.
27
+
28
+ ## 1.2.1 (2009-12-07)
29
+
30
+ * Added `rake resque:workers` task for starting multiple workers.
31
+ * 1.9.x compatibility
32
+ * Bugfix: Yajl decoder doesn't care about valid UTF-8
33
+ * config.ru loads RESQUECONFIG if the ENV variable is set.
34
+ * `resque-web` now sets RESQUECONFIG
35
+ * Job objects know if they are equal.
36
+ * Jobs can be re-queued using `Job#recreate`
37
+
38
+ ## 1.2.0 (2009-11-25)
39
+
40
+ * If USR1 is sent and no child is found, shutdown.
41
+ * Raise when a job class does not respond to `perform`.
42
+ * Added `Resque.remove_queue` for deleting a queue
43
+
44
+ ## 1.1.0 (2009-11-04)
45
+
46
+ * Bugfix: Broken ERB tag in failure UI
47
+ * Bugfix: Save the worker's ID, not the worker itself, in the failure module
48
+ * Redesigned the sinatra web interface
49
+ * Added option to clear failed jobs
50
+
51
+ ## 1.0.0 (2009-11-03)
52
+
53
+ * First release.
data/LICENSE ADDED
@@ -0,0 +1,20 @@
1
+ Copyright (c) 2009 Chris Wanstrath
2
+
3
+ Permission is hereby granted, free of charge, to any person obtaining
4
+ a copy of this software and associated documentation files (the
5
+ "Software"), to deal in the Software without restriction, including
6
+ without limitation the rights to use, copy, modify, merge, publish,
7
+ distribute, sublicense, and/or sell copies of the Software, and to
8
+ permit persons to whom the Software is furnished to do so, subject to
9
+ the following conditions:
10
+
11
+ The above copyright notice and this permission notice shall be
12
+ included in all copies or substantial portions of the Software.
13
+
14
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
15
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
16
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
17
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
18
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
19
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
20
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@@ -0,0 +1,755 @@
1
+ Resque-mongo
2
+ ============
3
+ Resque-mongo is a fork of [Resque][resque] that uses MongoDB as a
4
+ backend instead of Redis. This fork is a work in progress, all the
5
+ library functionality has been ported (all tests pass) and the
6
+ monitoring sinatra app works except the "stats" panel, although there are
7
+ a lot of details that need to been taken care of.
8
+
9
+ Resque-mongo uses a fairly new feature of mongo, [findAndModify()][fnr].
10
+ findAndModify is not yet supported by the ruby mongo driver because the
11
+ command's api might change a bit. You can use a [patched version
12
+ mongo-ruby-driver][fnr-ruby] that supports the current implementation.
13
+
14
+ Also, check your mongo version: 1.3.0 or higher
15
+
16
+ [fnr]: http://www.mongodb.org/display/DOCS/findandmodify+Command
17
+ [fnr-ruby]: http://github.com/ctrochalakis/mongo-ruby-driver/tree/find_replace
18
+
19
+ Resque
20
+ ======
21
+
22
+ Resque is a Redis-backed library for creating background jobs, placing
23
+ those jobs on multiple queues, and processing them later.
24
+
25
+ Background jobs can be any Ruby class or module that responds to
26
+ `perform`. Your existing classes can easily be converted to background
27
+ jobs or you can create new classes specifically to do work. Or, you
28
+ can do both.
29
+
30
+ Resque is heavily inspired by DelayedJob (which rocks) and is
31
+ comprised of three parts:
32
+
33
+ 1. A Ruby library for creating, querying, and processing jobs
34
+ 2. A Rake task for starting a worker which processes jobs
35
+ 3. A Sinatra app for monitoring queues, jobs, and workers.
36
+
37
+ Resque workers can be distributed between multiple machines,
38
+ support priorities, are resilient to memory bloat / "leaks," are
39
+ optimized for REE (but work on MRI and JRuby), tell you what they're
40
+ doing, and expect failure.
41
+
42
+ Resque queues are persistent; support constant time, atomic push and
43
+ pop (thanks to Redis); provide visibility into their contents; and
44
+ store jobs as simple JSON packages.
45
+
46
+ The Resque frontend tells you what workers are doing, what workers are
47
+ not doing, what queues you're using, what's in those queues, provides
48
+ general usage stats, and helps you track failures.
49
+
50
+
51
+ The Blog Post
52
+ -------------
53
+
54
+ For the backstory, philosophy, and history of Resque's beginnings,
55
+ please see [the blog post][0].
56
+
57
+
58
+ Overview
59
+ --------
60
+
61
+ Resque allows you to create jobs and place them on a queue, then,
62
+ later, pull those jobs off the queue and process them.
63
+
64
+ Resque jobs are Ruby classes (or modules) which respond to the
65
+ `perform` method. Here's an example:
66
+
67
+ class Archive
68
+ @queue = :file_serve
69
+
70
+ def self.perform(repo_id, branch = 'master')
71
+ repo = Repository.find(repo_id)
72
+ repo.create_archive(branch)
73
+ end
74
+ end
75
+
76
+ The `@queue` class instance variable determines which queue `Archive`
77
+ jobs will be placed in. Queues are arbitrary and created on the fly -
78
+ you can name them whatever you want and have as many as you want.
79
+
80
+ To place an `Archive` job on the `file_serve` queue, we might add this
81
+ to our application's pre-existing `Repository` class:
82
+
83
+ class Repository
84
+ def async_create_archive(branch)
85
+ Resque.enqueue(Archive, self.id, branch)
86
+ end
87
+ end
88
+
89
+ Now when we call `repo.async_create_archive('masterbrew')` in our
90
+ application, a job will be created and placed on the `file_serve`
91
+ queue.
92
+
93
+ Later, a worker will run something like this code to process the job:
94
+
95
+ klass, args = Resque.reserve(:file_serve)
96
+ klass.perform(*args) if klass.respond_to? :perform
97
+
98
+ Which translates to:
99
+
100
+ Archive.perform(44, 'masterbrew')
101
+
102
+ Let's start a worker to run `file_serve` jobs:
103
+
104
+ $ cd app_root
105
+ $ QUEUE=file_serve rake resque:work
106
+
107
+ This starts one Resque worker and tells it to work off the
108
+ `file_serve` queue. As soon as it's ready it'll try to run the
109
+ `Resque.reserve` code snippet above and process jobs until it can't
110
+ find any more, at which point it will sleep for a small period and
111
+ repeatedly poll the queue for more jobs.
112
+
113
+ Workers can be given multiple queues (a "queue list") and run on
114
+ multiple machines. In fact they can be run anywhere with network
115
+ access to the Redis server.
116
+
117
+
118
+ Jobs
119
+ ----
120
+
121
+ What should you run in the background? Anything that takes any time at
122
+ all. Slow INSERT statements, disk manipulating, data processing, etc.
123
+
124
+ At GitHub we use Resque to process the following types of jobs:
125
+
126
+ * Warming caches
127
+ * Counting disk usage
128
+ * Building tarballs
129
+ * Building Rubygems
130
+ * Firing off web hooks
131
+ * Creating events in the db and pre-caching them
132
+ * Building graphs
133
+ * Deleting users
134
+ * Updating our search index
135
+
136
+ As of writing we have about 35 different types of background jobs.
137
+
138
+ Keep in mind that you don't need a web app to use Resque - we just
139
+ mention "foreground" and "background" because they make conceptual
140
+ sense. You could easily be spidering sites and sticking data which
141
+ needs to be crunched later into a queue.
142
+
143
+
144
+ ### Persistence
145
+
146
+ Jobs are persisted to queues as JSON objects. Let's take our `Archive`
147
+ example from above. We'll run the following code to create a job:
148
+
149
+ repo = Repository.find(44)
150
+ repo.async_create_archive('masterbrew')
151
+
152
+ The following JSON will be stored in the `file_serve` queue:
153
+
154
+ {
155
+ 'class': 'Archive',
156
+ 'args': [ 44, 'masterbrew' ]
157
+ }
158
+
159
+ Because of this your jobs must only accept arguments that can be JSON encoded.
160
+
161
+ So instead of doing this:
162
+
163
+ Resque.enqueue(Archive, self, branch)
164
+
165
+ do this:
166
+
167
+ Resque.enqueue(Archive, self.id, branch)
168
+
169
+ This is why our above example (and all the examples in `examples/`)
170
+ uses object IDs instead of passing around the objects.
171
+
172
+ While this is less convenient than just sticking a marshaled object
173
+ in the database, it gives you a slight advantage: your jobs will be
174
+ run against the most recent version of an object because they need to
175
+ pull from the DB or cache.
176
+
177
+ If your jobs were run against marshaled objects, they could
178
+ potentially be operating on a stale record with out-of-date information.
179
+
180
+
181
+ ### send_later / async
182
+
183
+ Want something like DelayedJob's `send_later` or the ability to use
184
+ instance methods instead of just methods for jobs? See the `examples/`
185
+ directory for goodies.
186
+
187
+ We plan to provide first class `async` support in a future release.
188
+
189
+
190
+ ### Failure
191
+
192
+ If a job raises an exception, it is logged and handed off to the
193
+ `Resque::Failure` module. Failures are logged either locally in Redis
194
+ or using some different backend.
195
+
196
+ For example, Resque ships with Hoptoad support.
197
+
198
+ Keep this in mind when writing your jobs: you may want to throw
199
+ exceptions you would not normally throw in order to assist debugging.
200
+
201
+
202
+ Workers
203
+ -------
204
+
205
+ Resque workers are rake tasks that run forever. They basically do this:
206
+
207
+ start
208
+ loop do
209
+ if job = reserve
210
+ job.process
211
+ else
212
+ sleep 5
213
+ end
214
+ end
215
+ shutdown
216
+
217
+ Starting a worker is simple. Here's our example from earlier:
218
+
219
+ $ QUEUE=file_serve rake resque:work
220
+
221
+ By default Resque won't know about your application's
222
+ environment. That is, it won't be able to find and run your jobs - it
223
+ needs to load your application into memory.
224
+
225
+ If we've installed Resque as a Rails plugin, we might run this command
226
+ from our RAILS_ROOT:
227
+
228
+ $ QUEUE=file_serve rake environment resque:work
229
+
230
+ This will load the environment before starting a worker. Alternately
231
+ we can define a `resque:setup` task with a dependency on the
232
+ `environment` rake task:
233
+
234
+ task "resque:setup" => :environment
235
+
236
+ GitHub's setup task looks like this:
237
+
238
+ task "resque:setup" => :environment do
239
+ Grit::Git.git_timeout = 10.minutes
240
+ end
241
+
242
+ We don't want the `git_timeout` as high as 10 minutes in our web app,
243
+ but in the Resque workers it's fine.
244
+
245
+
246
+ ### Logging
247
+
248
+ Workers support basic logging to STDOUT. If you start them with the
249
+ `VERBOSE` env variable set, they will print basic debugging
250
+ information. You can also set the `VVERBOSE` (very verbose) env
251
+ variable.
252
+
253
+ $ VVERBOSE=1 QUEUE=file_serve rake environment resque:work
254
+
255
+
256
+ ### Priorities and Queue Lists
257
+
258
+ Resque doesn't support numeric priorities but instead uses the order
259
+ of queues you give it. We call this list of queues the "queue list."
260
+
261
+ Let's say we add a `warm_cache` queue in addition to our `file_serve`
262
+ queue. We'd now start a worker like so:
263
+
264
+ $ QUEUES=file_serve,warm_cache rake resque:work
265
+
266
+ When the worker looks for new jobs, it will first check
267
+ `file_serve`. If it finds a job, it'll process it then check
268
+ `file_serve` again. It will keep checking `file_serve` until no more
269
+ jobs are available. At that point, it will check `warm_cache`. If it
270
+ finds a job it'll process it then check `file_serve` (repeating the
271
+ whole process).
272
+
273
+ In this way you can prioritize certain queues. At GitHub we start our
274
+ workers with something like this:
275
+
276
+ $ QUEUES=critical,archive,high,low rake resque:work
277
+
278
+ Notice the `archive` queue - it is specialized and in our future
279
+ architecture will only be run from a single machine.
280
+
281
+ At that point we'll start workers on our generalized background
282
+ machines with this command:
283
+
284
+ $ QUEUES=critical,high,low rake resque:work
285
+
286
+ And workers on our specialized archive machine with this command:
287
+
288
+ $ QUEUE=archive rake resque:work
289
+
290
+
291
+ ### Running All Queues
292
+
293
+ If you want your workers to work off of every queue, including new
294
+ queues created on the fly, you can use a splat:
295
+
296
+ $ QUEUE=* rake resque:work
297
+
298
+ Queues will be processed in alphabetical order.
299
+
300
+
301
+ ### Running Multiple Workers
302
+
303
+ At GitHub we use god to start and stop multiple workers. A sample god
304
+ configuration file is included under `examples/god`. We recommend this
305
+ method.
306
+
307
+ If you'd like to run multiple workers in development mode, you can do
308
+ so using the `resque:workers` rake task:
309
+
310
+ $ COUNT=5 QUEUE=* rake resque:workers
311
+
312
+ This will spawn five Resque workers, each in its own thread. Hitting
313
+ ctrl-c should be sufficient to stop them all.
314
+
315
+
316
+ ### Forking
317
+
318
+ On certain platforms, when a Resque worker reserves a job it
319
+ immediately forks a child process. The child processes the job then
320
+ exits. When the child has exited successfully, the worker reserves
321
+ another job and repeats the process.
322
+
323
+ Why?
324
+
325
+ Because Resque assumes chaos.
326
+
327
+ Resque assumes your background workers will lock up, run too long, or
328
+ have unwanted memory growth.
329
+
330
+ If Resque workers processed jobs themselves, it'd be hard to whip them
331
+ into shape. Let's say one is using too much memory: you send it a
332
+ signal that says "shutdown after you finish processing the current
333
+ job," and it does so. It then starts up again - loading your entire
334
+ application environment. This adds useless CPU cycles and causes a
335
+ delay in queue processing.
336
+
337
+ Plus, what if it's using too much memory and has stopped responding to
338
+ signals?
339
+
340
+ Thanks to Resque's parent / child architecture, jobs that use too much memory
341
+ release that memory upon completion. No unwanted growth.
342
+
343
+ And what if a job is running too long? You'd need to `kill -9` it then
344
+ start the worker again. With Resque's parent / child architecture you
345
+ can tell the parent to forcefully kill the child then immediately
346
+ start processing more jobs. No startup delay or wasted cycles.
347
+
348
+ The parent / child architecture helps us keep tabs on what workers are
349
+ doing, too. By eliminating the need to `kill -9` workers we can have
350
+ parents remove themselves from the global listing of workers. If we
351
+ just ruthlessly killed workers, we'd need a separate watchdog process
352
+ to add and remove them to the global listing - which becomes
353
+ complicated.
354
+
355
+ Workers instead handle their own state.
356
+
357
+
358
+ ### Parents and Children
359
+
360
+ Here's a parent / child pair doing some work:
361
+
362
+ $ ps -e -o pid,command | grep [r]esque
363
+ 92099 resque: Forked 92102 at 1253142769
364
+ 92102 resque: Processing file_serve since 1253142769
365
+
366
+ You can clearly see that process 92099 forked 92102, which has been
367
+ working since 1253142769.
368
+
369
+ (By advertising the time they began processing you can easily use monit
370
+ or god to kill stale workers.)
371
+
372
+ When a parent process is idle, it lets you know what queues it is
373
+ waiting for work on:
374
+
375
+ $ ps -e -o pid,command | grep [r]esque
376
+ 92099 resque: Waiting for file_serve,warm_cache
377
+
378
+
379
+ ### Signals
380
+
381
+ Resque workers respond to a few different signals:
382
+
383
+ * `QUIT` - Wait for child to finish processing then exit
384
+ * `TERM` / `INT` - Immediately kill child then exit
385
+ * `USR1` - Immediately kill child but don't exit
386
+
387
+ If you want to gracefully shutdown a Resque worker, use `QUIT`.
388
+
389
+ If you want to kill a stale or stuck child, use `USR1`. Processing
390
+ will continue as normal unless the child was not found. In that case
391
+ Resque assumes the parent process is in a bad state and shuts down.
392
+
393
+ If you want to kill a stale or stuck child and shutdown, use `TERM`
394
+
395
+ ### Mysql::Error: MySQL server has gone away
396
+
397
+ If your workers remain idle for too long they may lose their MySQL
398
+ connection. If that happens we recommend using [this
399
+ Gist](http://gist.github.com/238999).
400
+
401
+
402
+ The Front End
403
+ -------------
404
+
405
+ Resque comes with a Sinatra-based front end for seeing what's up with
406
+ your queue.
407
+
408
+ ![The Front End](http://img.skitch.com/20091104-tqh5pgkwgbskjbk7qbtmpesnyw.jpg)
409
+
410
+ ### Standalone
411
+
412
+ If you've installed Resque as a gem running the front end standalone is easy:
413
+
414
+ $ resque-web
415
+
416
+ It's a thin layer around `rackup` so it's configurable as well:
417
+
418
+ $ resque-web -p 8282
419
+
420
+ If you have a Resque config file you want evaluated just pass it to
421
+ the script as the final argument:
422
+
423
+ $ resque-web -p 8282 rails_root/config/initializers/resque.rb
424
+
425
+ ### Passenger
426
+
427
+ Using Passenger? Resque ships with a `config.ru` you can use. See
428
+ Phusion's guide:
429
+
430
+ <http://www.modrails.com/documentation/Users%20guide.html#_deploying_a_rack_based_ruby_application>
431
+
432
+ ### Rack::URLMap
433
+
434
+ If you want to load Resque on a subpath, possibly alongside other
435
+ apps, it's easy to do with Rack's `URLMap`:
436
+
437
+ require 'resque/server'
438
+
439
+ run Rack::URLMap.new \
440
+ "/" => Your::App.new,
441
+ "/resque" => Resque::Server.new
442
+
443
+ Check `examples/demo/config.ru` for a functional example (including
444
+ HTTP basic auth).
445
+
446
+
447
+ Resque vs DelayedJob
448
+ --------------------
449
+
450
+ How does Resque compare to DelayedJob, and why would you choose one
451
+ over the other?
452
+
453
+ * Resque supports multiple queues
454
+ * DelayedJob supports finer grained priorities
455
+ * Resque workers are resilient to memory leaks / bloat
456
+ * DelayedJob workers are extremely simple and easy to modify
457
+ * Resque requires Redis
458
+ * DelayedJob requires ActiveRecord
459
+ * Resque can only place JSONable Ruby objects on a queue as arguments
460
+ * DelayedJob can place _any_ Ruby object on its queue as arguments
461
+ * Resque includes a Sinatra app for monitoring what's going on
462
+ * DelayedJob can be queried from within your Rails app if you want to
463
+ add an interface
464
+
465
+ If you're doing Rails development, you already have a database and
466
+ ActiveRecord. DelayedJob is super easy to setup and works great.
467
+ GitHub used it for many months to process almost 200 million jobs.
468
+
469
+ Choose Resque if:
470
+
471
+ * You need multiple queues
472
+ * You don't care / dislike numeric priorities
473
+ * You don't need to persist every Ruby object ever
474
+ * You have potentially huge queues
475
+ * You want to see what's going on
476
+ * You expect a lot of failure / chaos
477
+ * You can setup Redis
478
+ * You're not running short on RAM
479
+
480
+ Choose DelayedJob if:
481
+
482
+ * You like numeric priorities
483
+ * You're not doing a gigantic amount of jobs each day
484
+ * Your queue stays small and nimble
485
+ * There is not a lot failure / chaos
486
+ * You want to easily throw anything on the queue
487
+ * You don't want to setup Redis
488
+
489
+ In no way is Resque a "better" DelayedJob, so make sure you pick the
490
+ tool that's best for your app.
491
+
492
+
493
+ Installing Redis
494
+ ----------------
495
+
496
+ Resque uses Redis' lists for its queues. It also stores worker state
497
+ data in Redis.
498
+
499
+ #### Homebrew
500
+
501
+ If you're on OS X, Homebrew is the simplest way to install Redis:
502
+
503
+ $ brew install redis
504
+ $ redis-server /usr/local/etc/redis.conf
505
+
506
+ You now have a Redis daemon running on 6379.
507
+
508
+ #### Via Resque
509
+
510
+ Resque includes Rake tasks (thanks to Ezra's redis-rb) that will
511
+ install and run Redis for you:
512
+
513
+ $ git clone git://github.com/defunkt/resque.git
514
+ $ cd resque
515
+ $ rake redis:install dtach:install
516
+ $ rake redis:start
517
+
518
+ Or, if you don't have admin access on your machine:
519
+
520
+ $ git clone git://github.com/defunkt/resque.git
521
+ $ cd resque
522
+ $ PREFIX=<your_prefix> rake redis:install dtach:install
523
+ $ rake redis:start
524
+
525
+ You now have Redis running on 6379. Wait a second then hit ctrl-\ to
526
+ detach and keep it running in the background.
527
+
528
+ The demo is probably the best way to figure out how to put the parts
529
+ together. But, it's not that hard.
530
+
531
+
532
+ Resque Dependencies
533
+ -------------------
534
+
535
+ gem install redis redis-namespace yajl-ruby
536
+
537
+ If you cannot install `yajl-ruby` (JRuby?), you can install the `json`
538
+ gem and Resque will use it instead.
539
+
540
+
541
+ Installing Resque
542
+ -----------------
543
+
544
+ ### In a Rack app, as a gem
545
+
546
+ First install the gem.
547
+
548
+ $ gem install resque
549
+
550
+ Next include it in your application.
551
+
552
+ require 'resque'
553
+
554
+ Now start your application:
555
+
556
+ rackup config.ru
557
+
558
+ That's it! You can now create Resque jobs from within your app.
559
+
560
+ To start a worker, create a Rakefile in your app's root (or add this
561
+ to an existing Rakefile):
562
+
563
+ require 'your/app'
564
+ require 'resque/tasks'
565
+
566
+ Now:
567
+
568
+ $ QUEUE=* rake resque:work
569
+
570
+ Alternately you can define a `resque:setup` hook in your Rakefile if you
571
+ don't want to load your app every time rake runs.
572
+
573
+
574
+ ### In a Rails app, as a gem
575
+
576
+ First install the gem.
577
+
578
+ $ gem install resque
579
+
580
+ Next include it in your application.
581
+
582
+ $ cat config/initializers/load_resque.rb
583
+ require 'resque'
584
+
585
+ Now start your application:
586
+
587
+ $ ./script/server
588
+
589
+ That's it! You can now create Resque jobs from within your app.
590
+
591
+ To start a worker, add this to your Rakefile in `RAILS_ROOT`:
592
+
593
+ require 'resque/tasks'
594
+
595
+ Now:
596
+
597
+ $ QUEUE=* rake environment resque:work
598
+
599
+ Don't forget you can define a `resque:setup` hook in
600
+ `lib/tasks/whatever.rake` that loads the `environment` task every time.
601
+
602
+
603
+ ### In a Rails app, as a plugin
604
+
605
+ $ ./script/plugin install git://github.com/defunkt/resque
606
+
607
+ That's it! Resque will automatically be available when your Rails app
608
+ loads.
609
+
610
+ To start a worker:
611
+
612
+ $ QUEUE=* rake environment resque:work
613
+
614
+ Don't forget you can define a `resque:setup` hook in
615
+ `lib/tasks/whatever.rake` that loads the `environment` task every time.
616
+
617
+
618
+ Configuration
619
+ -------------
620
+
621
+ You may want to change the Redis host and port Resque connects to, or
622
+ set various other options at startup.
623
+
624
+ Resque has a `redis` setter which can be given a string or a Redis
625
+ object. This means if you're already using Redis in your app, Resque
626
+ can re-use the existing connection.
627
+
628
+ String: `Resque.redis = 'localhost:6379'`
629
+
630
+ Redis: `Redus.redis = $redis`
631
+
632
+ For our rails app we have a `config/initializers/resque.rb` file where
633
+ we load `config/resque.yml` by hand and set the Redis information
634
+ appropriately.
635
+
636
+ Here's our `config/resque.yml`:
637
+
638
+ development: localhost:6379
639
+ test: localhost:6379
640
+ staging: redis1.se.github.com:6379
641
+ fi: localhost:6379
642
+ production: redis1.ae.github.com:6379
643
+
644
+ And our initializer:
645
+
646
+ rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..'
647
+ rails_env = ENV['RAILS_ENV'] || 'development'
648
+
649
+ resque_config = YAML.load_file(rails_root + '/config/resque.yml')
650
+ Resque.redis = resque_config[rails_env]
651
+
652
+ Easy peasy! Why not just use `RAILS_ROOT` and `RAILS_ENV`? Because
653
+ this way we can tell our Sinatra app about the config file:
654
+
655
+ $ RAILS_ENV=production resque-web rails_root/config/initializers/resque.rb
656
+
657
+ Now everyone is on the same page.
658
+
659
+
660
+ Demo
661
+ ----
662
+
663
+ Resque ships with a demo Sinatra app for creating jobs that are later
664
+ processed in the background.
665
+
666
+ Try it out by looking at the README, found at `examples/demo/README.markdown`.
667
+
668
+
669
+ Monitoring
670
+ ----------
671
+
672
+ If you're using god to monitor Resque, we have provided example
673
+ configs in `examples/god/`. One is for starting / stopping workers,
674
+ the other is for killing workers that have been running too long.
675
+
676
+
677
+ Development
678
+ -----------
679
+
680
+ Want to hack on Resque?
681
+
682
+ First clone the repo and run the tests:
683
+
684
+ git clone git://github.com/defunkt/resque.git
685
+ cd resque
686
+ rake test
687
+
688
+ If the tests do not pass make sure you have Redis installed
689
+ correctly (though we make an effort to tell you if we feel this is the
690
+ case). The tests attempt to start an isolated instance of Redis to
691
+ run against.
692
+
693
+ Also make sure you've installed all the dependencies correctly. For
694
+ example, try loading the `redis-namespace` gem after you've installed
695
+ it:
696
+
697
+ $ irb
698
+ >> require 'rubygems'
699
+ => true
700
+ >> require 'redis/namespace'
701
+ => true
702
+
703
+ If you get an error requiring any of the dependencies, you may have
704
+ failed to install them or be seeing load path issues.
705
+
706
+ Feel free to ping the mailing list with your problem and we'll try to
707
+ sort it out.
708
+
709
+
710
+ Contributing
711
+ ------------
712
+
713
+ Once you've made your great commits:
714
+
715
+ 1. [Fork][1] Resque
716
+ 2. Create a topic branch - `git checkout -b my_branch`
717
+ 3. Push to your branch - `git push origin my_branch`
718
+ 4. Create an [Issue][2] with a link to your branch
719
+ 5. That's it!
720
+
721
+
722
+ Mailing List
723
+ ------------
724
+
725
+ To join the list simply send an email to <resque@librelist.com>. This
726
+ will subscribe you and send you information about your subscription,
727
+ include unsubscribe information.
728
+
729
+ The archive can be found at <http://librelist.com/browser/>.
730
+
731
+
732
+ Meta
733
+ ----
734
+
735
+ * Code: `git clone git://github.com/defunkt/resque.git`
736
+ * Home: <http://github.com/defunkt/resque>
737
+ * Docs: <http://defunkt.github.com/resque/>
738
+ * Bugs: <http://github.com/defunkt/resque/issues>
739
+ * List: <resque@librelist.com>
740
+ * Chat: <irc://irc.freenode.net/resque>
741
+ * Gems: <http://gemcutter.org/gems/resque>
742
+
743
+ This project uses [Semantic Versioning][sv].
744
+
745
+
746
+ Author
747
+ ------
748
+
749
+ Chris Wanstrath :: chris@ozmm.org :: @defunkt
750
+
751
+ [0]: http://github.com/blog/542-introducing-resque
752
+ [1]: http://help.github.com/forking/
753
+ [2]: http://github.com/defunkt/resque/issues
754
+ [sv]: http://semver.org/
755
+ [resque]: http://github.com/defunct/resque