resqueue 1.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (60) hide show
  1. checksums.yaml +7 -0
  2. data/HISTORY.md +488 -0
  3. data/LICENSE +20 -0
  4. data/README.markdown +920 -0
  5. data/Rakefile +57 -0
  6. data/bin/resque +81 -0
  7. data/bin/resque-web +31 -0
  8. data/lib/resque.rb +578 -0
  9. data/lib/resque/data_store.rb +326 -0
  10. data/lib/resque/errors.rb +21 -0
  11. data/lib/resque/failure.rb +119 -0
  12. data/lib/resque/failure/airbrake.rb +33 -0
  13. data/lib/resque/failure/base.rb +73 -0
  14. data/lib/resque/failure/multiple.rb +68 -0
  15. data/lib/resque/failure/redis.rb +128 -0
  16. data/lib/resque/failure/redis_multi_queue.rb +104 -0
  17. data/lib/resque/helpers.rb +48 -0
  18. data/lib/resque/job.rb +296 -0
  19. data/lib/resque/log_formatters/quiet_formatter.rb +7 -0
  20. data/lib/resque/log_formatters/verbose_formatter.rb +7 -0
  21. data/lib/resque/log_formatters/very_verbose_formatter.rb +8 -0
  22. data/lib/resque/logging.rb +18 -0
  23. data/lib/resque/plugin.rb +78 -0
  24. data/lib/resque/server.rb +299 -0
  25. data/lib/resque/server/helpers.rb +64 -0
  26. data/lib/resque/server/public/favicon.ico +0 -0
  27. data/lib/resque/server/public/idle.png +0 -0
  28. data/lib/resque/server/public/jquery-1.12.4.min.js +5 -0
  29. data/lib/resque/server/public/jquery.relatize_date.js +95 -0
  30. data/lib/resque/server/public/poll.png +0 -0
  31. data/lib/resque/server/public/ranger.js +78 -0
  32. data/lib/resque/server/public/reset.css +44 -0
  33. data/lib/resque/server/public/style.css +91 -0
  34. data/lib/resque/server/public/working.png +0 -0
  35. data/lib/resque/server/test_helper.rb +19 -0
  36. data/lib/resque/server/views/error.erb +1 -0
  37. data/lib/resque/server/views/failed.erb +29 -0
  38. data/lib/resque/server/views/failed_job.erb +50 -0
  39. data/lib/resque/server/views/failed_queues_overview.erb +24 -0
  40. data/lib/resque/server/views/key_sets.erb +17 -0
  41. data/lib/resque/server/views/key_string.erb +11 -0
  42. data/lib/resque/server/views/layout.erb +44 -0
  43. data/lib/resque/server/views/next_more.erb +22 -0
  44. data/lib/resque/server/views/overview.erb +4 -0
  45. data/lib/resque/server/views/queues.erb +58 -0
  46. data/lib/resque/server/views/stats.erb +62 -0
  47. data/lib/resque/server/views/workers.erb +111 -0
  48. data/lib/resque/server/views/working.erb +72 -0
  49. data/lib/resque/stat.rb +58 -0
  50. data/lib/resque/tasks.rb +72 -0
  51. data/lib/resque/thread_signal.rb +45 -0
  52. data/lib/resque/vendor/utf8_util.rb +26 -0
  53. data/lib/resque/vendor/utf8_util/utf8_util_18.rb +91 -0
  54. data/lib/resque/vendor/utf8_util/utf8_util_19.rb +6 -0
  55. data/lib/resque/version.rb +3 -0
  56. data/lib/resque/worker.rb +892 -0
  57. data/lib/resqueue.rb +4 -0
  58. data/lib/tasks/redis.rake +161 -0
  59. data/lib/tasks/resque.rake +2 -0
  60. metadata +197 -0
data/LICENSE ADDED
@@ -0,0 +1,20 @@
1
+ Copyright (c) Chris Wanstrath
2
+
3
+ Permission is hereby granted, free of charge, to any person obtaining
4
+ a copy of this software and associated documentation files (the
5
+ "Software"), to deal in the Software without restriction, including
6
+ without limitation the rights to use, copy, modify, merge, publish,
7
+ distribute, sublicense, and/or sell copies of the Software, and to
8
+ permit persons to whom the Software is furnished to do so, subject to
9
+ the following conditions:
10
+
11
+ The above copyright notice and this permission notice shall be
12
+ included in all copies or substantial portions of the Software.
13
+
14
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
15
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
16
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
17
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
18
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
19
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
20
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@@ -0,0 +1,920 @@
1
+ Resque
2
+ ======
3
+
4
+ [![Gem Version](https://badge.fury.io/rb/resque.svg)](https://rubygems.org/gems/resque)
5
+ [![Build Status](https://travis-ci.org/resque/resque.svg)](https://travis-ci.org/resque/resque)
6
+ [![Coverage Status](https://coveralls.io/repos/github/resque/resque/badge.svg?branch=1-x-stable)](https://coveralls.io/r/resque/resque?branch=1-x-stable)
7
+
8
+ Resque (pronounced like "rescue") is a Redis-backed library for creating
9
+ background jobs, placing those jobs on multiple queues, and processing
10
+ them later.
11
+
12
+ Background jobs can be any Ruby class or module that responds to
13
+ `perform`. Your existing classes can easily be converted to background
14
+ jobs or you can create new classes specifically to do work. Or, you
15
+ can do both.
16
+
17
+ Resque is heavily inspired by DelayedJob (which rocks) and comprises
18
+ three parts:
19
+
20
+ 1. A Ruby library for creating, querying, and processing jobs
21
+ 2. A Rake task for starting a worker which processes jobs
22
+ 3. A Sinatra app for monitoring queues, jobs, and workers.
23
+
24
+ Resque workers can be distributed between multiple machines,
25
+ support priorities, are resilient to memory bloat / "leaks," are
26
+ optimized for REE (but work on MRI and JRuby), tell you what they're
27
+ doing, and expect failure.
28
+
29
+ Resque queues are persistent; support constant time, atomic push and
30
+ pop (thanks to Redis); provide visibility into their contents; and
31
+ store jobs as simple JSON packages.
32
+
33
+ The Resque frontend tells you what workers are doing, what workers are
34
+ not doing, what queues you're using, what's in those queues, provides
35
+ general usage stats, and helps you track failures.
36
+
37
+
38
+ The Blog Post
39
+ -------------
40
+
41
+ For the backstory, philosophy, and history of Resque's beginnings,
42
+ please see [the blog post][0].
43
+
44
+
45
+ Overview
46
+ --------
47
+
48
+ Resque allows you to create jobs and place them on a queue, then,
49
+ later, pull those jobs off the queue and process them.
50
+
51
+ Resque jobs are Ruby classes (or modules) which respond to the
52
+ `perform` method. Here's an example:
53
+
54
+
55
+ ``` ruby
56
+ class Archive
57
+ @queue = :file_serve
58
+
59
+ def self.perform(repo_id, branch = 'master')
60
+ repo = Repository.find(repo_id)
61
+ repo.create_archive(branch)
62
+ end
63
+ end
64
+ ```
65
+
66
+ The `@queue` class instance variable determines which queue `Archive`
67
+ jobs will be placed in. Queues are arbitrary and created on the fly -
68
+ you can name them whatever you want and have as many as you want.
69
+
70
+ To place an `Archive` job on the `file_serve` queue, we might add this
71
+ to our application's pre-existing `Repository` class:
72
+
73
+ ``` ruby
74
+ class Repository
75
+ def async_create_archive(branch)
76
+ Resque.enqueue(Archive, self.id, branch)
77
+ end
78
+ end
79
+ ```
80
+
81
+ Now when we call `repo.async_create_archive('masterbrew')` in our
82
+ application, a job will be created and placed on the `file_serve`
83
+ queue.
84
+
85
+ Later, a worker will run something like this code to process the job:
86
+
87
+ ``` ruby
88
+ klass, args = Resque.reserve(:file_serve)
89
+ klass.perform(*args) if klass.respond_to? :perform
90
+ ```
91
+
92
+ Which translates to:
93
+
94
+ ``` ruby
95
+ Archive.perform(44, 'masterbrew')
96
+ ```
97
+
98
+ Let's start a worker to run `file_serve` jobs:
99
+
100
+ $ cd app_root
101
+ $ QUEUE=file_serve rake resque:work
102
+
103
+ This starts one Resque worker and tells it to work off the
104
+ `file_serve` queue. As soon as it's ready it'll try to run the
105
+ `Resque.reserve` code snippet above and process jobs until it can't
106
+ find any more, at which point it will sleep for a small period and
107
+ repeatedly poll the queue for more jobs.
108
+
109
+ Workers can be given multiple queues (a "queue list") and run on
110
+ multiple machines. In fact they can be run anywhere with network
111
+ access to the Redis server.
112
+
113
+
114
+ Jobs
115
+ ----
116
+
117
+ What should you run in the background? Anything that takes any time at
118
+ all. Slow INSERT statements, disk manipulating, data processing, etc.
119
+
120
+ At GitHub we use Resque to process the following types of jobs:
121
+
122
+ * Warming caches
123
+ * Counting disk usage
124
+ * Building tarballs
125
+ * Building Rubygems
126
+ * Firing off web hooks
127
+ * Creating events in the db and pre-caching them
128
+ * Building graphs
129
+ * Deleting users
130
+ * Updating our search index
131
+
132
+ As of writing we have about 35 different types of background jobs.
133
+
134
+ Keep in mind that you don't need a web app to use Resque - we just
135
+ mention "foreground" and "background" because they make conceptual
136
+ sense. You could easily be spidering sites and sticking data which
137
+ needs to be crunched later into a queue.
138
+
139
+
140
+ ### Persistence
141
+
142
+ Jobs are persisted to queues as JSON objects. Let's take our `Archive`
143
+ example from above. We'll run the following code to create a job:
144
+
145
+ ``` ruby
146
+ repo = Repository.find(44)
147
+ repo.async_create_archive('masterbrew')
148
+ ```
149
+
150
+ The following JSON will be stored in the `file_serve` queue:
151
+
152
+ ``` javascript
153
+ {
154
+ 'class': 'Archive',
155
+ 'args': [ 44, 'masterbrew' ]
156
+ }
157
+ ```
158
+
159
+ Because of this your jobs must only accept arguments that can be JSON encoded.
160
+
161
+ So instead of doing this:
162
+
163
+ ``` ruby
164
+ Resque.enqueue(Archive, self, branch)
165
+ ```
166
+
167
+ do this:
168
+
169
+ ``` ruby
170
+ Resque.enqueue(Archive, self.id, branch)
171
+ ```
172
+
173
+ This is why our above example (and all the examples in `examples/`)
174
+ uses object IDs instead of passing around the objects.
175
+
176
+ While this is less convenient than just sticking a marshaled object
177
+ in the database, it gives you a slight advantage: your jobs will be
178
+ run against the most recent version of an object because they need to
179
+ pull from the DB or cache.
180
+
181
+ If your jobs were run against marshaled objects, they could
182
+ potentially be operating on a stale record with out-of-date information.
183
+
184
+
185
+ ### send_later / async
186
+
187
+ Want something like DelayedJob's `send_later` or the ability to use
188
+ instance methods instead of just methods for jobs? See the `examples/`
189
+ directory for goodies.
190
+
191
+ We plan to provide first class `async` support in a future release.
192
+
193
+
194
+ ### Failure
195
+
196
+ If a job raises an exception, it is logged and handed off to the
197
+ `Resque::Failure` module. Failures are logged either locally in Redis
198
+ or using some different backend.
199
+
200
+ For example, Resque ships with Airbrake support. To configure it, put
201
+ the following into an initialisation file or into your rake job:
202
+
203
+ ``` ruby
204
+ # send errors which occur in background jobs to redis and airbrake
205
+ require 'resque/failure/multiple'
206
+ require 'resque/failure/redis'
207
+ require 'resque/failure/airbrake'
208
+
209
+ Resque::Failure::Multiple.classes = [Resque::Failure::Redis, Resque::Failure::Airbrake]
210
+ Resque::Failure.backend = Resque::Failure::Multiple
211
+ ```
212
+
213
+ Keep this in mind when writing your jobs: you may want to throw
214
+ exceptions you would not normally throw in order to assist debugging.
215
+
216
+
217
+ Workers
218
+ -------
219
+
220
+ Resque workers are rake tasks that run forever. They basically do this:
221
+
222
+ ``` ruby
223
+ start
224
+ loop do
225
+ if job = reserve
226
+ job.process
227
+ else
228
+ sleep 5 # Polling frequency = 5
229
+ end
230
+ end
231
+ shutdown
232
+ ```
233
+
234
+ Starting a worker is simple. Here's our example from earlier:
235
+
236
+ $ QUEUE=file_serve rake resque:work
237
+
238
+ By default Resque won't know about your application's
239
+ environment. That is, it won't be able to find and run your jobs - it
240
+ needs to load your application into memory.
241
+
242
+ If we've installed Resque as a Rails plugin, we might run this command
243
+ from our RAILS_ROOT:
244
+
245
+ $ QUEUE=file_serve rake environment resque:work
246
+
247
+ This will load the environment before starting a worker. Alternately
248
+ we can define a `resque:setup` task with a dependency on the
249
+ `environment` rake task:
250
+
251
+ ``` ruby
252
+ task "resque:setup" => :environment
253
+ ```
254
+
255
+ GitHub's setup task looks like this:
256
+
257
+ ``` ruby
258
+ task "resque:setup" => :environment do
259
+ Grit::Git.git_timeout = 10.minutes
260
+ end
261
+ ```
262
+
263
+ We don't want the `git_timeout` as high as 10 minutes in our web app,
264
+ but in the Resque workers it's fine.
265
+
266
+
267
+ ### Logging
268
+
269
+ Workers support basic logging to STDOUT. If you start them with the
270
+ `VERBOSE` env variable set, they will print basic debugging
271
+ information. You can also set the `VVERBOSE` (very verbose) env
272
+ variable.
273
+
274
+ $ VVERBOSE=1 QUEUE=file_serve rake environment resque:work
275
+
276
+ If you want Resque to log to a file, in Rails do:
277
+
278
+ ```ruby
279
+ # config/initializers/resque.rb
280
+ Resque.logger = Logger.new(Rails.root.join('log', "#{Rails.env}_resque.log"))
281
+ ```
282
+
283
+ ### Process IDs (PIDs)
284
+
285
+ There are scenarios where it's helpful to record the PID of a resque
286
+ worker process. Use the PIDFILE option for easy access to the PID:
287
+
288
+ $ PIDFILE=./resque.pid QUEUE=file_serve rake environment resque:work
289
+
290
+ ### Running in the background
291
+
292
+ (Only supported with ruby >= 1.9). There are scenarios where it's helpful for
293
+ the resque worker to run itself in the background (usually in combination with
294
+ PIDFILE). Use the BACKGROUND option so that rake will return as soon as the
295
+ worker is started.
296
+
297
+ $ PIDFILE=./resque.pid BACKGROUND=yes QUEUE=file_serve \
298
+ rake environment resque:work
299
+
300
+ ### Polling frequency
301
+
302
+ You can pass an INTERVAL option which is a float representing the polling frequency.
303
+ The default is 5 seconds, but for a semi-active app you may want to use a smaller value.
304
+
305
+ $ INTERVAL=0.1 QUEUE=file_serve rake environment resque:work
306
+
307
+ ### Priorities and Queue Lists
308
+
309
+ Resque doesn't support numeric priorities but instead uses the order
310
+ of queues you give it. We call this list of queues the "queue list."
311
+
312
+ Let's say we add a `warm_cache` queue in addition to our `file_serve`
313
+ queue. We'd now start a worker like so:
314
+
315
+ $ QUEUES=file_serve,warm_cache rake resque:work
316
+
317
+ When the worker looks for new jobs, it will first check
318
+ `file_serve`. If it finds a job, it'll process it then check
319
+ `file_serve` again. It will keep checking `file_serve` until no more
320
+ jobs are available. At that point, it will check `warm_cache`. If it
321
+ finds a job it'll process it then check `file_serve` (repeating the
322
+ whole process).
323
+
324
+ In this way you can prioritize certain queues. At GitHub we start our
325
+ workers with something like this:
326
+
327
+ $ QUEUES=critical,archive,high,low rake resque:work
328
+
329
+ Notice the `archive` queue - it is specialized and in our future
330
+ architecture will only be run from a single machine.
331
+
332
+ At that point we'll start workers on our generalized background
333
+ machines with this command:
334
+
335
+ $ QUEUES=critical,high,low rake resque:work
336
+
337
+ And workers on our specialized archive machine with this command:
338
+
339
+ $ QUEUE=archive rake resque:work
340
+
341
+
342
+ ### Running All Queues
343
+
344
+ If you want your workers to work off of every queue, including new
345
+ queues created on the fly, you can use a splat:
346
+
347
+ $ QUEUE=* rake resque:work
348
+
349
+ Queues will be processed in alphabetical order.
350
+
351
+
352
+ ### Running Multiple Workers
353
+
354
+ At GitHub we use god to start and stop multiple workers. A sample god
355
+ configuration file is included under `examples/god`. We recommend this
356
+ method.
357
+
358
+ If you'd like to run multiple workers in development mode, you can do
359
+ so using the `resque:workers` rake task:
360
+
361
+ $ COUNT=5 QUEUE=* rake resque:workers
362
+
363
+ This will spawn five Resque workers, each in its own process. Hitting
364
+ ctrl-c should be sufficient to stop them all.
365
+
366
+
367
+ ### Forking
368
+
369
+ On certain platforms, when a Resque worker reserves a job it
370
+ immediately forks a child process. The child processes the job then
371
+ exits. When the child has exited successfully, the worker reserves
372
+ another job and repeats the process.
373
+
374
+ Why?
375
+
376
+ Because Resque assumes chaos.
377
+
378
+ Resque assumes your background workers will lock up, run too long, or
379
+ have unwanted memory growth.
380
+
381
+ If Resque workers processed jobs themselves, it'd be hard to whip them
382
+ into shape. Let's say one is using too much memory: you send it a
383
+ signal that says "shutdown after you finish processing the current
384
+ job," and it does so. It then starts up again - loading your entire
385
+ application environment. This adds useless CPU cycles and causes a
386
+ delay in queue processing.
387
+
388
+ Plus, what if it's using too much memory and has stopped responding to
389
+ signals?
390
+
391
+ Thanks to Resque's parent / child architecture, jobs that use too much memory
392
+ release that memory upon completion. No unwanted growth.
393
+
394
+ And what if a job is running too long? You'd need to `kill -9` it then
395
+ start the worker again. With Resque's parent / child architecture you
396
+ can tell the parent to forcefully kill the child then immediately
397
+ start processing more jobs. No startup delay or wasted cycles.
398
+
399
+ The parent / child architecture helps us keep tabs on what workers are
400
+ doing, too. By eliminating the need to `kill -9` workers we can have
401
+ parents remove themselves from the global listing of workers. If we
402
+ just ruthlessly killed workers, we'd need a separate watchdog process
403
+ to add and remove them to the global listing - which becomes
404
+ complicated.
405
+
406
+ Workers instead handle their own state.
407
+
408
+
409
+ ### Parents and Children
410
+
411
+ Here's a parent / child pair doing some work:
412
+
413
+ $ ps -e -o pid,command | grep [r]esque
414
+ 92099 resque: Forked 92102 at 1253142769
415
+ 92102 resque: Processing file_serve since 1253142769
416
+
417
+ You can clearly see that process 92099 forked 92102, which has been
418
+ working since 1253142769.
419
+
420
+ (By advertising the time they began processing you can easily use monit
421
+ or god to kill stale workers.)
422
+
423
+ When a parent process is idle, it lets you know what queues it is
424
+ waiting for work on:
425
+
426
+ $ ps -e -o pid,command | grep [r]esque
427
+ 92099 resque: Waiting for file_serve,warm_cache
428
+
429
+
430
+ ### Signals
431
+
432
+ Resque workers respond to a few different signals:
433
+
434
+ * `QUIT` - Wait for child to finish processing then exit
435
+ * `TERM` / `INT` - Immediately kill child then exit
436
+ * `USR1` - Immediately kill child but don't exit
437
+ * `USR2` - Don't start to process any new jobs
438
+ * `CONT` - Start to process new jobs again after a USR2
439
+
440
+ If you want to gracefully shutdown a Resque worker, use `QUIT`.
441
+
442
+ If you want to kill a stale or stuck child, use `USR1`. Processing
443
+ will continue as normal unless the child was not found. In that case
444
+ Resque assumes the parent process is in a bad state and shuts down.
445
+
446
+ If you want to kill a stale or stuck child and shutdown, use `TERM`
447
+
448
+ If you want to stop processing jobs, but want to leave the worker running
449
+ (for example, to temporarily alleviate load), use `USR2` to stop processing,
450
+ then `CONT` to start it again.
451
+
452
+ ### Mysql::Error: MySQL server has gone away
453
+
454
+ If your workers remain idle for too long they may lose their MySQL connection. Depending on your version of Rails, we recommend the following:
455
+
456
+ #### Rails 3.x
457
+ In your `perform` method, add the following line:
458
+
459
+ ``` ruby
460
+ class MyTask
461
+ def self.perform
462
+ ActiveRecord::Base.verify_active_connections!
463
+ # rest of your code
464
+ end
465
+ end
466
+ ```
467
+
468
+ The Rails doc says the following about `verify_active_connections!`:
469
+
470
+ Verify active connections and remove and disconnect connections associated with stale threads.
471
+
472
+ #### Rails 4.x
473
+
474
+ In your `perform` method, instead of `verify_active_connections!`, use:
475
+
476
+ ``` ruby
477
+ class MyTask
478
+ def self.perform
479
+ ActiveRecord::Base.clear_active_connections!
480
+ # rest of your code
481
+ end
482
+ end
483
+ ```
484
+
485
+ From the Rails docs on [`clear_active_connections!`](http://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/ConnectionHandler.html#method-i-clear_active_connections-21):
486
+
487
+ Returns any connections in use by the current thread back to the pool, and also returns connections to the pool cached by threads that are no longer alive.
488
+
489
+
490
+
491
+ The Front End
492
+ -------------
493
+
494
+ Resque comes with a Sinatra-based front end for seeing what's up with
495
+ your queue.
496
+
497
+ ![The Front End](https://camo.githubusercontent.com/64d150a243987ffbc33f588bd6d7722a0bb8d69a/687474703a2f2f7475746f7269616c732e6a756d7073746172746c61622e636f6d2f696d616765732f7265737175655f6f766572766965772e706e67)
498
+
499
+ ### Standalone
500
+
501
+ If you've installed Resque as a gem running the front end standalone is easy:
502
+
503
+ $ resque-web
504
+
505
+ It's a thin layer around `rackup` so it's configurable as well:
506
+
507
+ $ resque-web -p 8282
508
+
509
+ If you have a Resque config file you want evaluated just pass it to
510
+ the script as the final argument:
511
+
512
+ $ resque-web -p 8282 rails_root/config/initializers/resque.rb
513
+
514
+ You can also set the namespace directly using `resque-web`:
515
+
516
+ $ resque-web -p 8282 -N myapp
517
+
518
+ or set the Redis connection string if you need to do something like select a different database:
519
+
520
+ $ resque-web -p 8282 -r localhost:6379:2
521
+
522
+ ### Passenger
523
+
524
+ Using Passenger? Resque ships with a `config.ru` you can use. See
525
+ Phusion's guide:
526
+
527
+ Apache: <https://www.phusionpassenger.com/library/deploy/apache/deploy/ruby/>
528
+ Nginx: <https://www.phusionpassenger.com/library/deploy/nginx/deploy/ruby/>
529
+
530
+ ### Rack::URLMap
531
+
532
+ If you want to load Resque on a subpath, possibly alongside other
533
+ apps, it's easy to do with Rack's `URLMap`:
534
+
535
+ ``` ruby
536
+ require 'resque/server'
537
+
538
+ run Rack::URLMap.new \
539
+ "/" => Your::App.new,
540
+ "/resque" => Resque::Server.new
541
+ ```
542
+
543
+ Check `examples/demo/config.ru` for a functional example (including
544
+ HTTP basic auth).
545
+
546
+ ### Rails 3
547
+
548
+ You can also mount Resque on a subpath in your existing Rails 3 app by adding `require 'resque/server'` to the top of your routes file or in an initializer then adding this to `routes.rb`:
549
+
550
+ ``` ruby
551
+ mount Resque::Server.new, :at => "/resque"
552
+ ```
553
+
554
+
555
+ Resque vs DelayedJob
556
+ --------------------
557
+
558
+ How does Resque compare to DelayedJob, and why would you choose one
559
+ over the other?
560
+
561
+ * Resque supports multiple queues
562
+ * DelayedJob supports finer grained priorities
563
+ * Resque workers are resilient to memory leaks / bloat
564
+ * DelayedJob workers are extremely simple and easy to modify
565
+ * Resque requires Redis
566
+ * DelayedJob requires ActiveRecord
567
+ * Resque can only place JSONable Ruby objects on a queue as arguments
568
+ * DelayedJob can place _any_ Ruby object on its queue as arguments
569
+ * Resque includes a Sinatra app for monitoring what's going on
570
+ * DelayedJob can be queried from within your Rails app if you want to
571
+ add an interface
572
+
573
+ If you're doing Rails development, you already have a database and
574
+ ActiveRecord. DelayedJob is super easy to setup and works great.
575
+ GitHub used it for many months to process almost 200 million jobs.
576
+
577
+ Choose Resque if:
578
+
579
+ * You need multiple queues
580
+ * You don't care / dislike numeric priorities
581
+ * You don't need to persist every Ruby object ever
582
+ * You have potentially huge queues
583
+ * You want to see what's going on
584
+ * You expect a lot of failure / chaos
585
+ * You can setup Redis
586
+ * You're not running short on RAM
587
+
588
+ Choose DelayedJob if:
589
+
590
+ * You like numeric priorities
591
+ * You're not doing a gigantic amount of jobs each day
592
+ * Your queue stays small and nimble
593
+ * There is not a lot failure / chaos
594
+ * You want to easily throw anything on the queue
595
+ * You don't want to setup Redis
596
+
597
+ In no way is Resque a "better" DelayedJob, so make sure you pick the
598
+ tool that's best for your app.
599
+
600
+ Resque Dependencies
601
+ -------------------
602
+
603
+ $ gem install bundler
604
+ $ bundle install
605
+
606
+
607
+ Installing Resque
608
+ -----------------
609
+
610
+ ### In a Rack app, as a gem
611
+
612
+ First install the gem.
613
+
614
+ $ gem install resque
615
+
616
+ Next include it in your application.
617
+
618
+ ``` ruby
619
+ require 'resque'
620
+ ```
621
+
622
+ Now start your application:
623
+
624
+ rackup config.ru
625
+
626
+ That's it! You can now create Resque jobs from within your app.
627
+
628
+ To start a worker, create a Rakefile in your app's root (or add this
629
+ to an existing Rakefile):
630
+
631
+ ``` ruby
632
+ require 'your/app'
633
+ require 'resque/tasks'
634
+ ```
635
+
636
+ Now:
637
+
638
+ $ QUEUE=* rake resque:work
639
+
640
+ Alternately you can define a `resque:setup` hook in your Rakefile if you
641
+ don't want to load your app every time rake runs.
642
+
643
+
644
+ ### In a Rails 2.x app, as a gem
645
+
646
+ First install the gem.
647
+
648
+ $ gem install resque
649
+
650
+ Next include it in your application.
651
+
652
+ $ cat config/initializers/load_resque.rb
653
+ require 'resque'
654
+
655
+ Now start your application:
656
+
657
+ $ ./script/server
658
+
659
+ That's it! You can now create Resque jobs from within your app.
660
+
661
+ To start a worker, add this to your Rakefile in `RAILS_ROOT`:
662
+
663
+ ``` ruby
664
+ require 'resque/tasks'
665
+ ```
666
+
667
+ Now:
668
+
669
+ $ QUEUE=* rake environment resque:work
670
+
671
+ Don't forget you can define a `resque:setup` hook in
672
+ `lib/tasks/whatever.rake` that loads the `environment` task every time.
673
+
674
+
675
+ ### In a Rails 2.x app, as a plugin
676
+
677
+ $ ./script/plugin install git://github.com/resque/resque
678
+
679
+ That's it! Resque will automatically be available when your Rails app
680
+ loads.
681
+
682
+ To start a worker:
683
+
684
+ $ QUEUE=* rake environment resque:work
685
+
686
+ Don't forget you can define a `resque:setup` hook in
687
+ `lib/tasks/whatever.rake` that loads the `environment` task every time.
688
+
689
+
690
+ ### In a Rails 3.x or 4.x app, as a gem
691
+
692
+ First include it in your Gemfile.
693
+
694
+ $ cat Gemfile
695
+ ...
696
+ gem 'resque'
697
+ ...
698
+
699
+ Next install it with Bundler.
700
+
701
+ $ bundle install
702
+
703
+ Now start your application:
704
+
705
+ $ rails server
706
+
707
+ That's it! You can now create Resque jobs from within your app.
708
+
709
+ To start a worker, add this to a file in `lib/tasks` (ex:
710
+ `lib/tasks/resque.rake`):
711
+
712
+ ``` ruby
713
+ require 'resque/tasks'
714
+ ```
715
+
716
+ Now:
717
+
718
+ $ QUEUE=* rake environment resque:work
719
+
720
+ Don't forget you can define a `resque:setup` hook in
721
+ `lib/tasks/whatever.rake` that loads the `environment` task every time.
722
+
723
+
724
+ Configuration
725
+ -------------
726
+
727
+ You may want to change the Redis host and port Resque connects to, or
728
+ set various other options at startup.
729
+
730
+ Resque has a `redis` setter which can be given a string or a Redis
731
+ object. This means if you're already using Redis in your app, Resque
732
+ can re-use the existing connection.
733
+
734
+ String: `Resque.redis = 'localhost:6379'`
735
+
736
+ Redis: `Resque.redis = $redis`
737
+
738
+ For our rails app we have a `config/initializers/resque.rb` file where
739
+ we load `config/resque.yml` by hand and set the Redis information
740
+ appropriately.
741
+
742
+ Here's our `config/resque.yml`:
743
+
744
+ development: localhost:6379
745
+ test: localhost:6379
746
+ staging: redis1.se.github.com:6379
747
+ fi: localhost:6379
748
+ production: redis1.ae.github.com:6379
749
+
750
+ And our initializer:
751
+
752
+ ``` ruby
753
+ rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..'
754
+ rails_env = ENV['RAILS_ENV'] || 'development'
755
+
756
+ resque_config = YAML.load_file(rails_root + '/config/resque.yml')
757
+ Resque.redis = resque_config[rails_env]
758
+ ```
759
+
760
+ Easy peasy! Why not just use `RAILS_ROOT` and `RAILS_ENV`? Because
761
+ this way we can tell our Sinatra app about the config file:
762
+
763
+ $ RAILS_ENV=production resque-web rails_root/config/initializers/resque.rb
764
+
765
+ Now everyone is on the same page.
766
+
767
+ Also, you could disable jobs queueing by setting 'inline' attribute.
768
+ For example, if you want to run all jobs in the same process for cucumber, try:
769
+
770
+ ``` ruby
771
+ Resque.inline = ENV['RAILS_ENV'] == "cucumber"
772
+ ```
773
+
774
+
775
+ Plugins and Hooks
776
+ -----------------
777
+
778
+ For a list of available plugins see
779
+ <http://wiki.github.com/resque/resque/plugins>.
780
+
781
+ If you'd like to write your own plugin, or want to customize Resque
782
+ using hooks (such as `Resque.after_fork`), see
783
+ [docs/HOOKS.md](http://github.com/resque/resque/blob/master/docs/HOOKS.md).
784
+
785
+
786
+ Namespaces
787
+ ----------
788
+
789
+ If you're running multiple, separate instances of Resque you may want
790
+ to namespace the keyspaces so they do not overlap. This is not unlike
791
+ the approach taken by many memcached clients.
792
+
793
+ This feature is provided by the [redis-namespace][rs] library, which
794
+ Resque uses by default to separate the keys it manages from other keys
795
+ in your Redis server.
796
+
797
+ Simply use the `Resque.redis.namespace` accessor:
798
+
799
+ ``` ruby
800
+ Resque.redis.namespace = "resque:GitHub"
801
+ ```
802
+
803
+ We recommend sticking this in your initializer somewhere after Redis
804
+ is configured.
805
+
806
+
807
+ Demo
808
+ ----
809
+
810
+ Resque ships with a demo Sinatra app for creating jobs that are later
811
+ processed in the background.
812
+
813
+ Try it out by looking at the README, found at `examples/demo/README.markdown`.
814
+
815
+
816
+ Monitoring
817
+ ----------
818
+
819
+ ### god
820
+
821
+ If you're using god to monitor Resque, we have provided example
822
+ configs in `examples/god/`. One is for starting / stopping workers,
823
+ the other is for killing workers that have been running too long.
824
+
825
+ ### monit
826
+
827
+ If you're using monit, `examples/monit/resque.monit` is provided free
828
+ of charge. This is **not** used by GitHub in production, so please
829
+ send patches for any tweaks or improvements you can make to it.
830
+
831
+
832
+ Questions
833
+ ---------
834
+
835
+ Please add them to the [FAQ](https://github.com/resque/resque/wiki/FAQ) or
836
+ ask on the Mailing List. The Mailing List is explained further below
837
+
838
+
839
+ Development
840
+ -----------
841
+
842
+ Want to hack on Resque?
843
+
844
+ First clone the repo and run the tests:
845
+
846
+ git clone git://github.com/resque/resque.git
847
+ cd resque
848
+ rake test
849
+
850
+ If the tests do not pass make sure you have Redis installed
851
+ correctly (though we make an effort to tell you if we feel this is the
852
+ case). The tests attempt to start an isolated instance of Redis to
853
+ run against.
854
+
855
+ Also make sure you've installed all the dependencies correctly. For
856
+ example, try loading the `redis-namespace` gem after you've installed
857
+ it:
858
+
859
+ $ irb
860
+ >> require 'rubygems'
861
+ => true
862
+ >> require 'redis/namespace'
863
+ => true
864
+
865
+ If you get an error requiring any of the dependencies, you may have
866
+ failed to install them or be seeing load path issues.
867
+
868
+ Feel free to ping the mailing list with your problem and we'll try to
869
+ sort it out.
870
+
871
+
872
+ Contributing
873
+ ------------
874
+
875
+ Read [CONTRIBUTING.md](CONTRIBUTING.md) first.
876
+
877
+ Once you've made your great commits:
878
+
879
+ 1. [Fork][1] Resque
880
+ 2. Create a topic branch - `git checkout -b my_branch`
881
+ 3. Push to your branch - `git push origin my_branch`
882
+ 4. Create a [Pull Request](http://help.github.com/pull-requests/) from your branch
883
+ 5. That's it!
884
+
885
+
886
+ Mailing List
887
+ ------------
888
+
889
+ To join the list simply send an email to <resque@librelist.com>. This
890
+ will subscribe you and send you information about your subscription,
891
+ including unsubscribe information.
892
+
893
+ The archive can be found at <http://librelist.com/browser/resque/>.
894
+
895
+
896
+ Meta
897
+ ----
898
+
899
+ * Code: `git clone git://github.com/resque/resque.git`
900
+ * Home: <http://github.com/resque/resque>
901
+ * Docs: <http://rubydoc.info/gems/resque>
902
+ * Bugs: <http://github.com/resque/resque/issues>
903
+ * List: <resque@librelist.com>
904
+ * Chat: <irc://irc.freenode.net/resque>
905
+ * Gems: <http://gemcutter.org/gems/resque>
906
+
907
+ This project uses [Semantic Versioning][sv].
908
+
909
+
910
+ Author
911
+ ------
912
+
913
+ Chris Wanstrath :: chris@ozmm.org :: @defunkt
914
+
915
+ [0]: http://github.com/blog/542-introducing-resque
916
+ [1]: http://help.github.com/forking/
917
+ [2]: http://github.com/resque/resque/issues
918
+ [sv]: http://semver.org/
919
+ [rs]: http://github.com/resque/redis-namespace
920
+ [cb]: http://wiki.github.com/resque/resque/contributing