resqueue 1.0.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (60) hide show
  1. checksums.yaml +7 -0
  2. data/HISTORY.md +488 -0
  3. data/LICENSE +20 -0
  4. data/README.markdown +920 -0
  5. data/Rakefile +57 -0
  6. data/bin/resque +81 -0
  7. data/bin/resque-web +31 -0
  8. data/lib/resque.rb +578 -0
  9. data/lib/resque/data_store.rb +326 -0
  10. data/lib/resque/errors.rb +21 -0
  11. data/lib/resque/failure.rb +119 -0
  12. data/lib/resque/failure/airbrake.rb +33 -0
  13. data/lib/resque/failure/base.rb +73 -0
  14. data/lib/resque/failure/multiple.rb +68 -0
  15. data/lib/resque/failure/redis.rb +128 -0
  16. data/lib/resque/failure/redis_multi_queue.rb +104 -0
  17. data/lib/resque/helpers.rb +48 -0
  18. data/lib/resque/job.rb +296 -0
  19. data/lib/resque/log_formatters/quiet_formatter.rb +7 -0
  20. data/lib/resque/log_formatters/verbose_formatter.rb +7 -0
  21. data/lib/resque/log_formatters/very_verbose_formatter.rb +8 -0
  22. data/lib/resque/logging.rb +18 -0
  23. data/lib/resque/plugin.rb +78 -0
  24. data/lib/resque/server.rb +299 -0
  25. data/lib/resque/server/helpers.rb +64 -0
  26. data/lib/resque/server/public/favicon.ico +0 -0
  27. data/lib/resque/server/public/idle.png +0 -0
  28. data/lib/resque/server/public/jquery-1.12.4.min.js +5 -0
  29. data/lib/resque/server/public/jquery.relatize_date.js +95 -0
  30. data/lib/resque/server/public/poll.png +0 -0
  31. data/lib/resque/server/public/ranger.js +78 -0
  32. data/lib/resque/server/public/reset.css +44 -0
  33. data/lib/resque/server/public/style.css +91 -0
  34. data/lib/resque/server/public/working.png +0 -0
  35. data/lib/resque/server/test_helper.rb +19 -0
  36. data/lib/resque/server/views/error.erb +1 -0
  37. data/lib/resque/server/views/failed.erb +29 -0
  38. data/lib/resque/server/views/failed_job.erb +50 -0
  39. data/lib/resque/server/views/failed_queues_overview.erb +24 -0
  40. data/lib/resque/server/views/key_sets.erb +17 -0
  41. data/lib/resque/server/views/key_string.erb +11 -0
  42. data/lib/resque/server/views/layout.erb +44 -0
  43. data/lib/resque/server/views/next_more.erb +22 -0
  44. data/lib/resque/server/views/overview.erb +4 -0
  45. data/lib/resque/server/views/queues.erb +58 -0
  46. data/lib/resque/server/views/stats.erb +62 -0
  47. data/lib/resque/server/views/workers.erb +111 -0
  48. data/lib/resque/server/views/working.erb +72 -0
  49. data/lib/resque/stat.rb +58 -0
  50. data/lib/resque/tasks.rb +72 -0
  51. data/lib/resque/thread_signal.rb +45 -0
  52. data/lib/resque/vendor/utf8_util.rb +26 -0
  53. data/lib/resque/vendor/utf8_util/utf8_util_18.rb +91 -0
  54. data/lib/resque/vendor/utf8_util/utf8_util_19.rb +6 -0
  55. data/lib/resque/version.rb +3 -0
  56. data/lib/resque/worker.rb +892 -0
  57. data/lib/resqueue.rb +4 -0
  58. data/lib/tasks/redis.rake +161 -0
  59. data/lib/tasks/resque.rake +2 -0
  60. metadata +197 -0
data/LICENSE ADDED
@@ -0,0 +1,20 @@
1
+ Copyright (c) Chris Wanstrath
2
+
3
+ Permission is hereby granted, free of charge, to any person obtaining
4
+ a copy of this software and associated documentation files (the
5
+ "Software"), to deal in the Software without restriction, including
6
+ without limitation the rights to use, copy, modify, merge, publish,
7
+ distribute, sublicense, and/or sell copies of the Software, and to
8
+ permit persons to whom the Software is furnished to do so, subject to
9
+ the following conditions:
10
+
11
+ The above copyright notice and this permission notice shall be
12
+ included in all copies or substantial portions of the Software.
13
+
14
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
15
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
16
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
17
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
18
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
19
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
20
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@@ -0,0 +1,920 @@
1
+ Resque
2
+ ======
3
+
4
+ [![Gem Version](https://badge.fury.io/rb/resque.svg)](https://rubygems.org/gems/resque)
5
+ [![Build Status](https://travis-ci.org/resque/resque.svg)](https://travis-ci.org/resque/resque)
6
+ [![Coverage Status](https://coveralls.io/repos/github/resque/resque/badge.svg?branch=1-x-stable)](https://coveralls.io/r/resque/resque?branch=1-x-stable)
7
+
8
+ Resque (pronounced like "rescue") is a Redis-backed library for creating
9
+ background jobs, placing those jobs on multiple queues, and processing
10
+ them later.
11
+
12
+ Background jobs can be any Ruby class or module that responds to
13
+ `perform`. Your existing classes can easily be converted to background
14
+ jobs or you can create new classes specifically to do work. Or, you
15
+ can do both.
16
+
17
+ Resque is heavily inspired by DelayedJob (which rocks) and comprises
18
+ three parts:
19
+
20
+ 1. A Ruby library for creating, querying, and processing jobs
21
+ 2. A Rake task for starting a worker which processes jobs
22
+ 3. A Sinatra app for monitoring queues, jobs, and workers.
23
+
24
+ Resque workers can be distributed between multiple machines,
25
+ support priorities, are resilient to memory bloat / "leaks," are
26
+ optimized for REE (but work on MRI and JRuby), tell you what they're
27
+ doing, and expect failure.
28
+
29
+ Resque queues are persistent; support constant time, atomic push and
30
+ pop (thanks to Redis); provide visibility into their contents; and
31
+ store jobs as simple JSON packages.
32
+
33
+ The Resque frontend tells you what workers are doing, what workers are
34
+ not doing, what queues you're using, what's in those queues, provides
35
+ general usage stats, and helps you track failures.
36
+
37
+
38
+ The Blog Post
39
+ -------------
40
+
41
+ For the backstory, philosophy, and history of Resque's beginnings,
42
+ please see [the blog post][0].
43
+
44
+
45
+ Overview
46
+ --------
47
+
48
+ Resque allows you to create jobs and place them on a queue, then,
49
+ later, pull those jobs off the queue and process them.
50
+
51
+ Resque jobs are Ruby classes (or modules) which respond to the
52
+ `perform` method. Here's an example:
53
+
54
+
55
+ ``` ruby
56
+ class Archive
57
+ @queue = :file_serve
58
+
59
+ def self.perform(repo_id, branch = 'master')
60
+ repo = Repository.find(repo_id)
61
+ repo.create_archive(branch)
62
+ end
63
+ end
64
+ ```
65
+
66
+ The `@queue` class instance variable determines which queue `Archive`
67
+ jobs will be placed in. Queues are arbitrary and created on the fly -
68
+ you can name them whatever you want and have as many as you want.
69
+
70
+ To place an `Archive` job on the `file_serve` queue, we might add this
71
+ to our application's pre-existing `Repository` class:
72
+
73
+ ``` ruby
74
+ class Repository
75
+ def async_create_archive(branch)
76
+ Resque.enqueue(Archive, self.id, branch)
77
+ end
78
+ end
79
+ ```
80
+
81
+ Now when we call `repo.async_create_archive('masterbrew')` in our
82
+ application, a job will be created and placed on the `file_serve`
83
+ queue.
84
+
85
+ Later, a worker will run something like this code to process the job:
86
+
87
+ ``` ruby
88
+ klass, args = Resque.reserve(:file_serve)
89
+ klass.perform(*args) if klass.respond_to? :perform
90
+ ```
91
+
92
+ Which translates to:
93
+
94
+ ``` ruby
95
+ Archive.perform(44, 'masterbrew')
96
+ ```
97
+
98
+ Let's start a worker to run `file_serve` jobs:
99
+
100
+ $ cd app_root
101
+ $ QUEUE=file_serve rake resque:work
102
+
103
+ This starts one Resque worker and tells it to work off the
104
+ `file_serve` queue. As soon as it's ready it'll try to run the
105
+ `Resque.reserve` code snippet above and process jobs until it can't
106
+ find any more, at which point it will sleep for a small period and
107
+ repeatedly poll the queue for more jobs.
108
+
109
+ Workers can be given multiple queues (a "queue list") and run on
110
+ multiple machines. In fact they can be run anywhere with network
111
+ access to the Redis server.
112
+
113
+
114
+ Jobs
115
+ ----
116
+
117
+ What should you run in the background? Anything that takes any time at
118
+ all. Slow INSERT statements, disk manipulating, data processing, etc.
119
+
120
+ At GitHub we use Resque to process the following types of jobs:
121
+
122
+ * Warming caches
123
+ * Counting disk usage
124
+ * Building tarballs
125
+ * Building Rubygems
126
+ * Firing off web hooks
127
+ * Creating events in the db and pre-caching them
128
+ * Building graphs
129
+ * Deleting users
130
+ * Updating our search index
131
+
132
+ As of writing we have about 35 different types of background jobs.
133
+
134
+ Keep in mind that you don't need a web app to use Resque - we just
135
+ mention "foreground" and "background" because they make conceptual
136
+ sense. You could easily be spidering sites and sticking data which
137
+ needs to be crunched later into a queue.
138
+
139
+
140
+ ### Persistence
141
+
142
+ Jobs are persisted to queues as JSON objects. Let's take our `Archive`
143
+ example from above. We'll run the following code to create a job:
144
+
145
+ ``` ruby
146
+ repo = Repository.find(44)
147
+ repo.async_create_archive('masterbrew')
148
+ ```
149
+
150
+ The following JSON will be stored in the `file_serve` queue:
151
+
152
+ ``` javascript
153
+ {
154
+ 'class': 'Archive',
155
+ 'args': [ 44, 'masterbrew' ]
156
+ }
157
+ ```
158
+
159
+ Because of this your jobs must only accept arguments that can be JSON encoded.
160
+
161
+ So instead of doing this:
162
+
163
+ ``` ruby
164
+ Resque.enqueue(Archive, self, branch)
165
+ ```
166
+
167
+ do this:
168
+
169
+ ``` ruby
170
+ Resque.enqueue(Archive, self.id, branch)
171
+ ```
172
+
173
+ This is why our above example (and all the examples in `examples/`)
174
+ uses object IDs instead of passing around the objects.
175
+
176
+ While this is less convenient than just sticking a marshaled object
177
+ in the database, it gives you a slight advantage: your jobs will be
178
+ run against the most recent version of an object because they need to
179
+ pull from the DB or cache.
180
+
181
+ If your jobs were run against marshaled objects, they could
182
+ potentially be operating on a stale record with out-of-date information.
183
+
184
+
185
+ ### send_later / async
186
+
187
+ Want something like DelayedJob's `send_later` or the ability to use
188
+ instance methods instead of just methods for jobs? See the `examples/`
189
+ directory for goodies.
190
+
191
+ We plan to provide first class `async` support in a future release.
192
+
193
+
194
+ ### Failure
195
+
196
+ If a job raises an exception, it is logged and handed off to the
197
+ `Resque::Failure` module. Failures are logged either locally in Redis
198
+ or using some different backend.
199
+
200
+ For example, Resque ships with Airbrake support. To configure it, put
201
+ the following into an initialisation file or into your rake job:
202
+
203
+ ``` ruby
204
+ # send errors which occur in background jobs to redis and airbrake
205
+ require 'resque/failure/multiple'
206
+ require 'resque/failure/redis'
207
+ require 'resque/failure/airbrake'
208
+
209
+ Resque::Failure::Multiple.classes = [Resque::Failure::Redis, Resque::Failure::Airbrake]
210
+ Resque::Failure.backend = Resque::Failure::Multiple
211
+ ```
212
+
213
+ Keep this in mind when writing your jobs: you may want to throw
214
+ exceptions you would not normally throw in order to assist debugging.
215
+
216
+
217
+ Workers
218
+ -------
219
+
220
+ Resque workers are rake tasks that run forever. They basically do this:
221
+
222
+ ``` ruby
223
+ start
224
+ loop do
225
+ if job = reserve
226
+ job.process
227
+ else
228
+ sleep 5 # Polling frequency = 5
229
+ end
230
+ end
231
+ shutdown
232
+ ```
233
+
234
+ Starting a worker is simple. Here's our example from earlier:
235
+
236
+ $ QUEUE=file_serve rake resque:work
237
+
238
+ By default Resque won't know about your application's
239
+ environment. That is, it won't be able to find and run your jobs - it
240
+ needs to load your application into memory.
241
+
242
+ If we've installed Resque as a Rails plugin, we might run this command
243
+ from our RAILS_ROOT:
244
+
245
+ $ QUEUE=file_serve rake environment resque:work
246
+
247
+ This will load the environment before starting a worker. Alternately
248
+ we can define a `resque:setup` task with a dependency on the
249
+ `environment` rake task:
250
+
251
+ ``` ruby
252
+ task "resque:setup" => :environment
253
+ ```
254
+
255
+ GitHub's setup task looks like this:
256
+
257
+ ``` ruby
258
+ task "resque:setup" => :environment do
259
+ Grit::Git.git_timeout = 10.minutes
260
+ end
261
+ ```
262
+
263
+ We don't want the `git_timeout` as high as 10 minutes in our web app,
264
+ but in the Resque workers it's fine.
265
+
266
+
267
+ ### Logging
268
+
269
+ Workers support basic logging to STDOUT. If you start them with the
270
+ `VERBOSE` env variable set, they will print basic debugging
271
+ information. You can also set the `VVERBOSE` (very verbose) env
272
+ variable.
273
+
274
+ $ VVERBOSE=1 QUEUE=file_serve rake environment resque:work
275
+
276
+ If you want Resque to log to a file, in Rails do:
277
+
278
+ ```ruby
279
+ # config/initializers/resque.rb
280
+ Resque.logger = Logger.new(Rails.root.join('log', "#{Rails.env}_resque.log"))
281
+ ```
282
+
283
+ ### Process IDs (PIDs)
284
+
285
+ There are scenarios where it's helpful to record the PID of a resque
286
+ worker process. Use the PIDFILE option for easy access to the PID:
287
+
288
+ $ PIDFILE=./resque.pid QUEUE=file_serve rake environment resque:work
289
+
290
+ ### Running in the background
291
+
292
+ (Only supported with ruby >= 1.9). There are scenarios where it's helpful for
293
+ the resque worker to run itself in the background (usually in combination with
294
+ PIDFILE). Use the BACKGROUND option so that rake will return as soon as the
295
+ worker is started.
296
+
297
+ $ PIDFILE=./resque.pid BACKGROUND=yes QUEUE=file_serve \
298
+ rake environment resque:work
299
+
300
+ ### Polling frequency
301
+
302
+ You can pass an INTERVAL option which is a float representing the polling frequency.
303
+ The default is 5 seconds, but for a semi-active app you may want to use a smaller value.
304
+
305
+ $ INTERVAL=0.1 QUEUE=file_serve rake environment resque:work
306
+
307
+ ### Priorities and Queue Lists
308
+
309
+ Resque doesn't support numeric priorities but instead uses the order
310
+ of queues you give it. We call this list of queues the "queue list."
311
+
312
+ Let's say we add a `warm_cache` queue in addition to our `file_serve`
313
+ queue. We'd now start a worker like so:
314
+
315
+ $ QUEUES=file_serve,warm_cache rake resque:work
316
+
317
+ When the worker looks for new jobs, it will first check
318
+ `file_serve`. If it finds a job, it'll process it then check
319
+ `file_serve` again. It will keep checking `file_serve` until no more
320
+ jobs are available. At that point, it will check `warm_cache`. If it
321
+ finds a job it'll process it then check `file_serve` (repeating the
322
+ whole process).
323
+
324
+ In this way you can prioritize certain queues. At GitHub we start our
325
+ workers with something like this:
326
+
327
+ $ QUEUES=critical,archive,high,low rake resque:work
328
+
329
+ Notice the `archive` queue - it is specialized and in our future
330
+ architecture will only be run from a single machine.
331
+
332
+ At that point we'll start workers on our generalized background
333
+ machines with this command:
334
+
335
+ $ QUEUES=critical,high,low rake resque:work
336
+
337
+ And workers on our specialized archive machine with this command:
338
+
339
+ $ QUEUE=archive rake resque:work
340
+
341
+
342
+ ### Running All Queues
343
+
344
+ If you want your workers to work off of every queue, including new
345
+ queues created on the fly, you can use a splat:
346
+
347
+ $ QUEUE=* rake resque:work
348
+
349
+ Queues will be processed in alphabetical order.
350
+
351
+
352
+ ### Running Multiple Workers
353
+
354
+ At GitHub we use god to start and stop multiple workers. A sample god
355
+ configuration file is included under `examples/god`. We recommend this
356
+ method.
357
+
358
+ If you'd like to run multiple workers in development mode, you can do
359
+ so using the `resque:workers` rake task:
360
+
361
+ $ COUNT=5 QUEUE=* rake resque:workers
362
+
363
+ This will spawn five Resque workers, each in its own process. Hitting
364
+ ctrl-c should be sufficient to stop them all.
365
+
366
+
367
+ ### Forking
368
+
369
+ On certain platforms, when a Resque worker reserves a job it
370
+ immediately forks a child process. The child processes the job then
371
+ exits. When the child has exited successfully, the worker reserves
372
+ another job and repeats the process.
373
+
374
+ Why?
375
+
376
+ Because Resque assumes chaos.
377
+
378
+ Resque assumes your background workers will lock up, run too long, or
379
+ have unwanted memory growth.
380
+
381
+ If Resque workers processed jobs themselves, it'd be hard to whip them
382
+ into shape. Let's say one is using too much memory: you send it a
383
+ signal that says "shutdown after you finish processing the current
384
+ job," and it does so. It then starts up again - loading your entire
385
+ application environment. This adds useless CPU cycles and causes a
386
+ delay in queue processing.
387
+
388
+ Plus, what if it's using too much memory and has stopped responding to
389
+ signals?
390
+
391
+ Thanks to Resque's parent / child architecture, jobs that use too much memory
392
+ release that memory upon completion. No unwanted growth.
393
+
394
+ And what if a job is running too long? You'd need to `kill -9` it then
395
+ start the worker again. With Resque's parent / child architecture you
396
+ can tell the parent to forcefully kill the child then immediately
397
+ start processing more jobs. No startup delay or wasted cycles.
398
+
399
+ The parent / child architecture helps us keep tabs on what workers are
400
+ doing, too. By eliminating the need to `kill -9` workers we can have
401
+ parents remove themselves from the global listing of workers. If we
402
+ just ruthlessly killed workers, we'd need a separate watchdog process
403
+ to add and remove them to the global listing - which becomes
404
+ complicated.
405
+
406
+ Workers instead handle their own state.
407
+
408
+
409
+ ### Parents and Children
410
+
411
+ Here's a parent / child pair doing some work:
412
+
413
+ $ ps -e -o pid,command | grep [r]esque
414
+ 92099 resque: Forked 92102 at 1253142769
415
+ 92102 resque: Processing file_serve since 1253142769
416
+
417
+ You can clearly see that process 92099 forked 92102, which has been
418
+ working since 1253142769.
419
+
420
+ (By advertising the time they began processing you can easily use monit
421
+ or god to kill stale workers.)
422
+
423
+ When a parent process is idle, it lets you know what queues it is
424
+ waiting for work on:
425
+
426
+ $ ps -e -o pid,command | grep [r]esque
427
+ 92099 resque: Waiting for file_serve,warm_cache
428
+
429
+
430
+ ### Signals
431
+
432
+ Resque workers respond to a few different signals:
433
+
434
+ * `QUIT` - Wait for child to finish processing then exit
435
+ * `TERM` / `INT` - Immediately kill child then exit
436
+ * `USR1` - Immediately kill child but don't exit
437
+ * `USR2` - Don't start to process any new jobs
438
+ * `CONT` - Start to process new jobs again after a USR2
439
+
440
+ If you want to gracefully shutdown a Resque worker, use `QUIT`.
441
+
442
+ If you want to kill a stale or stuck child, use `USR1`. Processing
443
+ will continue as normal unless the child was not found. In that case
444
+ Resque assumes the parent process is in a bad state and shuts down.
445
+
446
+ If you want to kill a stale or stuck child and shutdown, use `TERM`
447
+
448
+ If you want to stop processing jobs, but want to leave the worker running
449
+ (for example, to temporarily alleviate load), use `USR2` to stop processing,
450
+ then `CONT` to start it again.
451
+
452
+ ### Mysql::Error: MySQL server has gone away
453
+
454
+ If your workers remain idle for too long they may lose their MySQL connection. Depending on your version of Rails, we recommend the following:
455
+
456
+ #### Rails 3.x
457
+ In your `perform` method, add the following line:
458
+
459
+ ``` ruby
460
+ class MyTask
461
+ def self.perform
462
+ ActiveRecord::Base.verify_active_connections!
463
+ # rest of your code
464
+ end
465
+ end
466
+ ```
467
+
468
+ The Rails doc says the following about `verify_active_connections!`:
469
+
470
+ Verify active connections and remove and disconnect connections associated with stale threads.
471
+
472
+ #### Rails 4.x
473
+
474
+ In your `perform` method, instead of `verify_active_connections!`, use:
475
+
476
+ ``` ruby
477
+ class MyTask
478
+ def self.perform
479
+ ActiveRecord::Base.clear_active_connections!
480
+ # rest of your code
481
+ end
482
+ end
483
+ ```
484
+
485
+ From the Rails docs on [`clear_active_connections!`](http://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/ConnectionHandler.html#method-i-clear_active_connections-21):
486
+
487
+ Returns any connections in use by the current thread back to the pool, and also returns connections to the pool cached by threads that are no longer alive.
488
+
489
+
490
+
491
+ The Front End
492
+ -------------
493
+
494
+ Resque comes with a Sinatra-based front end for seeing what's up with
495
+ your queue.
496
+
497
+ ![The Front End](https://camo.githubusercontent.com/64d150a243987ffbc33f588bd6d7722a0bb8d69a/687474703a2f2f7475746f7269616c732e6a756d7073746172746c61622e636f6d2f696d616765732f7265737175655f6f766572766965772e706e67)
498
+
499
+ ### Standalone
500
+
501
+ If you've installed Resque as a gem running the front end standalone is easy:
502
+
503
+ $ resque-web
504
+
505
+ It's a thin layer around `rackup` so it's configurable as well:
506
+
507
+ $ resque-web -p 8282
508
+
509
+ If you have a Resque config file you want evaluated just pass it to
510
+ the script as the final argument:
511
+
512
+ $ resque-web -p 8282 rails_root/config/initializers/resque.rb
513
+
514
+ You can also set the namespace directly using `resque-web`:
515
+
516
+ $ resque-web -p 8282 -N myapp
517
+
518
+ or set the Redis connection string if you need to do something like select a different database:
519
+
520
+ $ resque-web -p 8282 -r localhost:6379:2
521
+
522
+ ### Passenger
523
+
524
+ Using Passenger? Resque ships with a `config.ru` you can use. See
525
+ Phusion's guide:
526
+
527
+ Apache: <https://www.phusionpassenger.com/library/deploy/apache/deploy/ruby/>
528
+ Nginx: <https://www.phusionpassenger.com/library/deploy/nginx/deploy/ruby/>
529
+
530
+ ### Rack::URLMap
531
+
532
+ If you want to load Resque on a subpath, possibly alongside other
533
+ apps, it's easy to do with Rack's `URLMap`:
534
+
535
+ ``` ruby
536
+ require 'resque/server'
537
+
538
+ run Rack::URLMap.new \
539
+ "/" => Your::App.new,
540
+ "/resque" => Resque::Server.new
541
+ ```
542
+
543
+ Check `examples/demo/config.ru` for a functional example (including
544
+ HTTP basic auth).
545
+
546
+ ### Rails 3
547
+
548
+ You can also mount Resque on a subpath in your existing Rails 3 app by adding `require 'resque/server'` to the top of your routes file or in an initializer then adding this to `routes.rb`:
549
+
550
+ ``` ruby
551
+ mount Resque::Server.new, :at => "/resque"
552
+ ```
553
+
554
+
555
+ Resque vs DelayedJob
556
+ --------------------
557
+
558
+ How does Resque compare to DelayedJob, and why would you choose one
559
+ over the other?
560
+
561
+ * Resque supports multiple queues
562
+ * DelayedJob supports finer grained priorities
563
+ * Resque workers are resilient to memory leaks / bloat
564
+ * DelayedJob workers are extremely simple and easy to modify
565
+ * Resque requires Redis
566
+ * DelayedJob requires ActiveRecord
567
+ * Resque can only place JSONable Ruby objects on a queue as arguments
568
+ * DelayedJob can place _any_ Ruby object on its queue as arguments
569
+ * Resque includes a Sinatra app for monitoring what's going on
570
+ * DelayedJob can be queried from within your Rails app if you want to
571
+ add an interface
572
+
573
+ If you're doing Rails development, you already have a database and
574
+ ActiveRecord. DelayedJob is super easy to setup and works great.
575
+ GitHub used it for many months to process almost 200 million jobs.
576
+
577
+ Choose Resque if:
578
+
579
+ * You need multiple queues
580
+ * You don't care / dislike numeric priorities
581
+ * You don't need to persist every Ruby object ever
582
+ * You have potentially huge queues
583
+ * You want to see what's going on
584
+ * You expect a lot of failure / chaos
585
+ * You can setup Redis
586
+ * You're not running short on RAM
587
+
588
+ Choose DelayedJob if:
589
+
590
+ * You like numeric priorities
591
+ * You're not doing a gigantic amount of jobs each day
592
+ * Your queue stays small and nimble
593
+ * There is not a lot failure / chaos
594
+ * You want to easily throw anything on the queue
595
+ * You don't want to setup Redis
596
+
597
+ In no way is Resque a "better" DelayedJob, so make sure you pick the
598
+ tool that's best for your app.
599
+
600
+ Resque Dependencies
601
+ -------------------
602
+
603
+ $ gem install bundler
604
+ $ bundle install
605
+
606
+
607
+ Installing Resque
608
+ -----------------
609
+
610
+ ### In a Rack app, as a gem
611
+
612
+ First install the gem.
613
+
614
+ $ gem install resque
615
+
616
+ Next include it in your application.
617
+
618
+ ``` ruby
619
+ require 'resque'
620
+ ```
621
+
622
+ Now start your application:
623
+
624
+ rackup config.ru
625
+
626
+ That's it! You can now create Resque jobs from within your app.
627
+
628
+ To start a worker, create a Rakefile in your app's root (or add this
629
+ to an existing Rakefile):
630
+
631
+ ``` ruby
632
+ require 'your/app'
633
+ require 'resque/tasks'
634
+ ```
635
+
636
+ Now:
637
+
638
+ $ QUEUE=* rake resque:work
639
+
640
+ Alternately you can define a `resque:setup` hook in your Rakefile if you
641
+ don't want to load your app every time rake runs.
642
+
643
+
644
+ ### In a Rails 2.x app, as a gem
645
+
646
+ First install the gem.
647
+
648
+ $ gem install resque
649
+
650
+ Next include it in your application.
651
+
652
+ $ cat config/initializers/load_resque.rb
653
+ require 'resque'
654
+
655
+ Now start your application:
656
+
657
+ $ ./script/server
658
+
659
+ That's it! You can now create Resque jobs from within your app.
660
+
661
+ To start a worker, add this to your Rakefile in `RAILS_ROOT`:
662
+
663
+ ``` ruby
664
+ require 'resque/tasks'
665
+ ```
666
+
667
+ Now:
668
+
669
+ $ QUEUE=* rake environment resque:work
670
+
671
+ Don't forget you can define a `resque:setup` hook in
672
+ `lib/tasks/whatever.rake` that loads the `environment` task every time.
673
+
674
+
675
+ ### In a Rails 2.x app, as a plugin
676
+
677
+ $ ./script/plugin install git://github.com/resque/resque
678
+
679
+ That's it! Resque will automatically be available when your Rails app
680
+ loads.
681
+
682
+ To start a worker:
683
+
684
+ $ QUEUE=* rake environment resque:work
685
+
686
+ Don't forget you can define a `resque:setup` hook in
687
+ `lib/tasks/whatever.rake` that loads the `environment` task every time.
688
+
689
+
690
+ ### In a Rails 3.x or 4.x app, as a gem
691
+
692
+ First include it in your Gemfile.
693
+
694
+ $ cat Gemfile
695
+ ...
696
+ gem 'resque'
697
+ ...
698
+
699
+ Next install it with Bundler.
700
+
701
+ $ bundle install
702
+
703
+ Now start your application:
704
+
705
+ $ rails server
706
+
707
+ That's it! You can now create Resque jobs from within your app.
708
+
709
+ To start a worker, add this to a file in `lib/tasks` (ex:
710
+ `lib/tasks/resque.rake`):
711
+
712
+ ``` ruby
713
+ require 'resque/tasks'
714
+ ```
715
+
716
+ Now:
717
+
718
+ $ QUEUE=* rake environment resque:work
719
+
720
+ Don't forget you can define a `resque:setup` hook in
721
+ `lib/tasks/whatever.rake` that loads the `environment` task every time.
722
+
723
+
724
+ Configuration
725
+ -------------
726
+
727
+ You may want to change the Redis host and port Resque connects to, or
728
+ set various other options at startup.
729
+
730
+ Resque has a `redis` setter which can be given a string or a Redis
731
+ object. This means if you're already using Redis in your app, Resque
732
+ can re-use the existing connection.
733
+
734
+ String: `Resque.redis = 'localhost:6379'`
735
+
736
+ Redis: `Resque.redis = $redis`
737
+
738
+ For our rails app we have a `config/initializers/resque.rb` file where
739
+ we load `config/resque.yml` by hand and set the Redis information
740
+ appropriately.
741
+
742
+ Here's our `config/resque.yml`:
743
+
744
+ development: localhost:6379
745
+ test: localhost:6379
746
+ staging: redis1.se.github.com:6379
747
+ fi: localhost:6379
748
+ production: redis1.ae.github.com:6379
749
+
750
+ And our initializer:
751
+
752
+ ``` ruby
753
+ rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..'
754
+ rails_env = ENV['RAILS_ENV'] || 'development'
755
+
756
+ resque_config = YAML.load_file(rails_root + '/config/resque.yml')
757
+ Resque.redis = resque_config[rails_env]
758
+ ```
759
+
760
+ Easy peasy! Why not just use `RAILS_ROOT` and `RAILS_ENV`? Because
761
+ this way we can tell our Sinatra app about the config file:
762
+
763
+ $ RAILS_ENV=production resque-web rails_root/config/initializers/resque.rb
764
+
765
+ Now everyone is on the same page.
766
+
767
+ Also, you could disable jobs queueing by setting 'inline' attribute.
768
+ For example, if you want to run all jobs in the same process for cucumber, try:
769
+
770
+ ``` ruby
771
+ Resque.inline = ENV['RAILS_ENV'] == "cucumber"
772
+ ```
773
+
774
+
775
+ Plugins and Hooks
776
+ -----------------
777
+
778
+ For a list of available plugins see
779
+ <http://wiki.github.com/resque/resque/plugins>.
780
+
781
+ If you'd like to write your own plugin, or want to customize Resque
782
+ using hooks (such as `Resque.after_fork`), see
783
+ [docs/HOOKS.md](http://github.com/resque/resque/blob/master/docs/HOOKS.md).
784
+
785
+
786
+ Namespaces
787
+ ----------
788
+
789
+ If you're running multiple, separate instances of Resque you may want
790
+ to namespace the keyspaces so they do not overlap. This is not unlike
791
+ the approach taken by many memcached clients.
792
+
793
+ This feature is provided by the [redis-namespace][rs] library, which
794
+ Resque uses by default to separate the keys it manages from other keys
795
+ in your Redis server.
796
+
797
+ Simply use the `Resque.redis.namespace` accessor:
798
+
799
+ ``` ruby
800
+ Resque.redis.namespace = "resque:GitHub"
801
+ ```
802
+
803
+ We recommend sticking this in your initializer somewhere after Redis
804
+ is configured.
805
+
806
+
807
+ Demo
808
+ ----
809
+
810
+ Resque ships with a demo Sinatra app for creating jobs that are later
811
+ processed in the background.
812
+
813
+ Try it out by looking at the README, found at `examples/demo/README.markdown`.
814
+
815
+
816
+ Monitoring
817
+ ----------
818
+
819
+ ### god
820
+
821
+ If you're using god to monitor Resque, we have provided example
822
+ configs in `examples/god/`. One is for starting / stopping workers,
823
+ the other is for killing workers that have been running too long.
824
+
825
+ ### monit
826
+
827
+ If you're using monit, `examples/monit/resque.monit` is provided free
828
+ of charge. This is **not** used by GitHub in production, so please
829
+ send patches for any tweaks or improvements you can make to it.
830
+
831
+
832
+ Questions
833
+ ---------
834
+
835
+ Please add them to the [FAQ](https://github.com/resque/resque/wiki/FAQ) or
836
+ ask on the Mailing List. The Mailing List is explained further below
837
+
838
+
839
+ Development
840
+ -----------
841
+
842
+ Want to hack on Resque?
843
+
844
+ First clone the repo and run the tests:
845
+
846
+ git clone git://github.com/resque/resque.git
847
+ cd resque
848
+ rake test
849
+
850
+ If the tests do not pass make sure you have Redis installed
851
+ correctly (though we make an effort to tell you if we feel this is the
852
+ case). The tests attempt to start an isolated instance of Redis to
853
+ run against.
854
+
855
+ Also make sure you've installed all the dependencies correctly. For
856
+ example, try loading the `redis-namespace` gem after you've installed
857
+ it:
858
+
859
+ $ irb
860
+ >> require 'rubygems'
861
+ => true
862
+ >> require 'redis/namespace'
863
+ => true
864
+
865
+ If you get an error requiring any of the dependencies, you may have
866
+ failed to install them or be seeing load path issues.
867
+
868
+ Feel free to ping the mailing list with your problem and we'll try to
869
+ sort it out.
870
+
871
+
872
+ Contributing
873
+ ------------
874
+
875
+ Read [CONTRIBUTING.md](CONTRIBUTING.md) first.
876
+
877
+ Once you've made your great commits:
878
+
879
+ 1. [Fork][1] Resque
880
+ 2. Create a topic branch - `git checkout -b my_branch`
881
+ 3. Push to your branch - `git push origin my_branch`
882
+ 4. Create a [Pull Request](http://help.github.com/pull-requests/) from your branch
883
+ 5. That's it!
884
+
885
+
886
+ Mailing List
887
+ ------------
888
+
889
+ To join the list simply send an email to <resque@librelist.com>. This
890
+ will subscribe you and send you information about your subscription,
891
+ including unsubscribe information.
892
+
893
+ The archive can be found at <http://librelist.com/browser/resque/>.
894
+
895
+
896
+ Meta
897
+ ----
898
+
899
+ * Code: `git clone git://github.com/resque/resque.git`
900
+ * Home: <http://github.com/resque/resque>
901
+ * Docs: <http://rubydoc.info/gems/resque>
902
+ * Bugs: <http://github.com/resque/resque/issues>
903
+ * List: <resque@librelist.com>
904
+ * Chat: <irc://irc.freenode.net/resque>
905
+ * Gems: <http://gemcutter.org/gems/resque>
906
+
907
+ This project uses [Semantic Versioning][sv].
908
+
909
+
910
+ Author
911
+ ------
912
+
913
+ Chris Wanstrath :: chris@ozmm.org :: @defunkt
914
+
915
+ [0]: http://github.com/blog/542-introducing-resque
916
+ [1]: http://help.github.com/forking/
917
+ [2]: http://github.com/resque/resque/issues
918
+ [sv]: http://semver.org/
919
+ [rs]: http://github.com/resque/redis-namespace
920
+ [cb]: http://wiki.github.com/resque/resque/contributing