resque_signal_from_child 1.25.1

Sign up to get free protection for your applications and to get access to all the features.
Files changed (72) hide show
  1. checksums.yaml +15 -0
  2. data/HISTORY.md +455 -0
  3. data/LICENSE +20 -0
  4. data/README.markdown +866 -0
  5. data/Rakefile +70 -0
  6. data/bin/resque +81 -0
  7. data/bin/resque-web +27 -0
  8. data/lib/resque.rb +487 -0
  9. data/lib/resque/errors.rb +13 -0
  10. data/lib/resque/failure.rb +113 -0
  11. data/lib/resque/failure/airbrake.rb +33 -0
  12. data/lib/resque/failure/base.rb +73 -0
  13. data/lib/resque/failure/multiple.rb +59 -0
  14. data/lib/resque/failure/redis.rb +108 -0
  15. data/lib/resque/failure/redis_multi_queue.rb +89 -0
  16. data/lib/resque/helpers.rb +101 -0
  17. data/lib/resque/job.rb +346 -0
  18. data/lib/resque/log_formatters/quiet_formatter.rb +7 -0
  19. data/lib/resque/log_formatters/verbose_formatter.rb +7 -0
  20. data/lib/resque/log_formatters/very_verbose_formatter.rb +8 -0
  21. data/lib/resque/logging.rb +18 -0
  22. data/lib/resque/plugin.rb +66 -0
  23. data/lib/resque/server.rb +271 -0
  24. data/lib/resque/server/helpers.rb +52 -0
  25. data/lib/resque/server/public/favicon.ico +0 -0
  26. data/lib/resque/server/public/idle.png +0 -0
  27. data/lib/resque/server/public/jquery-1.3.2.min.js +19 -0
  28. data/lib/resque/server/public/jquery.relatize_date.js +95 -0
  29. data/lib/resque/server/public/poll.png +0 -0
  30. data/lib/resque/server/public/ranger.js +78 -0
  31. data/lib/resque/server/public/reset.css +44 -0
  32. data/lib/resque/server/public/style.css +91 -0
  33. data/lib/resque/server/public/working.png +0 -0
  34. data/lib/resque/server/test_helper.rb +19 -0
  35. data/lib/resque/server/views/error.erb +1 -0
  36. data/lib/resque/server/views/failed.erb +29 -0
  37. data/lib/resque/server/views/failed_job.erb +50 -0
  38. data/lib/resque/server/views/failed_queues_overview.erb +24 -0
  39. data/lib/resque/server/views/key_sets.erb +19 -0
  40. data/lib/resque/server/views/key_string.erb +11 -0
  41. data/lib/resque/server/views/layout.erb +44 -0
  42. data/lib/resque/server/views/next_more.erb +22 -0
  43. data/lib/resque/server/views/overview.erb +4 -0
  44. data/lib/resque/server/views/queues.erb +58 -0
  45. data/lib/resque/server/views/stats.erb +62 -0
  46. data/lib/resque/server/views/workers.erb +109 -0
  47. data/lib/resque/server/views/working.erb +72 -0
  48. data/lib/resque/stat.rb +57 -0
  49. data/lib/resque/tasks.rb +83 -0
  50. data/lib/resque/vendor/utf8_util.rb +26 -0
  51. data/lib/resque/vendor/utf8_util/utf8_util_18.rb +91 -0
  52. data/lib/resque/vendor/utf8_util/utf8_util_19.rb +5 -0
  53. data/lib/resque/version.rb +3 -0
  54. data/lib/resque/worker.rb +772 -0
  55. data/lib/tasks/redis.rake +161 -0
  56. data/lib/tasks/resque.rake +2 -0
  57. data/test/airbrake_test.rb +27 -0
  58. data/test/dump.rdb +0 -0
  59. data/test/failure_base_test.rb +15 -0
  60. data/test/job_hooks_test.rb +464 -0
  61. data/test/job_plugins_test.rb +230 -0
  62. data/test/logging_test.rb +24 -0
  63. data/test/plugin_test.rb +116 -0
  64. data/test/redis-test-cluster.conf +115 -0
  65. data/test/redis-test.conf +115 -0
  66. data/test/resque-web_test.rb +59 -0
  67. data/test/resque_failure_redis_test.rb +19 -0
  68. data/test/resque_hook_test.rb +190 -0
  69. data/test/resque_test.rb +278 -0
  70. data/test/test_helper.rb +198 -0
  71. data/test/worker_test.rb +1015 -0
  72. metadata +196 -0
data/LICENSE ADDED
@@ -0,0 +1,20 @@
1
+ Copyright (c) Chris Wanstrath
2
+
3
+ Permission is hereby granted, free of charge, to any person obtaining
4
+ a copy of this software and associated documentation files (the
5
+ "Software"), to deal in the Software without restriction, including
6
+ without limitation the rights to use, copy, modify, merge, publish,
7
+ distribute, sublicense, and/or sell copies of the Software, and to
8
+ permit persons to whom the Software is furnished to do so, subject to
9
+ the following conditions:
10
+
11
+ The above copyright notice and this permission notice shall be
12
+ included in all copies or substantial portions of the Software.
13
+
14
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
15
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
16
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
17
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
18
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
19
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
20
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
data/README.markdown ADDED
@@ -0,0 +1,866 @@
1
+ Resque
2
+ ======
3
+
4
+ Resque (pronounced like "rescue") is a Redis-backed library for creating
5
+ background jobs, placing those jobs on multiple queues, and processing
6
+ them later.
7
+
8
+ Background jobs can be any Ruby class or module that responds to
9
+ `perform`. Your existing classes can easily be converted to background
10
+ jobs or you can create new classes specifically to do work. Or, you
11
+ can do both.
12
+
13
+ Resque is heavily inspired by DelayedJob (which rocks) and comprises
14
+ three parts:
15
+
16
+ 1. A Ruby library for creating, querying, and processing jobs
17
+ 2. A Rake task for starting a worker which processes jobs
18
+ 3. A Sinatra app for monitoring queues, jobs, and workers.
19
+
20
+ Resque workers can be distributed between multiple machines,
21
+ support priorities, are resilient to memory bloat / "leaks," are
22
+ optimized for REE (but work on MRI and JRuby), tell you what they're
23
+ doing, and expect failure.
24
+
25
+ Resque queues are persistent; support constant time, atomic push and
26
+ pop (thanks to Redis); provide visibility into their contents; and
27
+ store jobs as simple JSON packages.
28
+
29
+ The Resque frontend tells you what workers are doing, what workers are
30
+ not doing, what queues you're using, what's in those queues, provides
31
+ general usage stats, and helps you track failures.
32
+
33
+
34
+ The Blog Post
35
+ -------------
36
+
37
+ For the backstory, philosophy, and history of Resque's beginnings,
38
+ please see [the blog post][0].
39
+
40
+
41
+ Overview
42
+ --------
43
+
44
+ Resque allows you to create jobs and place them on a queue, then,
45
+ later, pull those jobs off the queue and process them.
46
+
47
+ Resque jobs are Ruby classes (or modules) which respond to the
48
+ `perform` method. Here's an example:
49
+
50
+
51
+ ``` ruby
52
+ class Archive
53
+ @queue = :file_serve
54
+
55
+ def self.perform(repo_id, branch = 'master')
56
+ repo = Repository.find(repo_id)
57
+ repo.create_archive(branch)
58
+ end
59
+ end
60
+ ```
61
+
62
+ The `@queue` class instance variable determines which queue `Archive`
63
+ jobs will be placed in. Queues are arbitrary and created on the fly -
64
+ you can name them whatever you want and have as many as you want.
65
+
66
+ To place an `Archive` job on the `file_serve` queue, we might add this
67
+ to our application's pre-existing `Repository` class:
68
+
69
+ ``` ruby
70
+ class Repository
71
+ def async_create_archive(branch)
72
+ Resque.enqueue(Archive, self.id, branch)
73
+ end
74
+ end
75
+ ```
76
+
77
+ Now when we call `repo.async_create_archive('masterbrew')` in our
78
+ application, a job will be created and placed on the `file_serve`
79
+ queue.
80
+
81
+ Later, a worker will run something like this code to process the job:
82
+
83
+ ``` ruby
84
+ klass, args = Resque.reserve(:file_serve)
85
+ klass.perform(*args) if klass.respond_to? :perform
86
+ ```
87
+
88
+ Which translates to:
89
+
90
+ ``` ruby
91
+ Archive.perform(44, 'masterbrew')
92
+ ```
93
+
94
+ Let's start a worker to run `file_serve` jobs:
95
+
96
+ $ cd app_root
97
+ $ QUEUE=file_serve rake resque:work
98
+
99
+ This starts one Resque worker and tells it to work off the
100
+ `file_serve` queue. As soon as it's ready it'll try to run the
101
+ `Resque.reserve` code snippet above and process jobs until it can't
102
+ find any more, at which point it will sleep for a small period and
103
+ repeatedly poll the queue for more jobs.
104
+
105
+ Workers can be given multiple queues (a "queue list") and run on
106
+ multiple machines. In fact they can be run anywhere with network
107
+ access to the Redis server.
108
+
109
+
110
+ Jobs
111
+ ----
112
+
113
+ What should you run in the background? Anything that takes any time at
114
+ all. Slow INSERT statements, disk manipulating, data processing, etc.
115
+
116
+ At GitHub we use Resque to process the following types of jobs:
117
+
118
+ * Warming caches
119
+ * Counting disk usage
120
+ * Building tarballs
121
+ * Building Rubygems
122
+ * Firing off web hooks
123
+ * Creating events in the db and pre-caching them
124
+ * Building graphs
125
+ * Deleting users
126
+ * Updating our search index
127
+
128
+ As of writing we have about 35 different types of background jobs.
129
+
130
+ Keep in mind that you don't need a web app to use Resque - we just
131
+ mention "foreground" and "background" because they make conceptual
132
+ sense. You could easily be spidering sites and sticking data which
133
+ needs to be crunched later into a queue.
134
+
135
+
136
+ ### Persistence
137
+
138
+ Jobs are persisted to queues as JSON objects. Let's take our `Archive`
139
+ example from above. We'll run the following code to create a job:
140
+
141
+ ``` ruby
142
+ repo = Repository.find(44)
143
+ repo.async_create_archive('masterbrew')
144
+ ```
145
+
146
+ The following JSON will be stored in the `file_serve` queue:
147
+
148
+ ``` javascript
149
+ {
150
+ 'class': 'Archive',
151
+ 'args': [ 44, 'masterbrew' ]
152
+ }
153
+ ```
154
+
155
+ Because of this your jobs must only accept arguments that can be JSON encoded.
156
+
157
+ So instead of doing this:
158
+
159
+ ``` ruby
160
+ Resque.enqueue(Archive, self, branch)
161
+ ```
162
+
163
+ do this:
164
+
165
+ ``` ruby
166
+ Resque.enqueue(Archive, self.id, branch)
167
+ ```
168
+
169
+ This is why our above example (and all the examples in `examples/`)
170
+ uses object IDs instead of passing around the objects.
171
+
172
+ While this is less convenient than just sticking a marshaled object
173
+ in the database, it gives you a slight advantage: your jobs will be
174
+ run against the most recent version of an object because they need to
175
+ pull from the DB or cache.
176
+
177
+ If your jobs were run against marshaled objects, they could
178
+ potentially be operating on a stale record with out-of-date information.
179
+
180
+
181
+ ### send_later / async
182
+
183
+ Want something like DelayedJob's `send_later` or the ability to use
184
+ instance methods instead of just methods for jobs? See the `examples/`
185
+ directory for goodies.
186
+
187
+ We plan to provide first class `async` support in a future release.
188
+
189
+
190
+ ### Failure
191
+
192
+ If a job raises an exception, it is logged and handed off to the
193
+ `Resque::Failure` module. Failures are logged either locally in Redis
194
+ or using some different backend.
195
+
196
+ For example, Resque ships with Airbrake support.
197
+
198
+ Keep this in mind when writing your jobs: you may want to throw
199
+ exceptions you would not normally throw in order to assist debugging.
200
+
201
+
202
+ Workers
203
+ -------
204
+
205
+ Resque workers are rake tasks that run forever. They basically do this:
206
+
207
+ ``` ruby
208
+ start
209
+ loop do
210
+ if job = reserve
211
+ job.process
212
+ else
213
+ sleep 5 # Polling frequency = 5
214
+ end
215
+ end
216
+ shutdown
217
+ ```
218
+
219
+ Starting a worker is simple. Here's our example from earlier:
220
+
221
+ $ QUEUE=file_serve rake resque:work
222
+
223
+ By default Resque won't know about your application's
224
+ environment. That is, it won't be able to find and run your jobs - it
225
+ needs to load your application into memory.
226
+
227
+ If we've installed Resque as a Rails plugin, we might run this command
228
+ from our RAILS_ROOT:
229
+
230
+ $ QUEUE=file_serve rake environment resque:work
231
+
232
+ This will load the environment before starting a worker. Alternately
233
+ we can define a `resque:setup` task with a dependency on the
234
+ `environment` rake task:
235
+
236
+ ``` ruby
237
+ task "resque:setup" => :environment
238
+ ```
239
+
240
+ GitHub's setup task looks like this:
241
+
242
+ ``` ruby
243
+ task "resque:setup" => :environment do
244
+ Grit::Git.git_timeout = 10.minutes
245
+ end
246
+ ```
247
+
248
+ We don't want the `git_timeout` as high as 10 minutes in our web app,
249
+ but in the Resque workers it's fine.
250
+
251
+
252
+ ### Logging
253
+
254
+ Workers support basic logging to STDOUT. If you start them with the
255
+ `VERBOSE` env variable set, they will print basic debugging
256
+ information. You can also set the `VVERBOSE` (very verbose) env
257
+ variable.
258
+
259
+ $ VVERBOSE=1 QUEUE=file_serve rake environment resque:work
260
+
261
+ ### Process IDs (PIDs)
262
+
263
+ There are scenarios where it's helpful to record the PID of a resque
264
+ worker process. Use the PIDFILE option for easy access to the PID:
265
+
266
+ $ PIDFILE=./resque.pid QUEUE=file_serve rake environment resque:work
267
+
268
+ ### Running in the background
269
+
270
+ (Only supported with ruby >= 1.9). There are scenarios where it's helpful for
271
+ the resque worker to run itself in the background (usually in combination with
272
+ PIDFILE). Use the BACKGROUND option so that rake will return as soon as the
273
+ worker is started.
274
+
275
+ $ PIDFILE=./resque.pid BACKGROUND=yes QUEUE=file_serve \
276
+ rake environment resque:work
277
+
278
+ ### Polling frequency
279
+
280
+ You can pass an INTERVAL option which is a float representing the polling frequency.
281
+ The default is 5 seconds, but for a semi-active app you may want to use a smaller value.
282
+
283
+ $ INTERVAL=0.1 QUEUE=file_serve rake environment resque:work
284
+
285
+ ### Priorities and Queue Lists
286
+
287
+ Resque doesn't support numeric priorities but instead uses the order
288
+ of queues you give it. We call this list of queues the "queue list."
289
+
290
+ Let's say we add a `warm_cache` queue in addition to our `file_serve`
291
+ queue. We'd now start a worker like so:
292
+
293
+ $ QUEUES=file_serve,warm_cache rake resque:work
294
+
295
+ When the worker looks for new jobs, it will first check
296
+ `file_serve`. If it finds a job, it'll process it then check
297
+ `file_serve` again. It will keep checking `file_serve` until no more
298
+ jobs are available. At that point, it will check `warm_cache`. If it
299
+ finds a job it'll process it then check `file_serve` (repeating the
300
+ whole process).
301
+
302
+ In this way you can prioritize certain queues. At GitHub we start our
303
+ workers with something like this:
304
+
305
+ $ QUEUES=critical,archive,high,low rake resque:work
306
+
307
+ Notice the `archive` queue - it is specialized and in our future
308
+ architecture will only be run from a single machine.
309
+
310
+ At that point we'll start workers on our generalized background
311
+ machines with this command:
312
+
313
+ $ QUEUES=critical,high,low rake resque:work
314
+
315
+ And workers on our specialized archive machine with this command:
316
+
317
+ $ QUEUE=archive rake resque:work
318
+
319
+
320
+ ### Running All Queues
321
+
322
+ If you want your workers to work off of every queue, including new
323
+ queues created on the fly, you can use a splat:
324
+
325
+ $ QUEUE=* rake resque:work
326
+
327
+ Queues will be processed in alphabetical order.
328
+
329
+
330
+ ### Running Multiple Workers
331
+
332
+ At GitHub we use god to start and stop multiple workers. A sample god
333
+ configuration file is included under `examples/god`. We recommend this
334
+ method.
335
+
336
+ If you'd like to run multiple workers in development mode, you can do
337
+ so using the `resque:workers` rake task:
338
+
339
+ $ COUNT=5 QUEUE=* rake resque:workers
340
+
341
+ This will spawn five Resque workers, each in its own process. Hitting
342
+ ctrl-c should be sufficient to stop them all.
343
+
344
+
345
+ ### Forking
346
+
347
+ On certain platforms, when a Resque worker reserves a job it
348
+ immediately forks a child process. The child processes the job then
349
+ exits. When the child has exited successfully, the worker reserves
350
+ another job and repeats the process.
351
+
352
+ Why?
353
+
354
+ Because Resque assumes chaos.
355
+
356
+ Resque assumes your background workers will lock up, run too long, or
357
+ have unwanted memory growth.
358
+
359
+ If Resque workers processed jobs themselves, it'd be hard to whip them
360
+ into shape. Let's say one is using too much memory: you send it a
361
+ signal that says "shutdown after you finish processing the current
362
+ job," and it does so. It then starts up again - loading your entire
363
+ application environment. This adds useless CPU cycles and causes a
364
+ delay in queue processing.
365
+
366
+ Plus, what if it's using too much memory and has stopped responding to
367
+ signals?
368
+
369
+ Thanks to Resque's parent / child architecture, jobs that use too much memory
370
+ release that memory upon completion. No unwanted growth.
371
+
372
+ And what if a job is running too long? You'd need to `kill -9` it then
373
+ start the worker again. With Resque's parent / child architecture you
374
+ can tell the parent to forcefully kill the child then immediately
375
+ start processing more jobs. No startup delay or wasted cycles.
376
+
377
+ The parent / child architecture helps us keep tabs on what workers are
378
+ doing, too. By eliminating the need to `kill -9` workers we can have
379
+ parents remove themselves from the global listing of workers. If we
380
+ just ruthlessly killed workers, we'd need a separate watchdog process
381
+ to add and remove them to the global listing - which becomes
382
+ complicated.
383
+
384
+ Workers instead handle their own state.
385
+
386
+
387
+ ### Parents and Children
388
+
389
+ Here's a parent / child pair doing some work:
390
+
391
+ $ ps -e -o pid,command | grep [r]esque
392
+ 92099 resque: Forked 92102 at 1253142769
393
+ 92102 resque: Processing file_serve since 1253142769
394
+
395
+ You can clearly see that process 92099 forked 92102, which has been
396
+ working since 1253142769.
397
+
398
+ (By advertising the time they began processing you can easily use monit
399
+ or god to kill stale workers.)
400
+
401
+ When a parent process is idle, it lets you know what queues it is
402
+ waiting for work on:
403
+
404
+ $ ps -e -o pid,command | grep [r]esque
405
+ 92099 resque: Waiting for file_serve,warm_cache
406
+
407
+
408
+ ### Signals
409
+
410
+ Resque workers respond to a few different signals:
411
+
412
+ * `QUIT` - Wait for child to finish processing then exit
413
+ * `TERM` / `INT` - Immediately kill child then exit
414
+ * `USR1` - Immediately kill child but don't exit
415
+ * `USR2` - Don't start to process any new jobs
416
+ * `CONT` - Start to process new jobs again after a USR2
417
+
418
+ If you want to gracefully shutdown a Resque worker, use `QUIT`.
419
+
420
+ If you want to kill a stale or stuck child, use `USR1`. Processing
421
+ will continue as normal unless the child was not found. In that case
422
+ Resque assumes the parent process is in a bad state and shuts down.
423
+
424
+ If you want to kill a stale or stuck child and shutdown, use `TERM`
425
+
426
+ If you want to stop processing jobs, but want to leave the worker running
427
+ (for example, to temporarily alleviate load), use `USR2` to stop processing,
428
+ then `CONT` to start it again.
429
+
430
+ ### Mysql::Error: MySQL server has gone away
431
+
432
+ If your workers remain idle for too long they may lose their MySQL
433
+ connection. If that happens we recommend using [this
434
+ Gist](http://gist.github.com/238999).
435
+
436
+
437
+ The Front End
438
+ -------------
439
+
440
+ Resque comes with a Sinatra-based front end for seeing what's up with
441
+ your queue.
442
+
443
+ ![The Front End](https://img.skitch.com/20110528-pc67a8qsfapgjxf5gagxd92fcu.png)
444
+
445
+ ### Standalone
446
+
447
+ If you've installed Resque as a gem running the front end standalone is easy:
448
+
449
+ $ resque-web
450
+
451
+ It's a thin layer around `rackup` so it's configurable as well:
452
+
453
+ $ resque-web -p 8282
454
+
455
+ If you have a Resque config file you want evaluated just pass it to
456
+ the script as the final argument:
457
+
458
+ $ resque-web -p 8282 rails_root/config/initializers/resque.rb
459
+
460
+ You can also set the namespace directly using `resque-web`:
461
+
462
+ $ resque-web -p 8282 -N myapp
463
+
464
+ or set the Redis connection string if you need to do something like select a different database:
465
+
466
+ $ resque-web -p 8282 -r localhost:6379:2
467
+
468
+ ### Passenger
469
+
470
+ Using Passenger? Resque ships with a `config.ru` you can use. See
471
+ Phusion's guide:
472
+
473
+ Apache: <http://www.modrails.com/documentation/Users%20guide%20Apache.html#_deploying_a_rack_based_ruby_application>
474
+ Nginx: <http://www.modrails.com/documentation/Users%20guide%20Nginx.html#deploying_a_rack_app>
475
+
476
+ ### Rack::URLMap
477
+
478
+ If you want to load Resque on a subpath, possibly alongside other
479
+ apps, it's easy to do with Rack's `URLMap`:
480
+
481
+ ``` ruby
482
+ require 'resque/server'
483
+
484
+ run Rack::URLMap.new \
485
+ "/" => Your::App.new,
486
+ "/resque" => Resque::Server.new
487
+ ```
488
+
489
+ Check `examples/demo/config.ru` for a functional example (including
490
+ HTTP basic auth).
491
+
492
+ ### Rails 3
493
+
494
+ You can also mount Resque on a subpath in your existing Rails 3 app by adding `require 'resque/server'` to the top of your routes file or in an initializer then adding this to `routes.rb`:
495
+
496
+ ``` ruby
497
+ mount Resque::Server.new, :at => "/resque"
498
+ ```
499
+
500
+
501
+ Resque vs DelayedJob
502
+ --------------------
503
+
504
+ How does Resque compare to DelayedJob, and why would you choose one
505
+ over the other?
506
+
507
+ * Resque supports multiple queues
508
+ * DelayedJob supports finer grained priorities
509
+ * Resque workers are resilient to memory leaks / bloat
510
+ * DelayedJob workers are extremely simple and easy to modify
511
+ * Resque requires Redis
512
+ * DelayedJob requires ActiveRecord
513
+ * Resque can only place JSONable Ruby objects on a queue as arguments
514
+ * DelayedJob can place _any_ Ruby object on its queue as arguments
515
+ * Resque includes a Sinatra app for monitoring what's going on
516
+ * DelayedJob can be queried from within your Rails app if you want to
517
+ add an interface
518
+
519
+ If you're doing Rails development, you already have a database and
520
+ ActiveRecord. DelayedJob is super easy to setup and works great.
521
+ GitHub used it for many months to process almost 200 million jobs.
522
+
523
+ Choose Resque if:
524
+
525
+ * You need multiple queues
526
+ * You don't care / dislike numeric priorities
527
+ * You don't need to persist every Ruby object ever
528
+ * You have potentially huge queues
529
+ * You want to see what's going on
530
+ * You expect a lot of failure / chaos
531
+ * You can setup Redis
532
+ * You're not running short on RAM
533
+
534
+ Choose DelayedJob if:
535
+
536
+ * You like numeric priorities
537
+ * You're not doing a gigantic amount of jobs each day
538
+ * Your queue stays small and nimble
539
+ * There is not a lot failure / chaos
540
+ * You want to easily throw anything on the queue
541
+ * You don't want to setup Redis
542
+
543
+ In no way is Resque a "better" DelayedJob, so make sure you pick the
544
+ tool that's best for your app.
545
+
546
+ Resque Dependencies
547
+ -------------------
548
+
549
+ $ gem install bundler
550
+ $ bundle install
551
+
552
+
553
+ Installing Resque
554
+ -----------------
555
+
556
+ ### In a Rack app, as a gem
557
+
558
+ First install the gem.
559
+
560
+ $ gem install resque
561
+
562
+ Next include it in your application.
563
+
564
+ ``` ruby
565
+ require 'resque'
566
+ ```
567
+
568
+ Now start your application:
569
+
570
+ rackup config.ru
571
+
572
+ That's it! You can now create Resque jobs from within your app.
573
+
574
+ To start a worker, create a Rakefile in your app's root (or add this
575
+ to an existing Rakefile):
576
+
577
+ ``` ruby
578
+ require 'your/app'
579
+ require 'resque/tasks'
580
+ ```
581
+
582
+ Now:
583
+
584
+ $ QUEUE=* rake resque:work
585
+
586
+ Alternately you can define a `resque:setup` hook in your Rakefile if you
587
+ don't want to load your app every time rake runs.
588
+
589
+
590
+ ### In a Rails 2.x app, as a gem
591
+
592
+ First install the gem.
593
+
594
+ $ gem install resque
595
+
596
+ Next include it in your application.
597
+
598
+ $ cat config/initializers/load_resque.rb
599
+ require 'resque'
600
+
601
+ Now start your application:
602
+
603
+ $ ./script/server
604
+
605
+ That's it! You can now create Resque jobs from within your app.
606
+
607
+ To start a worker, add this to your Rakefile in `RAILS_ROOT`:
608
+
609
+ ``` ruby
610
+ require 'resque/tasks'
611
+ ```
612
+
613
+ Now:
614
+
615
+ $ QUEUE=* rake environment resque:work
616
+
617
+ Don't forget you can define a `resque:setup` hook in
618
+ `lib/tasks/whatever.rake` that loads the `environment` task every time.
619
+
620
+
621
+ ### In a Rails 2.x app, as a plugin
622
+
623
+ $ ./script/plugin install git://github.com/resque/resque
624
+
625
+ That's it! Resque will automatically be available when your Rails app
626
+ loads.
627
+
628
+ To start a worker:
629
+
630
+ $ QUEUE=* rake environment resque:work
631
+
632
+ Don't forget you can define a `resque:setup` hook in
633
+ `lib/tasks/whatever.rake` that loads the `environment` task every time.
634
+
635
+
636
+ ### In a Rails 3 app, as a gem
637
+
638
+ First include it in your Gemfile.
639
+
640
+ $ cat Gemfile
641
+ ...
642
+ gem 'resque'
643
+ ...
644
+
645
+ Next install it with Bundler.
646
+
647
+ $ bundle install
648
+
649
+ Now start your application:
650
+
651
+ $ rails server
652
+
653
+ That's it! You can now create Resque jobs from within your app.
654
+
655
+ To start a worker, add this to a file in `lib/tasks` (ex:
656
+ `lib/tasks/resque.rake`):
657
+
658
+ ``` ruby
659
+ require 'resque/tasks'
660
+ ```
661
+
662
+ Now:
663
+
664
+ $ QUEUE=* rake environment resque:work
665
+
666
+ Don't forget you can define a `resque:setup` hook in
667
+ `lib/tasks/whatever.rake` that loads the `environment` task every time.
668
+
669
+
670
+ Configuration
671
+ -------------
672
+
673
+ You may want to change the Redis host and port Resque connects to, or
674
+ set various other options at startup.
675
+
676
+ Resque has a `redis` setter which can be given a string or a Redis
677
+ object. This means if you're already using Redis in your app, Resque
678
+ can re-use the existing connection.
679
+
680
+ String: `Resque.redis = 'localhost:6379'`
681
+
682
+ Redis: `Resque.redis = $redis`
683
+
684
+ For our rails app we have a `config/initializers/resque.rb` file where
685
+ we load `config/resque.yml` by hand and set the Redis information
686
+ appropriately.
687
+
688
+ Here's our `config/resque.yml`:
689
+
690
+ development: localhost:6379
691
+ test: localhost:6379
692
+ staging: redis1.se.github.com:6379
693
+ fi: localhost:6379
694
+ production: redis1.ae.github.com:6379
695
+
696
+ And our initializer:
697
+
698
+ ``` ruby
699
+ rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..'
700
+ rails_env = ENV['RAILS_ENV'] || 'development'
701
+
702
+ resque_config = YAML.load_file(rails_root + '/config/resque.yml')
703
+ Resque.redis = resque_config[rails_env]
704
+ ```
705
+
706
+ Easy peasy! Why not just use `RAILS_ROOT` and `RAILS_ENV`? Because
707
+ this way we can tell our Sinatra app about the config file:
708
+
709
+ $ RAILS_ENV=production resque-web rails_root/config/initializers/resque.rb
710
+
711
+ Now everyone is on the same page.
712
+
713
+ Also, you could disable jobs queueing by setting 'inline' attribute.
714
+ For example, if you want to run all jobs in the same process for cucumber, try:
715
+
716
+ ``` ruby
717
+ Resque.inline = ENV['RAILS_ENV'] == "cucumber"
718
+ ```
719
+
720
+
721
+ Plugins and Hooks
722
+ -----------------
723
+
724
+ For a list of available plugins see
725
+ <http://wiki.github.com/resque/resque/plugins>.
726
+
727
+ If you'd like to write your own plugin, or want to customize Resque
728
+ using hooks (such as `Resque.after_fork`), see
729
+ [docs/HOOKS.md](http://github.com/resque/resque/blob/master/docs/HOOKS.md).
730
+
731
+
732
+ Namespaces
733
+ ----------
734
+
735
+ If you're running multiple, separate instances of Resque you may want
736
+ to namespace the keyspaces so they do not overlap. This is not unlike
737
+ the approach taken by many memcached clients.
738
+
739
+ This feature is provided by the [redis-namespace][rs] library, which
740
+ Resque uses by default to separate the keys it manages from other keys
741
+ in your Redis server.
742
+
743
+ Simply use the `Resque.redis.namespace` accessor:
744
+
745
+ ``` ruby
746
+ Resque.redis.namespace = "resque:GitHub"
747
+ ```
748
+
749
+ We recommend sticking this in your initializer somewhere after Redis
750
+ is configured.
751
+
752
+
753
+ Demo
754
+ ----
755
+
756
+ Resque ships with a demo Sinatra app for creating jobs that are later
757
+ processed in the background.
758
+
759
+ Try it out by looking at the README, found at `examples/demo/README.markdown`.
760
+
761
+
762
+ Monitoring
763
+ ----------
764
+
765
+ ### god
766
+
767
+ If you're using god to monitor Resque, we have provided example
768
+ configs in `examples/god/`. One is for starting / stopping workers,
769
+ the other is for killing workers that have been running too long.
770
+
771
+ ### monit
772
+
773
+ If you're using monit, `examples/monit/resque.monit` is provided free
774
+ of charge. This is **not** used by GitHub in production, so please
775
+ send patches for any tweaks or improvements you can make to it.
776
+
777
+
778
+ Questions
779
+ ---------
780
+
781
+ Please add them to the [FAQ](https://github.com/resque/resque/wiki/FAQ) or
782
+ ask on the Mailing List. The Mailing List is explained further below
783
+
784
+
785
+ Development
786
+ -----------
787
+
788
+ Want to hack on Resque?
789
+
790
+ First clone the repo and run the tests:
791
+
792
+ git clone git://github.com/resque/resque.git
793
+ cd resque
794
+ rake test
795
+
796
+ If the tests do not pass make sure you have Redis installed
797
+ correctly (though we make an effort to tell you if we feel this is the
798
+ case). The tests attempt to start an isolated instance of Redis to
799
+ run against.
800
+
801
+ Also make sure you've installed all the dependencies correctly. For
802
+ example, try loading the `redis-namespace` gem after you've installed
803
+ it:
804
+
805
+ $ irb
806
+ >> require 'rubygems'
807
+ => true
808
+ >> require 'redis/namespace'
809
+ => true
810
+
811
+ If you get an error requiring any of the dependencies, you may have
812
+ failed to install them or be seeing load path issues.
813
+
814
+ Feel free to ping the mailing list with your problem and we'll try to
815
+ sort it out.
816
+
817
+
818
+ Contributing
819
+ ------------
820
+
821
+ Read the [Contributing][cb] wiki page first.
822
+
823
+ Once you've made your great commits:
824
+
825
+ 1. [Fork][1] Resque
826
+ 2. Create a topic branch - `git checkout -b my_branch`
827
+ 3. Push to your branch - `git push origin my_branch`
828
+ 4. Create a [Pull Request](http://help.github.com/pull-requests/) from your branch
829
+ 5. That's it!
830
+
831
+
832
+ Mailing List
833
+ ------------
834
+
835
+ To join the list simply send an email to <resque@librelist.com>. This
836
+ will subscribe you and send you information about your subscription,
837
+ including unsubscribe information.
838
+
839
+ The archive can be found at <http://librelist.com/browser/resque/>.
840
+
841
+
842
+ Meta
843
+ ----
844
+
845
+ * Code: `git clone git://github.com/resque/resque.git`
846
+ * Home: <http://github.com/resque/resque>
847
+ * Docs: <http://resque.github.com/resque/>
848
+ * Bugs: <http://github.com/resque/resque/issues>
849
+ * List: <resque@librelist.com>
850
+ * Chat: <irc://irc.freenode.net/resque>
851
+ * Gems: <http://gemcutter.org/gems/resque>
852
+
853
+ This project uses [Semantic Versioning][sv].
854
+
855
+
856
+ Author
857
+ ------
858
+
859
+ Chris Wanstrath :: chris@ozmm.org :: @defunkt
860
+
861
+ [0]: http://github.com/blog/542-introducing-resque
862
+ [1]: http://help.github.com/forking/
863
+ [2]: http://github.com/resque/resque/issues
864
+ [sv]: http://semver.org/
865
+ [rs]: http://github.com/resque/redis-namespace
866
+ [cb]: http://wiki.github.com/resque/resque/contributing