resque 1.27.4 → 2.5.0

Sign up to get free protection for your applications and to get access to all the features.

Potentially problematic release.


This version of resque might be problematic. Click here for more details.

Files changed (45) hide show
  1. checksums.yaml +5 -5
  2. data/HISTORY.md +109 -3
  3. data/README.markdown +438 -500
  4. data/bin/resque-web +10 -26
  5. data/lib/resque/data_store.rb +52 -58
  6. data/lib/resque/errors.rb +7 -1
  7. data/lib/resque/failure/airbrake.rb +19 -7
  8. data/lib/resque/failure/multiple.rb +6 -2
  9. data/lib/resque/failure/redis.rb +1 -1
  10. data/lib/resque/failure/redis_multi_queue.rb +1 -1
  11. data/lib/resque/failure.rb +7 -0
  12. data/lib/resque/job.rb +2 -2
  13. data/lib/resque/logging.rb +1 -1
  14. data/lib/resque/railtie.rb +10 -0
  15. data/lib/resque/server/public/jquery-3.6.0.min.js +2 -0
  16. data/lib/resque/server/public/main.js +3 -0
  17. data/lib/resque/server/public/ranger.js +7 -4
  18. data/lib/resque/server/public/style.css +3 -3
  19. data/lib/resque/server/views/error.erb +1 -1
  20. data/lib/resque/server/views/failed.erb +9 -3
  21. data/lib/resque/server/views/failed_job.erb +2 -2
  22. data/lib/resque/server/views/job_class.erb +3 -1
  23. data/lib/resque/server/views/key_string.erb +1 -1
  24. data/lib/resque/server/views/layout.erb +5 -4
  25. data/lib/resque/server/views/next_more.erb +14 -14
  26. data/lib/resque/server/views/queues.erb +6 -6
  27. data/lib/resque/server/views/stats.erb +5 -5
  28. data/lib/resque/server/views/working.erb +6 -6
  29. data/lib/resque/server.rb +11 -119
  30. data/lib/resque/server_helper.rb +185 -0
  31. data/lib/resque/stat.rb +16 -9
  32. data/lib/resque/tasks.rb +3 -11
  33. data/lib/resque/thread_signal.rb +13 -34
  34. data/lib/resque/vendor/utf8_util.rb +2 -8
  35. data/lib/resque/version.rb +1 -1
  36. data/lib/resque/web_runner.rb +374 -0
  37. data/lib/resque/worker.rb +76 -48
  38. data/lib/resque.rb +100 -27
  39. data/lib/tasks/redis.rake +10 -10
  40. metadata +47 -31
  41. data/lib/resque/server/helpers.rb +0 -64
  42. data/lib/resque/server/public/jquery-1.12.4.min.js +0 -5
  43. data/lib/resque/server/test_helper.rb +0 -19
  44. data/lib/resque/vendor/utf8_util/utf8_util_18.rb +0 -91
  45. data/lib/resque/vendor/utf8_util/utf8_util_19.rb +0 -6
data/README.markdown CHANGED
@@ -2,13 +2,17 @@ Resque
2
2
  ======
3
3
 
4
4
  [![Gem Version](https://badge.fury.io/rb/resque.svg)](https://rubygems.org/gems/resque)
5
- [![Build Status](https://travis-ci.org/resque/resque.svg)](https://travis-ci.org/resque/resque)
6
- [![Coverage Status](https://coveralls.io/repos/github/resque/resque/badge.svg?branch=1-x-stable)](https://coveralls.io/r/resque/resque?branch=1-x-stable)
5
+ [![Build Status](https://github.com/resque/resque/actions/workflows/ci.yml/badge.svg)](https://github.com/resque/resque/actions/workflows/ci.yml)
6
+
7
+ Introduction
8
+ ------------
7
9
 
8
10
  Resque (pronounced like "rescue") is a Redis-backed library for creating
9
11
  background jobs, placing those jobs on multiple queues, and processing
10
12
  them later.
11
13
 
14
+ For the backstory, philosophy, and history of Resque's beginnings, please see [the blog post](http://github.com/blog/542-introducing-resque) (2009).
15
+
12
16
  Background jobs can be any Ruby class or module that responds to
13
17
  `perform`. Your existing classes can easily be converted to background
14
18
  jobs or you can create new classes specifically to do work. Or, you
@@ -21,10 +25,11 @@ three parts:
21
25
  2. A Rake task for starting a worker which processes jobs
22
26
  3. A Sinatra app for monitoring queues, jobs, and workers.
23
27
 
24
- Resque workers can be distributed between multiple machines,
25
- support priorities, are resilient to memory bloat / "leaks," are
26
- optimized for REE (but work on MRI and JRuby), tell you what they're
27
- doing, and expect failure.
28
+ Resque workers can be given multiple queues (a "queue list"),
29
+ distributed between multiple machines,
30
+ run anywhere with network access to the Redis server,
31
+ support priorities, are resilient to memory bloat / "leaks,"
32
+ tell you what they're doing, and expect failure.
28
33
 
29
34
  Resque queues are persistent; support constant time, atomic push and
30
35
  pop (thanks to Redis); provide visibility into their contents; and
@@ -34,19 +39,18 @@ The Resque frontend tells you what workers are doing, what workers are
34
39
  not doing, what queues you're using, what's in those queues, provides
35
40
  general usage stats, and helps you track failures.
36
41
 
42
+ Resque now supports Ruby 2.3.0 and above.
43
+ We will also only be supporting Redis 3.0 and above going forward.
37
44
 
38
- The Blog Post
39
- -------------
40
-
41
- For the backstory, philosophy, and history of Resque's beginnings,
42
- please see [the blog post][0].
43
-
45
+ ### Note on the future of Resque
44
46
 
45
- Overview
46
- --------
47
+ Would you like to be involved in Resque? Do you have thoughts about what
48
+ Resque should be and do going forward? There's currently an [open discussion here](https://github.com/resque/resque/issues/1759)
49
+ on just that topic, so please feel free to join in. We'd love to hear your thoughts
50
+ and/or have people volunteer to be a part of the project!
47
51
 
48
- Resque allows you to create jobs and place them on a queue, then,
49
- later, pull those jobs off the queue and process them.
52
+ Example
53
+ -------
50
54
 
51
55
  Resque jobs are Ruby classes (or modules) which respond to the
52
56
  `perform` method. Here's an example:
@@ -106,10 +110,216 @@ This starts one Resque worker and tells it to work off the
106
110
  find any more, at which point it will sleep for a small period and
107
111
  repeatedly poll the queue for more jobs.
108
112
 
109
- Workers can be given multiple queues (a "queue list") and run on
110
- multiple machines. In fact they can be run anywhere with network
111
- access to the Redis server.
113
+ Installation
114
+ ------------
115
+
116
+ Add the gem to your Gemfile:
117
+
118
+ gem 'resque'
119
+
120
+ Next, install it with Bundler:
121
+
122
+ $ bundle
123
+
124
+ #### Rack
125
+
126
+ In your Rakefile, or some other file in `lib/tasks` (ex: `lib/tasks/resque.rake`), load the resque rake tasks:
127
+
128
+ ``` ruby
129
+ require 'resque'
130
+ require 'resque/tasks'
131
+ require 'your/app' # Include this line if you want your workers to have access to your application
132
+ ```
133
+
134
+ #### Rails
135
+
136
+ To make resque specific changes, you can override the `resque:setup` job in `lib/tasks` (ex: `lib/tasks/resque.rake`). GitHub's setup task looks like this:
137
+
138
+ ``` ruby
139
+ task "resque:setup" => :environment do
140
+ Grit::Git.git_timeout = 10.minutes
141
+ end
142
+ ```
143
+
144
+ We don't want the `git_timeout` as high as 10 minutes in our web app,
145
+ but in the Resque workers it's fine.
146
+
147
+ Running Workers
148
+ ---------------
149
+
150
+ Resque workers are rake tasks that run forever. They basically do this:
151
+
152
+ ``` ruby
153
+ start
154
+ loop do
155
+ if job = reserve
156
+ job.process
157
+ else
158
+ sleep 5 # Polling frequency = 5
159
+ end
160
+ end
161
+ shutdown
162
+ ```
163
+
164
+ Starting a worker is simple:
165
+
166
+ $ QUEUE=* rake resque:work
167
+
168
+ Or, you can start multiple workers:
169
+
170
+ $ COUNT=2 QUEUE=* rake resque:workers
171
+
172
+ This will spawn two Resque workers, each in its own process. Hitting
173
+ ctrl-c should be sufficient to stop them all.
174
+
175
+ #### Priorities and Queue Lists
176
+
177
+ Resque doesn't support numeric priorities but instead uses the order
178
+ of queues you give it. We call this list of queues the "queue list."
179
+
180
+ Let's say we add a `warm_cache` queue in addition to our `file_serve`
181
+ queue. We'd now start a worker like so:
182
+
183
+ $ QUEUES=file_serve,warm_cache rake resque:work
184
+
185
+ When the worker looks for new jobs, it will first check
186
+ `file_serve`. If it finds a job, it'll process it then check
187
+ `file_serve` again. It will keep checking `file_serve` until no more
188
+ jobs are available. At that point, it will check `warm_cache`. If it
189
+ finds a job it'll process it then check `file_serve` (repeating the
190
+ whole process).
191
+
192
+ In this way you can prioritize certain queues. At GitHub we start our
193
+ workers with something like this:
194
+
195
+ $ QUEUES=critical,archive,high,low rake resque:work
196
+
197
+ Notice the `archive` queue - it is specialized and in our future
198
+ architecture will only be run from a single machine.
199
+
200
+ At that point we'll start workers on our generalized background
201
+ machines with this command:
202
+
203
+ $ QUEUES=critical,high,low rake resque:work
204
+
205
+ And workers on our specialized archive machine with this command:
206
+
207
+ $ QUEUE=archive rake resque:work
208
+
209
+ #### Running All Queues
210
+
211
+ If you want your workers to work off of every queue, including new
212
+ queues created on the fly, you can use a splat:
213
+
214
+ $ QUEUE=* rake resque:work
215
+
216
+ Queues will be processed in alphabetical order.
217
+
218
+ Or, prioritize some queues above `*`:
219
+
220
+ # QUEUE=critical,* rake resque:work
221
+
222
+ #### Running All Queues Except for Some
223
+
224
+ If you want your workers to work off of all queues except for some,
225
+ you can use negation:
226
+
227
+ $ QUEUE=*,!low rake resque:work
228
+
229
+ Negated globs also work. The following will instruct workers to work
230
+ off of all queues except those beginning with `file_`:
231
+
232
+ $ QUEUE=*,!file_* rake resque:work
233
+
234
+ Note that the order in which negated queues are specified does not
235
+ matter, so `QUEUE=*,!file_*` and `QUEUE=!file_*,*` will have the same
236
+ effect.
237
+
238
+ #### Process IDs (PIDs)
239
+
240
+ There are scenarios where it's helpful to record the PID of a resque
241
+ worker process. Use the PIDFILE option for easy access to the PID:
242
+
243
+ $ PIDFILE=./resque.pid QUEUE=file_serve rake resque:work
244
+
245
+ #### Running in the background
246
+
247
+ There are scenarios where it's helpful for
248
+ the resque worker to run itself in the background (usually in combination with
249
+ PIDFILE). Use the BACKGROUND option so that rake will return as soon as the
250
+ worker is started.
251
+
252
+ $ PIDFILE=./resque.pid BACKGROUND=yes QUEUE=file_serve rake resque:work
253
+
254
+ #### Polling frequency
255
+
256
+ You can pass an INTERVAL option which is a float representing the polling frequency.
257
+ The default is 5 seconds, but for a semi-active app you may want to use a smaller value.
258
+
259
+ $ INTERVAL=0.1 QUEUE=file_serve rake resque:work
260
+
261
+ The Front End
262
+ -------------
263
+
264
+ Resque comes with a Sinatra-based front end for seeing what's up with
265
+ your queue.
266
+
267
+ ![The Front End](https://camo.githubusercontent.com/64d150a243987ffbc33f588bd6d7722a0bb8d69a/687474703a2f2f7475746f7269616c732e6a756d7073746172746c61622e636f6d2f696d616765732f7265737175655f6f766572766965772e706e67)
268
+
269
+ #### Standalone
270
+
271
+ If you've installed Resque as a gem running the front end standalone is easy:
272
+
273
+ $ resque-web
274
+
275
+ It's a thin layer around `rackup` so it's configurable as well:
276
+
277
+ $ resque-web -p 8282
278
+
279
+ If you have a Resque config file you want evaluated just pass it to
280
+ the script as the final argument:
281
+
282
+ $ resque-web -p 8282 rails_root/config/initializers/resque.rb
283
+
284
+ You can also set the namespace directly using `resque-web`:
285
+
286
+ $ resque-web -p 8282 -N myapp
287
+
288
+ or set the Redis connection string if you need to do something like select a different database:
289
+
290
+ $ resque-web -p 8282 -r localhost:6379:2
291
+
292
+ #### Passenger
293
+
294
+ Using Passenger? Resque ships with a `config.ru` you can use. See
295
+ Phusion's guide:
296
+
297
+ Apache: <https://www.phusionpassenger.com/library/deploy/apache/deploy/ruby/>
298
+ Nginx: <https://www.phusionpassenger.com/library/deploy/nginx/deploy/ruby/>
299
+
300
+ #### Rack::URLMap
301
+
302
+ If you want to load Resque on a subpath, possibly alongside other
303
+ apps, it's easy to do with Rack's `URLMap`:
304
+
305
+ ``` ruby
306
+ require 'resque/server'
307
+
308
+ run Rack::URLMap.new \
309
+ "/" => Your::App.new,
310
+ "/resque" => Resque::Server.new
311
+ ```
312
+
313
+ Check `examples/demo/config.ru` for a functional example (including
314
+ HTTP basic auth).
315
+
316
+ #### Rails
317
+
318
+ You can also mount Resque on a subpath in your existing Rails app by adding `require 'resque/server'` to the top of your routes file or in an initializer then adding this to `routes.rb`:
112
319
 
320
+ ``` ruby
321
+ mount Resque::Server.new, :at => "/resque"
322
+ ```
113
323
 
114
324
  Jobs
115
325
  ----
@@ -136,8 +346,7 @@ mention "foreground" and "background" because they make conceptual
136
346
  sense. You could easily be spidering sites and sticking data which
137
347
  needs to be crunched later into a queue.
138
348
 
139
-
140
- ### Persistence
349
+ #### Persistence
141
350
 
142
351
  Jobs are persisted to queues as JSON objects. Let's take our `Archive`
143
352
  example from above. We'll run the following code to create a job:
@@ -182,7 +391,7 @@ If your jobs were run against marshaled objects, they could
182
391
  potentially be operating on a stale record with out-of-date information.
183
392
 
184
393
 
185
- ### send_later / async
394
+ #### send_later / async
186
395
 
187
396
  Want something like DelayedJob's `send_later` or the ability to use
188
397
  instance methods instead of just methods for jobs? See the `examples/`
@@ -191,12 +400,12 @@ directory for goodies.
191
400
  We plan to provide first class `async` support in a future release.
192
401
 
193
402
 
194
- ### Failure
403
+ #### Failure
195
404
 
196
405
  If a job raises an exception, it is logged and handed off to the
197
406
  `Resque::Failure` module. Failures are logged either locally in Redis
198
407
  or using some different backend. To see exceptions while developing,
199
- use VERBOSE env variable, see details below under Logging.
408
+ see details below under Logging.
200
409
 
201
410
  For example, Resque ships with Airbrake support. To configure it, put
202
411
  the following into an initialisation file or into your rake job:
@@ -215,64 +424,98 @@ Keep this in mind when writing your jobs: you may want to throw
215
424
  exceptions you would not normally throw in order to assist debugging.
216
425
 
217
426
 
218
- Workers
219
- -------
427
+ #### Rails example
220
428
 
221
- Resque workers are rake tasks that run forever. They basically do this:
429
+ If you are using ActiveJob here's how your job definition will look:
222
430
 
223
431
  ``` ruby
224
- start
225
- loop do
226
- if job = reserve
227
- job.process
228
- else
229
- sleep 5 # Polling frequency = 5
432
+ class ArchiveJob < ApplicationJob
433
+ queue_as :file_serve
434
+
435
+ def perform(repo_id, branch = 'master')
436
+ repo = Repository.find(repo_id)
437
+ repo.create_archive(branch)
438
+ end
439
+ end
440
+ ```
441
+
442
+ ``` ruby
443
+ class Repository
444
+ def async_create_archive(branch)
445
+ ArchiveJob.perform_later(self.id, branch)
230
446
  end
231
447
  end
232
- shutdown
233
448
  ```
234
449
 
235
- Starting a worker is simple. Here's our example from earlier:
450
+ It is important to run `ArchiveJob.perform_later(self.id, branch)` rather than `Resque.enqueue(Archive, self.id, branch)`.
451
+ Otherwise Resque will process the job without actually doing anything.
452
+ Even if you put an obviously buggy line like `0/0` in the `perform` method,
453
+ the job will still succeed.
236
454
 
237
- $ QUEUE=file_serve rake resque:work
238
455
 
239
- By default Resque won't know about your application's
240
- environment. That is, it won't be able to find and run your jobs - it
241
- needs to load your application into memory.
456
+ Configuration
457
+ -------------
458
+
459
+ #### Redis
460
+
461
+ You may want to change the Redis host and port Resque connects to, or
462
+ set various other options at startup.
463
+
464
+ Resque has a `redis` setter which can be given a string or a Redis
465
+ object. This means if you're already using Redis in your app, Resque
466
+ can re-use the existing connection.
467
+
468
+ String: `Resque.redis = 'localhost:6379'`
469
+
470
+ Redis: `Resque.redis = $redis`
471
+
472
+ For our rails app we have a `config/initializers/resque.rb` file where
473
+ we load `config/resque.yml` by hand and set the Redis information
474
+ appropriately.
242
475
 
243
- If we've installed Resque as a Rails plugin, we might run this command
244
- from our RAILS_ROOT:
476
+ Here's our `config/resque.yml`:
245
477
 
246
- $ QUEUE=file_serve rake environment resque:work
478
+ development: localhost:6379
479
+ test: localhost:6379
480
+ staging: redis1.se.github.com:6379
481
+ fi: localhost:6379
482
+ production: <%= ENV['REDIS_URL'] %>
247
483
 
248
- This will load the environment before starting a worker. Alternately
249
- we can define a `resque:setup` task with a dependency on the
250
- `environment` rake task:
484
+ And our initializer:
251
485
 
252
486
  ``` ruby
253
- task "resque:setup" => :environment
487
+ rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..'
488
+ rails_env = ENV['RAILS_ENV'] || 'development'
489
+ config_file = rails_root + '/config/resque.yml'
490
+
491
+ resque_config = YAML::load(ERB.new(IO.read(config_file)).result)
492
+ Resque.redis = resque_config[rails_env]
254
493
  ```
255
494
 
256
- GitHub's setup task looks like this:
495
+ Easy peasy! Why not just use `RAILS_ROOT` and `RAILS_ENV`? Because
496
+ this way we can tell our Sinatra app about the config file:
497
+
498
+ $ RAILS_ENV=production resque-web rails_root/config/initializers/resque.rb
499
+
500
+ Now everyone is on the same page.
501
+
502
+ Also, you could disable jobs queueing by setting 'inline' attribute.
503
+ For example, if you want to run all jobs in the same process for cucumber, try:
257
504
 
258
505
  ``` ruby
259
- task "resque:setup" => :environment do
260
- Grit::Git.git_timeout = 10.minutes
261
- end
506
+ Resque.inline = ENV['RAILS_ENV'] == "cucumber"
262
507
  ```
263
508
 
264
- We don't want the `git_timeout` as high as 10 minutes in our web app,
265
- but in the Resque workers it's fine.
266
-
509
+ #### Logging
267
510
 
268
- ### Logging
511
+ Workers support basic logging to STDOUT.
269
512
 
270
- Workers support basic logging to STDOUT. If you start them with the
271
- `VERBOSE` env variable set, they will print basic debugging
272
- information. You can also set the `VVERBOSE` (very verbose) env
273
- variable.
513
+ You can control the logging threshold using `Resque.logger.level`:
274
514
 
275
- $ VVERBOSE=1 QUEUE=file_serve rake environment resque:work
515
+ ```ruby
516
+ # config/initializers/resque.rb
517
+ Resque.logger.level = Logger::DEBUG
518
+ ```
276
519
 
277
520
  If you want Resque to log to a file, in Rails do:
278
521
 
@@ -281,91 +524,111 @@ If you want Resque to log to a file, in Rails do:
281
524
  Resque.logger = Logger.new(Rails.root.join('log', "#{Rails.env}_resque.log"))
282
525
  ```
283
526
 
284
- ### Process IDs (PIDs)
285
-
286
- There are scenarios where it's helpful to record the PID of a resque
287
- worker process. Use the PIDFILE option for easy access to the PID:
527
+ #### Namespaces
288
528
 
289
- $ PIDFILE=./resque.pid QUEUE=file_serve rake environment resque:work
290
-
291
- ### Running in the background
529
+ If you're running multiple, separate instances of Resque you may want
530
+ to namespace the keyspaces so they do not overlap. This is not unlike
531
+ the approach taken by many memcached clients.
292
532
 
293
- (Only supported with ruby >= 1.9). There are scenarios where it's helpful for
294
- the resque worker to run itself in the background (usually in combination with
295
- PIDFILE). Use the BACKGROUND option so that rake will return as soon as the
296
- worker is started.
533
+ This feature is provided by the [redis-namespace](http://github.com/resque/redis-namespace) library, which
534
+ Resque uses by default to separate the keys it manages from other keys
535
+ in your Redis server.
297
536
 
298
- $ PIDFILE=./resque.pid BACKGROUND=yes QUEUE=file_serve \
299
- rake environment resque:work
537
+ Simply use the `Resque.redis.namespace` accessor:
300
538
 
301
- ### Polling frequency
302
-
303
- You can pass an INTERVAL option which is a float representing the polling frequency.
304
- The default is 5 seconds, but for a semi-active app you may want to use a smaller value.
539
+ ``` ruby
540
+ Resque.redis.namespace = "resque:GitHub"
541
+ ```
305
542
 
306
- $ INTERVAL=0.1 QUEUE=file_serve rake environment resque:work
543
+ We recommend sticking this in your initializer somewhere after Redis
544
+ is configured.
307
545
 
308
- ### Priorities and Queue Lists
546
+ #### Storing Statistics
547
+ Resque allows to store count of processed and failed jobs.
309
548
 
310
- Resque doesn't support numeric priorities but instead uses the order
311
- of queues you give it. We call this list of queues the "queue list."
549
+ By default it will store it in Redis using the keys `stats:processed` and `stats:failed`.
312
550
 
313
- Let's say we add a `warm_cache` queue in addition to our `file_serve`
314
- queue. We'd now start a worker like so:
551
+ Some apps would want another stats store, or even a null store:
315
552
 
316
- $ QUEUES=file_serve,warm_cache rake resque:work
317
-
318
- When the worker looks for new jobs, it will first check
319
- `file_serve`. If it finds a job, it'll process it then check
320
- `file_serve` again. It will keep checking `file_serve` until no more
321
- jobs are available. At that point, it will check `warm_cache`. If it
322
- finds a job it'll process it then check `file_serve` (repeating the
323
- whole process).
324
-
325
- In this way you can prioritize certain queues. At GitHub we start our
326
- workers with something like this:
553
+ ```ruby
554
+ # config/initializers/resque.rb
555
+ class NullDataStore
556
+ def stat(stat)
557
+ 0
558
+ end
327
559
 
328
- $ QUEUES=critical,archive,high,low rake resque:work
560
+ def increment_stat(stat, by)
561
+ end
329
562
 
330
- Notice the `archive` queue - it is specialized and in our future
331
- architecture will only be run from a single machine.
563
+ def decrement_stat(stat, by)
564
+ end
332
565
 
333
- At that point we'll start workers on our generalized background
334
- machines with this command:
566
+ def clear_stat(stat)
567
+ end
568
+ end
335
569
 
336
- $ QUEUES=critical,high,low rake resque:work
570
+ Resque.stat_data_store = NullDataStore.new
571
+ ```
337
572
 
338
- And workers on our specialized archive machine with this command:
573
+ Plugins and Hooks
574
+ -----------------
339
575
 
340
- $ QUEUE=archive rake resque:work
576
+ For a list of available plugins see
577
+ <https://github.com/resque/resque/wiki/plugins>.
341
578
 
579
+ If you'd like to write your own plugin, or want to customize Resque
580
+ using hooks (such as `Resque.after_fork`), see
581
+ [docs/HOOKS.md](http://github.com/resque/resque/blob/master/docs/HOOKS.md).
342
582
 
343
- ### Running All Queues
344
583
 
345
- If you want your workers to work off of every queue, including new
346
- queues created on the fly, you can use a splat:
584
+ Additional Information
585
+ ----------------------
347
586
 
348
- $ QUEUE=* rake resque:work
587
+ #### Resque vs DelayedJob
349
588
 
350
- Queues will be processed in alphabetical order.
589
+ How does Resque compare to DelayedJob, and why would you choose one
590
+ over the other?
351
591
 
592
+ * Resque supports multiple queues
593
+ * DelayedJob supports finer grained priorities
594
+ * Resque workers are resilient to memory leaks / bloat
595
+ * DelayedJob workers are extremely simple and easy to modify
596
+ * Resque requires Redis
597
+ * DelayedJob requires ActiveRecord
598
+ * Resque can only place JSONable Ruby objects on a queue as arguments
599
+ * DelayedJob can place _any_ Ruby object on its queue as arguments
600
+ * Resque includes a Sinatra app for monitoring what's going on
601
+ * DelayedJob can be queried from within your Rails app if you want to
602
+ add an interface
352
603
 
353
- ### Running Multiple Workers
604
+ If you're doing Rails development, you already have a database and
605
+ ActiveRecord. DelayedJob is super easy to setup and works great.
606
+ GitHub used it for many months to process almost 200 million jobs.
354
607
 
355
- At GitHub we use god to start and stop multiple workers. A sample god
356
- configuration file is included under `examples/god`. We recommend this
357
- method.
608
+ Choose Resque if:
358
609
 
359
- If you'd like to run multiple workers in development mode, you can do
360
- so using the `resque:workers` rake task:
610
+ * You need multiple queues
611
+ * You don't care / dislike numeric priorities
612
+ * You don't need to persist every Ruby object ever
613
+ * You have potentially huge queues
614
+ * You want to see what's going on
615
+ * You expect a lot of failure / chaos
616
+ * You can setup Redis
617
+ * You're not running short on RAM
361
618
 
362
- $ COUNT=5 QUEUE=* rake resque:workers
619
+ Choose DelayedJob if:
363
620
 
364
- This will spawn five Resque workers, each in its own process. Hitting
365
- ctrl-c should be sufficient to stop them all.
621
+ * You like numeric priorities
622
+ * You're not doing a gigantic amount of jobs each day
623
+ * Your queue stays small and nimble
624
+ * There is not a lot failure / chaos
625
+ * You want to easily throw anything on the queue
626
+ * You don't want to setup Redis
366
627
 
628
+ In no way is Resque a "better" DelayedJob, so make sure you pick the
629
+ tool that's best for your app.
367
630
 
368
- ### Forking
631
+ #### Forking
369
632
 
370
633
  On certain platforms, when a Resque worker reserves a job it
371
634
  immediately forks a child process. The child processes the job then
@@ -406,8 +669,13 @@ complicated.
406
669
 
407
670
  Workers instead handle their own state.
408
671
 
672
+ #### `at_exit` Callbacks
673
+
674
+ Resque uses `Kernel#exit!` for exiting its workers' child processes. So any `at_exit` callback defined in your application won't be executed when the job is finished and the child process exits.
675
+
676
+ You can alter this behavior by setting the `RUN_AT_EXIT_HOOKS` environment variable.
409
677
 
410
- ### Parents and Children
678
+ #### Parents and Children
411
679
 
412
680
  Here's a parent / child pair doing some work:
413
681
 
@@ -428,7 +696,7 @@ waiting for work on:
428
696
  92099 resque: Waiting for file_serve,warm_cache
429
697
 
430
698
 
431
- ### Signals
699
+ #### Signals
432
700
 
433
701
  Resque workers respond to a few different signals:
434
702
 
@@ -448,9 +716,9 @@ If you want to kill a stale or stuck child and shutdown, use `TERM`
448
716
 
449
717
  If you want to stop processing jobs, but want to leave the worker running
450
718
  (for example, to temporarily alleviate load), use `USR2` to stop processing,
451
- then `CONT` to start it again.
719
+ then `CONT` to start it again. It's also possible to [pause all workers](#pausing-all-workers).
452
720
 
453
- #### Signals on Heroku
721
+ #### Heroku
454
722
 
455
723
  When shutting down processes, Heroku sends every process a TERM signal at the
456
724
  same time. By default this causes an immediate shutdown of any running job
@@ -458,6 +726,11 @@ leading to frequent `Resque::TermException` errors. For short running jobs, a s
458
726
  solution is to give a small amount of time for the job to finish
459
727
  before killing it.
460
728
 
729
+ Resque doesn't handle this out of the box (for both cedar-14 and heroku-16), you need to
730
+ install the [`resque-heroku-signals`](https://github.com/iloveitaly/resque-heroku-signals)
731
+ addon which adds the required signal handling to make the behavior described above work.
732
+ Related issue: https://github.com/resque/resque/issues/1559
733
+
461
734
  To accomplish this set the following environment variables:
462
735
 
463
736
  * `RESQUE_PRE_SHUTDOWN_TIMEOUT` - The time between the parent receiving a shutdown signal (TERM by default) and it sending that signal on to the child process. Designed to give the child process
@@ -467,11 +740,41 @@ time to complete before being forced to die.
467
740
 
468
741
  * `RESQUE_TERM_TIMEOUT` - By default you have a few seconds to handle `Resque::TermException` in your job. `RESQUE_TERM_TIMEOUT` and `RESQUE_PRE_SHUTDOWN_TIMEOUT` must be lower than the [heroku dyno timeout](https://devcenter.heroku.com/articles/limits#exit-timeout).
469
742
 
470
- ### Mysql::Error: MySQL server has gone away
743
+ #### Pausing all workers
744
+
745
+ Workers will not process pending jobs if the Redis key `pause-all-workers` is set with the string value "true".
746
+
747
+ ``` ruby
748
+ Resque.redis.set('pause-all-workers', 'true')
749
+ ```
750
+
751
+ Nothing happens to jobs that are already being processed by workers.
752
+
753
+ Unpause by removing the Redis key `pause-all-workers`.
754
+
755
+ ``` ruby
756
+ Resque.redis.del('pause-all-workers')
757
+ ```
758
+
759
+ #### Monitoring
760
+
761
+ ##### god
762
+
763
+ If you're using god to monitor Resque, we have provided example
764
+ configs in `examples/god/`. One is for starting / stopping workers,
765
+ the other is for killing workers that have been running too long.
766
+
767
+ ##### monit
768
+
769
+ If you're using monit, `examples/monit/resque.monit` is provided free
770
+ of charge. This is **not** used by GitHub in production, so please
771
+ send patches for any tweaks or improvements you can make to it.
772
+
773
+ #### Mysql::Error: MySQL server has gone away
471
774
 
472
775
  If your workers remain idle for too long they may lose their MySQL connection. Depending on your version of Rails, we recommend the following:
473
776
 
474
- #### Rails 3.x
777
+ ##### Rails
475
778
  In your `perform` method, add the following line:
476
779
 
477
780
  ``` ruby
@@ -487,7 +790,7 @@ The Rails doc says the following about `verify_active_connections!`:
487
790
 
488
791
  Verify active connections and remove and disconnect connections associated with stale threads.
489
792
 
490
- #### Rails 4.x
793
+ ##### Rails 4.x
491
794
 
492
795
  In your `perform` method, instead of `verify_active_connections!`, use:
493
796
 
@@ -504,362 +807,6 @@ From the Rails docs on [`clear_active_connections!`](http://api.rubyonrails.org/
504
807
 
505
808
  Returns any connections in use by the current thread back to the pool, and also returns connections to the pool cached by threads that are no longer alive.
506
809
 
507
-
508
-
509
- The Front End
510
- -------------
511
-
512
- Resque comes with a Sinatra-based front end for seeing what's up with
513
- your queue.
514
-
515
- ![The Front End](https://camo.githubusercontent.com/64d150a243987ffbc33f588bd6d7722a0bb8d69a/687474703a2f2f7475746f7269616c732e6a756d7073746172746c61622e636f6d2f696d616765732f7265737175655f6f766572766965772e706e67)
516
-
517
- ### Standalone
518
-
519
- If you've installed Resque as a gem running the front end standalone is easy:
520
-
521
- $ resque-web
522
-
523
- It's a thin layer around `rackup` so it's configurable as well:
524
-
525
- $ resque-web -p 8282
526
-
527
- If you have a Resque config file you want evaluated just pass it to
528
- the script as the final argument:
529
-
530
- $ resque-web -p 8282 rails_root/config/initializers/resque.rb
531
-
532
- You can also set the namespace directly using `resque-web`:
533
-
534
- $ resque-web -p 8282 -N myapp
535
-
536
- or set the Redis connection string if you need to do something like select a different database:
537
-
538
- $ resque-web -p 8282 -r localhost:6379:2
539
-
540
- ### Passenger
541
-
542
- Using Passenger? Resque ships with a `config.ru` you can use. See
543
- Phusion's guide:
544
-
545
- Apache: <https://www.phusionpassenger.com/library/deploy/apache/deploy/ruby/>
546
- Nginx: <https://www.phusionpassenger.com/library/deploy/nginx/deploy/ruby/>
547
-
548
- ### Rack::URLMap
549
-
550
- If you want to load Resque on a subpath, possibly alongside other
551
- apps, it's easy to do with Rack's `URLMap`:
552
-
553
- ``` ruby
554
- require 'resque/server'
555
-
556
- run Rack::URLMap.new \
557
- "/" => Your::App.new,
558
- "/resque" => Resque::Server.new
559
- ```
560
-
561
- Check `examples/demo/config.ru` for a functional example (including
562
- HTTP basic auth).
563
-
564
- ### Rails 3
565
-
566
- You can also mount Resque on a subpath in your existing Rails 3 app by adding `require 'resque/server'` to the top of your routes file or in an initializer then adding this to `routes.rb`:
567
-
568
- ``` ruby
569
- mount Resque::Server.new, :at => "/resque"
570
- ```
571
-
572
-
573
- Resque vs DelayedJob
574
- --------------------
575
-
576
- How does Resque compare to DelayedJob, and why would you choose one
577
- over the other?
578
-
579
- * Resque supports multiple queues
580
- * DelayedJob supports finer grained priorities
581
- * Resque workers are resilient to memory leaks / bloat
582
- * DelayedJob workers are extremely simple and easy to modify
583
- * Resque requires Redis
584
- * DelayedJob requires ActiveRecord
585
- * Resque can only place JSONable Ruby objects on a queue as arguments
586
- * DelayedJob can place _any_ Ruby object on its queue as arguments
587
- * Resque includes a Sinatra app for monitoring what's going on
588
- * DelayedJob can be queried from within your Rails app if you want to
589
- add an interface
590
-
591
- If you're doing Rails development, you already have a database and
592
- ActiveRecord. DelayedJob is super easy to setup and works great.
593
- GitHub used it for many months to process almost 200 million jobs.
594
-
595
- Choose Resque if:
596
-
597
- * You need multiple queues
598
- * You don't care / dislike numeric priorities
599
- * You don't need to persist every Ruby object ever
600
- * You have potentially huge queues
601
- * You want to see what's going on
602
- * You expect a lot of failure / chaos
603
- * You can setup Redis
604
- * You're not running short on RAM
605
-
606
- Choose DelayedJob if:
607
-
608
- * You like numeric priorities
609
- * You're not doing a gigantic amount of jobs each day
610
- * Your queue stays small and nimble
611
- * There is not a lot failure / chaos
612
- * You want to easily throw anything on the queue
613
- * You don't want to setup Redis
614
-
615
- In no way is Resque a "better" DelayedJob, so make sure you pick the
616
- tool that's best for your app.
617
-
618
- Resque Dependencies
619
- -------------------
620
-
621
- $ gem install bundler
622
- $ bundle install
623
-
624
-
625
- Installing Resque
626
- -----------------
627
-
628
- ### In a Rack app, as a gem
629
-
630
- First install the gem.
631
-
632
- $ gem install resque
633
-
634
- Next include it in your application.
635
-
636
- ``` ruby
637
- require 'resque'
638
- ```
639
-
640
- Now start your application:
641
-
642
- rackup config.ru
643
-
644
- That's it! You can now create Resque jobs from within your app.
645
-
646
- To start a worker, create a Rakefile in your app's root (or add this
647
- to an existing Rakefile):
648
-
649
- ``` ruby
650
- require 'your/app'
651
- require 'resque/tasks'
652
- ```
653
-
654
- If you're using Rails 5.x, include the following in lib/tasks/resque.rb:
655
-
656
- ```ruby
657
- require 'resque/tasks'
658
- task 'resque:setup' => :environment
659
- ```
660
-
661
- Now:
662
-
663
- $ QUEUE=* rake resque:work
664
-
665
- Alternately you can define a `resque:setup` hook in your Rakefile if you
666
- don't want to load your app every time rake runs.
667
-
668
-
669
- ### In a Rails 2.x app, as a gem
670
-
671
- First install the gem.
672
-
673
- $ gem install resque
674
-
675
- Next include it in your application.
676
-
677
- $ cat config/initializers/load_resque.rb
678
- require 'resque'
679
-
680
- Now start your application:
681
-
682
- $ ./script/server
683
-
684
- That's it! You can now create Resque jobs from within your app.
685
-
686
- To start a worker, add this to your Rakefile in `RAILS_ROOT`:
687
-
688
- ``` ruby
689
- require 'resque/tasks'
690
- ```
691
-
692
- Now:
693
-
694
- $ QUEUE=* rake environment resque:work
695
-
696
- Don't forget you can define a `resque:setup` hook in
697
- `lib/tasks/whatever.rake` that loads the `environment` task every time.
698
-
699
-
700
- ### In a Rails 2.x app, as a plugin
701
-
702
- $ ./script/plugin install git://github.com/resque/resque
703
-
704
- That's it! Resque will automatically be available when your Rails app
705
- loads.
706
-
707
- To start a worker:
708
-
709
- $ QUEUE=* rake environment resque:work
710
-
711
- Don't forget you can define a `resque:setup` hook in
712
- `lib/tasks/whatever.rake` that loads the `environment` task every time.
713
-
714
-
715
- ### In a Rails 3.x or 4.x app, as a gem
716
-
717
- First include it in your Gemfile.
718
-
719
- $ cat Gemfile
720
- ...
721
- gem 'resque'
722
- ...
723
-
724
- Next install it with Bundler.
725
-
726
- $ bundle install
727
-
728
- Now start your application:
729
-
730
- $ rails server
731
-
732
- That's it! You can now create Resque jobs from within your app.
733
-
734
- To start a worker, add this to a file in `lib/tasks` (ex:
735
- `lib/tasks/resque.rake`):
736
-
737
- ``` ruby
738
- require 'resque/tasks'
739
- ```
740
-
741
- Now:
742
-
743
- $ QUEUE=* rake environment resque:work
744
-
745
- Don't forget you can define a `resque:setup` hook in
746
- `lib/tasks/whatever.rake` that loads the `environment` task every time.
747
-
748
-
749
- Configuration
750
- -------------
751
-
752
- You may want to change the Redis host and port Resque connects to, or
753
- set various other options at startup.
754
-
755
- Resque has a `redis` setter which can be given a string or a Redis
756
- object. This means if you're already using Redis in your app, Resque
757
- can re-use the existing connection.
758
-
759
- String: `Resque.redis = 'localhost:6379'`
760
-
761
- Redis: `Resque.redis = $redis`
762
-
763
- For our rails app we have a `config/initializers/resque.rb` file where
764
- we load `config/resque.yml` by hand and set the Redis information
765
- appropriately.
766
-
767
- Here's our `config/resque.yml`:
768
-
769
- development: localhost:6379
770
- test: localhost:6379
771
- staging: redis1.se.github.com:6379
772
- fi: localhost:6379
773
- production: redis1.ae.github.com:6379
774
-
775
- And our initializer:
776
-
777
- ``` ruby
778
- rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..'
779
- rails_env = ENV['RAILS_ENV'] || 'development'
780
-
781
- resque_config = YAML.load_file(rails_root + '/config/resque.yml')
782
- Resque.redis = resque_config[rails_env]
783
- ```
784
-
785
- Easy peasy! Why not just use `RAILS_ROOT` and `RAILS_ENV`? Because
786
- this way we can tell our Sinatra app about the config file:
787
-
788
- $ RAILS_ENV=production resque-web rails_root/config/initializers/resque.rb
789
-
790
- Now everyone is on the same page.
791
-
792
- Also, you could disable jobs queueing by setting 'inline' attribute.
793
- For example, if you want to run all jobs in the same process for cucumber, try:
794
-
795
- ``` ruby
796
- Resque.inline = ENV['RAILS_ENV'] == "cucumber"
797
- ```
798
-
799
-
800
- Plugins and Hooks
801
- -----------------
802
-
803
- For a list of available plugins see
804
- <http://wiki.github.com/resque/resque/plugins>.
805
-
806
- If you'd like to write your own plugin, or want to customize Resque
807
- using hooks (such as `Resque.after_fork`), see
808
- [docs/HOOKS.md](http://github.com/resque/resque/blob/master/docs/HOOKS.md).
809
-
810
-
811
- Namespaces
812
- ----------
813
-
814
- If you're running multiple, separate instances of Resque you may want
815
- to namespace the keyspaces so they do not overlap. This is not unlike
816
- the approach taken by many memcached clients.
817
-
818
- This feature is provided by the [redis-namespace][rs] library, which
819
- Resque uses by default to separate the keys it manages from other keys
820
- in your Redis server.
821
-
822
- Simply use the `Resque.redis.namespace` accessor:
823
-
824
- ``` ruby
825
- Resque.redis.namespace = "resque:GitHub"
826
- ```
827
-
828
- We recommend sticking this in your initializer somewhere after Redis
829
- is configured.
830
-
831
-
832
- Demo
833
- ----
834
-
835
- Resque ships with a demo Sinatra app for creating jobs that are later
836
- processed in the background.
837
-
838
- Try it out by looking at the README, found at `examples/demo/README.markdown`.
839
-
840
-
841
- Monitoring
842
- ----------
843
-
844
- ### god
845
-
846
- If you're using god to monitor Resque, we have provided example
847
- configs in `examples/god/`. One is for starting / stopping workers,
848
- the other is for killing workers that have been running too long.
849
-
850
- ### monit
851
-
852
- If you're using monit, `examples/monit/resque.monit` is provided free
853
- of charge. This is **not** used by GitHub in production, so please
854
- send patches for any tweaks or improvements you can make to it.
855
-
856
-
857
- Questions
858
- ---------
859
-
860
- Please add them to the [FAQ](https://github.com/resque/resque/wiki/FAQ) or open an issue on this repo.
861
-
862
-
863
810
  Development
864
811
  -----------
865
812
 
@@ -889,26 +836,27 @@ it:
889
836
  If you get an error requiring any of the dependencies, you may have
890
837
  failed to install them or be seeing load path issues.
891
838
 
839
+ #### Demo
840
+ Resque ships with a demo Sinatra app for creating jobs that are later
841
+ processed in the background.
892
842
 
893
- Contributing
894
- ------------
843
+ Try it out by looking at the README, found at `examples/demo/README.markdown`.
844
+
845
+ #### Contributing
895
846
 
896
847
  Read [CONTRIBUTING.md](CONTRIBUTING.md) first.
897
848
 
898
849
  Once you've made your great commits:
899
850
 
900
- 1. [Fork][1] Resque
851
+ 1. [Fork](http://help.github.com/forking/) Resque
901
852
  2. Create a topic branch - `git checkout -b my_branch`
902
853
  3. Push to your branch - `git push origin my_branch`
903
854
  4. Create a [Pull Request](http://help.github.com/pull-requests/) from your branch
904
- 5. That's it!
905
-
906
-
907
- Mailing List
908
- ------------
909
855
 
910
- This mailing list is no longer maintained. The archive can be found at <http://librelist.com/browser/resque/>.
856
+ Questions
857
+ ---------
911
858
 
859
+ Please add them to the [FAQ](https://github.com/resque/resque/wiki/FAQ) or open an issue on this repo.
912
860
 
913
861
  Meta
914
862
  ----
@@ -917,21 +865,11 @@ Meta
917
865
  * Home: <http://github.com/resque/resque>
918
866
  * Docs: <http://rubydoc.info/gems/resque>
919
867
  * Bugs: <http://github.com/resque/resque/issues>
920
- * List: <resque@librelist.com>
921
- * Chat: <irc://irc.freenode.net/resque>
922
- * Gems: <http://gemcutter.org/gems/resque>
923
-
924
- This project uses [Semantic Versioning][sv].
868
+ * Gems: <https://rubygems.org/gems/resque>
925
869
 
870
+ This project uses [Semantic Versioning](http://semver.org/)
926
871
 
927
872
  Author
928
873
  ------
929
874
 
930
875
  Chris Wanstrath :: chris@ozmm.org :: @defunkt
931
-
932
- [0]: http://github.com/blog/542-introducing-resque
933
- [1]: http://help.github.com/forking/
934
- [2]: http://github.com/resque/resque/issues
935
- [sv]: http://semver.org/
936
- [rs]: http://github.com/resque/redis-namespace
937
- [cb]: http://wiki.github.com/resque/resque/contributing