resque 1.27.2 → 2.1.0

Sign up to get free protection for your applications and to get access to all the features.

Potentially problematic release.


This version of resque might be problematic. Click here for more details.

Files changed (38) hide show
  1. checksums.yaml +5 -5
  2. data/HISTORY.md +67 -1
  3. data/README.markdown +436 -492
  4. data/bin/resque-web +1 -1
  5. data/lib/resque.rb +92 -18
  6. data/lib/resque/data_store.rb +21 -22
  7. data/lib/resque/errors.rb +7 -1
  8. data/lib/resque/failure.rb +1 -0
  9. data/lib/resque/failure/airbrake.rb +19 -7
  10. data/lib/resque/failure/multiple.rb +6 -2
  11. data/lib/resque/failure/redis.rb +1 -1
  12. data/lib/resque/failure/redis_multi_queue.rb +3 -3
  13. data/lib/resque/job.rb +2 -2
  14. data/lib/resque/logging.rb +1 -1
  15. data/lib/resque/railtie.rb +10 -0
  16. data/lib/resque/server.rb +7 -9
  17. data/lib/resque/server/public/jquery-3.6.0.min.js +2 -0
  18. data/lib/resque/server/public/main.js +3 -0
  19. data/lib/resque/server/public/ranger.js +7 -4
  20. data/lib/resque/server/public/style.css +3 -3
  21. data/lib/resque/server/test_helper.rb +1 -1
  22. data/lib/resque/server/views/failed.erb +2 -2
  23. data/lib/resque/server/views/failed_job.erb +2 -2
  24. data/lib/resque/server/views/layout.erb +3 -2
  25. data/lib/resque/server/views/next_more.erb +14 -14
  26. data/lib/resque/server/views/queues.erb +6 -6
  27. data/lib/resque/server/views/stats.erb +1 -1
  28. data/lib/resque/server/views/working.erb +6 -6
  29. data/lib/resque/stat.rb +12 -5
  30. data/lib/resque/tasks.rb +1 -9
  31. data/lib/resque/thread_signal.rb +13 -34
  32. data/lib/resque/vendor/utf8_util.rb +2 -8
  33. data/lib/resque/version.rb +1 -1
  34. data/lib/resque/worker.rb +90 -44
  35. metadata +12 -13
  36. data/lib/resque/server/public/jquery-1.12.4.min.js +0 -5
  37. data/lib/resque/vendor/utf8_util/utf8_util_18.rb +0 -91
  38. data/lib/resque/vendor/utf8_util/utf8_util_19.rb +0 -6
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
- SHA1:
3
- metadata.gz: c9757198382c0b8e5cf383fc49fe82ce4f59933e
4
- data.tar.gz: 6dbda5f17bf9cb44f99c18724b0cb6a6e1422e73
2
+ SHA256:
3
+ metadata.gz: b930a952313756e06d51bb6daa493ebeed0c4498940e46d24300f7fafdac36b3
4
+ data.tar.gz: ea821158f150dbf5740dff8cb210be0f7507c1e183acd3a902308f0ef91e7f2c
5
5
  SHA512:
6
- metadata.gz: 45cc843a75953f9b5e08ebcded0caafa7da8a833489d6c48922b68d6fecc3192ddd4dbce3f77dffc45cf187a89e6a1e79509a2043e9ce11081996e8466561d75
7
- data.tar.gz: 6d5c2fdc451fd17b50b1d82a7f7ed94a25aad40af6cc0515e0cace4304944b0539fee3cc42c64a8608db64135fec5d79db0f9fb6c5f1d13531794923078c4f96
6
+ metadata.gz: d266f8019b93add5de76d0d3e5a460c2a03c3fc04b3d71c65f296269c7e286875b6b1925d31d01699c952d14319164cab39062f9310314fed0b4229b92f47821
7
+ data.tar.gz: d375348987a655202483adb5eccb253ed07eea17d88749c04837e0ec05dad948805cb371500ac42b11650595327875f0af56e9c3501261e8223f6c1fbfff5f5f
data/HISTORY.md CHANGED
@@ -1,6 +1,72 @@
1
1
  ## Unreleased
2
2
 
3
- Nothing yet!
3
+ ### Added
4
+
5
+ *
6
+ *
7
+
8
+ ### Fixed
9
+
10
+ *
11
+ *
12
+
13
+ ## 2.1.0
14
+
15
+ ### Security
16
+
17
+ * Fix XSS via URL path in admin web UI queues view #1687
18
+ * Replace onclick handlers in server code to support Content Security Policies that don't allow 'unsafe-inline'
19
+ * Update jQuery from 1.12.4 to 3.6.0
20
+
21
+ ### Added
22
+
23
+ * Add requeue_queue method to Resque::Failure::Multiple #1659
24
+ * Confirmation prompt in admin front-end before submitting the retry of all failed jobs. #1753
25
+ * Railtie for default rake task setup when in Rails. #1715
26
+ * Added two new hooks.
27
+ - `queue_empty` when the job queue empties and the worker becomes idle
28
+ - `worker_exit` when the worker exits
29
+
30
+ See [docs/HOOKS.md](http://github.com/resque/resque/blob/master/docs/HOOKS.md) for
31
+ further details. (@jeremywadsack)
32
+
33
+ ### Fixed
34
+
35
+ * live poller shouldn't restart itself until it successds or fails. #1740
36
+ * Fix parsing worker_id when queue name includes colon. #1691
37
+ * Prune workers which haven't been registered but have set a heartbeat. #1751
38
+ * `Resque::Failure::Multiple.remove` did not pass on the queue parameter
39
+
40
+ ## 2.0.0 (2018-11-06)
41
+
42
+ ### Fixed
43
+ * Fix Airbrake failure backend
44
+ * Fix failed jobs page "argument out of range" error
45
+
46
+ ### Changed
47
+ * Remove support for Rubies < 2.3
48
+ * Remove support to Rails < 4
49
+ * Reduce the number of redis calls when trying to get the list of queues
50
+ * Only run `eager_load!` if `Rails.application.config.eager_load` is true
51
+ * Don't display log message if running without hooks
52
+ * Add Support to Redis 4.0
53
+ * Drop complex Redis identifier logic in favor of simple inspect
54
+ * When a child process is killed, report the signal it was terminated with
55
+ * Report a job that pruned worker was processing
56
+
57
+ ### Added
58
+
59
+ * Allow to configure statistic data store
60
+
61
+ ## 1.27.4 (2017-04-15)
62
+
63
+ ### Fixed
64
+ * Fix issue where removing a failure from Resque web didn't work when using `RedisMultiQueue` backend.
65
+
66
+ ## 1.27.3 (2017-04-10)
67
+
68
+ ### Fixed
69
+ * Fix issue where initializing a data store would attempt to hit Redis, even when Resque.inline = true
4
70
 
5
71
  ## 1.27.2 (2017-02-20)
6
72
 
data/README.markdown CHANGED
@@ -2,13 +2,21 @@ Resque
2
2
  ======
3
3
 
4
4
  [![Gem Version](https://badge.fury.io/rb/resque.svg)](https://rubygems.org/gems/resque)
5
- [![Build Status](https://travis-ci.org/resque/resque.svg)](https://travis-ci.org/resque/resque)
6
- [![Coverage Status](https://coveralls.io/repos/github/resque/resque/badge.svg?branch=1-x-stable)](https://coveralls.io/r/resque/resque?branch=1-x-stable)
5
+ [![Build Status](https://github.com/resque/resque/actions/workflows/ci.yml/badge.svg)](https://github.com/resque/resque/actions/workflows/ci.yml)
7
6
 
8
7
  Resque (pronounced like "rescue") is a Redis-backed library for creating
9
8
  background jobs, placing those jobs on multiple queues, and processing
10
9
  them later.
11
10
 
11
+ ---------
12
+ ### Note on the future of this repo
13
+
14
+ Would you like to be involved in Resque? Do you have thoughts about what
15
+ Resque should be and do going forward? There's currently an [open discussion here](https://github.com/resque/resque/issues/1759)
16
+ on just that topic, so please feel free to join in. We'd love to hear your thoughts
17
+ and/or have people volunteer to be a part of the project!
18
+
19
+ ---------
12
20
  Background jobs can be any Ruby class or module that responds to
13
21
  `perform`. Your existing classes can easily be converted to background
14
22
  jobs or you can create new classes specifically to do work. Or, you
@@ -34,17 +42,43 @@ The Resque frontend tells you what workers are doing, what workers are
34
42
  not doing, what queues you're using, what's in those queues, provides
35
43
  general usage stats, and helps you track failures.
36
44
 
45
+ Resque now supports Ruby 2.3.0 and above.
46
+ We will also only be supporting Redis 3.0 and above going forward.
37
47
 
38
- The Blog Post
39
- -------------
40
48
 
41
- For the backstory, philosophy, and history of Resque's beginnings,
42
- please see [the blog post][0].
49
+ Table of Contents
50
+ -----------------
43
51
 
52
+ * [Overview](#overview)
53
+ * [Installation](#installation)
54
+ * [Running Workers](#running-workers)
55
+ * [The Front End](#the-front-end)
56
+ * [Jobs](#jobs)
57
+ * [Configuration](#configuration)
58
+ * * [Redis](#redis)
59
+ * * [Logging](#logging)
60
+ * * [Namespaces](#namespaces)
61
+ * * [Storing Statistics](#storing-statistics)
62
+ * [Plugins and Hooks](#plugins-and-hooks)
63
+ * [Additional Information](#additional-information)
64
+ * * [Resque vs DelayedJob](#resque-vs-delayedjob)
65
+ * * [Forking](#forking)
66
+ * * [Signals](#signals)
67
+ * * [Heroku](#heroku)
68
+ * * [Monitoring](#monitoring)
69
+ * * [Mysql::Error](#mysqlerror-mysql-server-has-gone-away)
70
+ * [Development](#development)
71
+ * * [Demo](#demo)
72
+ * * [Contributing](#contributing)
73
+ * [Questions](#questions)
74
+ * [Meta](#meta)
75
+ * [Author](#author)
44
76
 
45
77
  Overview
46
78
  --------
47
79
 
80
+ For the backstory, philosophy, and history of Resque's beginnings, please see [the blog post](http://github.com/blog/542-introducing-resque).
81
+
48
82
  Resque allows you to create jobs and place them on a queue, then,
49
83
  later, pull those jobs off the queue and process them.
50
84
 
@@ -110,6 +144,200 @@ Workers can be given multiple queues (a "queue list") and run on
110
144
  multiple machines. In fact they can be run anywhere with network
111
145
  access to the Redis server.
112
146
 
147
+ Installation
148
+ ------------
149
+
150
+ Add the gem to your Gemfile:
151
+
152
+ gem 'resque'
153
+
154
+ Next, install it with Bundler:
155
+
156
+ $ bundle
157
+
158
+ #### Rack
159
+
160
+ In your Rakefile, or some other file in `lib/tasks` (ex: `lib/tasks/resque.rake`), load the resque rake tasks:
161
+
162
+ ``` ruby
163
+ require 'resque'
164
+ require 'resque/tasks'
165
+ require 'your/app' # Include this line if you want your workers to have access to your application
166
+ ```
167
+
168
+ #### Rails 3+
169
+
170
+ To make resque specific changes, you can override the `resque:setup` job in `lib/tasks` (ex: `lib/tasks/resque.rake`). GitHub's setup task looks like this:
171
+
172
+ ``` ruby
173
+ task "resque:setup" => :environment do
174
+ Grit::Git.git_timeout = 10.minutes
175
+ end
176
+ ```
177
+
178
+ We don't want the `git_timeout` as high as 10 minutes in our web app,
179
+ but in the Resque workers it's fine.
180
+
181
+ Running Workers
182
+ ---------------
183
+
184
+ Resque workers are rake tasks that run forever. They basically do this:
185
+
186
+ ``` ruby
187
+ start
188
+ loop do
189
+ if job = reserve
190
+ job.process
191
+ else
192
+ sleep 5 # Polling frequency = 5
193
+ end
194
+ end
195
+ shutdown
196
+ ```
197
+
198
+ Starting a worker is simple:
199
+
200
+ $ QUEUE=* rake resque:work
201
+
202
+ Or, you can start multiple workers:
203
+
204
+ $ COUNT=2 QUEUE=* rake resque:workers
205
+
206
+ This will spawn two Resque workers, each in its own process. Hitting
207
+ ctrl-c should be sufficient to stop them all.
208
+
209
+ #### Priorities and Queue Lists
210
+
211
+ Resque doesn't support numeric priorities but instead uses the order
212
+ of queues you give it. We call this list of queues the "queue list."
213
+
214
+ Let's say we add a `warm_cache` queue in addition to our `file_serve`
215
+ queue. We'd now start a worker like so:
216
+
217
+ $ QUEUES=file_serve,warm_cache rake resque:work
218
+
219
+ When the worker looks for new jobs, it will first check
220
+ `file_serve`. If it finds a job, it'll process it then check
221
+ `file_serve` again. It will keep checking `file_serve` until no more
222
+ jobs are available. At that point, it will check `warm_cache`. If it
223
+ finds a job it'll process it then check `file_serve` (repeating the
224
+ whole process).
225
+
226
+ In this way you can prioritize certain queues. At GitHub we start our
227
+ workers with something like this:
228
+
229
+ $ QUEUES=critical,archive,high,low rake resque:work
230
+
231
+ Notice the `archive` queue - it is specialized and in our future
232
+ architecture will only be run from a single machine.
233
+
234
+ At that point we'll start workers on our generalized background
235
+ machines with this command:
236
+
237
+ $ QUEUES=critical,high,low rake resque:work
238
+
239
+ And workers on our specialized archive machine with this command:
240
+
241
+ $ QUEUE=archive rake resque:work
242
+
243
+ #### Running All Queues
244
+
245
+ If you want your workers to work off of every queue, including new
246
+ queues created on the fly, you can use a splat:
247
+
248
+ $ QUEUE=* rake resque:work
249
+
250
+ Queues will be processed in alphabetical order.
251
+
252
+ Or, prioritize some queues above `*`:
253
+
254
+ # QUEUE=critical,* rake resque:work
255
+
256
+ #### Process IDs (PIDs)
257
+
258
+ There are scenarios where it's helpful to record the PID of a resque
259
+ worker process. Use the PIDFILE option for easy access to the PID:
260
+
261
+ $ PIDFILE=./resque.pid QUEUE=file_serve rake resque:work
262
+
263
+ #### Running in the background
264
+
265
+ There are scenarios where it's helpful for
266
+ the resque worker to run itself in the background (usually in combination with
267
+ PIDFILE). Use the BACKGROUND option so that rake will return as soon as the
268
+ worker is started.
269
+
270
+ $ PIDFILE=./resque.pid BACKGROUND=yes QUEUE=file_serve rake resque:work
271
+
272
+ #### Polling frequency
273
+
274
+ You can pass an INTERVAL option which is a float representing the polling frequency.
275
+ The default is 5 seconds, but for a semi-active app you may want to use a smaller value.
276
+
277
+ $ INTERVAL=0.1 QUEUE=file_serve rake resque:work
278
+
279
+ The Front End
280
+ -------------
281
+
282
+ Resque comes with a Sinatra-based front end for seeing what's up with
283
+ your queue.
284
+
285
+ ![The Front End](https://camo.githubusercontent.com/64d150a243987ffbc33f588bd6d7722a0bb8d69a/687474703a2f2f7475746f7269616c732e6a756d7073746172746c61622e636f6d2f696d616765732f7265737175655f6f766572766965772e706e67)
286
+
287
+ #### Standalone
288
+
289
+ If you've installed Resque as a gem running the front end standalone is easy:
290
+
291
+ $ resque-web
292
+
293
+ It's a thin layer around `rackup` so it's configurable as well:
294
+
295
+ $ resque-web -p 8282
296
+
297
+ If you have a Resque config file you want evaluated just pass it to
298
+ the script as the final argument:
299
+
300
+ $ resque-web -p 8282 rails_root/config/initializers/resque.rb
301
+
302
+ You can also set the namespace directly using `resque-web`:
303
+
304
+ $ resque-web -p 8282 -N myapp
305
+
306
+ or set the Redis connection string if you need to do something like select a different database:
307
+
308
+ $ resque-web -p 8282 -r localhost:6379:2
309
+
310
+ #### Passenger
311
+
312
+ Using Passenger? Resque ships with a `config.ru` you can use. See
313
+ Phusion's guide:
314
+
315
+ Apache: <https://www.phusionpassenger.com/library/deploy/apache/deploy/ruby/>
316
+ Nginx: <https://www.phusionpassenger.com/library/deploy/nginx/deploy/ruby/>
317
+
318
+ #### Rack::URLMap
319
+
320
+ If you want to load Resque on a subpath, possibly alongside other
321
+ apps, it's easy to do with Rack's `URLMap`:
322
+
323
+ ``` ruby
324
+ require 'resque/server'
325
+
326
+ run Rack::URLMap.new \
327
+ "/" => Your::App.new,
328
+ "/resque" => Resque::Server.new
329
+ ```
330
+
331
+ Check `examples/demo/config.ru` for a functional example (including
332
+ HTTP basic auth).
333
+
334
+ #### Rails 3+
335
+
336
+ You can also mount Resque on a subpath in your existing Rails 3 app by adding `require 'resque/server'` to the top of your routes file or in an initializer then adding this to `routes.rb`:
337
+
338
+ ``` ruby
339
+ mount Resque::Server.new, :at => "/resque"
340
+ ```
113
341
 
114
342
  Jobs
115
343
  ----
@@ -136,8 +364,7 @@ mention "foreground" and "background" because they make conceptual
136
364
  sense. You could easily be spidering sites and sticking data which
137
365
  needs to be crunched later into a queue.
138
366
 
139
-
140
- ### Persistence
367
+ #### Persistence
141
368
 
142
369
  Jobs are persisted to queues as JSON objects. Let's take our `Archive`
143
370
  example from above. We'll run the following code to create a job:
@@ -182,7 +409,7 @@ If your jobs were run against marshaled objects, they could
182
409
  potentially be operating on a stale record with out-of-date information.
183
410
 
184
411
 
185
- ### send_later / async
412
+ #### send_later / async
186
413
 
187
414
  Want something like DelayedJob's `send_later` or the ability to use
188
415
  instance methods instead of just methods for jobs? See the `examples/`
@@ -191,12 +418,12 @@ directory for goodies.
191
418
  We plan to provide first class `async` support in a future release.
192
419
 
193
420
 
194
- ### Failure
421
+ #### Failure
195
422
 
196
423
  If a job raises an exception, it is logged and handed off to the
197
424
  `Resque::Failure` module. Failures are logged either locally in Redis
198
425
  or using some different backend. To see exceptions while developing,
199
- use VERBOSE env variable, see details below under Logging.
426
+ see details below under Logging.
200
427
 
201
428
  For example, Resque ships with Airbrake support. To configure it, put
202
429
  the following into an initialisation file or into your rake job:
@@ -215,64 +442,98 @@ Keep this in mind when writing your jobs: you may want to throw
215
442
  exceptions you would not normally throw in order to assist debugging.
216
443
 
217
444
 
218
- Workers
219
- -------
445
+ #### Rails example
220
446
 
221
- Resque workers are rake tasks that run forever. They basically do this:
447
+ If you are using ActiveJob here's how your job definition will look:
222
448
 
223
449
  ``` ruby
224
- start
225
- loop do
226
- if job = reserve
227
- job.process
228
- else
229
- sleep 5 # Polling frequency = 5
450
+ class ArchiveJob < ApplicationJob
451
+ queue_as :file_serve
452
+
453
+ def perform(repo_id, branch = 'master')
454
+ repo = Repository.find(repo_id)
455
+ repo.create_archive(branch)
456
+ end
457
+ end
458
+ ```
459
+
460
+ ``` ruby
461
+ class Repository
462
+ def async_create_archive(branch)
463
+ ArchiveJob.perform_later(self.id, branch)
230
464
  end
231
465
  end
232
- shutdown
233
466
  ```
234
467
 
235
- Starting a worker is simple. Here's our example from earlier:
468
+ It is important to run `ArchiveJob.perform_later(self.id, branch)` rather than `Resque.enqueue(Archive, self.id, branch)`.
469
+ Otherwise Resque will process the job without actually doing anything.
470
+ Even if you put an obviously buggy line like `0/0` in the `perform` method,
471
+ the job will still succeed.
236
472
 
237
- $ QUEUE=file_serve rake resque:work
238
473
 
239
- By default Resque won't know about your application's
240
- environment. That is, it won't be able to find and run your jobs - it
241
- needs to load your application into memory.
474
+ Configuration
475
+ -------------
476
+
477
+ #### Redis
478
+
479
+ You may want to change the Redis host and port Resque connects to, or
480
+ set various other options at startup.
481
+
482
+ Resque has a `redis` setter which can be given a string or a Redis
483
+ object. This means if you're already using Redis in your app, Resque
484
+ can re-use the existing connection.
485
+
486
+ String: `Resque.redis = 'localhost:6379'`
487
+
488
+ Redis: `Resque.redis = $redis`
489
+
490
+ For our rails app we have a `config/initializers/resque.rb` file where
491
+ we load `config/resque.yml` by hand and set the Redis information
492
+ appropriately.
242
493
 
243
- If we've installed Resque as a Rails plugin, we might run this command
244
- from our RAILS_ROOT:
494
+ Here's our `config/resque.yml`:
245
495
 
246
- $ QUEUE=file_serve rake environment resque:work
496
+ development: localhost:6379
497
+ test: localhost:6379
498
+ staging: redis1.se.github.com:6379
499
+ fi: localhost:6379
500
+ production: <%= ENV['REDIS_URL'] %>
247
501
 
248
- This will load the environment before starting a worker. Alternately
249
- we can define a `resque:setup` task with a dependency on the
250
- `environment` rake task:
502
+ And our initializer:
251
503
 
252
504
  ``` ruby
253
- task "resque:setup" => :environment
505
+ rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..'
506
+ rails_env = ENV['RAILS_ENV'] || 'development'
507
+ config_file = rails_root + '/config/resque.yml'
508
+
509
+ resque_config = YAML::load(ERB.new(IO.read(config_file)).result)
510
+ Resque.redis = resque_config[rails_env]
254
511
  ```
255
512
 
256
- GitHub's setup task looks like this:
513
+ Easy peasy! Why not just use `RAILS_ROOT` and `RAILS_ENV`? Because
514
+ this way we can tell our Sinatra app about the config file:
515
+
516
+ $ RAILS_ENV=production resque-web rails_root/config/initializers/resque.rb
517
+
518
+ Now everyone is on the same page.
519
+
520
+ Also, you could disable jobs queueing by setting 'inline' attribute.
521
+ For example, if you want to run all jobs in the same process for cucumber, try:
257
522
 
258
523
  ``` ruby
259
- task "resque:setup" => :environment do
260
- Grit::Git.git_timeout = 10.minutes
261
- end
524
+ Resque.inline = ENV['RAILS_ENV'] == "cucumber"
262
525
  ```
263
526
 
264
- We don't want the `git_timeout` as high as 10 minutes in our web app,
265
- but in the Resque workers it's fine.
527
+ #### Logging
266
528
 
529
+ Workers support basic logging to STDOUT.
267
530
 
268
- ### Logging
531
+ You can control the logging threshold using `Resque.logger.level`:
269
532
 
270
- Workers support basic logging to STDOUT. If you start them with the
271
- `VERBOSE` env variable set, they will print basic debugging
272
- information. You can also set the `VVERBOSE` (very verbose) env
273
- variable.
274
-
275
- $ VVERBOSE=1 QUEUE=file_serve rake environment resque:work
533
+ ```ruby
534
+ # config/initializers/resque.rb
535
+ Resque.logger.level = Logger::DEBUG
536
+ ```
276
537
 
277
538
  If you want Resque to log to a file, in Rails do:
278
539
 
@@ -281,91 +542,111 @@ If you want Resque to log to a file, in Rails do:
281
542
  Resque.logger = Logger.new(Rails.root.join('log', "#{Rails.env}_resque.log"))
282
543
  ```
283
544
 
284
- ### Process IDs (PIDs)
545
+ #### Namespaces
285
546
 
286
- There are scenarios where it's helpful to record the PID of a resque
287
- worker process. Use the PIDFILE option for easy access to the PID:
547
+ If you're running multiple, separate instances of Resque you may want
548
+ to namespace the keyspaces so they do not overlap. This is not unlike
549
+ the approach taken by many memcached clients.
288
550
 
289
- $ PIDFILE=./resque.pid QUEUE=file_serve rake environment resque:work
551
+ This feature is provided by the [redis-namespace](http://github.com/resque/redis-namespace) library, which
552
+ Resque uses by default to separate the keys it manages from other keys
553
+ in your Redis server.
290
554
 
291
- ### Running in the background
555
+ Simply use the `Resque.redis.namespace` accessor:
292
556
 
293
- (Only supported with ruby >= 1.9). There are scenarios where it's helpful for
294
- the resque worker to run itself in the background (usually in combination with
295
- PIDFILE). Use the BACKGROUND option so that rake will return as soon as the
296
- worker is started.
557
+ ``` ruby
558
+ Resque.redis.namespace = "resque:GitHub"
559
+ ```
297
560
 
298
- $ PIDFILE=./resque.pid BACKGROUND=yes QUEUE=file_serve \
299
- rake environment resque:work
561
+ We recommend sticking this in your initializer somewhere after Redis
562
+ is configured.
300
563
 
301
- ### Polling frequency
564
+ #### Storing Statistics
565
+ Resque allows to store count of processed and failed jobs.
302
566
 
303
- You can pass an INTERVAL option which is a float representing the polling frequency.
304
- The default is 5 seconds, but for a semi-active app you may want to use a smaller value.
567
+ By default it will store it in Redis using the keys `stats:processed` and `stats:failed`.
305
568
 
306
- $ INTERVAL=0.1 QUEUE=file_serve rake environment resque:work
569
+ Some apps would want another stats store, or even a null store:
307
570
 
308
- ### Priorities and Queue Lists
309
-
310
- Resque doesn't support numeric priorities but instead uses the order
311
- of queues you give it. We call this list of queues the "queue list."
312
-
313
- Let's say we add a `warm_cache` queue in addition to our `file_serve`
314
- queue. We'd now start a worker like so:
315
-
316
- $ QUEUES=file_serve,warm_cache rake resque:work
317
-
318
- When the worker looks for new jobs, it will first check
319
- `file_serve`. If it finds a job, it'll process it then check
320
- `file_serve` again. It will keep checking `file_serve` until no more
321
- jobs are available. At that point, it will check `warm_cache`. If it
322
- finds a job it'll process it then check `file_serve` (repeating the
323
- whole process).
324
-
325
- In this way you can prioritize certain queues. At GitHub we start our
326
- workers with something like this:
571
+ ```ruby
572
+ # config/initializers/resque.rb
573
+ class NullDataStore
574
+ def stat(stat)
575
+ 0
576
+ end
327
577
 
328
- $ QUEUES=critical,archive,high,low rake resque:work
578
+ def increment_stat(stat, by)
579
+ end
329
580
 
330
- Notice the `archive` queue - it is specialized and in our future
331
- architecture will only be run from a single machine.
581
+ def decrement_stat(stat, by)
582
+ end
332
583
 
333
- At that point we'll start workers on our generalized background
334
- machines with this command:
584
+ def clear_stat(stat)
585
+ end
586
+ end
335
587
 
336
- $ QUEUES=critical,high,low rake resque:work
588
+ Resque.stat_data_store = NullDataStore.new
589
+ ```
337
590
 
338
- And workers on our specialized archive machine with this command:
591
+ Plugins and Hooks
592
+ -----------------
339
593
 
340
- $ QUEUE=archive rake resque:work
594
+ For a list of available plugins see
595
+ <https://github.com/resque/resque/wiki/plugins>.
341
596
 
597
+ If you'd like to write your own plugin, or want to customize Resque
598
+ using hooks (such as `Resque.after_fork`), see
599
+ [docs/HOOKS.md](http://github.com/resque/resque/blob/master/docs/HOOKS.md).
342
600
 
343
- ### Running All Queues
344
601
 
345
- If you want your workers to work off of every queue, including new
346
- queues created on the fly, you can use a splat:
602
+ Additional Information
603
+ ----------------------
347
604
 
348
- $ QUEUE=* rake resque:work
605
+ #### Resque vs DelayedJob
349
606
 
350
- Queues will be processed in alphabetical order.
607
+ How does Resque compare to DelayedJob, and why would you choose one
608
+ over the other?
351
609
 
610
+ * Resque supports multiple queues
611
+ * DelayedJob supports finer grained priorities
612
+ * Resque workers are resilient to memory leaks / bloat
613
+ * DelayedJob workers are extremely simple and easy to modify
614
+ * Resque requires Redis
615
+ * DelayedJob requires ActiveRecord
616
+ * Resque can only place JSONable Ruby objects on a queue as arguments
617
+ * DelayedJob can place _any_ Ruby object on its queue as arguments
618
+ * Resque includes a Sinatra app for monitoring what's going on
619
+ * DelayedJob can be queried from within your Rails app if you want to
620
+ add an interface
352
621
 
353
- ### Running Multiple Workers
622
+ If you're doing Rails development, you already have a database and
623
+ ActiveRecord. DelayedJob is super easy to setup and works great.
624
+ GitHub used it for many months to process almost 200 million jobs.
354
625
 
355
- At GitHub we use god to start and stop multiple workers. A sample god
356
- configuration file is included under `examples/god`. We recommend this
357
- method.
626
+ Choose Resque if:
358
627
 
359
- If you'd like to run multiple workers in development mode, you can do
360
- so using the `resque:workers` rake task:
628
+ * You need multiple queues
629
+ * You don't care / dislike numeric priorities
630
+ * You don't need to persist every Ruby object ever
631
+ * You have potentially huge queues
632
+ * You want to see what's going on
633
+ * You expect a lot of failure / chaos
634
+ * You can setup Redis
635
+ * You're not running short on RAM
361
636
 
362
- $ COUNT=5 QUEUE=* rake resque:workers
637
+ Choose DelayedJob if:
363
638
 
364
- This will spawn five Resque workers, each in its own process. Hitting
365
- ctrl-c should be sufficient to stop them all.
639
+ * You like numeric priorities
640
+ * You're not doing a gigantic amount of jobs each day
641
+ * Your queue stays small and nimble
642
+ * There is not a lot failure / chaos
643
+ * You want to easily throw anything on the queue
644
+ * You don't want to setup Redis
366
645
 
646
+ In no way is Resque a "better" DelayedJob, so make sure you pick the
647
+ tool that's best for your app.
367
648
 
368
- ### Forking
649
+ #### Forking
369
650
 
370
651
  On certain platforms, when a Resque worker reserves a job it
371
652
  immediately forks a child process. The child processes the job then
@@ -406,8 +687,7 @@ complicated.
406
687
 
407
688
  Workers instead handle their own state.
408
689
 
409
-
410
- ### Parents and Children
690
+ #### Parents and Children
411
691
 
412
692
  Here's a parent / child pair doing some work:
413
693
 
@@ -428,7 +708,7 @@ waiting for work on:
428
708
  92099 resque: Waiting for file_serve,warm_cache
429
709
 
430
710
 
431
- ### Signals
711
+ #### Signals
432
712
 
433
713
  Resque workers respond to a few different signals:
434
714
 
@@ -450,11 +730,47 @@ If you want to stop processing jobs, but want to leave the worker running
450
730
  (for example, to temporarily alleviate load), use `USR2` to stop processing,
451
731
  then `CONT` to start it again.
452
732
 
453
- ### Mysql::Error: MySQL server has gone away
733
+ #### Heroku
734
+
735
+ When shutting down processes, Heroku sends every process a TERM signal at the
736
+ same time. By default this causes an immediate shutdown of any running job
737
+ leading to frequent `Resque::TermException` errors. For short running jobs, a simple
738
+ solution is to give a small amount of time for the job to finish
739
+ before killing it.
740
+
741
+ Resque doesn't handle this out of the box (for both cedar-14 and heroku-16), you need to
742
+ install the [`resque-heroku-signals`](https://github.com/iloveitaly/resque-heroku-signals)
743
+ addon which adds the required signal handling to make the behavior described above work.
744
+ Related issue: https://github.com/resque/resque/issues/1559
745
+
746
+ To accomplish this set the following environment variables:
747
+
748
+ * `RESQUE_PRE_SHUTDOWN_TIMEOUT` - The time between the parent receiving a shutdown signal (TERM by default) and it sending that signal on to the child process. Designed to give the child process
749
+ time to complete before being forced to die.
750
+
751
+ * `TERM_CHILD` - Must be set for `RESQUE_PRE_SHUTDOWN_TIMEOUT` to be used. After the timeout, if the child is still running it will raise a `Resque::TermException` and exit.
752
+
753
+ * `RESQUE_TERM_TIMEOUT` - By default you have a few seconds to handle `Resque::TermException` in your job. `RESQUE_TERM_TIMEOUT` and `RESQUE_PRE_SHUTDOWN_TIMEOUT` must be lower than the [heroku dyno timeout](https://devcenter.heroku.com/articles/limits#exit-timeout).
754
+
755
+ #### Monitoring
756
+
757
+ ##### god
758
+
759
+ If you're using god to monitor Resque, we have provided example
760
+ configs in `examples/god/`. One is for starting / stopping workers,
761
+ the other is for killing workers that have been running too long.
762
+
763
+ ##### monit
764
+
765
+ If you're using monit, `examples/monit/resque.monit` is provided free
766
+ of charge. This is **not** used by GitHub in production, so please
767
+ send patches for any tweaks or improvements you can make to it.
768
+
769
+ #### Mysql::Error: MySQL server has gone away
454
770
 
455
771
  If your workers remain idle for too long they may lose their MySQL connection. Depending on your version of Rails, we recommend the following:
456
772
 
457
- #### Rails 3.x
773
+ ##### Rails 3.x
458
774
  In your `perform` method, add the following line:
459
775
 
460
776
  ``` ruby
@@ -470,7 +786,7 @@ The Rails doc says the following about `verify_active_connections!`:
470
786
 
471
787
  Verify active connections and remove and disconnect connections associated with stale threads.
472
788
 
473
- #### Rails 4.x
789
+ ##### Rails 4.x
474
790
 
475
791
  In your `perform` method, instead of `verify_active_connections!`, use:
476
792
 
@@ -487,363 +803,6 @@ From the Rails docs on [`clear_active_connections!`](http://api.rubyonrails.org/
487
803
 
488
804
  Returns any connections in use by the current thread back to the pool, and also returns connections to the pool cached by threads that are no longer alive.
489
805
 
490
-
491
-
492
- The Front End
493
- -------------
494
-
495
- Resque comes with a Sinatra-based front end for seeing what's up with
496
- your queue.
497
-
498
- ![The Front End](https://camo.githubusercontent.com/64d150a243987ffbc33f588bd6d7722a0bb8d69a/687474703a2f2f7475746f7269616c732e6a756d7073746172746c61622e636f6d2f696d616765732f7265737175655f6f766572766965772e706e67)
499
-
500
- ### Standalone
501
-
502
- If you've installed Resque as a gem running the front end standalone is easy:
503
-
504
- $ resque-web
505
-
506
- It's a thin layer around `rackup` so it's configurable as well:
507
-
508
- $ resque-web -p 8282
509
-
510
- If you have a Resque config file you want evaluated just pass it to
511
- the script as the final argument:
512
-
513
- $ resque-web -p 8282 rails_root/config/initializers/resque.rb
514
-
515
- You can also set the namespace directly using `resque-web`:
516
-
517
- $ resque-web -p 8282 -N myapp
518
-
519
- or set the Redis connection string if you need to do something like select a different database:
520
-
521
- $ resque-web -p 8282 -r localhost:6379:2
522
-
523
- ### Passenger
524
-
525
- Using Passenger? Resque ships with a `config.ru` you can use. See
526
- Phusion's guide:
527
-
528
- Apache: <https://www.phusionpassenger.com/library/deploy/apache/deploy/ruby/>
529
- Nginx: <https://www.phusionpassenger.com/library/deploy/nginx/deploy/ruby/>
530
-
531
- ### Rack::URLMap
532
-
533
- If you want to load Resque on a subpath, possibly alongside other
534
- apps, it's easy to do with Rack's `URLMap`:
535
-
536
- ``` ruby
537
- require 'resque/server'
538
-
539
- run Rack::URLMap.new \
540
- "/" => Your::App.new,
541
- "/resque" => Resque::Server.new
542
- ```
543
-
544
- Check `examples/demo/config.ru` for a functional example (including
545
- HTTP basic auth).
546
-
547
- ### Rails 3
548
-
549
- You can also mount Resque on a subpath in your existing Rails 3 app by adding `require 'resque/server'` to the top of your routes file or in an initializer then adding this to `routes.rb`:
550
-
551
- ``` ruby
552
- mount Resque::Server.new, :at => "/resque"
553
- ```
554
-
555
-
556
- Resque vs DelayedJob
557
- --------------------
558
-
559
- How does Resque compare to DelayedJob, and why would you choose one
560
- over the other?
561
-
562
- * Resque supports multiple queues
563
- * DelayedJob supports finer grained priorities
564
- * Resque workers are resilient to memory leaks / bloat
565
- * DelayedJob workers are extremely simple and easy to modify
566
- * Resque requires Redis
567
- * DelayedJob requires ActiveRecord
568
- * Resque can only place JSONable Ruby objects on a queue as arguments
569
- * DelayedJob can place _any_ Ruby object on its queue as arguments
570
- * Resque includes a Sinatra app for monitoring what's going on
571
- * DelayedJob can be queried from within your Rails app if you want to
572
- add an interface
573
-
574
- If you're doing Rails development, you already have a database and
575
- ActiveRecord. DelayedJob is super easy to setup and works great.
576
- GitHub used it for many months to process almost 200 million jobs.
577
-
578
- Choose Resque if:
579
-
580
- * You need multiple queues
581
- * You don't care / dislike numeric priorities
582
- * You don't need to persist every Ruby object ever
583
- * You have potentially huge queues
584
- * You want to see what's going on
585
- * You expect a lot of failure / chaos
586
- * You can setup Redis
587
- * You're not running short on RAM
588
-
589
- Choose DelayedJob if:
590
-
591
- * You like numeric priorities
592
- * You're not doing a gigantic amount of jobs each day
593
- * Your queue stays small and nimble
594
- * There is not a lot failure / chaos
595
- * You want to easily throw anything on the queue
596
- * You don't want to setup Redis
597
-
598
- In no way is Resque a "better" DelayedJob, so make sure you pick the
599
- tool that's best for your app.
600
-
601
- Resque Dependencies
602
- -------------------
603
-
604
- $ gem install bundler
605
- $ bundle install
606
-
607
-
608
- Installing Resque
609
- -----------------
610
-
611
- ### In a Rack app, as a gem
612
-
613
- First install the gem.
614
-
615
- $ gem install resque
616
-
617
- Next include it in your application.
618
-
619
- ``` ruby
620
- require 'resque'
621
- ```
622
-
623
- Now start your application:
624
-
625
- rackup config.ru
626
-
627
- That's it! You can now create Resque jobs from within your app.
628
-
629
- To start a worker, create a Rakefile in your app's root (or add this
630
- to an existing Rakefile):
631
-
632
- ``` ruby
633
- require 'your/app'
634
- require 'resque/tasks'
635
- ```
636
-
637
- If you're using Rails 5.x, include the following in lib/tasks/resque.rb:
638
-
639
- ```ruby
640
- require 'resque/tasks'
641
- task 'resque:setup' => :environment
642
- ```
643
-
644
- Now:
645
-
646
- $ QUEUE=* rake resque:work
647
-
648
- Alternately you can define a `resque:setup` hook in your Rakefile if you
649
- don't want to load your app every time rake runs.
650
-
651
-
652
- ### In a Rails 2.x app, as a gem
653
-
654
- First install the gem.
655
-
656
- $ gem install resque
657
-
658
- Next include it in your application.
659
-
660
- $ cat config/initializers/load_resque.rb
661
- require 'resque'
662
-
663
- Now start your application:
664
-
665
- $ ./script/server
666
-
667
- That's it! You can now create Resque jobs from within your app.
668
-
669
- To start a worker, add this to your Rakefile in `RAILS_ROOT`:
670
-
671
- ``` ruby
672
- require 'resque/tasks'
673
- ```
674
-
675
- Now:
676
-
677
- $ QUEUE=* rake environment resque:work
678
-
679
- Don't forget you can define a `resque:setup` hook in
680
- `lib/tasks/whatever.rake` that loads the `environment` task every time.
681
-
682
-
683
- ### In a Rails 2.x app, as a plugin
684
-
685
- $ ./script/plugin install git://github.com/resque/resque
686
-
687
- That's it! Resque will automatically be available when your Rails app
688
- loads.
689
-
690
- To start a worker:
691
-
692
- $ QUEUE=* rake environment resque:work
693
-
694
- Don't forget you can define a `resque:setup` hook in
695
- `lib/tasks/whatever.rake` that loads the `environment` task every time.
696
-
697
-
698
- ### In a Rails 3.x or 4.x app, as a gem
699
-
700
- First include it in your Gemfile.
701
-
702
- $ cat Gemfile
703
- ...
704
- gem 'resque'
705
- ...
706
-
707
- Next install it with Bundler.
708
-
709
- $ bundle install
710
-
711
- Now start your application:
712
-
713
- $ rails server
714
-
715
- That's it! You can now create Resque jobs from within your app.
716
-
717
- To start a worker, add this to a file in `lib/tasks` (ex:
718
- `lib/tasks/resque.rake`):
719
-
720
- ``` ruby
721
- require 'resque/tasks'
722
- ```
723
-
724
- Now:
725
-
726
- $ QUEUE=* rake environment resque:work
727
-
728
- Don't forget you can define a `resque:setup` hook in
729
- `lib/tasks/whatever.rake` that loads the `environment` task every time.
730
-
731
-
732
- Configuration
733
- -------------
734
-
735
- You may want to change the Redis host and port Resque connects to, or
736
- set various other options at startup.
737
-
738
- Resque has a `redis` setter which can be given a string or a Redis
739
- object. This means if you're already using Redis in your app, Resque
740
- can re-use the existing connection.
741
-
742
- String: `Resque.redis = 'localhost:6379'`
743
-
744
- Redis: `Resque.redis = $redis`
745
-
746
- For our rails app we have a `config/initializers/resque.rb` file where
747
- we load `config/resque.yml` by hand and set the Redis information
748
- appropriately.
749
-
750
- Here's our `config/resque.yml`:
751
-
752
- development: localhost:6379
753
- test: localhost:6379
754
- staging: redis1.se.github.com:6379
755
- fi: localhost:6379
756
- production: redis1.ae.github.com:6379
757
-
758
- And our initializer:
759
-
760
- ``` ruby
761
- rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..'
762
- rails_env = ENV['RAILS_ENV'] || 'development'
763
-
764
- resque_config = YAML.load_file(rails_root + '/config/resque.yml')
765
- Resque.redis = resque_config[rails_env]
766
- ```
767
-
768
- Easy peasy! Why not just use `RAILS_ROOT` and `RAILS_ENV`? Because
769
- this way we can tell our Sinatra app about the config file:
770
-
771
- $ RAILS_ENV=production resque-web rails_root/config/initializers/resque.rb
772
-
773
- Now everyone is on the same page.
774
-
775
- Also, you could disable jobs queueing by setting 'inline' attribute.
776
- For example, if you want to run all jobs in the same process for cucumber, try:
777
-
778
- ``` ruby
779
- Resque.inline = ENV['RAILS_ENV'] == "cucumber"
780
- ```
781
-
782
-
783
- Plugins and Hooks
784
- -----------------
785
-
786
- For a list of available plugins see
787
- <http://wiki.github.com/resque/resque/plugins>.
788
-
789
- If you'd like to write your own plugin, or want to customize Resque
790
- using hooks (such as `Resque.after_fork`), see
791
- [docs/HOOKS.md](http://github.com/resque/resque/blob/master/docs/HOOKS.md).
792
-
793
-
794
- Namespaces
795
- ----------
796
-
797
- If you're running multiple, separate instances of Resque you may want
798
- to namespace the keyspaces so they do not overlap. This is not unlike
799
- the approach taken by many memcached clients.
800
-
801
- This feature is provided by the [redis-namespace][rs] library, which
802
- Resque uses by default to separate the keys it manages from other keys
803
- in your Redis server.
804
-
805
- Simply use the `Resque.redis.namespace` accessor:
806
-
807
- ``` ruby
808
- Resque.redis.namespace = "resque:GitHub"
809
- ```
810
-
811
- We recommend sticking this in your initializer somewhere after Redis
812
- is configured.
813
-
814
-
815
- Demo
816
- ----
817
-
818
- Resque ships with a demo Sinatra app for creating jobs that are later
819
- processed in the background.
820
-
821
- Try it out by looking at the README, found at `examples/demo/README.markdown`.
822
-
823
-
824
- Monitoring
825
- ----------
826
-
827
- ### god
828
-
829
- If you're using god to monitor Resque, we have provided example
830
- configs in `examples/god/`. One is for starting / stopping workers,
831
- the other is for killing workers that have been running too long.
832
-
833
- ### monit
834
-
835
- If you're using monit, `examples/monit/resque.monit` is provided free
836
- of charge. This is **not** used by GitHub in production, so please
837
- send patches for any tweaks or improvements you can make to it.
838
-
839
-
840
- Questions
841
- ---------
842
-
843
- Please add them to the [FAQ](https://github.com/resque/resque/wiki/FAQ) or
844
- ask on the Mailing List. The Mailing List is explained further below
845
-
846
-
847
806
  Development
848
807
  -----------
849
808
 
@@ -873,33 +832,27 @@ it:
873
832
  If you get an error requiring any of the dependencies, you may have
874
833
  failed to install them or be seeing load path issues.
875
834
 
876
- Feel free to ping the mailing list with your problem and we'll try to
877
- sort it out.
835
+ #### Demo
836
+ Resque ships with a demo Sinatra app for creating jobs that are later
837
+ processed in the background.
878
838
 
839
+ Try it out by looking at the README, found at `examples/demo/README.markdown`.
879
840
 
880
- Contributing
881
- ------------
841
+ #### Contributing
882
842
 
883
843
  Read [CONTRIBUTING.md](CONTRIBUTING.md) first.
884
844
 
885
845
  Once you've made your great commits:
886
846
 
887
- 1. [Fork][1] Resque
847
+ 1. [Fork](http://help.github.com/forking/) Resque
888
848
  2. Create a topic branch - `git checkout -b my_branch`
889
849
  3. Push to your branch - `git push origin my_branch`
890
850
  4. Create a [Pull Request](http://help.github.com/pull-requests/) from your branch
891
- 5. That's it!
892
-
893
-
894
- Mailing List
895
- ------------
896
-
897
- To join the list simply send an email to <resque@librelist.com>. This
898
- will subscribe you and send you information about your subscription,
899
- including unsubscribe information.
900
851
 
901
- The archive can be found at <http://librelist.com/browser/resque/>.
852
+ Questions
853
+ ---------
902
854
 
855
+ Please add them to the [FAQ](https://github.com/resque/resque/wiki/FAQ) or open an issue on this repo.
903
856
 
904
857
  Meta
905
858
  ----
@@ -908,21 +861,12 @@ Meta
908
861
  * Home: <http://github.com/resque/resque>
909
862
  * Docs: <http://rubydoc.info/gems/resque>
910
863
  * Bugs: <http://github.com/resque/resque/issues>
911
- * List: <resque@librelist.com>
912
864
  * Chat: <irc://irc.freenode.net/resque>
913
- * Gems: <http://gemcutter.org/gems/resque>
914
-
915
- This project uses [Semantic Versioning][sv].
865
+ * Gems: <https://rubygems.org/gems/resque>
916
866
 
867
+ This project uses [Semantic Versioning](http://semver.org/)
917
868
 
918
869
  Author
919
870
  ------
920
871
 
921
872
  Chris Wanstrath :: chris@ozmm.org :: @defunkt
922
-
923
- [0]: http://github.com/blog/542-introducing-resque
924
- [1]: http://help.github.com/forking/
925
- [2]: http://github.com/resque/resque/issues
926
- [sv]: http://semver.org/
927
- [rs]: http://github.com/resque/redis-namespace
928
- [cb]: http://wiki.github.com/resque/resque/contributing