resque 2.0.0 → 2.1.0

Sign up to get free protection for your applications and to get access to all the features.

Potentially problematic release.


This version of resque might be problematic. Click here for more details.

checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 845d352e5d2e155c7a3c0c94b825abe83cc0daa617e70ac85fb6f2ec0fbad452
4
- data.tar.gz: 887666edf6aaa2b4bbc372001ece5b1fb354801be30786160ee0cde6b16828c0
3
+ metadata.gz: b930a952313756e06d51bb6daa493ebeed0c4498940e46d24300f7fafdac36b3
4
+ data.tar.gz: ea821158f150dbf5740dff8cb210be0f7507c1e183acd3a902308f0ef91e7f2c
5
5
  SHA512:
6
- metadata.gz: e99e247f115ce75ad125e7f4aa0bf1587e67426c424e3767ab856c4a03c9868ebb922a958aaff689f316beb9b2b07ca0d1818b592a300d6a2358af3b1312932a
7
- data.tar.gz: '081fe527bb158cee4c9e8c31d552904b20e1fa39984466a2f7d1851facdbb2f0eb1f3ec865ae6d9cafee79278a392316822d0cdc4aff7a5c3076f96f8907d26a'
6
+ metadata.gz: d266f8019b93add5de76d0d3e5a460c2a03c3fc04b3d71c65f296269c7e286875b6b1925d31d01699c952d14319164cab39062f9310314fed0b4229b92f47821
7
+ data.tar.gz: d375348987a655202483adb5eccb253ed07eea17d88749c04837e0ec05dad948805cb371500ac42b11650595327875f0af56e9c3501261e8223f6c1fbfff5f5f
data/HISTORY.md CHANGED
@@ -1,3 +1,42 @@
1
+ ## Unreleased
2
+
3
+ ### Added
4
+
5
+ *
6
+ *
7
+
8
+ ### Fixed
9
+
10
+ *
11
+ *
12
+
13
+ ## 2.1.0
14
+
15
+ ### Security
16
+
17
+ * Fix XSS via URL path in admin web UI queues view #1687
18
+ * Replace onclick handlers in server code to support Content Security Policies that don't allow 'unsafe-inline'
19
+ * Update jQuery from 1.12.4 to 3.6.0
20
+
21
+ ### Added
22
+
23
+ * Add requeue_queue method to Resque::Failure::Multiple #1659
24
+ * Confirmation prompt in admin front-end before submitting the retry of all failed jobs. #1753
25
+ * Railtie for default rake task setup when in Rails. #1715
26
+ * Added two new hooks.
27
+ - `queue_empty` when the job queue empties and the worker becomes idle
28
+ - `worker_exit` when the worker exits
29
+
30
+ See [docs/HOOKS.md](http://github.com/resque/resque/blob/master/docs/HOOKS.md) for
31
+ further details. (@jeremywadsack)
32
+
33
+ ### Fixed
34
+
35
+ * live poller shouldn't restart itself until it successds or fails. #1740
36
+ * Fix parsing worker_id when queue name includes colon. #1691
37
+ * Prune workers which haven't been registered but have set a heartbeat. #1751
38
+ * `Resque::Failure::Multiple.remove` did not pass on the queue parameter
39
+
1
40
  ## 2.0.0 (2018-11-06)
2
41
 
3
42
  ### Fixed
data/README.markdown CHANGED
@@ -2,12 +2,21 @@ Resque
2
2
  ======
3
3
 
4
4
  [![Gem Version](https://badge.fury.io/rb/resque.svg)](https://rubygems.org/gems/resque)
5
- [![Build Status](https://travis-ci.org/resque/resque.svg)](https://travis-ci.org/resque/resque)
5
+ [![Build Status](https://github.com/resque/resque/actions/workflows/ci.yml/badge.svg)](https://github.com/resque/resque/actions/workflows/ci.yml)
6
6
 
7
7
  Resque (pronounced like "rescue") is a Redis-backed library for creating
8
8
  background jobs, placing those jobs on multiple queues, and processing
9
9
  them later.
10
10
 
11
+ ---------
12
+ ### Note on the future of this repo
13
+
14
+ Would you like to be involved in Resque? Do you have thoughts about what
15
+ Resque should be and do going forward? There's currently an [open discussion here](https://github.com/resque/resque/issues/1759)
16
+ on just that topic, so please feel free to join in. We'd love to hear your thoughts
17
+ and/or have people volunteer to be a part of the project!
18
+
19
+ ---------
11
20
  Background jobs can be any Ruby class or module that responds to
12
21
  `perform`. Your existing classes can easily be converted to background
13
22
  jobs or you can create new classes specifically to do work. Or, you
@@ -37,16 +46,39 @@ Resque now supports Ruby 2.3.0 and above.
37
46
  We will also only be supporting Redis 3.0 and above going forward.
38
47
 
39
48
 
40
- The Blog Post
41
- -------------
42
-
43
- For the backstory, philosophy, and history of Resque's beginnings,
44
- please see [the blog post][0].
49
+ Table of Contents
50
+ -----------------
45
51
 
52
+ * [Overview](#overview)
53
+ * [Installation](#installation)
54
+ * [Running Workers](#running-workers)
55
+ * [The Front End](#the-front-end)
56
+ * [Jobs](#jobs)
57
+ * [Configuration](#configuration)
58
+ * * [Redis](#redis)
59
+ * * [Logging](#logging)
60
+ * * [Namespaces](#namespaces)
61
+ * * [Storing Statistics](#storing-statistics)
62
+ * [Plugins and Hooks](#plugins-and-hooks)
63
+ * [Additional Information](#additional-information)
64
+ * * [Resque vs DelayedJob](#resque-vs-delayedjob)
65
+ * * [Forking](#forking)
66
+ * * [Signals](#signals)
67
+ * * [Heroku](#heroku)
68
+ * * [Monitoring](#monitoring)
69
+ * * [Mysql::Error](#mysqlerror-mysql-server-has-gone-away)
70
+ * [Development](#development)
71
+ * * [Demo](#demo)
72
+ * * [Contributing](#contributing)
73
+ * [Questions](#questions)
74
+ * [Meta](#meta)
75
+ * [Author](#author)
46
76
 
47
77
  Overview
48
78
  --------
49
79
 
80
+ For the backstory, philosophy, and history of Resque's beginnings, please see [the blog post](http://github.com/blog/542-introducing-resque).
81
+
50
82
  Resque allows you to create jobs and place them on a queue, then,
51
83
  later, pull those jobs off the queue and process them.
52
84
 
@@ -112,6 +144,200 @@ Workers can be given multiple queues (a "queue list") and run on
112
144
  multiple machines. In fact they can be run anywhere with network
113
145
  access to the Redis server.
114
146
 
147
+ Installation
148
+ ------------
149
+
150
+ Add the gem to your Gemfile:
151
+
152
+ gem 'resque'
153
+
154
+ Next, install it with Bundler:
155
+
156
+ $ bundle
157
+
158
+ #### Rack
159
+
160
+ In your Rakefile, or some other file in `lib/tasks` (ex: `lib/tasks/resque.rake`), load the resque rake tasks:
161
+
162
+ ``` ruby
163
+ require 'resque'
164
+ require 'resque/tasks'
165
+ require 'your/app' # Include this line if you want your workers to have access to your application
166
+ ```
167
+
168
+ #### Rails 3+
169
+
170
+ To make resque specific changes, you can override the `resque:setup` job in `lib/tasks` (ex: `lib/tasks/resque.rake`). GitHub's setup task looks like this:
171
+
172
+ ``` ruby
173
+ task "resque:setup" => :environment do
174
+ Grit::Git.git_timeout = 10.minutes
175
+ end
176
+ ```
177
+
178
+ We don't want the `git_timeout` as high as 10 minutes in our web app,
179
+ but in the Resque workers it's fine.
180
+
181
+ Running Workers
182
+ ---------------
183
+
184
+ Resque workers are rake tasks that run forever. They basically do this:
185
+
186
+ ``` ruby
187
+ start
188
+ loop do
189
+ if job = reserve
190
+ job.process
191
+ else
192
+ sleep 5 # Polling frequency = 5
193
+ end
194
+ end
195
+ shutdown
196
+ ```
197
+
198
+ Starting a worker is simple:
199
+
200
+ $ QUEUE=* rake resque:work
201
+
202
+ Or, you can start multiple workers:
203
+
204
+ $ COUNT=2 QUEUE=* rake resque:workers
205
+
206
+ This will spawn two Resque workers, each in its own process. Hitting
207
+ ctrl-c should be sufficient to stop them all.
208
+
209
+ #### Priorities and Queue Lists
210
+
211
+ Resque doesn't support numeric priorities but instead uses the order
212
+ of queues you give it. We call this list of queues the "queue list."
213
+
214
+ Let's say we add a `warm_cache` queue in addition to our `file_serve`
215
+ queue. We'd now start a worker like so:
216
+
217
+ $ QUEUES=file_serve,warm_cache rake resque:work
218
+
219
+ When the worker looks for new jobs, it will first check
220
+ `file_serve`. If it finds a job, it'll process it then check
221
+ `file_serve` again. It will keep checking `file_serve` until no more
222
+ jobs are available. At that point, it will check `warm_cache`. If it
223
+ finds a job it'll process it then check `file_serve` (repeating the
224
+ whole process).
225
+
226
+ In this way you can prioritize certain queues. At GitHub we start our
227
+ workers with something like this:
228
+
229
+ $ QUEUES=critical,archive,high,low rake resque:work
230
+
231
+ Notice the `archive` queue - it is specialized and in our future
232
+ architecture will only be run from a single machine.
233
+
234
+ At that point we'll start workers on our generalized background
235
+ machines with this command:
236
+
237
+ $ QUEUES=critical,high,low rake resque:work
238
+
239
+ And workers on our specialized archive machine with this command:
240
+
241
+ $ QUEUE=archive rake resque:work
242
+
243
+ #### Running All Queues
244
+
245
+ If you want your workers to work off of every queue, including new
246
+ queues created on the fly, you can use a splat:
247
+
248
+ $ QUEUE=* rake resque:work
249
+
250
+ Queues will be processed in alphabetical order.
251
+
252
+ Or, prioritize some queues above `*`:
253
+
254
+ # QUEUE=critical,* rake resque:work
255
+
256
+ #### Process IDs (PIDs)
257
+
258
+ There are scenarios where it's helpful to record the PID of a resque
259
+ worker process. Use the PIDFILE option for easy access to the PID:
260
+
261
+ $ PIDFILE=./resque.pid QUEUE=file_serve rake resque:work
262
+
263
+ #### Running in the background
264
+
265
+ There are scenarios where it's helpful for
266
+ the resque worker to run itself in the background (usually in combination with
267
+ PIDFILE). Use the BACKGROUND option so that rake will return as soon as the
268
+ worker is started.
269
+
270
+ $ PIDFILE=./resque.pid BACKGROUND=yes QUEUE=file_serve rake resque:work
271
+
272
+ #### Polling frequency
273
+
274
+ You can pass an INTERVAL option which is a float representing the polling frequency.
275
+ The default is 5 seconds, but for a semi-active app you may want to use a smaller value.
276
+
277
+ $ INTERVAL=0.1 QUEUE=file_serve rake resque:work
278
+
279
+ The Front End
280
+ -------------
281
+
282
+ Resque comes with a Sinatra-based front end for seeing what's up with
283
+ your queue.
284
+
285
+ ![The Front End](https://camo.githubusercontent.com/64d150a243987ffbc33f588bd6d7722a0bb8d69a/687474703a2f2f7475746f7269616c732e6a756d7073746172746c61622e636f6d2f696d616765732f7265737175655f6f766572766965772e706e67)
286
+
287
+ #### Standalone
288
+
289
+ If you've installed Resque as a gem running the front end standalone is easy:
290
+
291
+ $ resque-web
292
+
293
+ It's a thin layer around `rackup` so it's configurable as well:
294
+
295
+ $ resque-web -p 8282
296
+
297
+ If you have a Resque config file you want evaluated just pass it to
298
+ the script as the final argument:
299
+
300
+ $ resque-web -p 8282 rails_root/config/initializers/resque.rb
301
+
302
+ You can also set the namespace directly using `resque-web`:
303
+
304
+ $ resque-web -p 8282 -N myapp
305
+
306
+ or set the Redis connection string if you need to do something like select a different database:
307
+
308
+ $ resque-web -p 8282 -r localhost:6379:2
309
+
310
+ #### Passenger
311
+
312
+ Using Passenger? Resque ships with a `config.ru` you can use. See
313
+ Phusion's guide:
314
+
315
+ Apache: <https://www.phusionpassenger.com/library/deploy/apache/deploy/ruby/>
316
+ Nginx: <https://www.phusionpassenger.com/library/deploy/nginx/deploy/ruby/>
317
+
318
+ #### Rack::URLMap
319
+
320
+ If you want to load Resque on a subpath, possibly alongside other
321
+ apps, it's easy to do with Rack's `URLMap`:
322
+
323
+ ``` ruby
324
+ require 'resque/server'
325
+
326
+ run Rack::URLMap.new \
327
+ "/" => Your::App.new,
328
+ "/resque" => Resque::Server.new
329
+ ```
330
+
331
+ Check `examples/demo/config.ru` for a functional example (including
332
+ HTTP basic auth).
333
+
334
+ #### Rails 3+
335
+
336
+ You can also mount Resque on a subpath in your existing Rails 3 app by adding `require 'resque/server'` to the top of your routes file or in an initializer then adding this to `routes.rb`:
337
+
338
+ ``` ruby
339
+ mount Resque::Server.new, :at => "/resque"
340
+ ```
115
341
 
116
342
  Jobs
117
343
  ----
@@ -138,8 +364,7 @@ mention "foreground" and "background" because they make conceptual
138
364
  sense. You could easily be spidering sites and sticking data which
139
365
  needs to be crunched later into a queue.
140
366
 
141
-
142
- ### Persistence
367
+ #### Persistence
143
368
 
144
369
  Jobs are persisted to queues as JSON objects. Let's take our `Archive`
145
370
  example from above. We'll run the following code to create a job:
@@ -184,7 +409,7 @@ If your jobs were run against marshaled objects, they could
184
409
  potentially be operating on a stale record with out-of-date information.
185
410
 
186
411
 
187
- ### send_later / async
412
+ #### send_later / async
188
413
 
189
414
  Want something like DelayedJob's `send_later` or the ability to use
190
415
  instance methods instead of just methods for jobs? See the `examples/`
@@ -193,7 +418,7 @@ directory for goodies.
193
418
  We plan to provide first class `async` support in a future release.
194
419
 
195
420
 
196
- ### Failure
421
+ #### Failure
197
422
 
198
423
  If a job raises an exception, it is logged and handed off to the
199
424
  `Resque::Failure` module. Failures are logged either locally in Redis
@@ -217,57 +442,89 @@ Keep this in mind when writing your jobs: you may want to throw
217
442
  exceptions you would not normally throw in order to assist debugging.
218
443
 
219
444
 
220
- Workers
221
- -------
445
+ #### Rails example
222
446
 
223
- Resque workers are rake tasks that run forever. They basically do this:
447
+ If you are using ActiveJob here's how your job definition will look:
224
448
 
225
449
  ``` ruby
226
- start
227
- loop do
228
- if job = reserve
229
- job.process
230
- else
231
- sleep 5 # Polling frequency = 5
450
+ class ArchiveJob < ApplicationJob
451
+ queue_as :file_serve
452
+
453
+ def perform(repo_id, branch = 'master')
454
+ repo = Repository.find(repo_id)
455
+ repo.create_archive(branch)
232
456
  end
233
457
  end
234
- shutdown
235
458
  ```
236
459
 
237
- Starting a worker is simple. Here's our example from earlier:
460
+ ``` ruby
461
+ class Repository
462
+ def async_create_archive(branch)
463
+ ArchiveJob.perform_later(self.id, branch)
464
+ end
465
+ end
466
+ ```
238
467
 
239
- $ QUEUE=file_serve rake resque:work
468
+ It is important to run `ArchiveJob.perform_later(self.id, branch)` rather than `Resque.enqueue(Archive, self.id, branch)`.
469
+ Otherwise Resque will process the job without actually doing anything.
470
+ Even if you put an obviously buggy line like `0/0` in the `perform` method,
471
+ the job will still succeed.
240
472
 
241
- By default Resque won't know about your application's
242
- environment. That is, it won't be able to find and run your jobs - it
243
- needs to load your application into memory.
244
473
 
245
- If we've installed Resque as a Rails plugin, we might run this command
246
- from our RAILS_ROOT:
474
+ Configuration
475
+ -------------
247
476
 
248
- $ QUEUE=file_serve rake environment resque:work
477
+ #### Redis
249
478
 
250
- This will load the environment before starting a worker. Alternately
251
- we can define a `resque:setup` task with a dependency on the
252
- `environment` rake task:
479
+ You may want to change the Redis host and port Resque connects to, or
480
+ set various other options at startup.
253
481
 
254
- ``` ruby
255
- task "resque:setup" => :environment
256
- ```
482
+ Resque has a `redis` setter which can be given a string or a Redis
483
+ object. This means if you're already using Redis in your app, Resque
484
+ can re-use the existing connection.
485
+
486
+ String: `Resque.redis = 'localhost:6379'`
487
+
488
+ Redis: `Resque.redis = $redis`
489
+
490
+ For our rails app we have a `config/initializers/resque.rb` file where
491
+ we load `config/resque.yml` by hand and set the Redis information
492
+ appropriately.
493
+
494
+ Here's our `config/resque.yml`:
495
+
496
+ development: localhost:6379
497
+ test: localhost:6379
498
+ staging: redis1.se.github.com:6379
499
+ fi: localhost:6379
500
+ production: <%= ENV['REDIS_URL'] %>
257
501
 
258
- GitHub's setup task looks like this:
502
+ And our initializer:
259
503
 
260
504
  ``` ruby
261
- task "resque:setup" => :environment do
262
- Grit::Git.git_timeout = 10.minutes
263
- end
505
+ rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..'
506
+ rails_env = ENV['RAILS_ENV'] || 'development'
507
+ config_file = rails_root + '/config/resque.yml'
508
+
509
+ resque_config = YAML::load(ERB.new(IO.read(config_file)).result)
510
+ Resque.redis = resque_config[rails_env]
264
511
  ```
265
512
 
266
- We don't want the `git_timeout` as high as 10 minutes in our web app,
267
- but in the Resque workers it's fine.
513
+ Easy peasy! Why not just use `RAILS_ROOT` and `RAILS_ENV`? Because
514
+ this way we can tell our Sinatra app about the config file:
515
+
516
+ $ RAILS_ENV=production resque-web rails_root/config/initializers/resque.rb
517
+
518
+ Now everyone is on the same page.
519
+
520
+ Also, you could disable jobs queueing by setting 'inline' attribute.
521
+ For example, if you want to run all jobs in the same process for cucumber, try:
268
522
 
523
+ ``` ruby
524
+ Resque.inline = ENV['RAILS_ENV'] == "cucumber"
525
+ ```
269
526
 
270
- ### Logging
527
+ #### Logging
271
528
 
272
529
  Workers support basic logging to STDOUT.
273
530
 
@@ -285,19 +542,38 @@ If you want Resque to log to a file, in Rails do:
285
542
  Resque.logger = Logger.new(Rails.root.join('log', "#{Rails.env}_resque.log"))
286
543
  ```
287
544
 
288
- ### Storing Statistics
289
- Resque allows to store count of processed and failed jobs.
545
+ #### Namespaces
290
546
 
291
- By default it will store it in Redis using the keys `stats:processed` and `stats:failed`.
547
+ If you're running multiple, separate instances of Resque you may want
548
+ to namespace the keyspaces so they do not overlap. This is not unlike
549
+ the approach taken by many memcached clients.
292
550
 
293
- Some apps would want another stats store, or even a null store:
551
+ This feature is provided by the [redis-namespace](http://github.com/resque/redis-namespace) library, which
552
+ Resque uses by default to separate the keys it manages from other keys
553
+ in your Redis server.
294
554
 
295
- ```ruby
296
- # config/initializers/resque.rb
297
- class NullDataStore
298
- def stat(stat)
299
- 0
300
- end
555
+ Simply use the `Resque.redis.namespace` accessor:
556
+
557
+ ``` ruby
558
+ Resque.redis.namespace = "resque:GitHub"
559
+ ```
560
+
561
+ We recommend sticking this in your initializer somewhere after Redis
562
+ is configured.
563
+
564
+ #### Storing Statistics
565
+ Resque allows to store count of processed and failed jobs.
566
+
567
+ By default it will store it in Redis using the keys `stats:processed` and `stats:failed`.
568
+
569
+ Some apps would want another stats store, or even a null store:
570
+
571
+ ```ruby
572
+ # config/initializers/resque.rb
573
+ class NullDataStore
574
+ def stat(stat)
575
+ 0
576
+ end
301
577
 
302
578
  def increment_stat(stat, by)
303
579
  end
@@ -312,91 +588,65 @@ end
312
588
  Resque.stat_data_store = NullDataStore.new
313
589
  ```
314
590
 
315
- ### Process IDs (PIDs)
316
-
317
- There are scenarios where it's helpful to record the PID of a resque
318
- worker process. Use the PIDFILE option for easy access to the PID:
319
-
320
- $ PIDFILE=./resque.pid QUEUE=file_serve rake environment resque:work
321
-
322
- ### Running in the background
323
-
324
- There are scenarios where it's helpful for
325
- the resque worker to run itself in the background (usually in combination with
326
- PIDFILE). Use the BACKGROUND option so that rake will return as soon as the
327
- worker is started.
328
-
329
- $ PIDFILE=./resque.pid BACKGROUND=yes QUEUE=file_serve \
330
- rake environment resque:work
331
-
332
- ### Polling frequency
333
-
334
- You can pass an INTERVAL option which is a float representing the polling frequency.
335
- The default is 5 seconds, but for a semi-active app you may want to use a smaller value.
336
-
337
- $ INTERVAL=0.1 QUEUE=file_serve rake environment resque:work
338
-
339
- ### Priorities and Queue Lists
340
-
341
- Resque doesn't support numeric priorities but instead uses the order
342
- of queues you give it. We call this list of queues the "queue list."
343
-
344
- Let's say we add a `warm_cache` queue in addition to our `file_serve`
345
- queue. We'd now start a worker like so:
346
-
347
- $ QUEUES=file_serve,warm_cache rake resque:work
348
-
349
- When the worker looks for new jobs, it will first check
350
- `file_serve`. If it finds a job, it'll process it then check
351
- `file_serve` again. It will keep checking `file_serve` until no more
352
- jobs are available. At that point, it will check `warm_cache`. If it
353
- finds a job it'll process it then check `file_serve` (repeating the
354
- whole process).
355
-
356
- In this way you can prioritize certain queues. At GitHub we start our
357
- workers with something like this:
358
-
359
- $ QUEUES=critical,archive,high,low rake resque:work
360
-
361
- Notice the `archive` queue - it is specialized and in our future
362
- architecture will only be run from a single machine.
363
-
364
- At that point we'll start workers on our generalized background
365
- machines with this command:
366
-
367
- $ QUEUES=critical,high,low rake resque:work
368
-
369
- And workers on our specialized archive machine with this command:
591
+ Plugins and Hooks
592
+ -----------------
370
593
 
371
- $ QUEUE=archive rake resque:work
594
+ For a list of available plugins see
595
+ <https://github.com/resque/resque/wiki/plugins>.
372
596
 
597
+ If you'd like to write your own plugin, or want to customize Resque
598
+ using hooks (such as `Resque.after_fork`), see
599
+ [docs/HOOKS.md](http://github.com/resque/resque/blob/master/docs/HOOKS.md).
373
600
 
374
- ### Running All Queues
375
601
 
376
- If you want your workers to work off of every queue, including new
377
- queues created on the fly, you can use a splat:
602
+ Additional Information
603
+ ----------------------
378
604
 
379
- $ QUEUE=* rake resque:work
605
+ #### Resque vs DelayedJob
380
606
 
381
- Queues will be processed in alphabetical order.
607
+ How does Resque compare to DelayedJob, and why would you choose one
608
+ over the other?
382
609
 
610
+ * Resque supports multiple queues
611
+ * DelayedJob supports finer grained priorities
612
+ * Resque workers are resilient to memory leaks / bloat
613
+ * DelayedJob workers are extremely simple and easy to modify
614
+ * Resque requires Redis
615
+ * DelayedJob requires ActiveRecord
616
+ * Resque can only place JSONable Ruby objects on a queue as arguments
617
+ * DelayedJob can place _any_ Ruby object on its queue as arguments
618
+ * Resque includes a Sinatra app for monitoring what's going on
619
+ * DelayedJob can be queried from within your Rails app if you want to
620
+ add an interface
383
621
 
384
- ### Running Multiple Workers
622
+ If you're doing Rails development, you already have a database and
623
+ ActiveRecord. DelayedJob is super easy to setup and works great.
624
+ GitHub used it for many months to process almost 200 million jobs.
385
625
 
386
- At GitHub we use god to start and stop multiple workers. A sample god
387
- configuration file is included under `examples/god`. We recommend this
388
- method.
626
+ Choose Resque if:
389
627
 
390
- If you'd like to run multiple workers in development mode, you can do
391
- so using the `resque:workers` rake task:
628
+ * You need multiple queues
629
+ * You don't care / dislike numeric priorities
630
+ * You don't need to persist every Ruby object ever
631
+ * You have potentially huge queues
632
+ * You want to see what's going on
633
+ * You expect a lot of failure / chaos
634
+ * You can setup Redis
635
+ * You're not running short on RAM
392
636
 
393
- $ COUNT=5 QUEUE=* rake resque:workers
637
+ Choose DelayedJob if:
394
638
 
395
- This will spawn five Resque workers, each in its own process. Hitting
396
- ctrl-c should be sufficient to stop them all.
639
+ * You like numeric priorities
640
+ * You're not doing a gigantic amount of jobs each day
641
+ * Your queue stays small and nimble
642
+ * There is not a lot failure / chaos
643
+ * You want to easily throw anything on the queue
644
+ * You don't want to setup Redis
397
645
 
646
+ In no way is Resque a "better" DelayedJob, so make sure you pick the
647
+ tool that's best for your app.
398
648
 
399
- ### Forking
649
+ #### Forking
400
650
 
401
651
  On certain platforms, when a Resque worker reserves a job it
402
652
  immediately forks a child process. The child processes the job then
@@ -437,8 +687,7 @@ complicated.
437
687
 
438
688
  Workers instead handle their own state.
439
689
 
440
-
441
- ### Parents and Children
690
+ #### Parents and Children
442
691
 
443
692
  Here's a parent / child pair doing some work:
444
693
 
@@ -459,7 +708,7 @@ waiting for work on:
459
708
  92099 resque: Waiting for file_serve,warm_cache
460
709
 
461
710
 
462
- ### Signals
711
+ #### Signals
463
712
 
464
713
  Resque workers respond to a few different signals:
465
714
 
@@ -481,7 +730,7 @@ If you want to stop processing jobs, but want to leave the worker running
481
730
  (for example, to temporarily alleviate load), use `USR2` to stop processing,
482
731
  then `CONT` to start it again.
483
732
 
484
- #### Signals on Heroku
733
+ #### Heroku
485
734
 
486
735
  When shutting down processes, Heroku sends every process a TERM signal at the
487
736
  same time. By default this causes an immediate shutdown of any running job
@@ -503,11 +752,25 @@ time to complete before being forced to die.
503
752
 
504
753
  * `RESQUE_TERM_TIMEOUT` - By default you have a few seconds to handle `Resque::TermException` in your job. `RESQUE_TERM_TIMEOUT` and `RESQUE_PRE_SHUTDOWN_TIMEOUT` must be lower than the [heroku dyno timeout](https://devcenter.heroku.com/articles/limits#exit-timeout).
505
754
 
506
- ### Mysql::Error: MySQL server has gone away
755
+ #### Monitoring
756
+
757
+ ##### god
758
+
759
+ If you're using god to monitor Resque, we have provided example
760
+ configs in `examples/god/`. One is for starting / stopping workers,
761
+ the other is for killing workers that have been running too long.
762
+
763
+ ##### monit
764
+
765
+ If you're using monit, `examples/monit/resque.monit` is provided free
766
+ of charge. This is **not** used by GitHub in production, so please
767
+ send patches for any tweaks or improvements you can make to it.
768
+
769
+ #### Mysql::Error: MySQL server has gone away
507
770
 
508
771
  If your workers remain idle for too long they may lose their MySQL connection. Depending on your version of Rails, we recommend the following:
509
772
 
510
- #### Rails 3.x
773
+ ##### Rails 3.x
511
774
  In your `perform` method, add the following line:
512
775
 
513
776
  ``` ruby
@@ -523,7 +786,7 @@ The Rails doc says the following about `verify_active_connections!`:
523
786
 
524
787
  Verify active connections and remove and disconnect connections associated with stale threads.
525
788
 
526
- #### Rails 4.x
789
+ ##### Rails 4.x
527
790
 
528
791
  In your `perform` method, instead of `verify_active_connections!`, use:
529
792
 
@@ -540,363 +803,6 @@ From the Rails docs on [`clear_active_connections!`](http://api.rubyonrails.org/
540
803
 
541
804
  Returns any connections in use by the current thread back to the pool, and also returns connections to the pool cached by threads that are no longer alive.
542
805
 
543
-
544
-
545
- The Front End
546
- -------------
547
-
548
- Resque comes with a Sinatra-based front end for seeing what's up with
549
- your queue.
550
-
551
- ![The Front End](https://camo.githubusercontent.com/64d150a243987ffbc33f588bd6d7722a0bb8d69a/687474703a2f2f7475746f7269616c732e6a756d7073746172746c61622e636f6d2f696d616765732f7265737175655f6f766572766965772e706e67)
552
-
553
- ### Standalone
554
-
555
- If you've installed Resque as a gem running the front end standalone is easy:
556
-
557
- $ resque-web
558
-
559
- It's a thin layer around `rackup` so it's configurable as well:
560
-
561
- $ resque-web -p 8282
562
-
563
- If you have a Resque config file you want evaluated just pass it to
564
- the script as the final argument:
565
-
566
- $ resque-web -p 8282 rails_root/config/initializers/resque.rb
567
-
568
- You can also set the namespace directly using `resque-web`:
569
-
570
- $ resque-web -p 8282 -N myapp
571
-
572
- or set the Redis connection string if you need to do something like select a different database:
573
-
574
- $ resque-web -p 8282 -r localhost:6379:2
575
-
576
- ### Passenger
577
-
578
- Using Passenger? Resque ships with a `config.ru` you can use. See
579
- Phusion's guide:
580
-
581
- Apache: <https://www.phusionpassenger.com/library/deploy/apache/deploy/ruby/>
582
- Nginx: <https://www.phusionpassenger.com/library/deploy/nginx/deploy/ruby/>
583
-
584
- ### Rack::URLMap
585
-
586
- If you want to load Resque on a subpath, possibly alongside other
587
- apps, it's easy to do with Rack's `URLMap`:
588
-
589
- ``` ruby
590
- require 'resque/server'
591
-
592
- run Rack::URLMap.new \
593
- "/" => Your::App.new,
594
- "/resque" => Resque::Server.new
595
- ```
596
-
597
- Check `examples/demo/config.ru` for a functional example (including
598
- HTTP basic auth).
599
-
600
- ### Rails 3
601
-
602
- You can also mount Resque on a subpath in your existing Rails 3 app by adding `require 'resque/server'` to the top of your routes file or in an initializer then adding this to `routes.rb`:
603
-
604
- ``` ruby
605
- mount Resque::Server.new, :at => "/resque"
606
- ```
607
-
608
-
609
- Resque vs DelayedJob
610
- --------------------
611
-
612
- How does Resque compare to DelayedJob, and why would you choose one
613
- over the other?
614
-
615
- * Resque supports multiple queues
616
- * DelayedJob supports finer grained priorities
617
- * Resque workers are resilient to memory leaks / bloat
618
- * DelayedJob workers are extremely simple and easy to modify
619
- * Resque requires Redis
620
- * DelayedJob requires ActiveRecord
621
- * Resque can only place JSONable Ruby objects on a queue as arguments
622
- * DelayedJob can place _any_ Ruby object on its queue as arguments
623
- * Resque includes a Sinatra app for monitoring what's going on
624
- * DelayedJob can be queried from within your Rails app if you want to
625
- add an interface
626
-
627
- If you're doing Rails development, you already have a database and
628
- ActiveRecord. DelayedJob is super easy to setup and works great.
629
- GitHub used it for many months to process almost 200 million jobs.
630
-
631
- Choose Resque if:
632
-
633
- * You need multiple queues
634
- * You don't care / dislike numeric priorities
635
- * You don't need to persist every Ruby object ever
636
- * You have potentially huge queues
637
- * You want to see what's going on
638
- * You expect a lot of failure / chaos
639
- * You can setup Redis
640
- * You're not running short on RAM
641
-
642
- Choose DelayedJob if:
643
-
644
- * You like numeric priorities
645
- * You're not doing a gigantic amount of jobs each day
646
- * Your queue stays small and nimble
647
- * There is not a lot failure / chaos
648
- * You want to easily throw anything on the queue
649
- * You don't want to setup Redis
650
-
651
- In no way is Resque a "better" DelayedJob, so make sure you pick the
652
- tool that's best for your app.
653
-
654
- Resque Dependencies
655
- -------------------
656
-
657
- $ gem install bundler
658
- $ bundle install
659
-
660
-
661
- Installing Resque
662
- -----------------
663
-
664
- ### In a Rack app, as a gem
665
-
666
- First install the gem.
667
-
668
- $ gem install resque
669
-
670
- Next include it in your application.
671
-
672
- ``` ruby
673
- require 'resque'
674
- ```
675
-
676
- Now start your application:
677
-
678
- rackup config.ru
679
-
680
- That's it! You can now create Resque jobs from within your app.
681
-
682
- To start a worker, create a Rakefile in your app's root (or add this
683
- to an existing Rakefile):
684
-
685
- ``` ruby
686
- require 'your/app'
687
- require 'resque/tasks'
688
- ```
689
-
690
- If you're using Rails 5.x, include the following in lib/tasks/resque.rake:
691
-
692
- ```ruby
693
- require 'resque/tasks'
694
- task 'resque:setup' => :environment
695
- ```
696
-
697
- Now:
698
-
699
- $ QUEUE=* rake resque:work
700
-
701
- Alternately you can define a `resque:setup` hook in your Rakefile if you
702
- don't want to load your app every time rake runs.
703
-
704
-
705
- ### In a Rails 2.x app, as a gem
706
-
707
- First install the gem.
708
-
709
- $ gem install resque
710
-
711
- Next include it in your application.
712
-
713
- $ cat config/initializers/load_resque.rb
714
- require 'resque'
715
-
716
- Now start your application:
717
-
718
- $ ./script/server
719
-
720
- That's it! You can now create Resque jobs from within your app.
721
-
722
- To start a worker, add this to your Rakefile in `RAILS_ROOT`:
723
-
724
- ``` ruby
725
- require 'resque/tasks'
726
- ```
727
-
728
- Now:
729
-
730
- $ QUEUE=* rake environment resque:work
731
-
732
- Don't forget you can define a `resque:setup` hook in
733
- `lib/tasks/whatever.rake` that loads the `environment` task every time.
734
-
735
-
736
- ### In a Rails 2.x app, as a plugin
737
-
738
- $ ./script/plugin install git://github.com/resque/resque
739
-
740
- That's it! Resque will automatically be available when your Rails app
741
- loads.
742
-
743
- To start a worker:
744
-
745
- $ QUEUE=* rake environment resque:work
746
-
747
- Don't forget you can define a `resque:setup` hook in
748
- `lib/tasks/whatever.rake` that loads the `environment` task every time.
749
-
750
-
751
- ### In a Rails 3.x or 4.x app, as a gem
752
-
753
- First include it in your Gemfile.
754
-
755
- $ cat Gemfile
756
- ...
757
- gem 'resque'
758
- ...
759
-
760
- Next install it with Bundler.
761
-
762
- $ bundle install
763
-
764
- Now start your application:
765
-
766
- $ rails server
767
-
768
- That's it! You can now create Resque jobs from within your app.
769
-
770
- To start a worker, add this to a file in `lib/tasks` (ex:
771
- `lib/tasks/resque.rake`):
772
-
773
- ``` ruby
774
- require 'resque/tasks'
775
- ```
776
-
777
- Now:
778
-
779
- $ QUEUE=* rake environment resque:work
780
-
781
- Don't forget you can define a `resque:setup` hook in
782
- `lib/tasks/whatever.rake` that loads the `environment` task every time.
783
-
784
-
785
- Configuration
786
- -------------
787
-
788
- You may want to change the Redis host and port Resque connects to, or
789
- set various other options at startup.
790
-
791
- Resque has a `redis` setter which can be given a string or a Redis
792
- object. This means if you're already using Redis in your app, Resque
793
- can re-use the existing connection.
794
-
795
- String: `Resque.redis = 'localhost:6379'`
796
-
797
- Redis: `Resque.redis = $redis`
798
-
799
- For our rails app we have a `config/initializers/resque.rb` file where
800
- we load `config/resque.yml` by hand and set the Redis information
801
- appropriately.
802
-
803
- Here's our `config/resque.yml`:
804
-
805
- development: localhost:6379
806
- test: localhost:6379
807
- staging: redis1.se.github.com:6379
808
- fi: localhost:6379
809
- production: <%= ENV['REDIS_URL'] %>
810
-
811
- And our initializer:
812
-
813
- ``` ruby
814
- rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..'
815
- rails_env = ENV['RAILS_ENV'] || 'development'
816
- config_file = rails_root + '/config/resque.yml'
817
-
818
- resque_config = YAML::load(ERB.new(IO.read(config_file)).result)
819
- Resque.redis = resque_config[rails_env]
820
- ```
821
-
822
- Easy peasy! Why not just use `RAILS_ROOT` and `RAILS_ENV`? Because
823
- this way we can tell our Sinatra app about the config file:
824
-
825
- $ RAILS_ENV=production resque-web rails_root/config/initializers/resque.rb
826
-
827
- Now everyone is on the same page.
828
-
829
- Also, you could disable jobs queueing by setting 'inline' attribute.
830
- For example, if you want to run all jobs in the same process for cucumber, try:
831
-
832
- ``` ruby
833
- Resque.inline = ENV['RAILS_ENV'] == "cucumber"
834
- ```
835
-
836
-
837
- Plugins and Hooks
838
- -----------------
839
-
840
- For a list of available plugins see
841
- <http://wiki.github.com/resque/resque/plugins>.
842
-
843
- If you'd like to write your own plugin, or want to customize Resque
844
- using hooks (such as `Resque.after_fork`), see
845
- [docs/HOOKS.md](http://github.com/resque/resque/blob/master/docs/HOOKS.md).
846
-
847
-
848
- Namespaces
849
- ----------
850
-
851
- If you're running multiple, separate instances of Resque you may want
852
- to namespace the keyspaces so they do not overlap. This is not unlike
853
- the approach taken by many memcached clients.
854
-
855
- This feature is provided by the [redis-namespace][rs] library, which
856
- Resque uses by default to separate the keys it manages from other keys
857
- in your Redis server.
858
-
859
- Simply use the `Resque.redis.namespace` accessor:
860
-
861
- ``` ruby
862
- Resque.redis.namespace = "resque:GitHub"
863
- ```
864
-
865
- We recommend sticking this in your initializer somewhere after Redis
866
- is configured.
867
-
868
-
869
- Demo
870
- ----
871
-
872
- Resque ships with a demo Sinatra app for creating jobs that are later
873
- processed in the background.
874
-
875
- Try it out by looking at the README, found at `examples/demo/README.markdown`.
876
-
877
-
878
- Monitoring
879
- ----------
880
-
881
- ### god
882
-
883
- If you're using god to monitor Resque, we have provided example
884
- configs in `examples/god/`. One is for starting / stopping workers,
885
- the other is for killing workers that have been running too long.
886
-
887
- ### monit
888
-
889
- If you're using monit, `examples/monit/resque.monit` is provided free
890
- of charge. This is **not** used by GitHub in production, so please
891
- send patches for any tweaks or improvements you can make to it.
892
-
893
-
894
- Questions
895
- ---------
896
-
897
- Please add them to the [FAQ](https://github.com/resque/resque/wiki/FAQ) or open an issue on this repo.
898
-
899
-
900
806
  Development
901
807
  -----------
902
808
 
@@ -926,26 +832,27 @@ it:
926
832
  If you get an error requiring any of the dependencies, you may have
927
833
  failed to install them or be seeing load path issues.
928
834
 
835
+ #### Demo
836
+ Resque ships with a demo Sinatra app for creating jobs that are later
837
+ processed in the background.
929
838
 
930
- Contributing
931
- ------------
839
+ Try it out by looking at the README, found at `examples/demo/README.markdown`.
840
+
841
+ #### Contributing
932
842
 
933
843
  Read [CONTRIBUTING.md](CONTRIBUTING.md) first.
934
844
 
935
845
  Once you've made your great commits:
936
846
 
937
- 1. [Fork][1] Resque
847
+ 1. [Fork](http://help.github.com/forking/) Resque
938
848
  2. Create a topic branch - `git checkout -b my_branch`
939
849
  3. Push to your branch - `git push origin my_branch`
940
850
  4. Create a [Pull Request](http://help.github.com/pull-requests/) from your branch
941
- 5. That's it!
942
-
943
851
 
944
- Mailing List
945
- ------------
946
-
947
- This mailing list is no longer maintained. The archive can be found at <http://librelist.com/browser/resque/>.
852
+ Questions
853
+ ---------
948
854
 
855
+ Please add them to the [FAQ](https://github.com/resque/resque/wiki/FAQ) or open an issue on this repo.
949
856
 
950
857
  Meta
951
858
  ----
@@ -954,21 +861,12 @@ Meta
954
861
  * Home: <http://github.com/resque/resque>
955
862
  * Docs: <http://rubydoc.info/gems/resque>
956
863
  * Bugs: <http://github.com/resque/resque/issues>
957
- * List: <resque@librelist.com>
958
864
  * Chat: <irc://irc.freenode.net/resque>
959
- * Gems: <http://gemcutter.org/gems/resque>
960
-
961
- This project uses [Semantic Versioning][sv].
865
+ * Gems: <https://rubygems.org/gems/resque>
962
866
 
867
+ This project uses [Semantic Versioning](http://semver.org/)
963
868
 
964
869
  Author
965
870
  ------
966
871
 
967
872
  Chris Wanstrath :: chris@ozmm.org :: @defunkt
968
-
969
- [0]: http://github.com/blog/542-introducing-resque
970
- [1]: http://help.github.com/forking/
971
- [2]: http://github.com/resque/resque/issues
972
- [sv]: http://semver.org/
973
- [rs]: http://github.com/resque/redis-namespace
974
- [cb]: http://wiki.github.com/resque/resque/contributing