resque 2.0.0 → 2.6.0

Sign up to get free protection for your applications and to get access to all the features.
data/README.markdown CHANGED
@@ -2,12 +2,17 @@ Resque
2
2
  ======
3
3
 
4
4
  [![Gem Version](https://badge.fury.io/rb/resque.svg)](https://rubygems.org/gems/resque)
5
- [![Build Status](https://travis-ci.org/resque/resque.svg)](https://travis-ci.org/resque/resque)
5
+ [![Build Status](https://github.com/resque/resque/actions/workflows/ci.yml/badge.svg)](https://github.com/resque/resque/actions/workflows/ci.yml)
6
+
7
+ Introduction
8
+ ------------
6
9
 
7
10
  Resque (pronounced like "rescue") is a Redis-backed library for creating
8
11
  background jobs, placing those jobs on multiple queues, and processing
9
12
  them later.
10
13
 
14
+ For the backstory, philosophy, and history of Resque's beginnings, please see [the blog post](http://github.com/blog/542-introducing-resque) (2009).
15
+
11
16
  Background jobs can be any Ruby class or module that responds to
12
17
  `perform`. Your existing classes can easily be converted to background
13
18
  jobs or you can create new classes specifically to do work. Or, you
@@ -20,10 +25,11 @@ three parts:
20
25
  2. A Rake task for starting a worker which processes jobs
21
26
  3. A Sinatra app for monitoring queues, jobs, and workers.
22
27
 
23
- Resque workers can be distributed between multiple machines,
24
- support priorities, are resilient to memory bloat / "leaks," are
25
- optimized for REE (but work on MRI and JRuby), tell you what they're
26
- doing, and expect failure.
28
+ Resque workers can be given multiple queues (a "queue list"),
29
+ distributed between multiple machines,
30
+ run anywhere with network access to the Redis server,
31
+ support priorities, are resilient to memory bloat / "leaks,"
32
+ tell you what they're doing, and expect failure.
27
33
 
28
34
  Resque queues are persistent; support constant time, atomic push and
29
35
  pop (thanks to Redis); provide visibility into their contents; and
@@ -36,19 +42,15 @@ general usage stats, and helps you track failures.
36
42
  Resque now supports Ruby 2.3.0 and above.
37
43
  We will also only be supporting Redis 3.0 and above going forward.
38
44
 
45
+ ### Note on the future of Resque
39
46
 
40
- The Blog Post
41
- -------------
42
-
43
- For the backstory, philosophy, and history of Resque's beginnings,
44
- please see [the blog post][0].
45
-
47
+ Would you like to be involved in Resque? Do you have thoughts about what
48
+ Resque should be and do going forward? There's currently an [open discussion here](https://github.com/resque/resque/issues/1759)
49
+ on just that topic, so please feel free to join in. We'd love to hear your thoughts
50
+ and/or have people volunteer to be a part of the project!
46
51
 
47
- Overview
48
- --------
49
-
50
- Resque allows you to create jobs and place them on a queue, then,
51
- later, pull those jobs off the queue and process them.
52
+ Example
53
+ -------
52
54
 
53
55
  Resque jobs are Ruby classes (or modules) which respond to the
54
56
  `perform` method. Here's an example:
@@ -108,10 +110,219 @@ This starts one Resque worker and tells it to work off the
108
110
  find any more, at which point it will sleep for a small period and
109
111
  repeatedly poll the queue for more jobs.
110
112
 
111
- Workers can be given multiple queues (a "queue list") and run on
112
- multiple machines. In fact they can be run anywhere with network
113
- access to the Redis server.
113
+ Installation
114
+ ------------
115
+
116
+ Add the gem to your Gemfile:
117
+
118
+ gem 'resque'
119
+
120
+ Next, install it with Bundler:
121
+
122
+ $ bundle
123
+
124
+ #### Rack
125
+
126
+ In your Rakefile, or some other file in `lib/tasks` (ex: `lib/tasks/resque.rake`), load the resque rake tasks:
127
+
128
+ ``` ruby
129
+ require 'resque'
130
+ require 'resque/tasks'
131
+ require 'your/app' # Include this line if you want your workers to have access to your application
132
+ ```
133
+
134
+ #### Rails
135
+
136
+ To make resque specific changes, you can override the `resque:setup` job in `lib/tasks` (ex: `lib/tasks/resque.rake`). GitHub's setup task looks like this:
137
+
138
+ ``` ruby
139
+ task "resque:setup" => :environment do
140
+ Grit::Git.git_timeout = 10.minutes
141
+ end
142
+ ```
143
+
144
+ We don't want the `git_timeout` as high as 10 minutes in our web app,
145
+ but in the Resque workers it's fine.
146
+
147
+ Running Workers
148
+ ---------------
149
+
150
+ Resque workers are rake tasks that run forever. They basically do this:
151
+
152
+ ``` ruby
153
+ start
154
+ loop do
155
+ if job = reserve
156
+ job.process
157
+ else
158
+ sleep 5 # Polling frequency = 5
159
+ end
160
+ end
161
+ shutdown
162
+ ```
163
+
164
+ Starting a worker is simple:
165
+
166
+ $ QUEUE=* rake resque:work
167
+
168
+ Or, you can start multiple workers:
169
+
170
+ $ COUNT=2 QUEUE=* rake resque:workers
171
+
172
+ This will spawn two Resque workers, each in its own process. Hitting
173
+ ctrl-c should be sufficient to stop them all.
174
+
175
+ #### Priorities and Queue Lists
176
+
177
+ Resque doesn't support numeric priorities but instead uses the order
178
+ of queues you give it. We call this list of queues the "queue list."
179
+
180
+ Let's say we add a `warm_cache` queue in addition to our `file_serve`
181
+ queue. We'd now start a worker like so:
182
+
183
+ $ QUEUES=file_serve,warm_cache rake resque:work
184
+
185
+ When the worker looks for new jobs, it will first check
186
+ `file_serve`. If it finds a job, it'll process it then check
187
+ `file_serve` again. It will keep checking `file_serve` until no more
188
+ jobs are available. At that point, it will check `warm_cache`. If it
189
+ finds a job it'll process it then check `file_serve` (repeating the
190
+ whole process).
191
+
192
+ In this way you can prioritize certain queues. At GitHub we start our
193
+ workers with something like this:
194
+
195
+ $ QUEUES=critical,archive,high,low rake resque:work
196
+
197
+ Notice the `archive` queue - it is specialized and in our future
198
+ architecture will only be run from a single machine.
199
+
200
+ At that point we'll start workers on our generalized background
201
+ machines with this command:
202
+
203
+ $ QUEUES=critical,high,low rake resque:work
204
+
205
+ And workers on our specialized archive machine with this command:
206
+
207
+ $ QUEUE=archive rake resque:work
208
+
209
+ #### Running All Queues
210
+
211
+ If you want your workers to work off of every queue, including new
212
+ queues created on the fly, you can use a splat:
213
+
214
+ $ QUEUE=* rake resque:work
215
+
216
+ Queues will be processed in alphabetical order.
217
+
218
+ Or, prioritize some queues above `*`:
219
+
220
+ # QUEUE=critical,* rake resque:work
221
+
222
+ #### Running All Queues Except for Some
223
+
224
+ If you want your workers to work off of all queues except for some,
225
+ you can use negation:
226
+
227
+ $ QUEUE=*,!low rake resque:work
228
+
229
+ Negated globs also work. The following will instruct workers to work
230
+ off of all queues except those beginning with `file_`:
231
+
232
+ $ QUEUE=*,!file_* rake resque:work
233
+
234
+ Note that the order in which negated queues are specified does not
235
+ matter, so `QUEUE=*,!file_*` and `QUEUE=!file_*,*` will have the same
236
+ effect.
237
+
238
+ #### Process IDs (PIDs)
239
+
240
+ There are scenarios where it's helpful to record the PID of a resque
241
+ worker process. Use the PIDFILE option for easy access to the PID:
242
+
243
+ $ PIDFILE=./resque.pid QUEUE=file_serve rake resque:work
244
+
245
+ #### Running in the background
246
+
247
+ There are scenarios where it's helpful for
248
+ the resque worker to run itself in the background (usually in combination with
249
+ PIDFILE). Use the BACKGROUND option so that rake will return as soon as the
250
+ worker is started.
251
+
252
+ $ PIDFILE=./resque.pid BACKGROUND=yes QUEUE=file_serve rake resque:work
253
+
254
+ #### Polling frequency
255
+
256
+ You can pass an INTERVAL option which is a float representing the polling frequency.
257
+ The default is 5 seconds, but for a semi-active app you may want to use a smaller value.
258
+
259
+ $ INTERVAL=0.1 QUEUE=file_serve rake resque:work
260
+
261
+ When INTERVAL is set to 0 it will run until the queue is empty and then
262
+ shutdown the worker, instead of waiting for new jobs.
263
+
264
+ The Front End
265
+ -------------
266
+
267
+ Resque comes with a Sinatra-based front end for seeing what's up with
268
+ your queue.
269
+
270
+ ![The Front End](https://camo.githubusercontent.com/64d150a243987ffbc33f588bd6d7722a0bb8d69a/687474703a2f2f7475746f7269616c732e6a756d7073746172746c61622e636f6d2f696d616765732f7265737175655f6f766572766965772e706e67)
271
+
272
+ #### Standalone
273
+
274
+ If you've installed Resque as a gem running the front end standalone is easy:
275
+
276
+ $ resque-web
277
+
278
+ It's a thin layer around `rackup` so it's configurable as well:
279
+
280
+ $ resque-web -p 8282
281
+
282
+ If you have a Resque config file you want evaluated just pass it to
283
+ the script as the final argument:
284
+
285
+ $ resque-web -p 8282 rails_root/config/initializers/resque.rb
286
+
287
+ You can also set the namespace directly using `resque-web`:
288
+
289
+ $ resque-web -p 8282 -N myapp
290
+
291
+ or set the Redis connection string if you need to do something like select a different database:
292
+
293
+ $ resque-web -p 8282 -r localhost:6379:2
294
+
295
+ #### Passenger
296
+
297
+ Using Passenger? Resque ships with a `config.ru` you can use. See
298
+ Phusion's guide:
299
+
300
+ Apache: <https://www.phusionpassenger.com/library/deploy/apache/deploy/ruby/>
301
+ Nginx: <https://www.phusionpassenger.com/library/deploy/nginx/deploy/ruby/>
302
+
303
+ #### Rack::URLMap
304
+
305
+ If you want to load Resque on a subpath, possibly alongside other
306
+ apps, it's easy to do with Rack's `URLMap`:
307
+
308
+ ``` ruby
309
+ require 'resque/server'
310
+
311
+ run Rack::URLMap.new \
312
+ "/" => Your::App.new,
313
+ "/resque" => Resque::Server.new
314
+ ```
315
+
316
+ Check `examples/demo/config.ru` for a functional example (including
317
+ HTTP basic auth).
318
+
319
+ #### Rails
114
320
 
321
+ You can also mount Resque on a subpath in your existing Rails app by adding `require 'resque/server'` to the top of your routes file or in an initializer then adding this to `routes.rb`:
322
+
323
+ ``` ruby
324
+ mount Resque::Server.new, :at => "/resque"
325
+ ```
115
326
 
116
327
  Jobs
117
328
  ----
@@ -138,8 +349,7 @@ mention "foreground" and "background" because they make conceptual
138
349
  sense. You could easily be spidering sites and sticking data which
139
350
  needs to be crunched later into a queue.
140
351
 
141
-
142
- ### Persistence
352
+ #### Persistence
143
353
 
144
354
  Jobs are persisted to queues as JSON objects. Let's take our `Archive`
145
355
  example from above. We'll run the following code to create a job:
@@ -184,7 +394,7 @@ If your jobs were run against marshaled objects, they could
184
394
  potentially be operating on a stale record with out-of-date information.
185
395
 
186
396
 
187
- ### send_later / async
397
+ #### send_later / async
188
398
 
189
399
  Want something like DelayedJob's `send_later` or the ability to use
190
400
  instance methods instead of just methods for jobs? See the `examples/`
@@ -193,7 +403,7 @@ directory for goodies.
193
403
  We plan to provide first class `async` support in a future release.
194
404
 
195
405
 
196
- ### Failure
406
+ #### Failure
197
407
 
198
408
  If a job raises an exception, it is logged and handed off to the
199
409
  `Resque::Failure` module. Failures are logged either locally in Redis
@@ -217,57 +427,89 @@ Keep this in mind when writing your jobs: you may want to throw
217
427
  exceptions you would not normally throw in order to assist debugging.
218
428
 
219
429
 
220
- Workers
221
- -------
430
+ #### Rails example
222
431
 
223
- Resque workers are rake tasks that run forever. They basically do this:
432
+ If you are using ActiveJob here's how your job definition will look:
224
433
 
225
434
  ``` ruby
226
- start
227
- loop do
228
- if job = reserve
229
- job.process
230
- else
231
- sleep 5 # Polling frequency = 5
435
+ class ArchiveJob < ApplicationJob
436
+ queue_as :file_serve
437
+
438
+ def perform(repo_id, branch = 'master')
439
+ repo = Repository.find(repo_id)
440
+ repo.create_archive(branch)
441
+ end
442
+ end
443
+ ```
444
+
445
+ ``` ruby
446
+ class Repository
447
+ def async_create_archive(branch)
448
+ ArchiveJob.perform_later(self.id, branch)
232
449
  end
233
450
  end
234
- shutdown
235
451
  ```
236
452
 
237
- Starting a worker is simple. Here's our example from earlier:
453
+ It is important to run `ArchiveJob.perform_later(self.id, branch)` rather than `Resque.enqueue(Archive, self.id, branch)`.
454
+ Otherwise Resque will process the job without actually doing anything.
455
+ Even if you put an obviously buggy line like `0/0` in the `perform` method,
456
+ the job will still succeed.
238
457
 
239
- $ QUEUE=file_serve rake resque:work
240
458
 
241
- By default Resque won't know about your application's
242
- environment. That is, it won't be able to find and run your jobs - it
243
- needs to load your application into memory.
459
+ Configuration
460
+ -------------
244
461
 
245
- If we've installed Resque as a Rails plugin, we might run this command
246
- from our RAILS_ROOT:
462
+ #### Redis
247
463
 
248
- $ QUEUE=file_serve rake environment resque:work
464
+ You may want to change the Redis host and port Resque connects to, or
465
+ set various other options at startup.
249
466
 
250
- This will load the environment before starting a worker. Alternately
251
- we can define a `resque:setup` task with a dependency on the
252
- `environment` rake task:
467
+ Resque has a `redis` setter which can be given a string or a Redis
468
+ object. This means if you're already using Redis in your app, Resque
469
+ can re-use the existing connection.
253
470
 
254
- ``` ruby
255
- task "resque:setup" => :environment
256
- ```
471
+ String: `Resque.redis = 'localhost:6379'`
472
+
473
+ Redis: `Resque.redis = $redis`
474
+
475
+ For our rails app we have a `config/initializers/resque.rb` file where
476
+ we load `config/resque.yml` by hand and set the Redis information
477
+ appropriately.
478
+
479
+ Here's our `config/resque.yml`:
480
+
481
+ development: localhost:6379
482
+ test: localhost:6379
483
+ staging: redis1.se.github.com:6379
484
+ fi: localhost:6379
485
+ production: <%= ENV['REDIS_URL'] %>
257
486
 
258
- GitHub's setup task looks like this:
487
+ And our initializer:
259
488
 
260
489
  ``` ruby
261
- task "resque:setup" => :environment do
262
- Grit::Git.git_timeout = 10.minutes
263
- end
490
+ rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..'
491
+ rails_env = ENV['RAILS_ENV'] || 'development'
492
+ config_file = rails_root + '/config/resque.yml'
493
+
494
+ resque_config = YAML::load(ERB.new(IO.read(config_file)).result)
495
+ Resque.redis = resque_config[rails_env]
264
496
  ```
265
497
 
266
- We don't want the `git_timeout` as high as 10 minutes in our web app,
267
- but in the Resque workers it's fine.
498
+ Easy peasy! Why not just use `RAILS_ROOT` and `RAILS_ENV`? Because
499
+ this way we can tell our Sinatra app about the config file:
500
+
501
+ $ RAILS_ENV=production resque-web rails_root/config/initializers/resque.rb
502
+
503
+ Now everyone is on the same page.
504
+
505
+ Also, you could disable jobs queueing by setting 'inline' attribute.
506
+ For example, if you want to run all jobs in the same process for cucumber, try:
268
507
 
508
+ ``` ruby
509
+ Resque.inline = ENV['RAILS_ENV'] == "cucumber"
510
+ ```
269
511
 
270
- ### Logging
512
+ #### Logging
271
513
 
272
514
  Workers support basic logging to STDOUT.
273
515
 
@@ -285,14 +527,33 @@ If you want Resque to log to a file, in Rails do:
285
527
  Resque.logger = Logger.new(Rails.root.join('log', "#{Rails.env}_resque.log"))
286
528
  ```
287
529
 
288
- ### Storing Statistics
289
- Resque allows to store count of processed and failed jobs.
530
+ #### Namespaces
290
531
 
291
- By default it will store it in Redis using the keys `stats:processed` and `stats:failed`.
532
+ If you're running multiple, separate instances of Resque you may want
533
+ to namespace the keyspaces so they do not overlap. This is not unlike
534
+ the approach taken by many memcached clients.
292
535
 
293
- Some apps would want another stats store, or even a null store:
536
+ This feature is provided by the [redis-namespace](http://github.com/resque/redis-namespace) library, which
537
+ Resque uses by default to separate the keys it manages from other keys
538
+ in your Redis server.
294
539
 
295
- ```ruby
540
+ Simply use the `Resque.redis.namespace` accessor:
541
+
542
+ ``` ruby
543
+ Resque.redis.namespace = "resque:GitHub"
544
+ ```
545
+
546
+ We recommend sticking this in your initializer somewhere after Redis
547
+ is configured.
548
+
549
+ #### Storing Statistics
550
+ Resque allows to store count of processed and failed jobs.
551
+
552
+ By default it will store it in Redis using the keys `stats:processed` and `stats:failed`.
553
+
554
+ Some apps would want another stats store, or even a null store:
555
+
556
+ ```ruby
296
557
  # config/initializers/resque.rb
297
558
  class NullDataStore
298
559
  def stat(stat)
@@ -312,91 +573,65 @@ end
312
573
  Resque.stat_data_store = NullDataStore.new
313
574
  ```
314
575
 
315
- ### Process IDs (PIDs)
316
-
317
- There are scenarios where it's helpful to record the PID of a resque
318
- worker process. Use the PIDFILE option for easy access to the PID:
319
-
320
- $ PIDFILE=./resque.pid QUEUE=file_serve rake environment resque:work
321
-
322
- ### Running in the background
323
-
324
- There are scenarios where it's helpful for
325
- the resque worker to run itself in the background (usually in combination with
326
- PIDFILE). Use the BACKGROUND option so that rake will return as soon as the
327
- worker is started.
328
-
329
- $ PIDFILE=./resque.pid BACKGROUND=yes QUEUE=file_serve \
330
- rake environment resque:work
331
-
332
- ### Polling frequency
333
-
334
- You can pass an INTERVAL option which is a float representing the polling frequency.
335
- The default is 5 seconds, but for a semi-active app you may want to use a smaller value.
336
-
337
- $ INTERVAL=0.1 QUEUE=file_serve rake environment resque:work
338
-
339
- ### Priorities and Queue Lists
340
-
341
- Resque doesn't support numeric priorities but instead uses the order
342
- of queues you give it. We call this list of queues the "queue list."
343
-
344
- Let's say we add a `warm_cache` queue in addition to our `file_serve`
345
- queue. We'd now start a worker like so:
346
-
347
- $ QUEUES=file_serve,warm_cache rake resque:work
348
-
349
- When the worker looks for new jobs, it will first check
350
- `file_serve`. If it finds a job, it'll process it then check
351
- `file_serve` again. It will keep checking `file_serve` until no more
352
- jobs are available. At that point, it will check `warm_cache`. If it
353
- finds a job it'll process it then check `file_serve` (repeating the
354
- whole process).
355
-
356
- In this way you can prioritize certain queues. At GitHub we start our
357
- workers with something like this:
358
-
359
- $ QUEUES=critical,archive,high,low rake resque:work
360
-
361
- Notice the `archive` queue - it is specialized and in our future
362
- architecture will only be run from a single machine.
363
-
364
- At that point we'll start workers on our generalized background
365
- machines with this command:
366
-
367
- $ QUEUES=critical,high,low rake resque:work
368
-
369
- And workers on our specialized archive machine with this command:
576
+ Plugins and Hooks
577
+ -----------------
370
578
 
371
- $ QUEUE=archive rake resque:work
579
+ For a list of available plugins see
580
+ <https://github.com/resque/resque/wiki/plugins>.
372
581
 
582
+ If you'd like to write your own plugin, or want to customize Resque
583
+ using hooks (such as `Resque.after_fork`), see
584
+ [docs/HOOKS.md](http://github.com/resque/resque/blob/master/docs/HOOKS.md).
373
585
 
374
- ### Running All Queues
375
586
 
376
- If you want your workers to work off of every queue, including new
377
- queues created on the fly, you can use a splat:
587
+ Additional Information
588
+ ----------------------
378
589
 
379
- $ QUEUE=* rake resque:work
590
+ #### Resque vs DelayedJob
380
591
 
381
- Queues will be processed in alphabetical order.
592
+ How does Resque compare to DelayedJob, and why would you choose one
593
+ over the other?
382
594
 
595
+ * Resque supports multiple queues
596
+ * DelayedJob supports finer grained priorities
597
+ * Resque workers are resilient to memory leaks / bloat
598
+ * DelayedJob workers are extremely simple and easy to modify
599
+ * Resque requires Redis
600
+ * DelayedJob requires ActiveRecord
601
+ * Resque can only place JSONable Ruby objects on a queue as arguments
602
+ * DelayedJob can place _any_ Ruby object on its queue as arguments
603
+ * Resque includes a Sinatra app for monitoring what's going on
604
+ * DelayedJob can be queried from within your Rails app if you want to
605
+ add an interface
383
606
 
384
- ### Running Multiple Workers
607
+ If you're doing Rails development, you already have a database and
608
+ ActiveRecord. DelayedJob is super easy to setup and works great.
609
+ GitHub used it for many months to process almost 200 million jobs.
385
610
 
386
- At GitHub we use god to start and stop multiple workers. A sample god
387
- configuration file is included under `examples/god`. We recommend this
388
- method.
611
+ Choose Resque if:
389
612
 
390
- If you'd like to run multiple workers in development mode, you can do
391
- so using the `resque:workers` rake task:
613
+ * You need multiple queues
614
+ * You don't care / dislike numeric priorities
615
+ * You don't need to persist every Ruby object ever
616
+ * You have potentially huge queues
617
+ * You want to see what's going on
618
+ * You expect a lot of failure / chaos
619
+ * You can setup Redis
620
+ * You're not running short on RAM
392
621
 
393
- $ COUNT=5 QUEUE=* rake resque:workers
622
+ Choose DelayedJob if:
394
623
 
395
- This will spawn five Resque workers, each in its own process. Hitting
396
- ctrl-c should be sufficient to stop them all.
624
+ * You like numeric priorities
625
+ * You're not doing a gigantic amount of jobs each day
626
+ * Your queue stays small and nimble
627
+ * There is not a lot failure / chaos
628
+ * You want to easily throw anything on the queue
629
+ * You don't want to setup Redis
397
630
 
631
+ In no way is Resque a "better" DelayedJob, so make sure you pick the
632
+ tool that's best for your app.
398
633
 
399
- ### Forking
634
+ #### Forking
400
635
 
401
636
  On certain platforms, when a Resque worker reserves a job it
402
637
  immediately forks a child process. The child processes the job then
@@ -437,8 +672,13 @@ complicated.
437
672
 
438
673
  Workers instead handle their own state.
439
674
 
675
+ #### `at_exit` Callbacks
676
+
677
+ Resque uses `Kernel#exit!` for exiting its workers' child processes. So any `at_exit` callback defined in your application won't be executed when the job is finished and the child process exits.
678
+
679
+ You can alter this behavior by setting the `RUN_AT_EXIT_HOOKS` environment variable.
440
680
 
441
- ### Parents and Children
681
+ #### Parents and Children
442
682
 
443
683
  Here's a parent / child pair doing some work:
444
684
 
@@ -459,7 +699,7 @@ waiting for work on:
459
699
  92099 resque: Waiting for file_serve,warm_cache
460
700
 
461
701
 
462
- ### Signals
702
+ #### Signals
463
703
 
464
704
  Resque workers respond to a few different signals:
465
705
 
@@ -479,9 +719,9 @@ If you want to kill a stale or stuck child and shutdown, use `TERM`
479
719
 
480
720
  If you want to stop processing jobs, but want to leave the worker running
481
721
  (for example, to temporarily alleviate load), use `USR2` to stop processing,
482
- then `CONT` to start it again.
722
+ then `CONT` to start it again. It's also possible to [pause all workers](#pausing-all-workers).
483
723
 
484
- #### Signals on Heroku
724
+ #### Heroku
485
725
 
486
726
  When shutting down processes, Heroku sends every process a TERM signal at the
487
727
  same time. By default this causes an immediate shutdown of any running job
@@ -503,11 +743,41 @@ time to complete before being forced to die.
503
743
 
504
744
  * `RESQUE_TERM_TIMEOUT` - By default you have a few seconds to handle `Resque::TermException` in your job. `RESQUE_TERM_TIMEOUT` and `RESQUE_PRE_SHUTDOWN_TIMEOUT` must be lower than the [heroku dyno timeout](https://devcenter.heroku.com/articles/limits#exit-timeout).
505
745
 
506
- ### Mysql::Error: MySQL server has gone away
746
+ #### Pausing all workers
747
+
748
+ Workers will not process pending jobs if the Redis key `pause-all-workers` is set with the string value "true".
749
+
750
+ ``` ruby
751
+ Resque.redis.set('pause-all-workers', 'true')
752
+ ```
753
+
754
+ Nothing happens to jobs that are already being processed by workers.
755
+
756
+ Unpause by removing the Redis key `pause-all-workers`.
757
+
758
+ ``` ruby
759
+ Resque.redis.del('pause-all-workers')
760
+ ```
761
+
762
+ #### Monitoring
763
+
764
+ ##### god
765
+
766
+ If you're using god to monitor Resque, we have provided example
767
+ configs in `examples/god/`. One is for starting / stopping workers,
768
+ the other is for killing workers that have been running too long.
769
+
770
+ ##### monit
771
+
772
+ If you're using monit, `examples/monit/resque.monit` is provided free
773
+ of charge. This is **not** used by GitHub in production, so please
774
+ send patches for any tweaks or improvements you can make to it.
775
+
776
+ #### Mysql::Error: MySQL server has gone away
507
777
 
508
778
  If your workers remain idle for too long they may lose their MySQL connection. Depending on your version of Rails, we recommend the following:
509
779
 
510
- #### Rails 3.x
780
+ ##### Rails
511
781
  In your `perform` method, add the following line:
512
782
 
513
783
  ``` ruby
@@ -523,7 +793,7 @@ The Rails doc says the following about `verify_active_connections!`:
523
793
 
524
794
  Verify active connections and remove and disconnect connections associated with stale threads.
525
795
 
526
- #### Rails 4.x
796
+ ##### Rails 4.x
527
797
 
528
798
  In your `perform` method, instead of `verify_active_connections!`, use:
529
799
 
@@ -540,363 +810,6 @@ From the Rails docs on [`clear_active_connections!`](http://api.rubyonrails.org/
540
810
 
541
811
  Returns any connections in use by the current thread back to the pool, and also returns connections to the pool cached by threads that are no longer alive.
542
812
 
543
-
544
-
545
- The Front End
546
- -------------
547
-
548
- Resque comes with a Sinatra-based front end for seeing what's up with
549
- your queue.
550
-
551
- ![The Front End](https://camo.githubusercontent.com/64d150a243987ffbc33f588bd6d7722a0bb8d69a/687474703a2f2f7475746f7269616c732e6a756d7073746172746c61622e636f6d2f696d616765732f7265737175655f6f766572766965772e706e67)
552
-
553
- ### Standalone
554
-
555
- If you've installed Resque as a gem running the front end standalone is easy:
556
-
557
- $ resque-web
558
-
559
- It's a thin layer around `rackup` so it's configurable as well:
560
-
561
- $ resque-web -p 8282
562
-
563
- If you have a Resque config file you want evaluated just pass it to
564
- the script as the final argument:
565
-
566
- $ resque-web -p 8282 rails_root/config/initializers/resque.rb
567
-
568
- You can also set the namespace directly using `resque-web`:
569
-
570
- $ resque-web -p 8282 -N myapp
571
-
572
- or set the Redis connection string if you need to do something like select a different database:
573
-
574
- $ resque-web -p 8282 -r localhost:6379:2
575
-
576
- ### Passenger
577
-
578
- Using Passenger? Resque ships with a `config.ru` you can use. See
579
- Phusion's guide:
580
-
581
- Apache: <https://www.phusionpassenger.com/library/deploy/apache/deploy/ruby/>
582
- Nginx: <https://www.phusionpassenger.com/library/deploy/nginx/deploy/ruby/>
583
-
584
- ### Rack::URLMap
585
-
586
- If you want to load Resque on a subpath, possibly alongside other
587
- apps, it's easy to do with Rack's `URLMap`:
588
-
589
- ``` ruby
590
- require 'resque/server'
591
-
592
- run Rack::URLMap.new \
593
- "/" => Your::App.new,
594
- "/resque" => Resque::Server.new
595
- ```
596
-
597
- Check `examples/demo/config.ru` for a functional example (including
598
- HTTP basic auth).
599
-
600
- ### Rails 3
601
-
602
- You can also mount Resque on a subpath in your existing Rails 3 app by adding `require 'resque/server'` to the top of your routes file or in an initializer then adding this to `routes.rb`:
603
-
604
- ``` ruby
605
- mount Resque::Server.new, :at => "/resque"
606
- ```
607
-
608
-
609
- Resque vs DelayedJob
610
- --------------------
611
-
612
- How does Resque compare to DelayedJob, and why would you choose one
613
- over the other?
614
-
615
- * Resque supports multiple queues
616
- * DelayedJob supports finer grained priorities
617
- * Resque workers are resilient to memory leaks / bloat
618
- * DelayedJob workers are extremely simple and easy to modify
619
- * Resque requires Redis
620
- * DelayedJob requires ActiveRecord
621
- * Resque can only place JSONable Ruby objects on a queue as arguments
622
- * DelayedJob can place _any_ Ruby object on its queue as arguments
623
- * Resque includes a Sinatra app for monitoring what's going on
624
- * DelayedJob can be queried from within your Rails app if you want to
625
- add an interface
626
-
627
- If you're doing Rails development, you already have a database and
628
- ActiveRecord. DelayedJob is super easy to setup and works great.
629
- GitHub used it for many months to process almost 200 million jobs.
630
-
631
- Choose Resque if:
632
-
633
- * You need multiple queues
634
- * You don't care / dislike numeric priorities
635
- * You don't need to persist every Ruby object ever
636
- * You have potentially huge queues
637
- * You want to see what's going on
638
- * You expect a lot of failure / chaos
639
- * You can setup Redis
640
- * You're not running short on RAM
641
-
642
- Choose DelayedJob if:
643
-
644
- * You like numeric priorities
645
- * You're not doing a gigantic amount of jobs each day
646
- * Your queue stays small and nimble
647
- * There is not a lot failure / chaos
648
- * You want to easily throw anything on the queue
649
- * You don't want to setup Redis
650
-
651
- In no way is Resque a "better" DelayedJob, so make sure you pick the
652
- tool that's best for your app.
653
-
654
- Resque Dependencies
655
- -------------------
656
-
657
- $ gem install bundler
658
- $ bundle install
659
-
660
-
661
- Installing Resque
662
- -----------------
663
-
664
- ### In a Rack app, as a gem
665
-
666
- First install the gem.
667
-
668
- $ gem install resque
669
-
670
- Next include it in your application.
671
-
672
- ``` ruby
673
- require 'resque'
674
- ```
675
-
676
- Now start your application:
677
-
678
- rackup config.ru
679
-
680
- That's it! You can now create Resque jobs from within your app.
681
-
682
- To start a worker, create a Rakefile in your app's root (or add this
683
- to an existing Rakefile):
684
-
685
- ``` ruby
686
- require 'your/app'
687
- require 'resque/tasks'
688
- ```
689
-
690
- If you're using Rails 5.x, include the following in lib/tasks/resque.rake:
691
-
692
- ```ruby
693
- require 'resque/tasks'
694
- task 'resque:setup' => :environment
695
- ```
696
-
697
- Now:
698
-
699
- $ QUEUE=* rake resque:work
700
-
701
- Alternately you can define a `resque:setup` hook in your Rakefile if you
702
- don't want to load your app every time rake runs.
703
-
704
-
705
- ### In a Rails 2.x app, as a gem
706
-
707
- First install the gem.
708
-
709
- $ gem install resque
710
-
711
- Next include it in your application.
712
-
713
- $ cat config/initializers/load_resque.rb
714
- require 'resque'
715
-
716
- Now start your application:
717
-
718
- $ ./script/server
719
-
720
- That's it! You can now create Resque jobs from within your app.
721
-
722
- To start a worker, add this to your Rakefile in `RAILS_ROOT`:
723
-
724
- ``` ruby
725
- require 'resque/tasks'
726
- ```
727
-
728
- Now:
729
-
730
- $ QUEUE=* rake environment resque:work
731
-
732
- Don't forget you can define a `resque:setup` hook in
733
- `lib/tasks/whatever.rake` that loads the `environment` task every time.
734
-
735
-
736
- ### In a Rails 2.x app, as a plugin
737
-
738
- $ ./script/plugin install git://github.com/resque/resque
739
-
740
- That's it! Resque will automatically be available when your Rails app
741
- loads.
742
-
743
- To start a worker:
744
-
745
- $ QUEUE=* rake environment resque:work
746
-
747
- Don't forget you can define a `resque:setup` hook in
748
- `lib/tasks/whatever.rake` that loads the `environment` task every time.
749
-
750
-
751
- ### In a Rails 3.x or 4.x app, as a gem
752
-
753
- First include it in your Gemfile.
754
-
755
- $ cat Gemfile
756
- ...
757
- gem 'resque'
758
- ...
759
-
760
- Next install it with Bundler.
761
-
762
- $ bundle install
763
-
764
- Now start your application:
765
-
766
- $ rails server
767
-
768
- That's it! You can now create Resque jobs from within your app.
769
-
770
- To start a worker, add this to a file in `lib/tasks` (ex:
771
- `lib/tasks/resque.rake`):
772
-
773
- ``` ruby
774
- require 'resque/tasks'
775
- ```
776
-
777
- Now:
778
-
779
- $ QUEUE=* rake environment resque:work
780
-
781
- Don't forget you can define a `resque:setup` hook in
782
- `lib/tasks/whatever.rake` that loads the `environment` task every time.
783
-
784
-
785
- Configuration
786
- -------------
787
-
788
- You may want to change the Redis host and port Resque connects to, or
789
- set various other options at startup.
790
-
791
- Resque has a `redis` setter which can be given a string or a Redis
792
- object. This means if you're already using Redis in your app, Resque
793
- can re-use the existing connection.
794
-
795
- String: `Resque.redis = 'localhost:6379'`
796
-
797
- Redis: `Resque.redis = $redis`
798
-
799
- For our rails app we have a `config/initializers/resque.rb` file where
800
- we load `config/resque.yml` by hand and set the Redis information
801
- appropriately.
802
-
803
- Here's our `config/resque.yml`:
804
-
805
- development: localhost:6379
806
- test: localhost:6379
807
- staging: redis1.se.github.com:6379
808
- fi: localhost:6379
809
- production: <%= ENV['REDIS_URL'] %>
810
-
811
- And our initializer:
812
-
813
- ``` ruby
814
- rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..'
815
- rails_env = ENV['RAILS_ENV'] || 'development'
816
- config_file = rails_root + '/config/resque.yml'
817
-
818
- resque_config = YAML::load(ERB.new(IO.read(config_file)).result)
819
- Resque.redis = resque_config[rails_env]
820
- ```
821
-
822
- Easy peasy! Why not just use `RAILS_ROOT` and `RAILS_ENV`? Because
823
- this way we can tell our Sinatra app about the config file:
824
-
825
- $ RAILS_ENV=production resque-web rails_root/config/initializers/resque.rb
826
-
827
- Now everyone is on the same page.
828
-
829
- Also, you could disable jobs queueing by setting 'inline' attribute.
830
- For example, if you want to run all jobs in the same process for cucumber, try:
831
-
832
- ``` ruby
833
- Resque.inline = ENV['RAILS_ENV'] == "cucumber"
834
- ```
835
-
836
-
837
- Plugins and Hooks
838
- -----------------
839
-
840
- For a list of available plugins see
841
- <http://wiki.github.com/resque/resque/plugins>.
842
-
843
- If you'd like to write your own plugin, or want to customize Resque
844
- using hooks (such as `Resque.after_fork`), see
845
- [docs/HOOKS.md](http://github.com/resque/resque/blob/master/docs/HOOKS.md).
846
-
847
-
848
- Namespaces
849
- ----------
850
-
851
- If you're running multiple, separate instances of Resque you may want
852
- to namespace the keyspaces so they do not overlap. This is not unlike
853
- the approach taken by many memcached clients.
854
-
855
- This feature is provided by the [redis-namespace][rs] library, which
856
- Resque uses by default to separate the keys it manages from other keys
857
- in your Redis server.
858
-
859
- Simply use the `Resque.redis.namespace` accessor:
860
-
861
- ``` ruby
862
- Resque.redis.namespace = "resque:GitHub"
863
- ```
864
-
865
- We recommend sticking this in your initializer somewhere after Redis
866
- is configured.
867
-
868
-
869
- Demo
870
- ----
871
-
872
- Resque ships with a demo Sinatra app for creating jobs that are later
873
- processed in the background.
874
-
875
- Try it out by looking at the README, found at `examples/demo/README.markdown`.
876
-
877
-
878
- Monitoring
879
- ----------
880
-
881
- ### god
882
-
883
- If you're using god to monitor Resque, we have provided example
884
- configs in `examples/god/`. One is for starting / stopping workers,
885
- the other is for killing workers that have been running too long.
886
-
887
- ### monit
888
-
889
- If you're using monit, `examples/monit/resque.monit` is provided free
890
- of charge. This is **not** used by GitHub in production, so please
891
- send patches for any tweaks or improvements you can make to it.
892
-
893
-
894
- Questions
895
- ---------
896
-
897
- Please add them to the [FAQ](https://github.com/resque/resque/wiki/FAQ) or open an issue on this repo.
898
-
899
-
900
813
  Development
901
814
  -----------
902
815
 
@@ -926,26 +839,27 @@ it:
926
839
  If you get an error requiring any of the dependencies, you may have
927
840
  failed to install them or be seeing load path issues.
928
841
 
842
+ #### Demo
843
+ Resque ships with a demo Sinatra app for creating jobs that are later
844
+ processed in the background.
929
845
 
930
- Contributing
931
- ------------
846
+ Try it out by looking at the README, found at `examples/demo/README.markdown`.
847
+
848
+ #### Contributing
932
849
 
933
850
  Read [CONTRIBUTING.md](CONTRIBUTING.md) first.
934
851
 
935
852
  Once you've made your great commits:
936
853
 
937
- 1. [Fork][1] Resque
854
+ 1. [Fork](http://help.github.com/forking/) Resque
938
855
  2. Create a topic branch - `git checkout -b my_branch`
939
856
  3. Push to your branch - `git push origin my_branch`
940
857
  4. Create a [Pull Request](http://help.github.com/pull-requests/) from your branch
941
- 5. That's it!
942
-
943
858
 
944
- Mailing List
945
- ------------
946
-
947
- This mailing list is no longer maintained. The archive can be found at <http://librelist.com/browser/resque/>.
859
+ Questions
860
+ ---------
948
861
 
862
+ Please add them to the [FAQ](https://github.com/resque/resque/wiki/FAQ) or open an issue on this repo.
949
863
 
950
864
  Meta
951
865
  ----
@@ -954,21 +868,11 @@ Meta
954
868
  * Home: <http://github.com/resque/resque>
955
869
  * Docs: <http://rubydoc.info/gems/resque>
956
870
  * Bugs: <http://github.com/resque/resque/issues>
957
- * List: <resque@librelist.com>
958
- * Chat: <irc://irc.freenode.net/resque>
959
- * Gems: <http://gemcutter.org/gems/resque>
960
-
961
- This project uses [Semantic Versioning][sv].
871
+ * Gems: <https://rubygems.org/gems/resque>
962
872
 
873
+ This project uses [Semantic Versioning](http://semver.org/)
963
874
 
964
875
  Author
965
876
  ------
966
877
 
967
878
  Chris Wanstrath :: chris@ozmm.org :: @defunkt
968
-
969
- [0]: http://github.com/blog/542-introducing-resque
970
- [1]: http://help.github.com/forking/
971
- [2]: http://github.com/resque/resque/issues
972
- [sv]: http://semver.org/
973
- [rs]: http://github.com/resque/redis-namespace
974
- [cb]: http://wiki.github.com/resque/resque/contributing