resqueue 1.0.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +7 -0
- data/HISTORY.md +488 -0
- data/LICENSE +20 -0
- data/README.markdown +920 -0
- data/Rakefile +57 -0
- data/bin/resque +81 -0
- data/bin/resque-web +31 -0
- data/lib/resque.rb +578 -0
- data/lib/resque/data_store.rb +326 -0
- data/lib/resque/errors.rb +21 -0
- data/lib/resque/failure.rb +119 -0
- data/lib/resque/failure/airbrake.rb +33 -0
- data/lib/resque/failure/base.rb +73 -0
- data/lib/resque/failure/multiple.rb +68 -0
- data/lib/resque/failure/redis.rb +128 -0
- data/lib/resque/failure/redis_multi_queue.rb +104 -0
- data/lib/resque/helpers.rb +48 -0
- data/lib/resque/job.rb +296 -0
- data/lib/resque/log_formatters/quiet_formatter.rb +7 -0
- data/lib/resque/log_formatters/verbose_formatter.rb +7 -0
- data/lib/resque/log_formatters/very_verbose_formatter.rb +8 -0
- data/lib/resque/logging.rb +18 -0
- data/lib/resque/plugin.rb +78 -0
- data/lib/resque/server.rb +299 -0
- data/lib/resque/server/helpers.rb +64 -0
- data/lib/resque/server/public/favicon.ico +0 -0
- data/lib/resque/server/public/idle.png +0 -0
- data/lib/resque/server/public/jquery-1.12.4.min.js +5 -0
- data/lib/resque/server/public/jquery.relatize_date.js +95 -0
- data/lib/resque/server/public/poll.png +0 -0
- data/lib/resque/server/public/ranger.js +78 -0
- data/lib/resque/server/public/reset.css +44 -0
- data/lib/resque/server/public/style.css +91 -0
- data/lib/resque/server/public/working.png +0 -0
- data/lib/resque/server/test_helper.rb +19 -0
- data/lib/resque/server/views/error.erb +1 -0
- data/lib/resque/server/views/failed.erb +29 -0
- data/lib/resque/server/views/failed_job.erb +50 -0
- data/lib/resque/server/views/failed_queues_overview.erb +24 -0
- data/lib/resque/server/views/key_sets.erb +17 -0
- data/lib/resque/server/views/key_string.erb +11 -0
- data/lib/resque/server/views/layout.erb +44 -0
- data/lib/resque/server/views/next_more.erb +22 -0
- data/lib/resque/server/views/overview.erb +4 -0
- data/lib/resque/server/views/queues.erb +58 -0
- data/lib/resque/server/views/stats.erb +62 -0
- data/lib/resque/server/views/workers.erb +111 -0
- data/lib/resque/server/views/working.erb +72 -0
- data/lib/resque/stat.rb +58 -0
- data/lib/resque/tasks.rb +72 -0
- data/lib/resque/thread_signal.rb +45 -0
- data/lib/resque/vendor/utf8_util.rb +26 -0
- data/lib/resque/vendor/utf8_util/utf8_util_18.rb +91 -0
- data/lib/resque/vendor/utf8_util/utf8_util_19.rb +6 -0
- data/lib/resque/version.rb +3 -0
- data/lib/resque/worker.rb +892 -0
- data/lib/resqueue.rb +4 -0
- data/lib/tasks/redis.rake +161 -0
- data/lib/tasks/resque.rake +2 -0
- metadata +197 -0
data/LICENSE
ADDED
@@ -0,0 +1,20 @@
|
|
1
|
+
Copyright (c) Chris Wanstrath
|
2
|
+
|
3
|
+
Permission is hereby granted, free of charge, to any person obtaining
|
4
|
+
a copy of this software and associated documentation files (the
|
5
|
+
"Software"), to deal in the Software without restriction, including
|
6
|
+
without limitation the rights to use, copy, modify, merge, publish,
|
7
|
+
distribute, sublicense, and/or sell copies of the Software, and to
|
8
|
+
permit persons to whom the Software is furnished to do so, subject to
|
9
|
+
the following conditions:
|
10
|
+
|
11
|
+
The above copyright notice and this permission notice shall be
|
12
|
+
included in all copies or substantial portions of the Software.
|
13
|
+
|
14
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
15
|
+
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
16
|
+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
17
|
+
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
18
|
+
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
19
|
+
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
20
|
+
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
data/README.markdown
ADDED
@@ -0,0 +1,920 @@
|
|
1
|
+
Resque
|
2
|
+
======
|
3
|
+
|
4
|
+
[![Gem Version](https://badge.fury.io/rb/resque.svg)](https://rubygems.org/gems/resque)
|
5
|
+
[![Build Status](https://travis-ci.org/resque/resque.svg)](https://travis-ci.org/resque/resque)
|
6
|
+
[![Coverage Status](https://coveralls.io/repos/github/resque/resque/badge.svg?branch=1-x-stable)](https://coveralls.io/r/resque/resque?branch=1-x-stable)
|
7
|
+
|
8
|
+
Resque (pronounced like "rescue") is a Redis-backed library for creating
|
9
|
+
background jobs, placing those jobs on multiple queues, and processing
|
10
|
+
them later.
|
11
|
+
|
12
|
+
Background jobs can be any Ruby class or module that responds to
|
13
|
+
`perform`. Your existing classes can easily be converted to background
|
14
|
+
jobs or you can create new classes specifically to do work. Or, you
|
15
|
+
can do both.
|
16
|
+
|
17
|
+
Resque is heavily inspired by DelayedJob (which rocks) and comprises
|
18
|
+
three parts:
|
19
|
+
|
20
|
+
1. A Ruby library for creating, querying, and processing jobs
|
21
|
+
2. A Rake task for starting a worker which processes jobs
|
22
|
+
3. A Sinatra app for monitoring queues, jobs, and workers.
|
23
|
+
|
24
|
+
Resque workers can be distributed between multiple machines,
|
25
|
+
support priorities, are resilient to memory bloat / "leaks," are
|
26
|
+
optimized for REE (but work on MRI and JRuby), tell you what they're
|
27
|
+
doing, and expect failure.
|
28
|
+
|
29
|
+
Resque queues are persistent; support constant time, atomic push and
|
30
|
+
pop (thanks to Redis); provide visibility into their contents; and
|
31
|
+
store jobs as simple JSON packages.
|
32
|
+
|
33
|
+
The Resque frontend tells you what workers are doing, what workers are
|
34
|
+
not doing, what queues you're using, what's in those queues, provides
|
35
|
+
general usage stats, and helps you track failures.
|
36
|
+
|
37
|
+
|
38
|
+
The Blog Post
|
39
|
+
-------------
|
40
|
+
|
41
|
+
For the backstory, philosophy, and history of Resque's beginnings,
|
42
|
+
please see [the blog post][0].
|
43
|
+
|
44
|
+
|
45
|
+
Overview
|
46
|
+
--------
|
47
|
+
|
48
|
+
Resque allows you to create jobs and place them on a queue, then,
|
49
|
+
later, pull those jobs off the queue and process them.
|
50
|
+
|
51
|
+
Resque jobs are Ruby classes (or modules) which respond to the
|
52
|
+
`perform` method. Here's an example:
|
53
|
+
|
54
|
+
|
55
|
+
``` ruby
|
56
|
+
class Archive
|
57
|
+
@queue = :file_serve
|
58
|
+
|
59
|
+
def self.perform(repo_id, branch = 'master')
|
60
|
+
repo = Repository.find(repo_id)
|
61
|
+
repo.create_archive(branch)
|
62
|
+
end
|
63
|
+
end
|
64
|
+
```
|
65
|
+
|
66
|
+
The `@queue` class instance variable determines which queue `Archive`
|
67
|
+
jobs will be placed in. Queues are arbitrary and created on the fly -
|
68
|
+
you can name them whatever you want and have as many as you want.
|
69
|
+
|
70
|
+
To place an `Archive` job on the `file_serve` queue, we might add this
|
71
|
+
to our application's pre-existing `Repository` class:
|
72
|
+
|
73
|
+
``` ruby
|
74
|
+
class Repository
|
75
|
+
def async_create_archive(branch)
|
76
|
+
Resque.enqueue(Archive, self.id, branch)
|
77
|
+
end
|
78
|
+
end
|
79
|
+
```
|
80
|
+
|
81
|
+
Now when we call `repo.async_create_archive('masterbrew')` in our
|
82
|
+
application, a job will be created and placed on the `file_serve`
|
83
|
+
queue.
|
84
|
+
|
85
|
+
Later, a worker will run something like this code to process the job:
|
86
|
+
|
87
|
+
``` ruby
|
88
|
+
klass, args = Resque.reserve(:file_serve)
|
89
|
+
klass.perform(*args) if klass.respond_to? :perform
|
90
|
+
```
|
91
|
+
|
92
|
+
Which translates to:
|
93
|
+
|
94
|
+
``` ruby
|
95
|
+
Archive.perform(44, 'masterbrew')
|
96
|
+
```
|
97
|
+
|
98
|
+
Let's start a worker to run `file_serve` jobs:
|
99
|
+
|
100
|
+
$ cd app_root
|
101
|
+
$ QUEUE=file_serve rake resque:work
|
102
|
+
|
103
|
+
This starts one Resque worker and tells it to work off the
|
104
|
+
`file_serve` queue. As soon as it's ready it'll try to run the
|
105
|
+
`Resque.reserve` code snippet above and process jobs until it can't
|
106
|
+
find any more, at which point it will sleep for a small period and
|
107
|
+
repeatedly poll the queue for more jobs.
|
108
|
+
|
109
|
+
Workers can be given multiple queues (a "queue list") and run on
|
110
|
+
multiple machines. In fact they can be run anywhere with network
|
111
|
+
access to the Redis server.
|
112
|
+
|
113
|
+
|
114
|
+
Jobs
|
115
|
+
----
|
116
|
+
|
117
|
+
What should you run in the background? Anything that takes any time at
|
118
|
+
all. Slow INSERT statements, disk manipulating, data processing, etc.
|
119
|
+
|
120
|
+
At GitHub we use Resque to process the following types of jobs:
|
121
|
+
|
122
|
+
* Warming caches
|
123
|
+
* Counting disk usage
|
124
|
+
* Building tarballs
|
125
|
+
* Building Rubygems
|
126
|
+
* Firing off web hooks
|
127
|
+
* Creating events in the db and pre-caching them
|
128
|
+
* Building graphs
|
129
|
+
* Deleting users
|
130
|
+
* Updating our search index
|
131
|
+
|
132
|
+
As of writing we have about 35 different types of background jobs.
|
133
|
+
|
134
|
+
Keep in mind that you don't need a web app to use Resque - we just
|
135
|
+
mention "foreground" and "background" because they make conceptual
|
136
|
+
sense. You could easily be spidering sites and sticking data which
|
137
|
+
needs to be crunched later into a queue.
|
138
|
+
|
139
|
+
|
140
|
+
### Persistence
|
141
|
+
|
142
|
+
Jobs are persisted to queues as JSON objects. Let's take our `Archive`
|
143
|
+
example from above. We'll run the following code to create a job:
|
144
|
+
|
145
|
+
``` ruby
|
146
|
+
repo = Repository.find(44)
|
147
|
+
repo.async_create_archive('masterbrew')
|
148
|
+
```
|
149
|
+
|
150
|
+
The following JSON will be stored in the `file_serve` queue:
|
151
|
+
|
152
|
+
``` javascript
|
153
|
+
{
|
154
|
+
'class': 'Archive',
|
155
|
+
'args': [ 44, 'masterbrew' ]
|
156
|
+
}
|
157
|
+
```
|
158
|
+
|
159
|
+
Because of this your jobs must only accept arguments that can be JSON encoded.
|
160
|
+
|
161
|
+
So instead of doing this:
|
162
|
+
|
163
|
+
``` ruby
|
164
|
+
Resque.enqueue(Archive, self, branch)
|
165
|
+
```
|
166
|
+
|
167
|
+
do this:
|
168
|
+
|
169
|
+
``` ruby
|
170
|
+
Resque.enqueue(Archive, self.id, branch)
|
171
|
+
```
|
172
|
+
|
173
|
+
This is why our above example (and all the examples in `examples/`)
|
174
|
+
uses object IDs instead of passing around the objects.
|
175
|
+
|
176
|
+
While this is less convenient than just sticking a marshaled object
|
177
|
+
in the database, it gives you a slight advantage: your jobs will be
|
178
|
+
run against the most recent version of an object because they need to
|
179
|
+
pull from the DB or cache.
|
180
|
+
|
181
|
+
If your jobs were run against marshaled objects, they could
|
182
|
+
potentially be operating on a stale record with out-of-date information.
|
183
|
+
|
184
|
+
|
185
|
+
### send_later / async
|
186
|
+
|
187
|
+
Want something like DelayedJob's `send_later` or the ability to use
|
188
|
+
instance methods instead of just methods for jobs? See the `examples/`
|
189
|
+
directory for goodies.
|
190
|
+
|
191
|
+
We plan to provide first class `async` support in a future release.
|
192
|
+
|
193
|
+
|
194
|
+
### Failure
|
195
|
+
|
196
|
+
If a job raises an exception, it is logged and handed off to the
|
197
|
+
`Resque::Failure` module. Failures are logged either locally in Redis
|
198
|
+
or using some different backend.
|
199
|
+
|
200
|
+
For example, Resque ships with Airbrake support. To configure it, put
|
201
|
+
the following into an initialisation file or into your rake job:
|
202
|
+
|
203
|
+
``` ruby
|
204
|
+
# send errors which occur in background jobs to redis and airbrake
|
205
|
+
require 'resque/failure/multiple'
|
206
|
+
require 'resque/failure/redis'
|
207
|
+
require 'resque/failure/airbrake'
|
208
|
+
|
209
|
+
Resque::Failure::Multiple.classes = [Resque::Failure::Redis, Resque::Failure::Airbrake]
|
210
|
+
Resque::Failure.backend = Resque::Failure::Multiple
|
211
|
+
```
|
212
|
+
|
213
|
+
Keep this in mind when writing your jobs: you may want to throw
|
214
|
+
exceptions you would not normally throw in order to assist debugging.
|
215
|
+
|
216
|
+
|
217
|
+
Workers
|
218
|
+
-------
|
219
|
+
|
220
|
+
Resque workers are rake tasks that run forever. They basically do this:
|
221
|
+
|
222
|
+
``` ruby
|
223
|
+
start
|
224
|
+
loop do
|
225
|
+
if job = reserve
|
226
|
+
job.process
|
227
|
+
else
|
228
|
+
sleep 5 # Polling frequency = 5
|
229
|
+
end
|
230
|
+
end
|
231
|
+
shutdown
|
232
|
+
```
|
233
|
+
|
234
|
+
Starting a worker is simple. Here's our example from earlier:
|
235
|
+
|
236
|
+
$ QUEUE=file_serve rake resque:work
|
237
|
+
|
238
|
+
By default Resque won't know about your application's
|
239
|
+
environment. That is, it won't be able to find and run your jobs - it
|
240
|
+
needs to load your application into memory.
|
241
|
+
|
242
|
+
If we've installed Resque as a Rails plugin, we might run this command
|
243
|
+
from our RAILS_ROOT:
|
244
|
+
|
245
|
+
$ QUEUE=file_serve rake environment resque:work
|
246
|
+
|
247
|
+
This will load the environment before starting a worker. Alternately
|
248
|
+
we can define a `resque:setup` task with a dependency on the
|
249
|
+
`environment` rake task:
|
250
|
+
|
251
|
+
``` ruby
|
252
|
+
task "resque:setup" => :environment
|
253
|
+
```
|
254
|
+
|
255
|
+
GitHub's setup task looks like this:
|
256
|
+
|
257
|
+
``` ruby
|
258
|
+
task "resque:setup" => :environment do
|
259
|
+
Grit::Git.git_timeout = 10.minutes
|
260
|
+
end
|
261
|
+
```
|
262
|
+
|
263
|
+
We don't want the `git_timeout` as high as 10 minutes in our web app,
|
264
|
+
but in the Resque workers it's fine.
|
265
|
+
|
266
|
+
|
267
|
+
### Logging
|
268
|
+
|
269
|
+
Workers support basic logging to STDOUT. If you start them with the
|
270
|
+
`VERBOSE` env variable set, they will print basic debugging
|
271
|
+
information. You can also set the `VVERBOSE` (very verbose) env
|
272
|
+
variable.
|
273
|
+
|
274
|
+
$ VVERBOSE=1 QUEUE=file_serve rake environment resque:work
|
275
|
+
|
276
|
+
If you want Resque to log to a file, in Rails do:
|
277
|
+
|
278
|
+
```ruby
|
279
|
+
# config/initializers/resque.rb
|
280
|
+
Resque.logger = Logger.new(Rails.root.join('log', "#{Rails.env}_resque.log"))
|
281
|
+
```
|
282
|
+
|
283
|
+
### Process IDs (PIDs)
|
284
|
+
|
285
|
+
There are scenarios where it's helpful to record the PID of a resque
|
286
|
+
worker process. Use the PIDFILE option for easy access to the PID:
|
287
|
+
|
288
|
+
$ PIDFILE=./resque.pid QUEUE=file_serve rake environment resque:work
|
289
|
+
|
290
|
+
### Running in the background
|
291
|
+
|
292
|
+
(Only supported with ruby >= 1.9). There are scenarios where it's helpful for
|
293
|
+
the resque worker to run itself in the background (usually in combination with
|
294
|
+
PIDFILE). Use the BACKGROUND option so that rake will return as soon as the
|
295
|
+
worker is started.
|
296
|
+
|
297
|
+
$ PIDFILE=./resque.pid BACKGROUND=yes QUEUE=file_serve \
|
298
|
+
rake environment resque:work
|
299
|
+
|
300
|
+
### Polling frequency
|
301
|
+
|
302
|
+
You can pass an INTERVAL option which is a float representing the polling frequency.
|
303
|
+
The default is 5 seconds, but for a semi-active app you may want to use a smaller value.
|
304
|
+
|
305
|
+
$ INTERVAL=0.1 QUEUE=file_serve rake environment resque:work
|
306
|
+
|
307
|
+
### Priorities and Queue Lists
|
308
|
+
|
309
|
+
Resque doesn't support numeric priorities but instead uses the order
|
310
|
+
of queues you give it. We call this list of queues the "queue list."
|
311
|
+
|
312
|
+
Let's say we add a `warm_cache` queue in addition to our `file_serve`
|
313
|
+
queue. We'd now start a worker like so:
|
314
|
+
|
315
|
+
$ QUEUES=file_serve,warm_cache rake resque:work
|
316
|
+
|
317
|
+
When the worker looks for new jobs, it will first check
|
318
|
+
`file_serve`. If it finds a job, it'll process it then check
|
319
|
+
`file_serve` again. It will keep checking `file_serve` until no more
|
320
|
+
jobs are available. At that point, it will check `warm_cache`. If it
|
321
|
+
finds a job it'll process it then check `file_serve` (repeating the
|
322
|
+
whole process).
|
323
|
+
|
324
|
+
In this way you can prioritize certain queues. At GitHub we start our
|
325
|
+
workers with something like this:
|
326
|
+
|
327
|
+
$ QUEUES=critical,archive,high,low rake resque:work
|
328
|
+
|
329
|
+
Notice the `archive` queue - it is specialized and in our future
|
330
|
+
architecture will only be run from a single machine.
|
331
|
+
|
332
|
+
At that point we'll start workers on our generalized background
|
333
|
+
machines with this command:
|
334
|
+
|
335
|
+
$ QUEUES=critical,high,low rake resque:work
|
336
|
+
|
337
|
+
And workers on our specialized archive machine with this command:
|
338
|
+
|
339
|
+
$ QUEUE=archive rake resque:work
|
340
|
+
|
341
|
+
|
342
|
+
### Running All Queues
|
343
|
+
|
344
|
+
If you want your workers to work off of every queue, including new
|
345
|
+
queues created on the fly, you can use a splat:
|
346
|
+
|
347
|
+
$ QUEUE=* rake resque:work
|
348
|
+
|
349
|
+
Queues will be processed in alphabetical order.
|
350
|
+
|
351
|
+
|
352
|
+
### Running Multiple Workers
|
353
|
+
|
354
|
+
At GitHub we use god to start and stop multiple workers. A sample god
|
355
|
+
configuration file is included under `examples/god`. We recommend this
|
356
|
+
method.
|
357
|
+
|
358
|
+
If you'd like to run multiple workers in development mode, you can do
|
359
|
+
so using the `resque:workers` rake task:
|
360
|
+
|
361
|
+
$ COUNT=5 QUEUE=* rake resque:workers
|
362
|
+
|
363
|
+
This will spawn five Resque workers, each in its own process. Hitting
|
364
|
+
ctrl-c should be sufficient to stop them all.
|
365
|
+
|
366
|
+
|
367
|
+
### Forking
|
368
|
+
|
369
|
+
On certain platforms, when a Resque worker reserves a job it
|
370
|
+
immediately forks a child process. The child processes the job then
|
371
|
+
exits. When the child has exited successfully, the worker reserves
|
372
|
+
another job and repeats the process.
|
373
|
+
|
374
|
+
Why?
|
375
|
+
|
376
|
+
Because Resque assumes chaos.
|
377
|
+
|
378
|
+
Resque assumes your background workers will lock up, run too long, or
|
379
|
+
have unwanted memory growth.
|
380
|
+
|
381
|
+
If Resque workers processed jobs themselves, it'd be hard to whip them
|
382
|
+
into shape. Let's say one is using too much memory: you send it a
|
383
|
+
signal that says "shutdown after you finish processing the current
|
384
|
+
job," and it does so. It then starts up again - loading your entire
|
385
|
+
application environment. This adds useless CPU cycles and causes a
|
386
|
+
delay in queue processing.
|
387
|
+
|
388
|
+
Plus, what if it's using too much memory and has stopped responding to
|
389
|
+
signals?
|
390
|
+
|
391
|
+
Thanks to Resque's parent / child architecture, jobs that use too much memory
|
392
|
+
release that memory upon completion. No unwanted growth.
|
393
|
+
|
394
|
+
And what if a job is running too long? You'd need to `kill -9` it then
|
395
|
+
start the worker again. With Resque's parent / child architecture you
|
396
|
+
can tell the parent to forcefully kill the child then immediately
|
397
|
+
start processing more jobs. No startup delay or wasted cycles.
|
398
|
+
|
399
|
+
The parent / child architecture helps us keep tabs on what workers are
|
400
|
+
doing, too. By eliminating the need to `kill -9` workers we can have
|
401
|
+
parents remove themselves from the global listing of workers. If we
|
402
|
+
just ruthlessly killed workers, we'd need a separate watchdog process
|
403
|
+
to add and remove them to the global listing - which becomes
|
404
|
+
complicated.
|
405
|
+
|
406
|
+
Workers instead handle their own state.
|
407
|
+
|
408
|
+
|
409
|
+
### Parents and Children
|
410
|
+
|
411
|
+
Here's a parent / child pair doing some work:
|
412
|
+
|
413
|
+
$ ps -e -o pid,command | grep [r]esque
|
414
|
+
92099 resque: Forked 92102 at 1253142769
|
415
|
+
92102 resque: Processing file_serve since 1253142769
|
416
|
+
|
417
|
+
You can clearly see that process 92099 forked 92102, which has been
|
418
|
+
working since 1253142769.
|
419
|
+
|
420
|
+
(By advertising the time they began processing you can easily use monit
|
421
|
+
or god to kill stale workers.)
|
422
|
+
|
423
|
+
When a parent process is idle, it lets you know what queues it is
|
424
|
+
waiting for work on:
|
425
|
+
|
426
|
+
$ ps -e -o pid,command | grep [r]esque
|
427
|
+
92099 resque: Waiting for file_serve,warm_cache
|
428
|
+
|
429
|
+
|
430
|
+
### Signals
|
431
|
+
|
432
|
+
Resque workers respond to a few different signals:
|
433
|
+
|
434
|
+
* `QUIT` - Wait for child to finish processing then exit
|
435
|
+
* `TERM` / `INT` - Immediately kill child then exit
|
436
|
+
* `USR1` - Immediately kill child but don't exit
|
437
|
+
* `USR2` - Don't start to process any new jobs
|
438
|
+
* `CONT` - Start to process new jobs again after a USR2
|
439
|
+
|
440
|
+
If you want to gracefully shutdown a Resque worker, use `QUIT`.
|
441
|
+
|
442
|
+
If you want to kill a stale or stuck child, use `USR1`. Processing
|
443
|
+
will continue as normal unless the child was not found. In that case
|
444
|
+
Resque assumes the parent process is in a bad state and shuts down.
|
445
|
+
|
446
|
+
If you want to kill a stale or stuck child and shutdown, use `TERM`
|
447
|
+
|
448
|
+
If you want to stop processing jobs, but want to leave the worker running
|
449
|
+
(for example, to temporarily alleviate load), use `USR2` to stop processing,
|
450
|
+
then `CONT` to start it again.
|
451
|
+
|
452
|
+
### Mysql::Error: MySQL server has gone away
|
453
|
+
|
454
|
+
If your workers remain idle for too long they may lose their MySQL connection. Depending on your version of Rails, we recommend the following:
|
455
|
+
|
456
|
+
#### Rails 3.x
|
457
|
+
In your `perform` method, add the following line:
|
458
|
+
|
459
|
+
``` ruby
|
460
|
+
class MyTask
|
461
|
+
def self.perform
|
462
|
+
ActiveRecord::Base.verify_active_connections!
|
463
|
+
# rest of your code
|
464
|
+
end
|
465
|
+
end
|
466
|
+
```
|
467
|
+
|
468
|
+
The Rails doc says the following about `verify_active_connections!`:
|
469
|
+
|
470
|
+
Verify active connections and remove and disconnect connections associated with stale threads.
|
471
|
+
|
472
|
+
#### Rails 4.x
|
473
|
+
|
474
|
+
In your `perform` method, instead of `verify_active_connections!`, use:
|
475
|
+
|
476
|
+
``` ruby
|
477
|
+
class MyTask
|
478
|
+
def self.perform
|
479
|
+
ActiveRecord::Base.clear_active_connections!
|
480
|
+
# rest of your code
|
481
|
+
end
|
482
|
+
end
|
483
|
+
```
|
484
|
+
|
485
|
+
From the Rails docs on [`clear_active_connections!`](http://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/ConnectionHandler.html#method-i-clear_active_connections-21):
|
486
|
+
|
487
|
+
Returns any connections in use by the current thread back to the pool, and also returns connections to the pool cached by threads that are no longer alive.
|
488
|
+
|
489
|
+
|
490
|
+
|
491
|
+
The Front End
|
492
|
+
-------------
|
493
|
+
|
494
|
+
Resque comes with a Sinatra-based front end for seeing what's up with
|
495
|
+
your queue.
|
496
|
+
|
497
|
+
![The Front End](https://camo.githubusercontent.com/64d150a243987ffbc33f588bd6d7722a0bb8d69a/687474703a2f2f7475746f7269616c732e6a756d7073746172746c61622e636f6d2f696d616765732f7265737175655f6f766572766965772e706e67)
|
498
|
+
|
499
|
+
### Standalone
|
500
|
+
|
501
|
+
If you've installed Resque as a gem running the front end standalone is easy:
|
502
|
+
|
503
|
+
$ resque-web
|
504
|
+
|
505
|
+
It's a thin layer around `rackup` so it's configurable as well:
|
506
|
+
|
507
|
+
$ resque-web -p 8282
|
508
|
+
|
509
|
+
If you have a Resque config file you want evaluated just pass it to
|
510
|
+
the script as the final argument:
|
511
|
+
|
512
|
+
$ resque-web -p 8282 rails_root/config/initializers/resque.rb
|
513
|
+
|
514
|
+
You can also set the namespace directly using `resque-web`:
|
515
|
+
|
516
|
+
$ resque-web -p 8282 -N myapp
|
517
|
+
|
518
|
+
or set the Redis connection string if you need to do something like select a different database:
|
519
|
+
|
520
|
+
$ resque-web -p 8282 -r localhost:6379:2
|
521
|
+
|
522
|
+
### Passenger
|
523
|
+
|
524
|
+
Using Passenger? Resque ships with a `config.ru` you can use. See
|
525
|
+
Phusion's guide:
|
526
|
+
|
527
|
+
Apache: <https://www.phusionpassenger.com/library/deploy/apache/deploy/ruby/>
|
528
|
+
Nginx: <https://www.phusionpassenger.com/library/deploy/nginx/deploy/ruby/>
|
529
|
+
|
530
|
+
### Rack::URLMap
|
531
|
+
|
532
|
+
If you want to load Resque on a subpath, possibly alongside other
|
533
|
+
apps, it's easy to do with Rack's `URLMap`:
|
534
|
+
|
535
|
+
``` ruby
|
536
|
+
require 'resque/server'
|
537
|
+
|
538
|
+
run Rack::URLMap.new \
|
539
|
+
"/" => Your::App.new,
|
540
|
+
"/resque" => Resque::Server.new
|
541
|
+
```
|
542
|
+
|
543
|
+
Check `examples/demo/config.ru` for a functional example (including
|
544
|
+
HTTP basic auth).
|
545
|
+
|
546
|
+
### Rails 3
|
547
|
+
|
548
|
+
You can also mount Resque on a subpath in your existing Rails 3 app by adding `require 'resque/server'` to the top of your routes file or in an initializer then adding this to `routes.rb`:
|
549
|
+
|
550
|
+
``` ruby
|
551
|
+
mount Resque::Server.new, :at => "/resque"
|
552
|
+
```
|
553
|
+
|
554
|
+
|
555
|
+
Resque vs DelayedJob
|
556
|
+
--------------------
|
557
|
+
|
558
|
+
How does Resque compare to DelayedJob, and why would you choose one
|
559
|
+
over the other?
|
560
|
+
|
561
|
+
* Resque supports multiple queues
|
562
|
+
* DelayedJob supports finer grained priorities
|
563
|
+
* Resque workers are resilient to memory leaks / bloat
|
564
|
+
* DelayedJob workers are extremely simple and easy to modify
|
565
|
+
* Resque requires Redis
|
566
|
+
* DelayedJob requires ActiveRecord
|
567
|
+
* Resque can only place JSONable Ruby objects on a queue as arguments
|
568
|
+
* DelayedJob can place _any_ Ruby object on its queue as arguments
|
569
|
+
* Resque includes a Sinatra app for monitoring what's going on
|
570
|
+
* DelayedJob can be queried from within your Rails app if you want to
|
571
|
+
add an interface
|
572
|
+
|
573
|
+
If you're doing Rails development, you already have a database and
|
574
|
+
ActiveRecord. DelayedJob is super easy to setup and works great.
|
575
|
+
GitHub used it for many months to process almost 200 million jobs.
|
576
|
+
|
577
|
+
Choose Resque if:
|
578
|
+
|
579
|
+
* You need multiple queues
|
580
|
+
* You don't care / dislike numeric priorities
|
581
|
+
* You don't need to persist every Ruby object ever
|
582
|
+
* You have potentially huge queues
|
583
|
+
* You want to see what's going on
|
584
|
+
* You expect a lot of failure / chaos
|
585
|
+
* You can setup Redis
|
586
|
+
* You're not running short on RAM
|
587
|
+
|
588
|
+
Choose DelayedJob if:
|
589
|
+
|
590
|
+
* You like numeric priorities
|
591
|
+
* You're not doing a gigantic amount of jobs each day
|
592
|
+
* Your queue stays small and nimble
|
593
|
+
* There is not a lot failure / chaos
|
594
|
+
* You want to easily throw anything on the queue
|
595
|
+
* You don't want to setup Redis
|
596
|
+
|
597
|
+
In no way is Resque a "better" DelayedJob, so make sure you pick the
|
598
|
+
tool that's best for your app.
|
599
|
+
|
600
|
+
Resque Dependencies
|
601
|
+
-------------------
|
602
|
+
|
603
|
+
$ gem install bundler
|
604
|
+
$ bundle install
|
605
|
+
|
606
|
+
|
607
|
+
Installing Resque
|
608
|
+
-----------------
|
609
|
+
|
610
|
+
### In a Rack app, as a gem
|
611
|
+
|
612
|
+
First install the gem.
|
613
|
+
|
614
|
+
$ gem install resque
|
615
|
+
|
616
|
+
Next include it in your application.
|
617
|
+
|
618
|
+
``` ruby
|
619
|
+
require 'resque'
|
620
|
+
```
|
621
|
+
|
622
|
+
Now start your application:
|
623
|
+
|
624
|
+
rackup config.ru
|
625
|
+
|
626
|
+
That's it! You can now create Resque jobs from within your app.
|
627
|
+
|
628
|
+
To start a worker, create a Rakefile in your app's root (or add this
|
629
|
+
to an existing Rakefile):
|
630
|
+
|
631
|
+
``` ruby
|
632
|
+
require 'your/app'
|
633
|
+
require 'resque/tasks'
|
634
|
+
```
|
635
|
+
|
636
|
+
Now:
|
637
|
+
|
638
|
+
$ QUEUE=* rake resque:work
|
639
|
+
|
640
|
+
Alternately you can define a `resque:setup` hook in your Rakefile if you
|
641
|
+
don't want to load your app every time rake runs.
|
642
|
+
|
643
|
+
|
644
|
+
### In a Rails 2.x app, as a gem
|
645
|
+
|
646
|
+
First install the gem.
|
647
|
+
|
648
|
+
$ gem install resque
|
649
|
+
|
650
|
+
Next include it in your application.
|
651
|
+
|
652
|
+
$ cat config/initializers/load_resque.rb
|
653
|
+
require 'resque'
|
654
|
+
|
655
|
+
Now start your application:
|
656
|
+
|
657
|
+
$ ./script/server
|
658
|
+
|
659
|
+
That's it! You can now create Resque jobs from within your app.
|
660
|
+
|
661
|
+
To start a worker, add this to your Rakefile in `RAILS_ROOT`:
|
662
|
+
|
663
|
+
``` ruby
|
664
|
+
require 'resque/tasks'
|
665
|
+
```
|
666
|
+
|
667
|
+
Now:
|
668
|
+
|
669
|
+
$ QUEUE=* rake environment resque:work
|
670
|
+
|
671
|
+
Don't forget you can define a `resque:setup` hook in
|
672
|
+
`lib/tasks/whatever.rake` that loads the `environment` task every time.
|
673
|
+
|
674
|
+
|
675
|
+
### In a Rails 2.x app, as a plugin
|
676
|
+
|
677
|
+
$ ./script/plugin install git://github.com/resque/resque
|
678
|
+
|
679
|
+
That's it! Resque will automatically be available when your Rails app
|
680
|
+
loads.
|
681
|
+
|
682
|
+
To start a worker:
|
683
|
+
|
684
|
+
$ QUEUE=* rake environment resque:work
|
685
|
+
|
686
|
+
Don't forget you can define a `resque:setup` hook in
|
687
|
+
`lib/tasks/whatever.rake` that loads the `environment` task every time.
|
688
|
+
|
689
|
+
|
690
|
+
### In a Rails 3.x or 4.x app, as a gem
|
691
|
+
|
692
|
+
First include it in your Gemfile.
|
693
|
+
|
694
|
+
$ cat Gemfile
|
695
|
+
...
|
696
|
+
gem 'resque'
|
697
|
+
...
|
698
|
+
|
699
|
+
Next install it with Bundler.
|
700
|
+
|
701
|
+
$ bundle install
|
702
|
+
|
703
|
+
Now start your application:
|
704
|
+
|
705
|
+
$ rails server
|
706
|
+
|
707
|
+
That's it! You can now create Resque jobs from within your app.
|
708
|
+
|
709
|
+
To start a worker, add this to a file in `lib/tasks` (ex:
|
710
|
+
`lib/tasks/resque.rake`):
|
711
|
+
|
712
|
+
``` ruby
|
713
|
+
require 'resque/tasks'
|
714
|
+
```
|
715
|
+
|
716
|
+
Now:
|
717
|
+
|
718
|
+
$ QUEUE=* rake environment resque:work
|
719
|
+
|
720
|
+
Don't forget you can define a `resque:setup` hook in
|
721
|
+
`lib/tasks/whatever.rake` that loads the `environment` task every time.
|
722
|
+
|
723
|
+
|
724
|
+
Configuration
|
725
|
+
-------------
|
726
|
+
|
727
|
+
You may want to change the Redis host and port Resque connects to, or
|
728
|
+
set various other options at startup.
|
729
|
+
|
730
|
+
Resque has a `redis` setter which can be given a string or a Redis
|
731
|
+
object. This means if you're already using Redis in your app, Resque
|
732
|
+
can re-use the existing connection.
|
733
|
+
|
734
|
+
String: `Resque.redis = 'localhost:6379'`
|
735
|
+
|
736
|
+
Redis: `Resque.redis = $redis`
|
737
|
+
|
738
|
+
For our rails app we have a `config/initializers/resque.rb` file where
|
739
|
+
we load `config/resque.yml` by hand and set the Redis information
|
740
|
+
appropriately.
|
741
|
+
|
742
|
+
Here's our `config/resque.yml`:
|
743
|
+
|
744
|
+
development: localhost:6379
|
745
|
+
test: localhost:6379
|
746
|
+
staging: redis1.se.github.com:6379
|
747
|
+
fi: localhost:6379
|
748
|
+
production: redis1.ae.github.com:6379
|
749
|
+
|
750
|
+
And our initializer:
|
751
|
+
|
752
|
+
``` ruby
|
753
|
+
rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..'
|
754
|
+
rails_env = ENV['RAILS_ENV'] || 'development'
|
755
|
+
|
756
|
+
resque_config = YAML.load_file(rails_root + '/config/resque.yml')
|
757
|
+
Resque.redis = resque_config[rails_env]
|
758
|
+
```
|
759
|
+
|
760
|
+
Easy peasy! Why not just use `RAILS_ROOT` and `RAILS_ENV`? Because
|
761
|
+
this way we can tell our Sinatra app about the config file:
|
762
|
+
|
763
|
+
$ RAILS_ENV=production resque-web rails_root/config/initializers/resque.rb
|
764
|
+
|
765
|
+
Now everyone is on the same page.
|
766
|
+
|
767
|
+
Also, you could disable jobs queueing by setting 'inline' attribute.
|
768
|
+
For example, if you want to run all jobs in the same process for cucumber, try:
|
769
|
+
|
770
|
+
``` ruby
|
771
|
+
Resque.inline = ENV['RAILS_ENV'] == "cucumber"
|
772
|
+
```
|
773
|
+
|
774
|
+
|
775
|
+
Plugins and Hooks
|
776
|
+
-----------------
|
777
|
+
|
778
|
+
For a list of available plugins see
|
779
|
+
<http://wiki.github.com/resque/resque/plugins>.
|
780
|
+
|
781
|
+
If you'd like to write your own plugin, or want to customize Resque
|
782
|
+
using hooks (such as `Resque.after_fork`), see
|
783
|
+
[docs/HOOKS.md](http://github.com/resque/resque/blob/master/docs/HOOKS.md).
|
784
|
+
|
785
|
+
|
786
|
+
Namespaces
|
787
|
+
----------
|
788
|
+
|
789
|
+
If you're running multiple, separate instances of Resque you may want
|
790
|
+
to namespace the keyspaces so they do not overlap. This is not unlike
|
791
|
+
the approach taken by many memcached clients.
|
792
|
+
|
793
|
+
This feature is provided by the [redis-namespace][rs] library, which
|
794
|
+
Resque uses by default to separate the keys it manages from other keys
|
795
|
+
in your Redis server.
|
796
|
+
|
797
|
+
Simply use the `Resque.redis.namespace` accessor:
|
798
|
+
|
799
|
+
``` ruby
|
800
|
+
Resque.redis.namespace = "resque:GitHub"
|
801
|
+
```
|
802
|
+
|
803
|
+
We recommend sticking this in your initializer somewhere after Redis
|
804
|
+
is configured.
|
805
|
+
|
806
|
+
|
807
|
+
Demo
|
808
|
+
----
|
809
|
+
|
810
|
+
Resque ships with a demo Sinatra app for creating jobs that are later
|
811
|
+
processed in the background.
|
812
|
+
|
813
|
+
Try it out by looking at the README, found at `examples/demo/README.markdown`.
|
814
|
+
|
815
|
+
|
816
|
+
Monitoring
|
817
|
+
----------
|
818
|
+
|
819
|
+
### god
|
820
|
+
|
821
|
+
If you're using god to monitor Resque, we have provided example
|
822
|
+
configs in `examples/god/`. One is for starting / stopping workers,
|
823
|
+
the other is for killing workers that have been running too long.
|
824
|
+
|
825
|
+
### monit
|
826
|
+
|
827
|
+
If you're using monit, `examples/monit/resque.monit` is provided free
|
828
|
+
of charge. This is **not** used by GitHub in production, so please
|
829
|
+
send patches for any tweaks or improvements you can make to it.
|
830
|
+
|
831
|
+
|
832
|
+
Questions
|
833
|
+
---------
|
834
|
+
|
835
|
+
Please add them to the [FAQ](https://github.com/resque/resque/wiki/FAQ) or
|
836
|
+
ask on the Mailing List. The Mailing List is explained further below
|
837
|
+
|
838
|
+
|
839
|
+
Development
|
840
|
+
-----------
|
841
|
+
|
842
|
+
Want to hack on Resque?
|
843
|
+
|
844
|
+
First clone the repo and run the tests:
|
845
|
+
|
846
|
+
git clone git://github.com/resque/resque.git
|
847
|
+
cd resque
|
848
|
+
rake test
|
849
|
+
|
850
|
+
If the tests do not pass make sure you have Redis installed
|
851
|
+
correctly (though we make an effort to tell you if we feel this is the
|
852
|
+
case). The tests attempt to start an isolated instance of Redis to
|
853
|
+
run against.
|
854
|
+
|
855
|
+
Also make sure you've installed all the dependencies correctly. For
|
856
|
+
example, try loading the `redis-namespace` gem after you've installed
|
857
|
+
it:
|
858
|
+
|
859
|
+
$ irb
|
860
|
+
>> require 'rubygems'
|
861
|
+
=> true
|
862
|
+
>> require 'redis/namespace'
|
863
|
+
=> true
|
864
|
+
|
865
|
+
If you get an error requiring any of the dependencies, you may have
|
866
|
+
failed to install them or be seeing load path issues.
|
867
|
+
|
868
|
+
Feel free to ping the mailing list with your problem and we'll try to
|
869
|
+
sort it out.
|
870
|
+
|
871
|
+
|
872
|
+
Contributing
|
873
|
+
------------
|
874
|
+
|
875
|
+
Read [CONTRIBUTING.md](CONTRIBUTING.md) first.
|
876
|
+
|
877
|
+
Once you've made your great commits:
|
878
|
+
|
879
|
+
1. [Fork][1] Resque
|
880
|
+
2. Create a topic branch - `git checkout -b my_branch`
|
881
|
+
3. Push to your branch - `git push origin my_branch`
|
882
|
+
4. Create a [Pull Request](http://help.github.com/pull-requests/) from your branch
|
883
|
+
5. That's it!
|
884
|
+
|
885
|
+
|
886
|
+
Mailing List
|
887
|
+
------------
|
888
|
+
|
889
|
+
To join the list simply send an email to <resque@librelist.com>. This
|
890
|
+
will subscribe you and send you information about your subscription,
|
891
|
+
including unsubscribe information.
|
892
|
+
|
893
|
+
The archive can be found at <http://librelist.com/browser/resque/>.
|
894
|
+
|
895
|
+
|
896
|
+
Meta
|
897
|
+
----
|
898
|
+
|
899
|
+
* Code: `git clone git://github.com/resque/resque.git`
|
900
|
+
* Home: <http://github.com/resque/resque>
|
901
|
+
* Docs: <http://rubydoc.info/gems/resque>
|
902
|
+
* Bugs: <http://github.com/resque/resque/issues>
|
903
|
+
* List: <resque@librelist.com>
|
904
|
+
* Chat: <irc://irc.freenode.net/resque>
|
905
|
+
* Gems: <http://gemcutter.org/gems/resque>
|
906
|
+
|
907
|
+
This project uses [Semantic Versioning][sv].
|
908
|
+
|
909
|
+
|
910
|
+
Author
|
911
|
+
------
|
912
|
+
|
913
|
+
Chris Wanstrath :: chris@ozmm.org :: @defunkt
|
914
|
+
|
915
|
+
[0]: http://github.com/blog/542-introducing-resque
|
916
|
+
[1]: http://help.github.com/forking/
|
917
|
+
[2]: http://github.com/resque/resque/issues
|
918
|
+
[sv]: http://semver.org/
|
919
|
+
[rs]: http://github.com/resque/redis-namespace
|
920
|
+
[cb]: http://wiki.github.com/resque/resque/contributing
|