resque_admin 0.2.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +7 -0
- data/HISTORY.md +530 -0
- data/LICENSE +20 -0
- data/README.markdown +957 -0
- data/Rakefile +57 -0
- data/bin/resque-admin +81 -0
- data/bin/resque-admin-web +31 -0
- data/lib/resque_admin.rb +578 -0
- data/lib/resque_admin/data_store.rb +325 -0
- data/lib/resque_admin/errors.rb +21 -0
- data/lib/resque_admin/failure.rb +119 -0
- data/lib/resque_admin/failure/airbrake.rb +33 -0
- data/lib/resque_admin/failure/base.rb +73 -0
- data/lib/resque_admin/failure/multiple.rb +68 -0
- data/lib/resque_admin/failure/redis.rb +128 -0
- data/lib/resque_admin/failure/redis_multi_queue.rb +104 -0
- data/lib/resque_admin/helpers.rb +48 -0
- data/lib/resque_admin/job.rb +296 -0
- data/lib/resque_admin/log_formatters/quiet_formatter.rb +7 -0
- data/lib/resque_admin/log_formatters/verbose_formatter.rb +7 -0
- data/lib/resque_admin/log_formatters/very_verbose_formatter.rb +8 -0
- data/lib/resque_admin/logging.rb +18 -0
- data/lib/resque_admin/plugin.rb +78 -0
- data/lib/resque_admin/server.rb +301 -0
- data/lib/resque_admin/server/helpers.rb +64 -0
- data/lib/resque_admin/server/public/favicon.ico +0 -0
- data/lib/resque_admin/server/public/idle.png +0 -0
- data/lib/resque_admin/server/public/jquery-1.12.4.min.js +5 -0
- data/lib/resque_admin/server/public/jquery.relatize_date.js +95 -0
- data/lib/resque_admin/server/public/poll.png +0 -0
- data/lib/resque_admin/server/public/ranger.js +78 -0
- data/lib/resque_admin/server/public/reset.css +44 -0
- data/lib/resque_admin/server/public/style.css +91 -0
- data/lib/resque_admin/server/public/working.png +0 -0
- data/lib/resque_admin/server/test_helper.rb +19 -0
- data/lib/resque_admin/server/views/error.erb +1 -0
- data/lib/resque_admin/server/views/failed.erb +29 -0
- data/lib/resque_admin/server/views/failed_job.erb +50 -0
- data/lib/resque_admin/server/views/failed_queues_overview.erb +24 -0
- data/lib/resque_admin/server/views/job_class.erb +6 -0
- data/lib/resque_admin/server/views/key_sets.erb +17 -0
- data/lib/resque_admin/server/views/key_string.erb +11 -0
- data/lib/resque_admin/server/views/layout.erb +44 -0
- data/lib/resque_admin/server/views/next_more.erb +22 -0
- data/lib/resque_admin/server/views/overview.erb +4 -0
- data/lib/resque_admin/server/views/processing.erb +2 -0
- data/lib/resque_admin/server/views/queues.erb +58 -0
- data/lib/resque_admin/server/views/stats.erb +62 -0
- data/lib/resque_admin/server/views/workers.erb +109 -0
- data/lib/resque_admin/server/views/working.erb +71 -0
- data/lib/resque_admin/stat.rb +58 -0
- data/lib/resque_admin/tasks.rb +72 -0
- data/lib/resque_admin/thread_signal.rb +24 -0
- data/lib/resque_admin/vendor/utf8_util.rb +24 -0
- data/lib/resque_admin/version.rb +3 -0
- data/lib/resque_admin/worker.rb +917 -0
- data/lib/tasks/redis.rake +161 -0
- data/lib/tasks/resque_admin.rake +2 -0
- metadata +191 -0
data/LICENSE
ADDED
@@ -0,0 +1,20 @@
|
|
1
|
+
Copyright (c) Chris Wanstrath
|
2
|
+
|
3
|
+
Permission is hereby granted, free of charge, to any person obtaining
|
4
|
+
a copy of this software and associated documentation files (the
|
5
|
+
"Software"), to deal in the Software without restriction, including
|
6
|
+
without limitation the rights to use, copy, modify, merge, publish,
|
7
|
+
distribute, sublicense, and/or sell copies of the Software, and to
|
8
|
+
permit persons to whom the Software is furnished to do so, subject to
|
9
|
+
the following conditions:
|
10
|
+
|
11
|
+
The above copyright notice and this permission notice shall be
|
12
|
+
included in all copies or substantial portions of the Software.
|
13
|
+
|
14
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
15
|
+
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
16
|
+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
17
|
+
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
18
|
+
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
19
|
+
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
20
|
+
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
data/README.markdown
ADDED
@@ -0,0 +1,957 @@
|
|
1
|
+
ResqueAdmin
|
2
|
+
======
|
3
|
+
|
4
|
+
[![Gem Version](https://badge.fury.io/rb/resque.svg)](https://rubygems.org/gems/resque)
|
5
|
+
[![Build Status](https://travis-ci.org/resque/resque.svg)](https://travis-ci.org/resque/resque)
|
6
|
+
[![Coverage Status](https://coveralls.io/repos/github/resque/resque/badge.svg?branch=1-x-stable)](https://coveralls.io/r/resque/resque?branch=1-x-stable)
|
7
|
+
|
8
|
+
ResqueAdmin (pronounced like "rescue") is a Redis-backed library for creating
|
9
|
+
background jobs, placing those jobs on multiple queues, and processing
|
10
|
+
them later.
|
11
|
+
|
12
|
+
Background jobs can be any Ruby class or module that responds to
|
13
|
+
`perform`. Your existing classes can easily be converted to background
|
14
|
+
jobs or you can create new classes specifically to do work. Or, you
|
15
|
+
can do both.
|
16
|
+
|
17
|
+
ResqueAdmin is heavily inspired by DelayedJob (which rocks) and comprises
|
18
|
+
three parts:
|
19
|
+
|
20
|
+
1. A Ruby library for creating, querying, and processing jobs
|
21
|
+
2. A Rake task for starting a worker which processes jobs
|
22
|
+
3. A Sinatra app for monitoring queues, jobs, and workers.
|
23
|
+
|
24
|
+
ResqueAdmin workers can be distributed between multiple machines,
|
25
|
+
support priorities, are resilient to memory bloat / "leaks," are
|
26
|
+
optimized for REE (but work on MRI and JRuby), tell you what they're
|
27
|
+
doing, and expect failure.
|
28
|
+
|
29
|
+
ResqueAdmin queues are persistent; support constant time, atomic push and
|
30
|
+
pop (thanks to Redis); provide visibility into their contents; and
|
31
|
+
store jobs as simple JSON packages.
|
32
|
+
|
33
|
+
The ResqueAdmin frontend tells you what workers are doing, what workers are
|
34
|
+
not doing, what queues you're using, what's in those queues, provides
|
35
|
+
general usage stats, and helps you track failures.
|
36
|
+
|
37
|
+
ResqueAdmin now supports Ruby 2.0.0 and above. Any future updates will not be
|
38
|
+
guaranteed to work without defects on any Rubies older than 2.0.0. We will also only be supporting Redis 3.0 and above going forward.
|
39
|
+
|
40
|
+
|
41
|
+
The Blog Post
|
42
|
+
-------------
|
43
|
+
|
44
|
+
For the backstory, philosophy, and history of ResqueAdmin's beginnings,
|
45
|
+
please see [the blog post][0].
|
46
|
+
|
47
|
+
|
48
|
+
Overview
|
49
|
+
--------
|
50
|
+
|
51
|
+
ResqueAdmin allows you to create jobs and place them on a queue, then,
|
52
|
+
later, pull those jobs off the queue and process them.
|
53
|
+
|
54
|
+
ResqueAdmin jobs are Ruby classes (or modules) which respond to the
|
55
|
+
`perform` method. Here's an example:
|
56
|
+
|
57
|
+
|
58
|
+
``` ruby
|
59
|
+
class Archive
|
60
|
+
@queue = :file_serve
|
61
|
+
|
62
|
+
def self.perform(repo_id, branch = 'master')
|
63
|
+
repo = Repository.find(repo_id)
|
64
|
+
repo.create_archive(branch)
|
65
|
+
end
|
66
|
+
end
|
67
|
+
```
|
68
|
+
|
69
|
+
The `@queue` class instance variable determines which queue `Archive`
|
70
|
+
jobs will be placed in. Queues are arbitrary and created on the fly -
|
71
|
+
you can name them whatever you want and have as many as you want.
|
72
|
+
|
73
|
+
To place an `Archive` job on the `file_serve` queue, we might add this
|
74
|
+
to our application's pre-existing `Repository` class:
|
75
|
+
|
76
|
+
``` ruby
|
77
|
+
class Repository
|
78
|
+
def async_create_archive(branch)
|
79
|
+
ResqueAdmin.enqueue(Archive, self.id, branch)
|
80
|
+
end
|
81
|
+
end
|
82
|
+
```
|
83
|
+
|
84
|
+
Now when we call `repo.async_create_archive('masterbrew')` in our
|
85
|
+
application, a job will be created and placed on the `file_serve`
|
86
|
+
queue.
|
87
|
+
|
88
|
+
Later, a worker will run something like this code to process the job:
|
89
|
+
|
90
|
+
``` ruby
|
91
|
+
klass, args = ResqueAdmin.reserve(:file_serve)
|
92
|
+
klass.perform(*args) if klass.respond_to? :perform
|
93
|
+
```
|
94
|
+
|
95
|
+
Which translates to:
|
96
|
+
|
97
|
+
``` ruby
|
98
|
+
Archive.perform(44, 'masterbrew')
|
99
|
+
```
|
100
|
+
|
101
|
+
Let's start a worker to run `file_serve` jobs:
|
102
|
+
|
103
|
+
$ cd app_root
|
104
|
+
$ QUEUE=file_serve rake resque:work
|
105
|
+
|
106
|
+
This starts one ResqueAdmin worker and tells it to work off the
|
107
|
+
`file_serve` queue. As soon as it's ready it'll try to run the
|
108
|
+
`ResqueAdmin.reserve` code snippet above and process jobs until it can't
|
109
|
+
find any more, at which point it will sleep for a small period and
|
110
|
+
repeatedly poll the queue for more jobs.
|
111
|
+
|
112
|
+
Workers can be given multiple queues (a "queue list") and run on
|
113
|
+
multiple machines. In fact they can be run anywhere with network
|
114
|
+
access to the Redis server.
|
115
|
+
|
116
|
+
|
117
|
+
Jobs
|
118
|
+
----
|
119
|
+
|
120
|
+
What should you run in the background? Anything that takes any time at
|
121
|
+
all. Slow INSERT statements, disk manipulating, data processing, etc.
|
122
|
+
|
123
|
+
At GitHub we use ResqueAdmin to process the following types of jobs:
|
124
|
+
|
125
|
+
* Warming caches
|
126
|
+
* Counting disk usage
|
127
|
+
* Building tarballs
|
128
|
+
* Building Rubygems
|
129
|
+
* Firing off web hooks
|
130
|
+
* Creating events in the db and pre-caching them
|
131
|
+
* Building graphs
|
132
|
+
* Deleting users
|
133
|
+
* Updating our search index
|
134
|
+
|
135
|
+
As of writing we have about 35 different types of background jobs.
|
136
|
+
|
137
|
+
Keep in mind that you don't need a web app to use ResqueAdmin - we just
|
138
|
+
mention "foreground" and "background" because they make conceptual
|
139
|
+
sense. You could easily be spidering sites and sticking data which
|
140
|
+
needs to be crunched later into a queue.
|
141
|
+
|
142
|
+
|
143
|
+
### Persistence
|
144
|
+
|
145
|
+
Jobs are persisted to queues as JSON objects. Let's take our `Archive`
|
146
|
+
example from above. We'll run the following code to create a job:
|
147
|
+
|
148
|
+
``` ruby
|
149
|
+
repo = Repository.find(44)
|
150
|
+
repo.async_create_archive('masterbrew')
|
151
|
+
```
|
152
|
+
|
153
|
+
The following JSON will be stored in the `file_serve` queue:
|
154
|
+
|
155
|
+
``` javascript
|
156
|
+
{
|
157
|
+
'class': 'Archive',
|
158
|
+
'args': [ 44, 'masterbrew' ]
|
159
|
+
}
|
160
|
+
```
|
161
|
+
|
162
|
+
Because of this your jobs must only accept arguments that can be JSON encoded.
|
163
|
+
|
164
|
+
So instead of doing this:
|
165
|
+
|
166
|
+
``` ruby
|
167
|
+
ResqueAdmin.enqueue(Archive, self, branch)
|
168
|
+
```
|
169
|
+
|
170
|
+
do this:
|
171
|
+
|
172
|
+
``` ruby
|
173
|
+
ResqueAdmin.enqueue(Archive, self.id, branch)
|
174
|
+
```
|
175
|
+
|
176
|
+
This is why our above example (and all the examples in `examples/`)
|
177
|
+
uses object IDs instead of passing around the objects.
|
178
|
+
|
179
|
+
While this is less convenient than just sticking a marshaled object
|
180
|
+
in the database, it gives you a slight advantage: your jobs will be
|
181
|
+
run against the most recent version of an object because they need to
|
182
|
+
pull from the DB or cache.
|
183
|
+
|
184
|
+
If your jobs were run against marshaled objects, they could
|
185
|
+
potentially be operating on a stale record with out-of-date information.
|
186
|
+
|
187
|
+
|
188
|
+
### send_later / async
|
189
|
+
|
190
|
+
Want something like DelayedJob's `send_later` or the ability to use
|
191
|
+
instance methods instead of just methods for jobs? See the `examples/`
|
192
|
+
directory for goodies.
|
193
|
+
|
194
|
+
We plan to provide first class `async` support in a future release.
|
195
|
+
|
196
|
+
|
197
|
+
### Failure
|
198
|
+
|
199
|
+
If a job raises an exception, it is logged and handed off to the
|
200
|
+
`ResqueAdmin::Failure` module. Failures are logged either locally in Redis
|
201
|
+
or using some different backend. To see exceptions while developing,
|
202
|
+
use VERBOSE env variable, see details below under Logging.
|
203
|
+
|
204
|
+
For example, ResqueAdmin ships with Airbrake support. To configure it, put
|
205
|
+
the following into an initialisation file or into your rake job:
|
206
|
+
|
207
|
+
``` ruby
|
208
|
+
# send errors which occur in background jobs to redis and airbrake
|
209
|
+
require 'resque/failure/multiple'
|
210
|
+
require 'resque/failure/redis'
|
211
|
+
require 'resque/failure/airbrake'
|
212
|
+
|
213
|
+
ResqueAdmin::Failure::Multiple.classes = [ResqueAdmin::Failure::Redis, ResqueAdmin::Failure::Airbrake]
|
214
|
+
ResqueAdmin::Failure.backend = ResqueAdmin::Failure::Multiple
|
215
|
+
```
|
216
|
+
|
217
|
+
Keep this in mind when writing your jobs: you may want to throw
|
218
|
+
exceptions you would not normally throw in order to assist debugging.
|
219
|
+
|
220
|
+
|
221
|
+
Workers
|
222
|
+
-------
|
223
|
+
|
224
|
+
ResqueAdmin workers are rake tasks that run forever. They basically do this:
|
225
|
+
|
226
|
+
``` ruby
|
227
|
+
start
|
228
|
+
loop do
|
229
|
+
if job = reserve
|
230
|
+
job.process
|
231
|
+
else
|
232
|
+
sleep 5 # Polling frequency = 5
|
233
|
+
end
|
234
|
+
end
|
235
|
+
shutdown
|
236
|
+
```
|
237
|
+
|
238
|
+
Starting a worker is simple. Here's our example from earlier:
|
239
|
+
|
240
|
+
$ QUEUE=file_serve rake resque:work
|
241
|
+
|
242
|
+
By default ResqueAdmin won't know about your application's
|
243
|
+
environment. That is, it won't be able to find and run your jobs - it
|
244
|
+
needs to load your application into memory.
|
245
|
+
|
246
|
+
If we've installed ResqueAdmin as a Rails plugin, we might run this command
|
247
|
+
from our RAILS_ROOT:
|
248
|
+
|
249
|
+
$ QUEUE=file_serve rake environment resque:work
|
250
|
+
|
251
|
+
This will load the environment before starting a worker. Alternately
|
252
|
+
we can define a `resque:setup` task with a dependency on the
|
253
|
+
`environment` rake task:
|
254
|
+
|
255
|
+
``` ruby
|
256
|
+
task "resque:setup" => :environment
|
257
|
+
```
|
258
|
+
|
259
|
+
GitHub's setup task looks like this:
|
260
|
+
|
261
|
+
``` ruby
|
262
|
+
task "resque:setup" => :environment do
|
263
|
+
Grit::Git.git_timeout = 10.minutes
|
264
|
+
end
|
265
|
+
```
|
266
|
+
|
267
|
+
We don't want the `git_timeout` as high as 10 minutes in our web app,
|
268
|
+
but in the ResqueAdmin workers it's fine.
|
269
|
+
|
270
|
+
|
271
|
+
### Logging
|
272
|
+
|
273
|
+
Workers support basic logging to STDOUT. If you start them with the
|
274
|
+
`VERBOSE` env variable set, they will print basic debugging
|
275
|
+
information. You can also set the `VVERBOSE` (very verbose) env
|
276
|
+
variable.
|
277
|
+
|
278
|
+
$ VVERBOSE=1 QUEUE=file_serve rake environment resque:work
|
279
|
+
|
280
|
+
If you want ResqueAdmin to log to a file, in Rails do:
|
281
|
+
|
282
|
+
```ruby
|
283
|
+
# config/initializers/resque.rb
|
284
|
+
ResqueAdmin.logger = Logger.new(Rails.root.join('log', "#{Rails.env}_resque.log"))
|
285
|
+
```
|
286
|
+
|
287
|
+
### Process IDs (PIDs)
|
288
|
+
|
289
|
+
There are scenarios where it's helpful to record the PID of a resque
|
290
|
+
worker process. Use the PIDFILE option for easy access to the PID:
|
291
|
+
|
292
|
+
$ PIDFILE=./resque.pid QUEUE=file_serve rake environment resque:work
|
293
|
+
|
294
|
+
### Running in the background
|
295
|
+
|
296
|
+
There are scenarios where it's helpful for
|
297
|
+
the resque worker to run itself in the background (usually in combination with
|
298
|
+
PIDFILE). Use the BACKGROUND option so that rake will return as soon as the
|
299
|
+
worker is started.
|
300
|
+
|
301
|
+
$ PIDFILE=./resque.pid BACKGROUND=yes QUEUE=file_serve \
|
302
|
+
rake environment resque:work
|
303
|
+
|
304
|
+
### Polling frequency
|
305
|
+
|
306
|
+
You can pass an INTERVAL option which is a float representing the polling frequency.
|
307
|
+
The default is 5 seconds, but for a semi-active app you may want to use a smaller value.
|
308
|
+
|
309
|
+
$ INTERVAL=0.1 QUEUE=file_serve rake environment resque:work
|
310
|
+
|
311
|
+
### Priorities and Queue Lists
|
312
|
+
|
313
|
+
ResqueAdmin doesn't support numeric priorities but instead uses the order
|
314
|
+
of queues you give it. We call this list of queues the "queue list."
|
315
|
+
|
316
|
+
Let's say we add a `warm_cache` queue in addition to our `file_serve`
|
317
|
+
queue. We'd now start a worker like so:
|
318
|
+
|
319
|
+
$ QUEUES=file_serve,warm_cache rake resque:work
|
320
|
+
|
321
|
+
When the worker looks for new jobs, it will first check
|
322
|
+
`file_serve`. If it finds a job, it'll process it then check
|
323
|
+
`file_serve` again. It will keep checking `file_serve` until no more
|
324
|
+
jobs are available. At that point, it will check `warm_cache`. If it
|
325
|
+
finds a job it'll process it then check `file_serve` (repeating the
|
326
|
+
whole process).
|
327
|
+
|
328
|
+
In this way you can prioritize certain queues. At GitHub we start our
|
329
|
+
workers with something like this:
|
330
|
+
|
331
|
+
$ QUEUES=critical,archive,high,low rake resque:work
|
332
|
+
|
333
|
+
Notice the `archive` queue - it is specialized and in our future
|
334
|
+
architecture will only be run from a single machine.
|
335
|
+
|
336
|
+
At that point we'll start workers on our generalized background
|
337
|
+
machines with this command:
|
338
|
+
|
339
|
+
$ QUEUES=critical,high,low rake resque:work
|
340
|
+
|
341
|
+
And workers on our specialized archive machine with this command:
|
342
|
+
|
343
|
+
$ QUEUE=archive rake resque:work
|
344
|
+
|
345
|
+
|
346
|
+
### Running All Queues
|
347
|
+
|
348
|
+
If you want your workers to work off of every queue, including new
|
349
|
+
queues created on the fly, you can use a splat:
|
350
|
+
|
351
|
+
$ QUEUE=* rake resque:work
|
352
|
+
|
353
|
+
Queues will be processed in alphabetical order.
|
354
|
+
|
355
|
+
|
356
|
+
### Running Multiple Workers
|
357
|
+
|
358
|
+
At GitHub we use god to start and stop multiple workers. A sample god
|
359
|
+
configuration file is included under `examples/god`. We recommend this
|
360
|
+
method.
|
361
|
+
|
362
|
+
If you'd like to run multiple workers in development mode, you can do
|
363
|
+
so using the `resque:workers` rake task:
|
364
|
+
|
365
|
+
$ COUNT=5 QUEUE=* rake resque:workers
|
366
|
+
|
367
|
+
This will spawn five ResqueAdmin workers, each in its own process. Hitting
|
368
|
+
ctrl-c should be sufficient to stop them all.
|
369
|
+
|
370
|
+
|
371
|
+
### Forking
|
372
|
+
|
373
|
+
On certain platforms, when a ResqueAdmin worker reserves a job it
|
374
|
+
immediately forks a child process. The child processes the job then
|
375
|
+
exits. When the child has exited successfully, the worker reserves
|
376
|
+
another job and repeats the process.
|
377
|
+
|
378
|
+
Why?
|
379
|
+
|
380
|
+
Because ResqueAdmin assumes chaos.
|
381
|
+
|
382
|
+
ResqueAdmin assumes your background workers will lock up, run too long, or
|
383
|
+
have unwanted memory growth.
|
384
|
+
|
385
|
+
If ResqueAdmin workers processed jobs themselves, it'd be hard to whip them
|
386
|
+
into shape. Let's say one is using too much memory: you send it a
|
387
|
+
signal that says "shutdown after you finish processing the current
|
388
|
+
job," and it does so. It then starts up again - loading your entire
|
389
|
+
application environment. This adds useless CPU cycles and causes a
|
390
|
+
delay in queue processing.
|
391
|
+
|
392
|
+
Plus, what if it's using too much memory and has stopped responding to
|
393
|
+
signals?
|
394
|
+
|
395
|
+
Thanks to ResqueAdmin's parent / child architecture, jobs that use too much memory
|
396
|
+
release that memory upon completion. No unwanted growth.
|
397
|
+
|
398
|
+
And what if a job is running too long? You'd need to `kill -9` it then
|
399
|
+
start the worker again. With ResqueAdmin's parent / child architecture you
|
400
|
+
can tell the parent to forcefully kill the child then immediately
|
401
|
+
start processing more jobs. No startup delay or wasted cycles.
|
402
|
+
|
403
|
+
The parent / child architecture helps us keep tabs on what workers are
|
404
|
+
doing, too. By eliminating the need to `kill -9` workers we can have
|
405
|
+
parents remove themselves from the global listing of workers. If we
|
406
|
+
just ruthlessly killed workers, we'd need a separate watchdog process
|
407
|
+
to add and remove them to the global listing - which becomes
|
408
|
+
complicated.
|
409
|
+
|
410
|
+
Workers instead handle their own state.
|
411
|
+
|
412
|
+
|
413
|
+
### Parents and Children
|
414
|
+
|
415
|
+
Here's a parent / child pair doing some work:
|
416
|
+
|
417
|
+
$ ps -e -o pid,command | grep [r]esque
|
418
|
+
92099 resque: Forked 92102 at 1253142769
|
419
|
+
92102 resque: Processing file_serve since 1253142769
|
420
|
+
|
421
|
+
You can clearly see that process 92099 forked 92102, which has been
|
422
|
+
working since 1253142769.
|
423
|
+
|
424
|
+
(By advertising the time they began processing you can easily use monit
|
425
|
+
or god to kill stale workers.)
|
426
|
+
|
427
|
+
When a parent process is idle, it lets you know what queues it is
|
428
|
+
waiting for work on:
|
429
|
+
|
430
|
+
$ ps -e -o pid,command | grep [r]esque
|
431
|
+
92099 resque: Waiting for file_serve,warm_cache
|
432
|
+
|
433
|
+
|
434
|
+
### Signals
|
435
|
+
|
436
|
+
ResqueAdmin workers respond to a few different signals:
|
437
|
+
|
438
|
+
* `QUIT` - Wait for child to finish processing then exit
|
439
|
+
* `TERM` / `INT` - Immediately kill child then exit
|
440
|
+
* `USR1` - Immediately kill child but don't exit
|
441
|
+
* `USR2` - Don't start to process any new jobs
|
442
|
+
* `CONT` - Start to process new jobs again after a USR2
|
443
|
+
|
444
|
+
If you want to gracefully shutdown a ResqueAdmin worker, use `QUIT`.
|
445
|
+
|
446
|
+
If you want to kill a stale or stuck child, use `USR1`. Processing
|
447
|
+
will continue as normal unless the child was not found. In that case
|
448
|
+
ResqueAdmin assumes the parent process is in a bad state and shuts down.
|
449
|
+
|
450
|
+
If you want to kill a stale or stuck child and shutdown, use `TERM`
|
451
|
+
|
452
|
+
If you want to stop processing jobs, but want to leave the worker running
|
453
|
+
(for example, to temporarily alleviate load), use `USR2` to stop processing,
|
454
|
+
then `CONT` to start it again.
|
455
|
+
|
456
|
+
#### Signals on Heroku
|
457
|
+
|
458
|
+
When shutting down processes, Heroku sends every process a TERM signal at the
|
459
|
+
same time. By default this causes an immediate shutdown of any running job
|
460
|
+
leading to frequent `ResqueAdmin::TermException` errors. For short running jobs, a simple
|
461
|
+
solution is to give a small amount of time for the job to finish
|
462
|
+
before killing it.
|
463
|
+
|
464
|
+
ResqueAdmin doesn't handle this out of the box (for both cedar-14 and heroku-16), you need to
|
465
|
+
install the [`resque-heroku-signals`](https://github.com/iloveitaly/resque-heroku-signals)
|
466
|
+
addon which adds the required signal handling to make the behavior described above work.
|
467
|
+
Related issue: https://github.com/resque/resque/issues/1559
|
468
|
+
|
469
|
+
To accomplish this set the following environment variables:
|
470
|
+
|
471
|
+
* `RESQUE_PRE_SHUTDOWN_TIMEOUT` - The time between the parent receiving a shutdown signal (TERM by default) and it sending that signal on to the child process. Designed to give the child process
|
472
|
+
time to complete before being forced to die.
|
473
|
+
|
474
|
+
* `TERM_CHILD` - Must be set for `RESQUE_PRE_SHUTDOWN_TIMEOUT` to be used. After the timeout, if the child is still running it will raise a `ResqueAdmin::TermException` and exit.
|
475
|
+
|
476
|
+
* `RESQUE_TERM_TIMEOUT` - By default you have a few seconds to handle `ResqueAdmin::TermException` in your job. `RESQUE_TERM_TIMEOUT` and `RESQUE_PRE_SHUTDOWN_TIMEOUT` must be lower than the [heroku dyno timeout](https://devcenter.heroku.com/articles/limits#exit-timeout).
|
477
|
+
|
478
|
+
### Mysql::Error: MySQL server has gone away
|
479
|
+
|
480
|
+
If your workers remain idle for too long they may lose their MySQL connection. Depending on your version of Rails, we recommend the following:
|
481
|
+
|
482
|
+
#### Rails 3.x
|
483
|
+
In your `perform` method, add the following line:
|
484
|
+
|
485
|
+
``` ruby
|
486
|
+
class MyTask
|
487
|
+
def self.perform
|
488
|
+
ActiveRecord::Base.verify_active_connections!
|
489
|
+
# rest of your code
|
490
|
+
end
|
491
|
+
end
|
492
|
+
```
|
493
|
+
|
494
|
+
The Rails doc says the following about `verify_active_connections!`:
|
495
|
+
|
496
|
+
Verify active connections and remove and disconnect connections associated with stale threads.
|
497
|
+
|
498
|
+
#### Rails 4.x
|
499
|
+
|
500
|
+
In your `perform` method, instead of `verify_active_connections!`, use:
|
501
|
+
|
502
|
+
``` ruby
|
503
|
+
class MyTask
|
504
|
+
def self.perform
|
505
|
+
ActiveRecord::Base.clear_active_connections!
|
506
|
+
# rest of your code
|
507
|
+
end
|
508
|
+
end
|
509
|
+
```
|
510
|
+
|
511
|
+
From the Rails docs on [`clear_active_connections!`](http://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/ConnectionHandler.html#method-i-clear_active_connections-21):
|
512
|
+
|
513
|
+
Returns any connections in use by the current thread back to the pool, and also returns connections to the pool cached by threads that are no longer alive.
|
514
|
+
|
515
|
+
|
516
|
+
#### ActiveJob
|
517
|
+
|
518
|
+
If you're going to need to use the database in a number of different jobs, consider adding this to your ApplicationJob file:
|
519
|
+
|
520
|
+
``` ruby
|
521
|
+
class ApplicationJob < ActiveJob::Base
|
522
|
+
before_perform do |job|
|
523
|
+
ActiveRecord::Base.clear_active_connections!
|
524
|
+
end
|
525
|
+
end
|
526
|
+
```
|
527
|
+
|
528
|
+
|
529
|
+
The Front End
|
530
|
+
-------------
|
531
|
+
|
532
|
+
ResqueAdmin comes with a Sinatra-based front end for seeing what's up with
|
533
|
+
your queue.
|
534
|
+
|
535
|
+
![The Front End](https://camo.githubusercontent.com/64d150a243987ffbc33f588bd6d7722a0bb8d69a/687474703a2f2f7475746f7269616c732e6a756d7073746172746c61622e636f6d2f696d616765732f7265737175655f6f766572766965772e706e67)
|
536
|
+
|
537
|
+
### Standalone
|
538
|
+
|
539
|
+
If you've installed ResqueAdmin as a gem running the front end standalone is easy:
|
540
|
+
|
541
|
+
$ resque-web
|
542
|
+
|
543
|
+
It's a thin layer around `rackup` so it's configurable as well:
|
544
|
+
|
545
|
+
$ resque-web -p 8282
|
546
|
+
|
547
|
+
If you have a ResqueAdmin config file you want evaluated just pass it to
|
548
|
+
the script as the final argument:
|
549
|
+
|
550
|
+
$ resque-web -p 8282 rails_root/config/initializers/resque.rb
|
551
|
+
|
552
|
+
You can also set the namespace directly using `resque-web`:
|
553
|
+
|
554
|
+
$ resque-web -p 8282 -N myapp
|
555
|
+
|
556
|
+
or set the Redis connection string if you need to do something like select a different database:
|
557
|
+
|
558
|
+
$ resque-web -p 8282 -r localhost:6379:2
|
559
|
+
|
560
|
+
### Passenger
|
561
|
+
|
562
|
+
Using Passenger? ResqueAdmin ships with a `config.ru` you can use. See
|
563
|
+
Phusion's guide:
|
564
|
+
|
565
|
+
Apache: <https://www.phusionpassenger.com/library/deploy/apache/deploy/ruby/>
|
566
|
+
Nginx: <https://www.phusionpassenger.com/library/deploy/nginx/deploy/ruby/>
|
567
|
+
|
568
|
+
### Rack::URLMap
|
569
|
+
|
570
|
+
If you want to load ResqueAdmin on a subpath, possibly alongside other
|
571
|
+
apps, it's easy to do with Rack's `URLMap`:
|
572
|
+
|
573
|
+
``` ruby
|
574
|
+
require 'resque/server'
|
575
|
+
|
576
|
+
run Rack::URLMap.new \
|
577
|
+
"/" => Your::App.new,
|
578
|
+
"/resque" => ResqueAdmin::Server.new
|
579
|
+
```
|
580
|
+
|
581
|
+
Check `examples/demo/config.ru` for a functional example (including
|
582
|
+
HTTP basic auth).
|
583
|
+
|
584
|
+
### Rails 3
|
585
|
+
|
586
|
+
You can also mount ResqueAdmin on a subpath in your existing Rails 3 app by adding `require 'resque/server'` to the top of your routes file or in an initializer then adding this to `routes.rb`:
|
587
|
+
|
588
|
+
``` ruby
|
589
|
+
mount ResqueAdmin::Server.new, :at => "/resque"
|
590
|
+
```
|
591
|
+
|
592
|
+
|
593
|
+
ResqueAdmin vs DelayedJob
|
594
|
+
--------------------
|
595
|
+
|
596
|
+
How does ResqueAdmin compare to DelayedJob, and why would you choose one
|
597
|
+
over the other?
|
598
|
+
|
599
|
+
* ResqueAdmin supports multiple queues
|
600
|
+
* DelayedJob supports finer grained priorities
|
601
|
+
* ResqueAdmin workers are resilient to memory leaks / bloat
|
602
|
+
* DelayedJob workers are extremely simple and easy to modify
|
603
|
+
* ResqueAdmin requires Redis
|
604
|
+
* DelayedJob requires ActiveRecord
|
605
|
+
* ResqueAdmin can only place JSONable Ruby objects on a queue as arguments
|
606
|
+
* DelayedJob can place _any_ Ruby object on its queue as arguments
|
607
|
+
* ResqueAdmin includes a Sinatra app for monitoring what's going on
|
608
|
+
* DelayedJob can be queried from within your Rails app if you want to
|
609
|
+
add an interface
|
610
|
+
|
611
|
+
If you're doing Rails development, you already have a database and
|
612
|
+
ActiveRecord. DelayedJob is super easy to setup and works great.
|
613
|
+
GitHub used it for many months to process almost 200 million jobs.
|
614
|
+
|
615
|
+
Choose ResqueAdmin if:
|
616
|
+
|
617
|
+
* You need multiple queues
|
618
|
+
* You don't care / dislike numeric priorities
|
619
|
+
* You don't need to persist every Ruby object ever
|
620
|
+
* You have potentially huge queues
|
621
|
+
* You want to see what's going on
|
622
|
+
* You expect a lot of failure / chaos
|
623
|
+
* You can setup Redis
|
624
|
+
* You're not running short on RAM
|
625
|
+
|
626
|
+
Choose DelayedJob if:
|
627
|
+
|
628
|
+
* You like numeric priorities
|
629
|
+
* You're not doing a gigantic amount of jobs each day
|
630
|
+
* Your queue stays small and nimble
|
631
|
+
* There is not a lot failure / chaos
|
632
|
+
* You want to easily throw anything on the queue
|
633
|
+
* You don't want to setup Redis
|
634
|
+
|
635
|
+
In no way is ResqueAdmin a "better" DelayedJob, so make sure you pick the
|
636
|
+
tool that's best for your app.
|
637
|
+
|
638
|
+
ResqueAdmin Dependencies
|
639
|
+
-------------------
|
640
|
+
|
641
|
+
$ gem install bundler
|
642
|
+
$ bundle install
|
643
|
+
|
644
|
+
|
645
|
+
Installing ResqueAdmin
|
646
|
+
-----------------
|
647
|
+
|
648
|
+
### In a Rack app, as a gem
|
649
|
+
|
650
|
+
First install the gem.
|
651
|
+
|
652
|
+
$ gem install resque
|
653
|
+
|
654
|
+
Next include it in your application.
|
655
|
+
|
656
|
+
``` ruby
|
657
|
+
require 'resque'
|
658
|
+
```
|
659
|
+
|
660
|
+
Now start your application:
|
661
|
+
|
662
|
+
rackup config.ru
|
663
|
+
|
664
|
+
That's it! You can now create ResqueAdmin jobs from within your app.
|
665
|
+
|
666
|
+
To start a worker, create a Rakefile in your app's root (or add this
|
667
|
+
to an existing Rakefile):
|
668
|
+
|
669
|
+
``` ruby
|
670
|
+
require 'your/app'
|
671
|
+
require 'resque/tasks'
|
672
|
+
```
|
673
|
+
|
674
|
+
If you're using Rails 5.x, include the following in lib/tasks/resque.rb:
|
675
|
+
|
676
|
+
```ruby
|
677
|
+
require 'resque/tasks'
|
678
|
+
task 'resque:setup' => :environment
|
679
|
+
```
|
680
|
+
|
681
|
+
Now:
|
682
|
+
|
683
|
+
$ QUEUE=* rake resque:work
|
684
|
+
|
685
|
+
Alternately you can define a `resque:setup` hook in your Rakefile if you
|
686
|
+
don't want to load your app every time rake runs.
|
687
|
+
|
688
|
+
|
689
|
+
### In a Rails 2.x app, as a gem
|
690
|
+
|
691
|
+
First install the gem.
|
692
|
+
|
693
|
+
$ gem install resque
|
694
|
+
|
695
|
+
Next include it in your application.
|
696
|
+
|
697
|
+
$ cat config/initializers/load_resque.rb
|
698
|
+
require 'resque'
|
699
|
+
|
700
|
+
Now start your application:
|
701
|
+
|
702
|
+
$ ./script/server
|
703
|
+
|
704
|
+
That's it! You can now create ResqueAdmin jobs from within your app.
|
705
|
+
|
706
|
+
To start a worker, add this to your Rakefile in `RAILS_ROOT`:
|
707
|
+
|
708
|
+
``` ruby
|
709
|
+
require 'resque/tasks'
|
710
|
+
```
|
711
|
+
|
712
|
+
Now:
|
713
|
+
|
714
|
+
$ QUEUE=* rake environment resque:work
|
715
|
+
|
716
|
+
Don't forget you can define a `resque:setup` hook in
|
717
|
+
`lib/tasks/whatever.rake` that loads the `environment` task every time.
|
718
|
+
|
719
|
+
|
720
|
+
### In a Rails 2.x app, as a plugin
|
721
|
+
|
722
|
+
$ ./script/plugin install git://github.com/resque/resque
|
723
|
+
|
724
|
+
That's it! ResqueAdmin will automatically be available when your Rails app
|
725
|
+
loads.
|
726
|
+
|
727
|
+
To start a worker:
|
728
|
+
|
729
|
+
$ QUEUE=* rake environment resque:work
|
730
|
+
|
731
|
+
Don't forget you can define a `resque:setup` hook in
|
732
|
+
`lib/tasks/whatever.rake` that loads the `environment` task every time.
|
733
|
+
|
734
|
+
|
735
|
+
### In a Rails 3.x or 4.x app, as a gem
|
736
|
+
|
737
|
+
First include it in your Gemfile.
|
738
|
+
|
739
|
+
$ cat Gemfile
|
740
|
+
...
|
741
|
+
gem 'resque'
|
742
|
+
...
|
743
|
+
|
744
|
+
Next install it with Bundler.
|
745
|
+
|
746
|
+
$ bundle install
|
747
|
+
|
748
|
+
Now start your application:
|
749
|
+
|
750
|
+
$ rails server
|
751
|
+
|
752
|
+
That's it! You can now create ResqueAdmin jobs from within your app.
|
753
|
+
|
754
|
+
To start a worker, add this to a file in `lib/tasks` (ex:
|
755
|
+
`lib/tasks/resque.rake`):
|
756
|
+
|
757
|
+
``` ruby
|
758
|
+
require 'resque/tasks'
|
759
|
+
```
|
760
|
+
|
761
|
+
Now:
|
762
|
+
|
763
|
+
$ QUEUE=* rake environment resque:work
|
764
|
+
|
765
|
+
Don't forget you can define a `resque:setup` hook in
|
766
|
+
`lib/tasks/whatever.rake` that loads the `environment` task every time.
|
767
|
+
|
768
|
+
|
769
|
+
Configuration
|
770
|
+
-------------
|
771
|
+
|
772
|
+
You may want to change the Redis host and port ResqueAdmin connects to, or
|
773
|
+
set various other options at startup.
|
774
|
+
|
775
|
+
ResqueAdmin has a `redis` setter which can be given a string or a Redis
|
776
|
+
object. This means if you're already using Redis in your app, ResqueAdmin
|
777
|
+
can re-use the existing connection.
|
778
|
+
|
779
|
+
String: `ResqueAdmin.redis = 'localhost:6379'`
|
780
|
+
|
781
|
+
Redis: `ResqueAdmin.redis = $redis`
|
782
|
+
|
783
|
+
For our rails app we have a `config/initializers/resque.rb` file where
|
784
|
+
we load `config/resque.yml` by hand and set the Redis information
|
785
|
+
appropriately.
|
786
|
+
|
787
|
+
Here's our `config/resque.yml`:
|
788
|
+
|
789
|
+
development: localhost:6379
|
790
|
+
test: localhost:6379
|
791
|
+
staging: redis1.se.github.com:6379
|
792
|
+
fi: localhost:6379
|
793
|
+
production: redis1.ae.github.com:6379
|
794
|
+
|
795
|
+
And our initializer:
|
796
|
+
|
797
|
+
``` ruby
|
798
|
+
rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..'
|
799
|
+
rails_env = ENV['RAILS_ENV'] || 'development'
|
800
|
+
|
801
|
+
resque_config = YAML.load_file(rails_root + '/config/resque.yml')
|
802
|
+
ResqueAdmin.redis = resque_config[rails_env]
|
803
|
+
```
|
804
|
+
|
805
|
+
Easy peasy! Why not just use `RAILS_ROOT` and `RAILS_ENV`? Because
|
806
|
+
this way we can tell our Sinatra app about the config file:
|
807
|
+
|
808
|
+
$ RAILS_ENV=production resque-web rails_root/config/initializers/resque.rb
|
809
|
+
|
810
|
+
Now everyone is on the same page.
|
811
|
+
|
812
|
+
Also, you could disable jobs queueing by setting 'inline' attribute.
|
813
|
+
For example, if you want to run all jobs in the same process for cucumber, try:
|
814
|
+
|
815
|
+
``` ruby
|
816
|
+
ResqueAdmin.inline = ENV['RAILS_ENV'] == "cucumber"
|
817
|
+
```
|
818
|
+
|
819
|
+
|
820
|
+
Plugins and Hooks
|
821
|
+
-----------------
|
822
|
+
|
823
|
+
For a list of available plugins see
|
824
|
+
<http://wiki.github.com/resque/resque/plugins>.
|
825
|
+
|
826
|
+
If you'd like to write your own plugin, or want to customize ResqueAdmin
|
827
|
+
using hooks (such as `ResqueAdmin.after_fork`), see
|
828
|
+
[docs/HOOKS.md](http://github.com/resque/resque/blob/master/docs/HOOKS.md).
|
829
|
+
|
830
|
+
|
831
|
+
Namespaces
|
832
|
+
----------
|
833
|
+
|
834
|
+
If you're running multiple, separate instances of ResqueAdmin you may want
|
835
|
+
to namespace the keyspaces so they do not overlap. This is not unlike
|
836
|
+
the approach taken by many memcached clients.
|
837
|
+
|
838
|
+
This feature is provided by the [redis-namespace][rs] library, which
|
839
|
+
ResqueAdmin uses by default to separate the keys it manages from other keys
|
840
|
+
in your Redis server.
|
841
|
+
|
842
|
+
Simply use the `ResqueAdmin.redis.namespace` accessor:
|
843
|
+
|
844
|
+
``` ruby
|
845
|
+
ResqueAdmin.redis.namespace = "resque:GitHub"
|
846
|
+
```
|
847
|
+
|
848
|
+
We recommend sticking this in your initializer somewhere after Redis
|
849
|
+
is configured.
|
850
|
+
|
851
|
+
|
852
|
+
Demo
|
853
|
+
----
|
854
|
+
|
855
|
+
ResqueAdmin ships with a demo Sinatra app for creating jobs that are later
|
856
|
+
processed in the background.
|
857
|
+
|
858
|
+
Try it out by looking at the README, found at `examples/demo/README.markdown`.
|
859
|
+
|
860
|
+
|
861
|
+
Monitoring
|
862
|
+
----------
|
863
|
+
|
864
|
+
### god
|
865
|
+
|
866
|
+
If you're using god to monitor ResqueAdmin, we have provided example
|
867
|
+
configs in `examples/god/`. One is for starting / stopping workers,
|
868
|
+
the other is for killing workers that have been running too long.
|
869
|
+
|
870
|
+
### monit
|
871
|
+
|
872
|
+
If you're using monit, `examples/monit/resque.monit` is provided free
|
873
|
+
of charge. This is **not** used by GitHub in production, so please
|
874
|
+
send patches for any tweaks or improvements you can make to it.
|
875
|
+
|
876
|
+
|
877
|
+
Questions
|
878
|
+
---------
|
879
|
+
|
880
|
+
Please add them to the [FAQ](https://github.com/resque/resque/wiki/FAQ) or open an issue on this repo.
|
881
|
+
|
882
|
+
|
883
|
+
Development
|
884
|
+
-----------
|
885
|
+
|
886
|
+
Want to hack on ResqueAdmin?
|
887
|
+
|
888
|
+
First clone the repo and run the tests:
|
889
|
+
|
890
|
+
git clone git://github.com/resque/resque.git
|
891
|
+
cd resque
|
892
|
+
rake test
|
893
|
+
|
894
|
+
If the tests do not pass make sure you have Redis installed
|
895
|
+
correctly (though we make an effort to tell you if we feel this is the
|
896
|
+
case). The tests attempt to start an isolated instance of Redis to
|
897
|
+
run against.
|
898
|
+
|
899
|
+
Also make sure you've installed all the dependencies correctly. For
|
900
|
+
example, try loading the `redis-namespace` gem after you've installed
|
901
|
+
it:
|
902
|
+
|
903
|
+
$ irb
|
904
|
+
>> require 'rubygems'
|
905
|
+
=> true
|
906
|
+
>> require 'redis/namespace'
|
907
|
+
=> true
|
908
|
+
|
909
|
+
If you get an error requiring any of the dependencies, you may have
|
910
|
+
failed to install them or be seeing load path issues.
|
911
|
+
|
912
|
+
|
913
|
+
Contributing
|
914
|
+
------------
|
915
|
+
|
916
|
+
Read [CONTRIBUTING.md](CONTRIBUTING.md) first.
|
917
|
+
|
918
|
+
Once you've made your great commits:
|
919
|
+
|
920
|
+
1. [Fork][1] ResqueAdmin
|
921
|
+
2. Create a topic branch - `git checkout -b my_branch`
|
922
|
+
3. Push to your branch - `git push origin my_branch`
|
923
|
+
4. Create a [Pull Request](http://help.github.com/pull-requests/) from your branch
|
924
|
+
5. That's it!
|
925
|
+
|
926
|
+
|
927
|
+
Mailing List
|
928
|
+
------------
|
929
|
+
|
930
|
+
This mailing list is no longer maintained. The archive can be found at <http://librelist.com/browser/resque/>.
|
931
|
+
|
932
|
+
|
933
|
+
Meta
|
934
|
+
----
|
935
|
+
|
936
|
+
* Code: `git clone git://github.com/resque/resque.git`
|
937
|
+
* Home: <http://github.com/resque/resque>
|
938
|
+
* Docs: <http://rubydoc.info/gems/resque>
|
939
|
+
* Bugs: <http://github.com/resque/resque/issues>
|
940
|
+
* List: <resque@librelist.com>
|
941
|
+
* Chat: <irc://irc.freenode.net/resque>
|
942
|
+
* Gems: <http://gemcutter.org/gems/resque>
|
943
|
+
|
944
|
+
This project uses [Semantic Versioning][sv].
|
945
|
+
|
946
|
+
|
947
|
+
Author
|
948
|
+
------
|
949
|
+
|
950
|
+
Chris Wanstrath :: chris@ozmm.org :: @defunkt
|
951
|
+
|
952
|
+
[0]: http://github.com/blog/542-introducing-resque
|
953
|
+
[1]: http://help.github.com/forking/
|
954
|
+
[2]: http://github.com/resque/resque/issues
|
955
|
+
[sv]: http://semver.org/
|
956
|
+
[rs]: http://github.com/resque/redis-namespace
|
957
|
+
[cb]: http://wiki.github.com/resque/resque/contributing
|