resque_admin 0.2.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (59) hide show
  1. checksums.yaml +7 -0
  2. data/HISTORY.md +530 -0
  3. data/LICENSE +20 -0
  4. data/README.markdown +957 -0
  5. data/Rakefile +57 -0
  6. data/bin/resque-admin +81 -0
  7. data/bin/resque-admin-web +31 -0
  8. data/lib/resque_admin.rb +578 -0
  9. data/lib/resque_admin/data_store.rb +325 -0
  10. data/lib/resque_admin/errors.rb +21 -0
  11. data/lib/resque_admin/failure.rb +119 -0
  12. data/lib/resque_admin/failure/airbrake.rb +33 -0
  13. data/lib/resque_admin/failure/base.rb +73 -0
  14. data/lib/resque_admin/failure/multiple.rb +68 -0
  15. data/lib/resque_admin/failure/redis.rb +128 -0
  16. data/lib/resque_admin/failure/redis_multi_queue.rb +104 -0
  17. data/lib/resque_admin/helpers.rb +48 -0
  18. data/lib/resque_admin/job.rb +296 -0
  19. data/lib/resque_admin/log_formatters/quiet_formatter.rb +7 -0
  20. data/lib/resque_admin/log_formatters/verbose_formatter.rb +7 -0
  21. data/lib/resque_admin/log_formatters/very_verbose_formatter.rb +8 -0
  22. data/lib/resque_admin/logging.rb +18 -0
  23. data/lib/resque_admin/plugin.rb +78 -0
  24. data/lib/resque_admin/server.rb +301 -0
  25. data/lib/resque_admin/server/helpers.rb +64 -0
  26. data/lib/resque_admin/server/public/favicon.ico +0 -0
  27. data/lib/resque_admin/server/public/idle.png +0 -0
  28. data/lib/resque_admin/server/public/jquery-1.12.4.min.js +5 -0
  29. data/lib/resque_admin/server/public/jquery.relatize_date.js +95 -0
  30. data/lib/resque_admin/server/public/poll.png +0 -0
  31. data/lib/resque_admin/server/public/ranger.js +78 -0
  32. data/lib/resque_admin/server/public/reset.css +44 -0
  33. data/lib/resque_admin/server/public/style.css +91 -0
  34. data/lib/resque_admin/server/public/working.png +0 -0
  35. data/lib/resque_admin/server/test_helper.rb +19 -0
  36. data/lib/resque_admin/server/views/error.erb +1 -0
  37. data/lib/resque_admin/server/views/failed.erb +29 -0
  38. data/lib/resque_admin/server/views/failed_job.erb +50 -0
  39. data/lib/resque_admin/server/views/failed_queues_overview.erb +24 -0
  40. data/lib/resque_admin/server/views/job_class.erb +6 -0
  41. data/lib/resque_admin/server/views/key_sets.erb +17 -0
  42. data/lib/resque_admin/server/views/key_string.erb +11 -0
  43. data/lib/resque_admin/server/views/layout.erb +44 -0
  44. data/lib/resque_admin/server/views/next_more.erb +22 -0
  45. data/lib/resque_admin/server/views/overview.erb +4 -0
  46. data/lib/resque_admin/server/views/processing.erb +2 -0
  47. data/lib/resque_admin/server/views/queues.erb +58 -0
  48. data/lib/resque_admin/server/views/stats.erb +62 -0
  49. data/lib/resque_admin/server/views/workers.erb +109 -0
  50. data/lib/resque_admin/server/views/working.erb +71 -0
  51. data/lib/resque_admin/stat.rb +58 -0
  52. data/lib/resque_admin/tasks.rb +72 -0
  53. data/lib/resque_admin/thread_signal.rb +24 -0
  54. data/lib/resque_admin/vendor/utf8_util.rb +24 -0
  55. data/lib/resque_admin/version.rb +3 -0
  56. data/lib/resque_admin/worker.rb +917 -0
  57. data/lib/tasks/redis.rake +161 -0
  58. data/lib/tasks/resque_admin.rake +2 -0
  59. metadata +191 -0
data/LICENSE ADDED
@@ -0,0 +1,20 @@
1
+ Copyright (c) Chris Wanstrath
2
+
3
+ Permission is hereby granted, free of charge, to any person obtaining
4
+ a copy of this software and associated documentation files (the
5
+ "Software"), to deal in the Software without restriction, including
6
+ without limitation the rights to use, copy, modify, merge, publish,
7
+ distribute, sublicense, and/or sell copies of the Software, and to
8
+ permit persons to whom the Software is furnished to do so, subject to
9
+ the following conditions:
10
+
11
+ The above copyright notice and this permission notice shall be
12
+ included in all copies or substantial portions of the Software.
13
+
14
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
15
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
16
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
17
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
18
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
19
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
20
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
data/README.markdown ADDED
@@ -0,0 +1,957 @@
1
+ ResqueAdmin
2
+ ======
3
+
4
+ [![Gem Version](https://badge.fury.io/rb/resque.svg)](https://rubygems.org/gems/resque)
5
+ [![Build Status](https://travis-ci.org/resque/resque.svg)](https://travis-ci.org/resque/resque)
6
+ [![Coverage Status](https://coveralls.io/repos/github/resque/resque/badge.svg?branch=1-x-stable)](https://coveralls.io/r/resque/resque?branch=1-x-stable)
7
+
8
+ ResqueAdmin (pronounced like "rescue") is a Redis-backed library for creating
9
+ background jobs, placing those jobs on multiple queues, and processing
10
+ them later.
11
+
12
+ Background jobs can be any Ruby class or module that responds to
13
+ `perform`. Your existing classes can easily be converted to background
14
+ jobs or you can create new classes specifically to do work. Or, you
15
+ can do both.
16
+
17
+ ResqueAdmin is heavily inspired by DelayedJob (which rocks) and comprises
18
+ three parts:
19
+
20
+ 1. A Ruby library for creating, querying, and processing jobs
21
+ 2. A Rake task for starting a worker which processes jobs
22
+ 3. A Sinatra app for monitoring queues, jobs, and workers.
23
+
24
+ ResqueAdmin workers can be distributed between multiple machines,
25
+ support priorities, are resilient to memory bloat / "leaks," are
26
+ optimized for REE (but work on MRI and JRuby), tell you what they're
27
+ doing, and expect failure.
28
+
29
+ ResqueAdmin queues are persistent; support constant time, atomic push and
30
+ pop (thanks to Redis); provide visibility into their contents; and
31
+ store jobs as simple JSON packages.
32
+
33
+ The ResqueAdmin frontend tells you what workers are doing, what workers are
34
+ not doing, what queues you're using, what's in those queues, provides
35
+ general usage stats, and helps you track failures.
36
+
37
+ ResqueAdmin now supports Ruby 2.0.0 and above. Any future updates will not be
38
+ guaranteed to work without defects on any Rubies older than 2.0.0. We will also only be supporting Redis 3.0 and above going forward.
39
+
40
+
41
+ The Blog Post
42
+ -------------
43
+
44
+ For the backstory, philosophy, and history of ResqueAdmin's beginnings,
45
+ please see [the blog post][0].
46
+
47
+
48
+ Overview
49
+ --------
50
+
51
+ ResqueAdmin allows you to create jobs and place them on a queue, then,
52
+ later, pull those jobs off the queue and process them.
53
+
54
+ ResqueAdmin jobs are Ruby classes (or modules) which respond to the
55
+ `perform` method. Here's an example:
56
+
57
+
58
+ ``` ruby
59
+ class Archive
60
+ @queue = :file_serve
61
+
62
+ def self.perform(repo_id, branch = 'master')
63
+ repo = Repository.find(repo_id)
64
+ repo.create_archive(branch)
65
+ end
66
+ end
67
+ ```
68
+
69
+ The `@queue` class instance variable determines which queue `Archive`
70
+ jobs will be placed in. Queues are arbitrary and created on the fly -
71
+ you can name them whatever you want and have as many as you want.
72
+
73
+ To place an `Archive` job on the `file_serve` queue, we might add this
74
+ to our application's pre-existing `Repository` class:
75
+
76
+ ``` ruby
77
+ class Repository
78
+ def async_create_archive(branch)
79
+ ResqueAdmin.enqueue(Archive, self.id, branch)
80
+ end
81
+ end
82
+ ```
83
+
84
+ Now when we call `repo.async_create_archive('masterbrew')` in our
85
+ application, a job will be created and placed on the `file_serve`
86
+ queue.
87
+
88
+ Later, a worker will run something like this code to process the job:
89
+
90
+ ``` ruby
91
+ klass, args = ResqueAdmin.reserve(:file_serve)
92
+ klass.perform(*args) if klass.respond_to? :perform
93
+ ```
94
+
95
+ Which translates to:
96
+
97
+ ``` ruby
98
+ Archive.perform(44, 'masterbrew')
99
+ ```
100
+
101
+ Let's start a worker to run `file_serve` jobs:
102
+
103
+ $ cd app_root
104
+ $ QUEUE=file_serve rake resque:work
105
+
106
+ This starts one ResqueAdmin worker and tells it to work off the
107
+ `file_serve` queue. As soon as it's ready it'll try to run the
108
+ `ResqueAdmin.reserve` code snippet above and process jobs until it can't
109
+ find any more, at which point it will sleep for a small period and
110
+ repeatedly poll the queue for more jobs.
111
+
112
+ Workers can be given multiple queues (a "queue list") and run on
113
+ multiple machines. In fact they can be run anywhere with network
114
+ access to the Redis server.
115
+
116
+
117
+ Jobs
118
+ ----
119
+
120
+ What should you run in the background? Anything that takes any time at
121
+ all. Slow INSERT statements, disk manipulating, data processing, etc.
122
+
123
+ At GitHub we use ResqueAdmin to process the following types of jobs:
124
+
125
+ * Warming caches
126
+ * Counting disk usage
127
+ * Building tarballs
128
+ * Building Rubygems
129
+ * Firing off web hooks
130
+ * Creating events in the db and pre-caching them
131
+ * Building graphs
132
+ * Deleting users
133
+ * Updating our search index
134
+
135
+ As of writing we have about 35 different types of background jobs.
136
+
137
+ Keep in mind that you don't need a web app to use ResqueAdmin - we just
138
+ mention "foreground" and "background" because they make conceptual
139
+ sense. You could easily be spidering sites and sticking data which
140
+ needs to be crunched later into a queue.
141
+
142
+
143
+ ### Persistence
144
+
145
+ Jobs are persisted to queues as JSON objects. Let's take our `Archive`
146
+ example from above. We'll run the following code to create a job:
147
+
148
+ ``` ruby
149
+ repo = Repository.find(44)
150
+ repo.async_create_archive('masterbrew')
151
+ ```
152
+
153
+ The following JSON will be stored in the `file_serve` queue:
154
+
155
+ ``` javascript
156
+ {
157
+ 'class': 'Archive',
158
+ 'args': [ 44, 'masterbrew' ]
159
+ }
160
+ ```
161
+
162
+ Because of this your jobs must only accept arguments that can be JSON encoded.
163
+
164
+ So instead of doing this:
165
+
166
+ ``` ruby
167
+ ResqueAdmin.enqueue(Archive, self, branch)
168
+ ```
169
+
170
+ do this:
171
+
172
+ ``` ruby
173
+ ResqueAdmin.enqueue(Archive, self.id, branch)
174
+ ```
175
+
176
+ This is why our above example (and all the examples in `examples/`)
177
+ uses object IDs instead of passing around the objects.
178
+
179
+ While this is less convenient than just sticking a marshaled object
180
+ in the database, it gives you a slight advantage: your jobs will be
181
+ run against the most recent version of an object because they need to
182
+ pull from the DB or cache.
183
+
184
+ If your jobs were run against marshaled objects, they could
185
+ potentially be operating on a stale record with out-of-date information.
186
+
187
+
188
+ ### send_later / async
189
+
190
+ Want something like DelayedJob's `send_later` or the ability to use
191
+ instance methods instead of just methods for jobs? See the `examples/`
192
+ directory for goodies.
193
+
194
+ We plan to provide first class `async` support in a future release.
195
+
196
+
197
+ ### Failure
198
+
199
+ If a job raises an exception, it is logged and handed off to the
200
+ `ResqueAdmin::Failure` module. Failures are logged either locally in Redis
201
+ or using some different backend. To see exceptions while developing,
202
+ use VERBOSE env variable, see details below under Logging.
203
+
204
+ For example, ResqueAdmin ships with Airbrake support. To configure it, put
205
+ the following into an initialisation file or into your rake job:
206
+
207
+ ``` ruby
208
+ # send errors which occur in background jobs to redis and airbrake
209
+ require 'resque/failure/multiple'
210
+ require 'resque/failure/redis'
211
+ require 'resque/failure/airbrake'
212
+
213
+ ResqueAdmin::Failure::Multiple.classes = [ResqueAdmin::Failure::Redis, ResqueAdmin::Failure::Airbrake]
214
+ ResqueAdmin::Failure.backend = ResqueAdmin::Failure::Multiple
215
+ ```
216
+
217
+ Keep this in mind when writing your jobs: you may want to throw
218
+ exceptions you would not normally throw in order to assist debugging.
219
+
220
+
221
+ Workers
222
+ -------
223
+
224
+ ResqueAdmin workers are rake tasks that run forever. They basically do this:
225
+
226
+ ``` ruby
227
+ start
228
+ loop do
229
+ if job = reserve
230
+ job.process
231
+ else
232
+ sleep 5 # Polling frequency = 5
233
+ end
234
+ end
235
+ shutdown
236
+ ```
237
+
238
+ Starting a worker is simple. Here's our example from earlier:
239
+
240
+ $ QUEUE=file_serve rake resque:work
241
+
242
+ By default ResqueAdmin won't know about your application's
243
+ environment. That is, it won't be able to find and run your jobs - it
244
+ needs to load your application into memory.
245
+
246
+ If we've installed ResqueAdmin as a Rails plugin, we might run this command
247
+ from our RAILS_ROOT:
248
+
249
+ $ QUEUE=file_serve rake environment resque:work
250
+
251
+ This will load the environment before starting a worker. Alternately
252
+ we can define a `resque:setup` task with a dependency on the
253
+ `environment` rake task:
254
+
255
+ ``` ruby
256
+ task "resque:setup" => :environment
257
+ ```
258
+
259
+ GitHub's setup task looks like this:
260
+
261
+ ``` ruby
262
+ task "resque:setup" => :environment do
263
+ Grit::Git.git_timeout = 10.minutes
264
+ end
265
+ ```
266
+
267
+ We don't want the `git_timeout` as high as 10 minutes in our web app,
268
+ but in the ResqueAdmin workers it's fine.
269
+
270
+
271
+ ### Logging
272
+
273
+ Workers support basic logging to STDOUT. If you start them with the
274
+ `VERBOSE` env variable set, they will print basic debugging
275
+ information. You can also set the `VVERBOSE` (very verbose) env
276
+ variable.
277
+
278
+ $ VVERBOSE=1 QUEUE=file_serve rake environment resque:work
279
+
280
+ If you want ResqueAdmin to log to a file, in Rails do:
281
+
282
+ ```ruby
283
+ # config/initializers/resque.rb
284
+ ResqueAdmin.logger = Logger.new(Rails.root.join('log', "#{Rails.env}_resque.log"))
285
+ ```
286
+
287
+ ### Process IDs (PIDs)
288
+
289
+ There are scenarios where it's helpful to record the PID of a resque
290
+ worker process. Use the PIDFILE option for easy access to the PID:
291
+
292
+ $ PIDFILE=./resque.pid QUEUE=file_serve rake environment resque:work
293
+
294
+ ### Running in the background
295
+
296
+ There are scenarios where it's helpful for
297
+ the resque worker to run itself in the background (usually in combination with
298
+ PIDFILE). Use the BACKGROUND option so that rake will return as soon as the
299
+ worker is started.
300
+
301
+ $ PIDFILE=./resque.pid BACKGROUND=yes QUEUE=file_serve \
302
+ rake environment resque:work
303
+
304
+ ### Polling frequency
305
+
306
+ You can pass an INTERVAL option which is a float representing the polling frequency.
307
+ The default is 5 seconds, but for a semi-active app you may want to use a smaller value.
308
+
309
+ $ INTERVAL=0.1 QUEUE=file_serve rake environment resque:work
310
+
311
+ ### Priorities and Queue Lists
312
+
313
+ ResqueAdmin doesn't support numeric priorities but instead uses the order
314
+ of queues you give it. We call this list of queues the "queue list."
315
+
316
+ Let's say we add a `warm_cache` queue in addition to our `file_serve`
317
+ queue. We'd now start a worker like so:
318
+
319
+ $ QUEUES=file_serve,warm_cache rake resque:work
320
+
321
+ When the worker looks for new jobs, it will first check
322
+ `file_serve`. If it finds a job, it'll process it then check
323
+ `file_serve` again. It will keep checking `file_serve` until no more
324
+ jobs are available. At that point, it will check `warm_cache`. If it
325
+ finds a job it'll process it then check `file_serve` (repeating the
326
+ whole process).
327
+
328
+ In this way you can prioritize certain queues. At GitHub we start our
329
+ workers with something like this:
330
+
331
+ $ QUEUES=critical,archive,high,low rake resque:work
332
+
333
+ Notice the `archive` queue - it is specialized and in our future
334
+ architecture will only be run from a single machine.
335
+
336
+ At that point we'll start workers on our generalized background
337
+ machines with this command:
338
+
339
+ $ QUEUES=critical,high,low rake resque:work
340
+
341
+ And workers on our specialized archive machine with this command:
342
+
343
+ $ QUEUE=archive rake resque:work
344
+
345
+
346
+ ### Running All Queues
347
+
348
+ If you want your workers to work off of every queue, including new
349
+ queues created on the fly, you can use a splat:
350
+
351
+ $ QUEUE=* rake resque:work
352
+
353
+ Queues will be processed in alphabetical order.
354
+
355
+
356
+ ### Running Multiple Workers
357
+
358
+ At GitHub we use god to start and stop multiple workers. A sample god
359
+ configuration file is included under `examples/god`. We recommend this
360
+ method.
361
+
362
+ If you'd like to run multiple workers in development mode, you can do
363
+ so using the `resque:workers` rake task:
364
+
365
+ $ COUNT=5 QUEUE=* rake resque:workers
366
+
367
+ This will spawn five ResqueAdmin workers, each in its own process. Hitting
368
+ ctrl-c should be sufficient to stop them all.
369
+
370
+
371
+ ### Forking
372
+
373
+ On certain platforms, when a ResqueAdmin worker reserves a job it
374
+ immediately forks a child process. The child processes the job then
375
+ exits. When the child has exited successfully, the worker reserves
376
+ another job and repeats the process.
377
+
378
+ Why?
379
+
380
+ Because ResqueAdmin assumes chaos.
381
+
382
+ ResqueAdmin assumes your background workers will lock up, run too long, or
383
+ have unwanted memory growth.
384
+
385
+ If ResqueAdmin workers processed jobs themselves, it'd be hard to whip them
386
+ into shape. Let's say one is using too much memory: you send it a
387
+ signal that says "shutdown after you finish processing the current
388
+ job," and it does so. It then starts up again - loading your entire
389
+ application environment. This adds useless CPU cycles and causes a
390
+ delay in queue processing.
391
+
392
+ Plus, what if it's using too much memory and has stopped responding to
393
+ signals?
394
+
395
+ Thanks to ResqueAdmin's parent / child architecture, jobs that use too much memory
396
+ release that memory upon completion. No unwanted growth.
397
+
398
+ And what if a job is running too long? You'd need to `kill -9` it then
399
+ start the worker again. With ResqueAdmin's parent / child architecture you
400
+ can tell the parent to forcefully kill the child then immediately
401
+ start processing more jobs. No startup delay or wasted cycles.
402
+
403
+ The parent / child architecture helps us keep tabs on what workers are
404
+ doing, too. By eliminating the need to `kill -9` workers we can have
405
+ parents remove themselves from the global listing of workers. If we
406
+ just ruthlessly killed workers, we'd need a separate watchdog process
407
+ to add and remove them to the global listing - which becomes
408
+ complicated.
409
+
410
+ Workers instead handle their own state.
411
+
412
+
413
+ ### Parents and Children
414
+
415
+ Here's a parent / child pair doing some work:
416
+
417
+ $ ps -e -o pid,command | grep [r]esque
418
+ 92099 resque: Forked 92102 at 1253142769
419
+ 92102 resque: Processing file_serve since 1253142769
420
+
421
+ You can clearly see that process 92099 forked 92102, which has been
422
+ working since 1253142769.
423
+
424
+ (By advertising the time they began processing you can easily use monit
425
+ or god to kill stale workers.)
426
+
427
+ When a parent process is idle, it lets you know what queues it is
428
+ waiting for work on:
429
+
430
+ $ ps -e -o pid,command | grep [r]esque
431
+ 92099 resque: Waiting for file_serve,warm_cache
432
+
433
+
434
+ ### Signals
435
+
436
+ ResqueAdmin workers respond to a few different signals:
437
+
438
+ * `QUIT` - Wait for child to finish processing then exit
439
+ * `TERM` / `INT` - Immediately kill child then exit
440
+ * `USR1` - Immediately kill child but don't exit
441
+ * `USR2` - Don't start to process any new jobs
442
+ * `CONT` - Start to process new jobs again after a USR2
443
+
444
+ If you want to gracefully shutdown a ResqueAdmin worker, use `QUIT`.
445
+
446
+ If you want to kill a stale or stuck child, use `USR1`. Processing
447
+ will continue as normal unless the child was not found. In that case
448
+ ResqueAdmin assumes the parent process is in a bad state and shuts down.
449
+
450
+ If you want to kill a stale or stuck child and shutdown, use `TERM`
451
+
452
+ If you want to stop processing jobs, but want to leave the worker running
453
+ (for example, to temporarily alleviate load), use `USR2` to stop processing,
454
+ then `CONT` to start it again.
455
+
456
+ #### Signals on Heroku
457
+
458
+ When shutting down processes, Heroku sends every process a TERM signal at the
459
+ same time. By default this causes an immediate shutdown of any running job
460
+ leading to frequent `ResqueAdmin::TermException` errors. For short running jobs, a simple
461
+ solution is to give a small amount of time for the job to finish
462
+ before killing it.
463
+
464
+ ResqueAdmin doesn't handle this out of the box (for both cedar-14 and heroku-16), you need to
465
+ install the [`resque-heroku-signals`](https://github.com/iloveitaly/resque-heroku-signals)
466
+ addon which adds the required signal handling to make the behavior described above work.
467
+ Related issue: https://github.com/resque/resque/issues/1559
468
+
469
+ To accomplish this set the following environment variables:
470
+
471
+ * `RESQUE_PRE_SHUTDOWN_TIMEOUT` - The time between the parent receiving a shutdown signal (TERM by default) and it sending that signal on to the child process. Designed to give the child process
472
+ time to complete before being forced to die.
473
+
474
+ * `TERM_CHILD` - Must be set for `RESQUE_PRE_SHUTDOWN_TIMEOUT` to be used. After the timeout, if the child is still running it will raise a `ResqueAdmin::TermException` and exit.
475
+
476
+ * `RESQUE_TERM_TIMEOUT` - By default you have a few seconds to handle `ResqueAdmin::TermException` in your job. `RESQUE_TERM_TIMEOUT` and `RESQUE_PRE_SHUTDOWN_TIMEOUT` must be lower than the [heroku dyno timeout](https://devcenter.heroku.com/articles/limits#exit-timeout).
477
+
478
+ ### Mysql::Error: MySQL server has gone away
479
+
480
+ If your workers remain idle for too long they may lose their MySQL connection. Depending on your version of Rails, we recommend the following:
481
+
482
+ #### Rails 3.x
483
+ In your `perform` method, add the following line:
484
+
485
+ ``` ruby
486
+ class MyTask
487
+ def self.perform
488
+ ActiveRecord::Base.verify_active_connections!
489
+ # rest of your code
490
+ end
491
+ end
492
+ ```
493
+
494
+ The Rails doc says the following about `verify_active_connections!`:
495
+
496
+ Verify active connections and remove and disconnect connections associated with stale threads.
497
+
498
+ #### Rails 4.x
499
+
500
+ In your `perform` method, instead of `verify_active_connections!`, use:
501
+
502
+ ``` ruby
503
+ class MyTask
504
+ def self.perform
505
+ ActiveRecord::Base.clear_active_connections!
506
+ # rest of your code
507
+ end
508
+ end
509
+ ```
510
+
511
+ From the Rails docs on [`clear_active_connections!`](http://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/ConnectionHandler.html#method-i-clear_active_connections-21):
512
+
513
+ Returns any connections in use by the current thread back to the pool, and also returns connections to the pool cached by threads that are no longer alive.
514
+
515
+
516
+ #### ActiveJob
517
+
518
+ If you're going to need to use the database in a number of different jobs, consider adding this to your ApplicationJob file:
519
+
520
+ ``` ruby
521
+ class ApplicationJob < ActiveJob::Base
522
+ before_perform do |job|
523
+ ActiveRecord::Base.clear_active_connections!
524
+ end
525
+ end
526
+ ```
527
+
528
+
529
+ The Front End
530
+ -------------
531
+
532
+ ResqueAdmin comes with a Sinatra-based front end for seeing what's up with
533
+ your queue.
534
+
535
+ ![The Front End](https://camo.githubusercontent.com/64d150a243987ffbc33f588bd6d7722a0bb8d69a/687474703a2f2f7475746f7269616c732e6a756d7073746172746c61622e636f6d2f696d616765732f7265737175655f6f766572766965772e706e67)
536
+
537
+ ### Standalone
538
+
539
+ If you've installed ResqueAdmin as a gem running the front end standalone is easy:
540
+
541
+ $ resque-web
542
+
543
+ It's a thin layer around `rackup` so it's configurable as well:
544
+
545
+ $ resque-web -p 8282
546
+
547
+ If you have a ResqueAdmin config file you want evaluated just pass it to
548
+ the script as the final argument:
549
+
550
+ $ resque-web -p 8282 rails_root/config/initializers/resque.rb
551
+
552
+ You can also set the namespace directly using `resque-web`:
553
+
554
+ $ resque-web -p 8282 -N myapp
555
+
556
+ or set the Redis connection string if you need to do something like select a different database:
557
+
558
+ $ resque-web -p 8282 -r localhost:6379:2
559
+
560
+ ### Passenger
561
+
562
+ Using Passenger? ResqueAdmin ships with a `config.ru` you can use. See
563
+ Phusion's guide:
564
+
565
+ Apache: <https://www.phusionpassenger.com/library/deploy/apache/deploy/ruby/>
566
+ Nginx: <https://www.phusionpassenger.com/library/deploy/nginx/deploy/ruby/>
567
+
568
+ ### Rack::URLMap
569
+
570
+ If you want to load ResqueAdmin on a subpath, possibly alongside other
571
+ apps, it's easy to do with Rack's `URLMap`:
572
+
573
+ ``` ruby
574
+ require 'resque/server'
575
+
576
+ run Rack::URLMap.new \
577
+ "/" => Your::App.new,
578
+ "/resque" => ResqueAdmin::Server.new
579
+ ```
580
+
581
+ Check `examples/demo/config.ru` for a functional example (including
582
+ HTTP basic auth).
583
+
584
+ ### Rails 3
585
+
586
+ You can also mount ResqueAdmin on a subpath in your existing Rails 3 app by adding `require 'resque/server'` to the top of your routes file or in an initializer then adding this to `routes.rb`:
587
+
588
+ ``` ruby
589
+ mount ResqueAdmin::Server.new, :at => "/resque"
590
+ ```
591
+
592
+
593
+ ResqueAdmin vs DelayedJob
594
+ --------------------
595
+
596
+ How does ResqueAdmin compare to DelayedJob, and why would you choose one
597
+ over the other?
598
+
599
+ * ResqueAdmin supports multiple queues
600
+ * DelayedJob supports finer grained priorities
601
+ * ResqueAdmin workers are resilient to memory leaks / bloat
602
+ * DelayedJob workers are extremely simple and easy to modify
603
+ * ResqueAdmin requires Redis
604
+ * DelayedJob requires ActiveRecord
605
+ * ResqueAdmin can only place JSONable Ruby objects on a queue as arguments
606
+ * DelayedJob can place _any_ Ruby object on its queue as arguments
607
+ * ResqueAdmin includes a Sinatra app for monitoring what's going on
608
+ * DelayedJob can be queried from within your Rails app if you want to
609
+ add an interface
610
+
611
+ If you're doing Rails development, you already have a database and
612
+ ActiveRecord. DelayedJob is super easy to setup and works great.
613
+ GitHub used it for many months to process almost 200 million jobs.
614
+
615
+ Choose ResqueAdmin if:
616
+
617
+ * You need multiple queues
618
+ * You don't care / dislike numeric priorities
619
+ * You don't need to persist every Ruby object ever
620
+ * You have potentially huge queues
621
+ * You want to see what's going on
622
+ * You expect a lot of failure / chaos
623
+ * You can setup Redis
624
+ * You're not running short on RAM
625
+
626
+ Choose DelayedJob if:
627
+
628
+ * You like numeric priorities
629
+ * You're not doing a gigantic amount of jobs each day
630
+ * Your queue stays small and nimble
631
+ * There is not a lot failure / chaos
632
+ * You want to easily throw anything on the queue
633
+ * You don't want to setup Redis
634
+
635
+ In no way is ResqueAdmin a "better" DelayedJob, so make sure you pick the
636
+ tool that's best for your app.
637
+
638
+ ResqueAdmin Dependencies
639
+ -------------------
640
+
641
+ $ gem install bundler
642
+ $ bundle install
643
+
644
+
645
+ Installing ResqueAdmin
646
+ -----------------
647
+
648
+ ### In a Rack app, as a gem
649
+
650
+ First install the gem.
651
+
652
+ $ gem install resque
653
+
654
+ Next include it in your application.
655
+
656
+ ``` ruby
657
+ require 'resque'
658
+ ```
659
+
660
+ Now start your application:
661
+
662
+ rackup config.ru
663
+
664
+ That's it! You can now create ResqueAdmin jobs from within your app.
665
+
666
+ To start a worker, create a Rakefile in your app's root (or add this
667
+ to an existing Rakefile):
668
+
669
+ ``` ruby
670
+ require 'your/app'
671
+ require 'resque/tasks'
672
+ ```
673
+
674
+ If you're using Rails 5.x, include the following in lib/tasks/resque.rb:
675
+
676
+ ```ruby
677
+ require 'resque/tasks'
678
+ task 'resque:setup' => :environment
679
+ ```
680
+
681
+ Now:
682
+
683
+ $ QUEUE=* rake resque:work
684
+
685
+ Alternately you can define a `resque:setup` hook in your Rakefile if you
686
+ don't want to load your app every time rake runs.
687
+
688
+
689
+ ### In a Rails 2.x app, as a gem
690
+
691
+ First install the gem.
692
+
693
+ $ gem install resque
694
+
695
+ Next include it in your application.
696
+
697
+ $ cat config/initializers/load_resque.rb
698
+ require 'resque'
699
+
700
+ Now start your application:
701
+
702
+ $ ./script/server
703
+
704
+ That's it! You can now create ResqueAdmin jobs from within your app.
705
+
706
+ To start a worker, add this to your Rakefile in `RAILS_ROOT`:
707
+
708
+ ``` ruby
709
+ require 'resque/tasks'
710
+ ```
711
+
712
+ Now:
713
+
714
+ $ QUEUE=* rake environment resque:work
715
+
716
+ Don't forget you can define a `resque:setup` hook in
717
+ `lib/tasks/whatever.rake` that loads the `environment` task every time.
718
+
719
+
720
+ ### In a Rails 2.x app, as a plugin
721
+
722
+ $ ./script/plugin install git://github.com/resque/resque
723
+
724
+ That's it! ResqueAdmin will automatically be available when your Rails app
725
+ loads.
726
+
727
+ To start a worker:
728
+
729
+ $ QUEUE=* rake environment resque:work
730
+
731
+ Don't forget you can define a `resque:setup` hook in
732
+ `lib/tasks/whatever.rake` that loads the `environment` task every time.
733
+
734
+
735
+ ### In a Rails 3.x or 4.x app, as a gem
736
+
737
+ First include it in your Gemfile.
738
+
739
+ $ cat Gemfile
740
+ ...
741
+ gem 'resque'
742
+ ...
743
+
744
+ Next install it with Bundler.
745
+
746
+ $ bundle install
747
+
748
+ Now start your application:
749
+
750
+ $ rails server
751
+
752
+ That's it! You can now create ResqueAdmin jobs from within your app.
753
+
754
+ To start a worker, add this to a file in `lib/tasks` (ex:
755
+ `lib/tasks/resque.rake`):
756
+
757
+ ``` ruby
758
+ require 'resque/tasks'
759
+ ```
760
+
761
+ Now:
762
+
763
+ $ QUEUE=* rake environment resque:work
764
+
765
+ Don't forget you can define a `resque:setup` hook in
766
+ `lib/tasks/whatever.rake` that loads the `environment` task every time.
767
+
768
+
769
+ Configuration
770
+ -------------
771
+
772
+ You may want to change the Redis host and port ResqueAdmin connects to, or
773
+ set various other options at startup.
774
+
775
+ ResqueAdmin has a `redis` setter which can be given a string or a Redis
776
+ object. This means if you're already using Redis in your app, ResqueAdmin
777
+ can re-use the existing connection.
778
+
779
+ String: `ResqueAdmin.redis = 'localhost:6379'`
780
+
781
+ Redis: `ResqueAdmin.redis = $redis`
782
+
783
+ For our rails app we have a `config/initializers/resque.rb` file where
784
+ we load `config/resque.yml` by hand and set the Redis information
785
+ appropriately.
786
+
787
+ Here's our `config/resque.yml`:
788
+
789
+ development: localhost:6379
790
+ test: localhost:6379
791
+ staging: redis1.se.github.com:6379
792
+ fi: localhost:6379
793
+ production: redis1.ae.github.com:6379
794
+
795
+ And our initializer:
796
+
797
+ ``` ruby
798
+ rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..'
799
+ rails_env = ENV['RAILS_ENV'] || 'development'
800
+
801
+ resque_config = YAML.load_file(rails_root + '/config/resque.yml')
802
+ ResqueAdmin.redis = resque_config[rails_env]
803
+ ```
804
+
805
+ Easy peasy! Why not just use `RAILS_ROOT` and `RAILS_ENV`? Because
806
+ this way we can tell our Sinatra app about the config file:
807
+
808
+ $ RAILS_ENV=production resque-web rails_root/config/initializers/resque.rb
809
+
810
+ Now everyone is on the same page.
811
+
812
+ Also, you could disable jobs queueing by setting 'inline' attribute.
813
+ For example, if you want to run all jobs in the same process for cucumber, try:
814
+
815
+ ``` ruby
816
+ ResqueAdmin.inline = ENV['RAILS_ENV'] == "cucumber"
817
+ ```
818
+
819
+
820
+ Plugins and Hooks
821
+ -----------------
822
+
823
+ For a list of available plugins see
824
+ <http://wiki.github.com/resque/resque/plugins>.
825
+
826
+ If you'd like to write your own plugin, or want to customize ResqueAdmin
827
+ using hooks (such as `ResqueAdmin.after_fork`), see
828
+ [docs/HOOKS.md](http://github.com/resque/resque/blob/master/docs/HOOKS.md).
829
+
830
+
831
+ Namespaces
832
+ ----------
833
+
834
+ If you're running multiple, separate instances of ResqueAdmin you may want
835
+ to namespace the keyspaces so they do not overlap. This is not unlike
836
+ the approach taken by many memcached clients.
837
+
838
+ This feature is provided by the [redis-namespace][rs] library, which
839
+ ResqueAdmin uses by default to separate the keys it manages from other keys
840
+ in your Redis server.
841
+
842
+ Simply use the `ResqueAdmin.redis.namespace` accessor:
843
+
844
+ ``` ruby
845
+ ResqueAdmin.redis.namespace = "resque:GitHub"
846
+ ```
847
+
848
+ We recommend sticking this in your initializer somewhere after Redis
849
+ is configured.
850
+
851
+
852
+ Demo
853
+ ----
854
+
855
+ ResqueAdmin ships with a demo Sinatra app for creating jobs that are later
856
+ processed in the background.
857
+
858
+ Try it out by looking at the README, found at `examples/demo/README.markdown`.
859
+
860
+
861
+ Monitoring
862
+ ----------
863
+
864
+ ### god
865
+
866
+ If you're using god to monitor ResqueAdmin, we have provided example
867
+ configs in `examples/god/`. One is for starting / stopping workers,
868
+ the other is for killing workers that have been running too long.
869
+
870
+ ### monit
871
+
872
+ If you're using monit, `examples/monit/resque.monit` is provided free
873
+ of charge. This is **not** used by GitHub in production, so please
874
+ send patches for any tweaks or improvements you can make to it.
875
+
876
+
877
+ Questions
878
+ ---------
879
+
880
+ Please add them to the [FAQ](https://github.com/resque/resque/wiki/FAQ) or open an issue on this repo.
881
+
882
+
883
+ Development
884
+ -----------
885
+
886
+ Want to hack on ResqueAdmin?
887
+
888
+ First clone the repo and run the tests:
889
+
890
+ git clone git://github.com/resque/resque.git
891
+ cd resque
892
+ rake test
893
+
894
+ If the tests do not pass make sure you have Redis installed
895
+ correctly (though we make an effort to tell you if we feel this is the
896
+ case). The tests attempt to start an isolated instance of Redis to
897
+ run against.
898
+
899
+ Also make sure you've installed all the dependencies correctly. For
900
+ example, try loading the `redis-namespace` gem after you've installed
901
+ it:
902
+
903
+ $ irb
904
+ >> require 'rubygems'
905
+ => true
906
+ >> require 'redis/namespace'
907
+ => true
908
+
909
+ If you get an error requiring any of the dependencies, you may have
910
+ failed to install them or be seeing load path issues.
911
+
912
+
913
+ Contributing
914
+ ------------
915
+
916
+ Read [CONTRIBUTING.md](CONTRIBUTING.md) first.
917
+
918
+ Once you've made your great commits:
919
+
920
+ 1. [Fork][1] ResqueAdmin
921
+ 2. Create a topic branch - `git checkout -b my_branch`
922
+ 3. Push to your branch - `git push origin my_branch`
923
+ 4. Create a [Pull Request](http://help.github.com/pull-requests/) from your branch
924
+ 5. That's it!
925
+
926
+
927
+ Mailing List
928
+ ------------
929
+
930
+ This mailing list is no longer maintained. The archive can be found at <http://librelist.com/browser/resque/>.
931
+
932
+
933
+ Meta
934
+ ----
935
+
936
+ * Code: `git clone git://github.com/resque/resque.git`
937
+ * Home: <http://github.com/resque/resque>
938
+ * Docs: <http://rubydoc.info/gems/resque>
939
+ * Bugs: <http://github.com/resque/resque/issues>
940
+ * List: <resque@librelist.com>
941
+ * Chat: <irc://irc.freenode.net/resque>
942
+ * Gems: <http://gemcutter.org/gems/resque>
943
+
944
+ This project uses [Semantic Versioning][sv].
945
+
946
+
947
+ Author
948
+ ------
949
+
950
+ Chris Wanstrath :: chris@ozmm.org :: @defunkt
951
+
952
+ [0]: http://github.com/blog/542-introducing-resque
953
+ [1]: http://help.github.com/forking/
954
+ [2]: http://github.com/resque/resque/issues
955
+ [sv]: http://semver.org/
956
+ [rs]: http://github.com/resque/redis-namespace
957
+ [cb]: http://wiki.github.com/resque/resque/contributing