steini-resque 1.18.5

Sign up to get free protection for your applications and to get access to all the features.
Files changed (54) hide show
  1. data/HISTORY.md +322 -0
  2. data/LICENSE +20 -0
  3. data/README.markdown +881 -0
  4. data/Rakefile +78 -0
  5. data/bin/resque +81 -0
  6. data/bin/resque-web +23 -0
  7. data/lib/resque.rb +352 -0
  8. data/lib/resque/errors.rb +10 -0
  9. data/lib/resque/failure.rb +70 -0
  10. data/lib/resque/failure/base.rb +64 -0
  11. data/lib/resque/failure/hoptoad.rb +48 -0
  12. data/lib/resque/failure/multiple.rb +54 -0
  13. data/lib/resque/failure/redis.rb +51 -0
  14. data/lib/resque/helpers.rb +63 -0
  15. data/lib/resque/job.rb +205 -0
  16. data/lib/resque/plugin.rb +56 -0
  17. data/lib/resque/server.rb +231 -0
  18. data/lib/resque/server/public/favicon.ico +0 -0
  19. data/lib/resque/server/public/idle.png +0 -0
  20. data/lib/resque/server/public/jquery-1.3.2.min.js +19 -0
  21. data/lib/resque/server/public/jquery.relatize_date.js +95 -0
  22. data/lib/resque/server/public/poll.png +0 -0
  23. data/lib/resque/server/public/ranger.js +73 -0
  24. data/lib/resque/server/public/reset.css +48 -0
  25. data/lib/resque/server/public/style.css +85 -0
  26. data/lib/resque/server/public/working.png +0 -0
  27. data/lib/resque/server/test_helper.rb +19 -0
  28. data/lib/resque/server/views/error.erb +1 -0
  29. data/lib/resque/server/views/failed.erb +64 -0
  30. data/lib/resque/server/views/key_sets.erb +19 -0
  31. data/lib/resque/server/views/key_string.erb +11 -0
  32. data/lib/resque/server/views/layout.erb +44 -0
  33. data/lib/resque/server/views/next_more.erb +10 -0
  34. data/lib/resque/server/views/overview.erb +4 -0
  35. data/lib/resque/server/views/queues.erb +49 -0
  36. data/lib/resque/server/views/stats.erb +62 -0
  37. data/lib/resque/server/views/workers.erb +109 -0
  38. data/lib/resque/server/views/working.erb +72 -0
  39. data/lib/resque/stat.rb +53 -0
  40. data/lib/resque/tasks.rb +51 -0
  41. data/lib/resque/version.rb +3 -0
  42. data/lib/resque/worker.rb +533 -0
  43. data/lib/tasks/redis.rake +161 -0
  44. data/lib/tasks/resque.rake +2 -0
  45. data/test/hoptoad_test.rb +25 -0
  46. data/test/job_hooks_test.rb +363 -0
  47. data/test/job_plugins_test.rb +230 -0
  48. data/test/plugin_test.rb +116 -0
  49. data/test/redis-test.conf +115 -0
  50. data/test/resque-web_test.rb +53 -0
  51. data/test/resque_test.rb +259 -0
  52. data/test/test_helper.rb +148 -0
  53. data/test/worker_test.rb +332 -0
  54. metadata +183 -0
@@ -0,0 +1,322 @@
1
+ ## 1.18.5 (2011-08-24)
2
+
3
+ * Added support for Travis CI
4
+ * Bugfix: preload only happens in production Rails environment
5
+
6
+ ## 1.18.4 (2011-08-23)
7
+
8
+ * Bugfix: preload task depends on setup
9
+
10
+ ## 1.18.3 (2011-08-23)
11
+
12
+ * Bugfix: Fix preloading on Rails 3.x.
13
+
14
+ ## 1.18.2 (2011-08-19)
15
+
16
+ * Fix RAILS_ROOT deprecation warning
17
+
18
+ ## 1.18.1 (2011-08-19)
19
+
20
+ * Bugfix: Use RAILS_ROOT in preload task
21
+
22
+ ## 1.18.0 (2011-08-18)
23
+
24
+ * Added before_enqueue hook.
25
+ * Resque workers now preload files under app/ in Rails
26
+ * Switch to MultiJSON
27
+ * Bugfix: Finding worker pids on Solaris
28
+ * Web UI: Fix NaN days ago for worker screens
29
+ * Web UI: Add Cache-Control header to prevent proxy caching
30
+ * Web UI: Update Resque.redis_id so it can be used in a distributed ring.
31
+
32
+ ## 1.17.1 (2011-05-27)
33
+
34
+ * Reverted `exit` change. Back to `exit!`.
35
+
36
+ ## 1.17.0 (2011-05-26)
37
+
38
+ * Workers exit with `exit` instead of `exit!`. This means you
39
+ can now use `at_exit` hooks inside workers.
40
+ * More monit typo fixes.
41
+ * Fixed bug in Hoptoad backend.
42
+ * Web UI: Wrap preformatted arguments.
43
+
44
+ ## 1.16.1 (2011-05-17)
45
+
46
+ * Bugfix: Resque::Failure::Hoptoad.configure works again
47
+ * Bugfix: Loading rake tasks
48
+
49
+ ## 1.16.0 (2011-05-16)
50
+
51
+ * Optional Hoptoad backend extracted into hoptoad_notifier. Install the gem to use it.
52
+ * Added `Worker#paused?` method
53
+ * Bugfix: Properly reseed random number generator after forking.
54
+ * Bugfix: Resque.redis=(<a Redis::Namespace>)
55
+ * Bugfix: Monit example stdout/stderr redirection
56
+ * Bugfix: Removing single failure now works with multiple failure backends
57
+ * Web: 'Remove Queue' now requires confirmation
58
+ * Web: Favicon!
59
+ * Web Bugfix: Dates display in Safari
60
+ * Web Bugfix: Dates display timezone
61
+ * Web Bugfix: Race condition querying working workers
62
+ * Web Bugfix: Fix polling /workers/all in resque-web
63
+
64
+ ## 1.15.0 (2011-03-18)
65
+
66
+ * Fallback to Redis.connect. Makes ENV variables and whatnot work.
67
+ * Fixed Sinatra 1.2 compatibility
68
+
69
+ ## 1.14.0 (2011-03-17)
70
+
71
+ * Sleep interval can now be a float
72
+ * Added Resque.inline to allow in-process performing of jobs (for testing)
73
+ * Fixed tests for Ruby 1.9.2
74
+ * Added Resque.validate(klass) to validate a Job
75
+ * Decode errors are no longer ignored to help debugging
76
+ * Web: Sinatra 1.2 compatibility
77
+ * Fixed after_enqueue hook to actually run in `Resque.enqueue`
78
+ * Fixed very_verbose timestamps to use 24 hour time (AM/PM wasn't included)
79
+ * Fixed monit example
80
+ * Fixed Worker#pid
81
+
82
+ ## 1.13.0 (2011-02-07)
83
+
84
+ * Depend on redis-namespace >= 0.10
85
+ * README tweaks
86
+ * Use thread_safe option when setting redis url
87
+ * Bugfix: worker pruning
88
+
89
+ ## 1.12.0 (2011-02-03)
90
+
91
+ * Added pidfile writing from `rake resque:work`
92
+ * Added Worker#pid method
93
+ * Added configurable location for `rake install`
94
+ * Bugfix: Errors in failure backend are rescue'd
95
+ * Bugfix: Non-working workers no longer counted in "working" count
96
+ * Bugfix: Don't think resque-web is a worker
97
+
98
+ ## 1.11.0 (2010-08-23)
99
+
100
+ * Web UI: Group /workers page by hostnames
101
+
102
+ ## 1.10.0 (2010-08-23)
103
+
104
+ * Support redis:// string format in `Resque.redis=`
105
+ * Using new cross-platform JSON gem.
106
+ * Added `after_enqueue` plugin hook.
107
+ * Added `shutdown?` method which can be overridden.
108
+ * Added support for the "leftright" gem when running tests.
109
+ * Grammarfix: In the README
110
+
111
+ ## 1.9.10 (2010-08-06)
112
+
113
+ * Bugfix: before_fork should get passed the job
114
+
115
+ ## 1.9.9 (2010-07-26)
116
+
117
+ * Depend on redis-namespace 0.8.0
118
+ * Depend on json_pure instead of json (for JRuby compat)
119
+ * Bugfix: rails_env display in stats view
120
+
121
+ ## 1.9.8 (2010-07-20)
122
+
123
+ * Bugfix: Worker.all should never return nil
124
+ * monit example: Fixed Syntax Error and adding environment to the rake task
125
+ * redis rake task: Fixed typo in copy command
126
+
127
+ ## 1.9.7 (2010-07-09)
128
+
129
+ * Improved memory usage in Job.destroy
130
+ * redis-namespace 0.7.0 now required
131
+ * Bugfix: Reverted $0 changes
132
+ * Web Bugfix: Payload-less failures in the web ui work
133
+
134
+ ## 1.9.6 (2010-06-22)
135
+
136
+ * Bugfix: Rakefile logging works the same as all the other logging
137
+
138
+ ## 1.9.5 (2010-06-16)
139
+
140
+ * Web Bugfix: Display the configured namespace on the stats page
141
+ * Revert Bugfix: Make ps -o more cross platform friendly
142
+
143
+ ## 1.9.4 (2010-06-14)
144
+
145
+ * Bugfix: Multiple failure backend gets exception information when created
146
+
147
+ ## 1.9.3 (2010-06-14)
148
+
149
+ * Bugfix: Resque#queues always returns an array
150
+
151
+ ## 1.9.2 (2010-06-13)
152
+
153
+ * Bugfix: Worker.all returning nil fix
154
+ * Bugfix: Make ps -o more cross platform friendly
155
+
156
+ ## 1.9.1 (2010-06-04)
157
+
158
+ * Less strict JSON dependency
159
+ * Included HISTORY.md in gem
160
+
161
+ ## 1.9.0 (2010-06-04)
162
+
163
+ * Redis 2 support
164
+ * Depend on redis-namespace 0.5.0
165
+ * Added Resque::VERSION constant (alias of Resque::Version)
166
+ * Bugfix: Specify JSON dependency
167
+ * Bugfix: Hoptoad plugin now works on 1.9
168
+
169
+ ## 1.8.5 (2010-05-18)
170
+
171
+ * Bugfix: Be more liberal in which Redis clients we accept.
172
+
173
+ ## 1.8.4 (2010-05-18)
174
+
175
+ * Try to resolve redis-namespace dependency issue
176
+
177
+ ## 1.8.3 (2010-05-17)
178
+
179
+ * Depend on redis-rb ~> 1.0.7
180
+
181
+ ## 1.8.2 (2010-05-03)
182
+
183
+ * Bugfix: Include "tasks/" dir in RubyGem
184
+
185
+ ## 1.8.1 (2010-04-29)
186
+
187
+ * Bugfix: Multiple failure backend did not support requeue-ing failed jobs
188
+ * Bugfix: Fix /failed when error has no backtrace
189
+ * Bugfix: Add `Redis::DistRedis` as a valid client
190
+
191
+ ## 1.8.0 (2010-04-07)
192
+
193
+ * Jobs that never complete due to killed worker are now failed.
194
+ * Worker "working" state is now maintained by the parent, not the child.
195
+ * Stopped using deprecated redis.rb methods
196
+ * `Worker.working` race condition fixed
197
+ * `Worker#process` has been deprecated.
198
+ * Monit example fixed
199
+ * Redis::Client and Redis::Namespace can be passed to `Resque.redis=`
200
+
201
+ ## 1.7.1 (2010-04-02)
202
+
203
+ * Bugfix: Make job hook execution order consistent
204
+ * Bugfix: stdout buffering in child process
205
+
206
+ ## 1.7.0 (2010-03-31)
207
+
208
+ * Job hooks API. See docs/HOOKS.md.
209
+ * web: Hovering over dates shows a timestamp
210
+ * web: AJAXify retry action for failed jobs
211
+ * web bugfix: Fix pagination bug
212
+
213
+ ## 1.6.1 (2010-03-25)
214
+
215
+ * Bugfix: Workers may not be clearing their state correctly on
216
+ shutdown
217
+ * Added example monit config.
218
+ * Exception class is now recorded when an error is raised in a
219
+ worker.
220
+ * web: Unit tests
221
+ * web: Show namespace in header and footer
222
+ * web: Remove a queue
223
+ * web: Retry failed jobs
224
+
225
+ ## 1.6.0 (2010-03-09)
226
+
227
+ * Added `before_first_fork`, `before_fork`, and `after_fork` hooks.
228
+ * Hoptoad: Added server_environment config setting
229
+ * Hoptoad bugfix: Don't depend on RAILS_ROOT
230
+ * 1.8.6 compat fixes
231
+
232
+ ## 1.5.2 (2010-03-03)
233
+
234
+ * Bugfix: JSON check was crazy.
235
+
236
+ ## 1.5.1 (2010-03-03)
237
+
238
+ * `Job.destroy` and `Resque.dequeue` return the # of destroyed jobs.
239
+ * Hoptoad notifier improvements
240
+ * Specify the namespace with `resque-web` by passing `-N namespace`
241
+ * Bugfix: Don't crash when trying to parse invalid JSON.
242
+ * Bugfix: Non-standard namespace support
243
+ * Web: Red backgound for queue "failed" only shown if there are failed jobs.
244
+ * Web bugfix: Tabs highlight properly now
245
+ * Web bugfix: ZSET partial support in stats
246
+ * Web bugfix: Deleting failed jobs works again
247
+ * Web bugfix: Sets (or zsets, lists, etc) now paginate.
248
+
249
+ ## 1.5.0 (2010-02-17)
250
+
251
+ * Version now included in procline, e.g. `resque-1.5.0: Message`
252
+ * Web bugfix: Ignore idle works in the "working" page
253
+ * Added `Resque::Job.destroy(queue, klass, *args)`
254
+ * Added `Resque.dequeue(klass, *args)`
255
+
256
+ ## 1.4.0 (2010-02-11)
257
+
258
+ * Fallback when unable to bind QUIT and USR1 for Windows and JRuby.
259
+ * Fallback when no `Kernel.fork` is provided (for IronRuby).
260
+ * Web: Rounded corners in Firefox
261
+ * Cut down system calls in `Worker#prune_dead_workers`
262
+ * Enable switching DB in a Redis server from config
263
+ * Support USR2 and CONT to stop and start job processing.
264
+ * Web: Add example failing job
265
+ * Bugfix: `Worker#unregister_worker` shouldn't call `done_working`
266
+ * Bugfix: Example god config now restarts Resque properly.
267
+ * Multiple failure backends now permitted.
268
+ * Hoptoad failure backend updated to new API
269
+
270
+ ## 1.3.1 (2010-01-11)
271
+
272
+ * Vegas bugfix: Don't error without a config
273
+
274
+ ## 1.3.0 (2010-01-11)
275
+
276
+ * Use Vegas for resque-web
277
+ * Web Bugfix: Show proper date/time value for failed_at on Failures
278
+ * Web Bugfix: Make the / route more flexible
279
+ * Add Resque::Server.tabs array (so plugins can add their own tabs)
280
+ * Start using [Semantic Versioning](http://semver.org/)
281
+
282
+ ## 1.2.4 (2009-12-15)
283
+
284
+ * Web Bugfix: fix key links on stat page
285
+
286
+ ## 1.2.3 (2009-12-15)
287
+
288
+ * Bugfix: Fixed `rand` seeding in child processes.
289
+ * Bugfix: Better JSON encoding/decoding without Yajl.
290
+ * Bugfix: Avoid `ps` flag error on Linux
291
+ * Add `PREFIX` observance to `rake` install tasks.
292
+
293
+ ## 1.2.2 (2009-12-08)
294
+
295
+ * Bugfix: Job equality was not properly implemented.
296
+
297
+ ## 1.2.1 (2009-12-07)
298
+
299
+ * Added `rake resque:workers` task for starting multiple workers.
300
+ * 1.9.x compatibility
301
+ * Bugfix: Yajl decoder doesn't care about valid UTF-8
302
+ * config.ru loads RESQUECONFIG if the ENV variable is set.
303
+ * `resque-web` now sets RESQUECONFIG
304
+ * Job objects know if they are equal.
305
+ * Jobs can be re-queued using `Job#recreate`
306
+
307
+ ## 1.2.0 (2009-11-25)
308
+
309
+ * If USR1 is sent and no child is found, shutdown.
310
+ * Raise when a job class does not respond to `perform`.
311
+ * Added `Resque.remove_queue` for deleting a queue
312
+
313
+ ## 1.1.0 (2009-11-04)
314
+
315
+ * Bugfix: Broken ERB tag in failure UI
316
+ * Bugfix: Save the worker's ID, not the worker itself, in the failure module
317
+ * Redesigned the sinatra web interface
318
+ * Added option to clear failed jobs
319
+
320
+ ## 1.0.0 (2009-11-03)
321
+
322
+ * First release.
data/LICENSE ADDED
@@ -0,0 +1,20 @@
1
+ Copyright (c) Chris Wanstrath
2
+
3
+ Permission is hereby granted, free of charge, to any person obtaining
4
+ a copy of this software and associated documentation files (the
5
+ "Software"), to deal in the Software without restriction, including
6
+ without limitation the rights to use, copy, modify, merge, publish,
7
+ distribute, sublicense, and/or sell copies of the Software, and to
8
+ permit persons to whom the Software is furnished to do so, subject to
9
+ the following conditions:
10
+
11
+ The above copyright notice and this permission notice shall be
12
+ included in all copies or substantial portions of the Software.
13
+
14
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
15
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
16
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
17
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
18
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
19
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
20
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@@ -0,0 +1,881 @@
1
+ Resque
2
+ ======
3
+
4
+ Resque (pronounced like "rescue") is a Redis-backed library for creating
5
+ background jobs, placing those jobs on multiple queues, and processing
6
+ them later.
7
+
8
+ Background jobs can be any Ruby class or module that responds to
9
+ `perform`. Your existing classes can easily be converted to background
10
+ jobs or you can create new classes specifically to do work. Or, you
11
+ can do both.
12
+
13
+ Resque is heavily inspired by DelayedJob (which rocks) and comprises
14
+ three parts:
15
+
16
+ 1. A Ruby library for creating, querying, and processing jobs
17
+ 2. A Rake task for starting a worker which processes jobs
18
+ 3. A Sinatra app for monitoring queues, jobs, and workers.
19
+
20
+ Resque workers can be distributed between multiple machines,
21
+ support priorities, are resilient to memory bloat / "leaks," are
22
+ optimized for REE (but work on MRI and JRuby), tell you what they're
23
+ doing, and expect failure.
24
+
25
+ Resque queues are persistent; support constant time, atomic push and
26
+ pop (thanks to Redis); provide visibility into their contents; and
27
+ store jobs as simple JSON packages.
28
+
29
+ The Resque frontend tells you what workers are doing, what workers are
30
+ not doing, what queues you're using, what's in those queues, provides
31
+ general usage stats, and helps you track failures.
32
+
33
+
34
+ The Blog Post
35
+ -------------
36
+
37
+ For the backstory, philosophy, and history of Resque's beginnings,
38
+ please see [the blog post][0].
39
+
40
+
41
+ Overview
42
+ --------
43
+
44
+ Resque allows you to create jobs and place them on a queue, then,
45
+ later, pull those jobs off the queue and process them.
46
+
47
+ Resque jobs are Ruby classes (or modules) which respond to the
48
+ `perform` method. Here's an example:
49
+
50
+
51
+ ``` ruby
52
+ class Archive
53
+ @queue = :file_serve
54
+
55
+ def self.perform(repo_id, branch = 'master')
56
+ repo = Repository.find(repo_id)
57
+ repo.create_archive(branch)
58
+ end
59
+ end
60
+ ```
61
+
62
+ The `@queue` class instance variable determines which queue `Archive`
63
+ jobs will be placed in. Queues are arbitrary and created on the fly -
64
+ you can name them whatever you want and have as many as you want.
65
+
66
+ To place an `Archive` job on the `file_serve` queue, we might add this
67
+ to our application's pre-existing `Repository` class:
68
+
69
+ ``` ruby
70
+ class Repository
71
+ def async_create_archive(branch)
72
+ Resque.enqueue(Archive, self.id, branch)
73
+ end
74
+ end
75
+ ```
76
+
77
+ Now when we call `repo.async_create_archive('masterbrew')` in our
78
+ application, a job will be created and placed on the `file_serve`
79
+ queue.
80
+
81
+ Later, a worker will run something like this code to process the job:
82
+
83
+ ``` ruby
84
+ klass, args = Resque.reserve(:file_serve)
85
+ klass.perform(*args) if klass.respond_to? :perform
86
+ ```
87
+
88
+ Which translates to:
89
+
90
+ ``` ruby
91
+ Archive.perform(44, 'masterbrew')
92
+ ```
93
+
94
+ Let's start a worker to run `file_serve` jobs:
95
+
96
+ $ cd app_root
97
+ $ QUEUE=file_serve rake resque:work
98
+
99
+ This starts one Resque worker and tells it to work off the
100
+ `file_serve` queue. As soon as it's ready it'll try to run the
101
+ `Resque.reserve` code snippet above and process jobs until it can't
102
+ find any more, at which point it will sleep for a small period and
103
+ repeatedly poll the queue for more jobs.
104
+
105
+ Workers can be given multiple queues (a "queue list") and run on
106
+ multiple machines. In fact they can be run anywhere with network
107
+ access to the Redis server.
108
+
109
+
110
+ Jobs
111
+ ----
112
+
113
+ What should you run in the background? Anything that takes any time at
114
+ all. Slow INSERT statements, disk manipulating, data processing, etc.
115
+
116
+ At GitHub we use Resque to process the following types of jobs:
117
+
118
+ * Warming caches
119
+ * Counting disk usage
120
+ * Building tarballs
121
+ * Building Rubygems
122
+ * Firing off web hooks
123
+ * Creating events in the db and pre-caching them
124
+ * Building graphs
125
+ * Deleting users
126
+ * Updating our search index
127
+
128
+ As of writing we have about 35 different types of background jobs.
129
+
130
+ Keep in mind that you don't need a web app to use Resque - we just
131
+ mention "foreground" and "background" because they make conceptual
132
+ sense. You could easily be spidering sites and sticking data which
133
+ needs to be crunched later into a queue.
134
+
135
+
136
+ ### Persistence
137
+
138
+ Jobs are persisted to queues as JSON objects. Let's take our `Archive`
139
+ example from above. We'll run the following code to create a job:
140
+
141
+ ``` ruby
142
+ repo = Repository.find(44)
143
+ repo.async_create_archive('masterbrew')
144
+ ```
145
+
146
+ The following JSON will be stored in the `file_serve` queue:
147
+
148
+ ``` javascript
149
+ {
150
+ 'class': 'Archive',
151
+ 'args': [ 44, 'masterbrew' ]
152
+ }
153
+ ```
154
+
155
+ Because of this your jobs must only accept arguments that can be JSON encoded.
156
+
157
+ So instead of doing this:
158
+
159
+ ``` ruby
160
+ Resque.enqueue(Archive, self, branch)
161
+ ```
162
+
163
+ do this:
164
+
165
+ ``` ruby
166
+ Resque.enqueue(Archive, self.id, branch)
167
+ ```
168
+
169
+ This is why our above example (and all the examples in `examples/`)
170
+ uses object IDs instead of passing around the objects.
171
+
172
+ While this is less convenient than just sticking a marshaled object
173
+ in the database, it gives you a slight advantage: your jobs will be
174
+ run against the most recent version of an object because they need to
175
+ pull from the DB or cache.
176
+
177
+ If your jobs were run against marshaled objects, they could
178
+ potentially be operating on a stale record with out-of-date information.
179
+
180
+
181
+ ### send_later / async
182
+
183
+ Want something like DelayedJob's `send_later` or the ability to use
184
+ instance methods instead of just methods for jobs? See the `examples/`
185
+ directory for goodies.
186
+
187
+ We plan to provide first class `async` support in a future release.
188
+
189
+
190
+ ### Failure
191
+
192
+ If a job raises an exception, it is logged and handed off to the
193
+ `Resque::Failure` module. Failures are logged either locally in Redis
194
+ or using some different backend.
195
+
196
+ For example, Resque ships with Hoptoad support.
197
+
198
+ Keep this in mind when writing your jobs: you may want to throw
199
+ exceptions you would not normally throw in order to assist debugging.
200
+
201
+
202
+ Workers
203
+ -------
204
+
205
+ Resque workers are rake tasks that run forever. They basically do this:
206
+
207
+ ``` ruby
208
+ start
209
+ loop do
210
+ if job = reserve
211
+ job.process
212
+ else
213
+ sleep 5
214
+ end
215
+ end
216
+ shutdown
217
+ ```
218
+
219
+ Starting a worker is simple. Here's our example from earlier:
220
+
221
+ $ QUEUE=file_serve rake resque:work
222
+
223
+ By default Resque won't know about your application's
224
+ environment. That is, it won't be able to find and run your jobs - it
225
+ needs to load your application into memory.
226
+
227
+ If we've installed Resque as a Rails plugin, we might run this command
228
+ from our RAILS_ROOT:
229
+
230
+ $ QUEUE=file_serve rake environment resque:work
231
+
232
+ This will load the environment before starting a worker. Alternately
233
+ we can define a `resque:setup` task with a dependency on the
234
+ `environment` rake task:
235
+
236
+ ``` ruby
237
+ task "resque:setup" => :environment
238
+ ```
239
+
240
+ GitHub's setup task looks like this:
241
+
242
+ ``` ruby
243
+ task "resque:setup" => :environment do
244
+ Grit::Git.git_timeout = 10.minutes
245
+ end
246
+ ```
247
+
248
+ We don't want the `git_timeout` as high as 10 minutes in our web app,
249
+ but in the Resque workers it's fine.
250
+
251
+
252
+ ### Logging
253
+
254
+ Workers support basic logging to STDOUT. If you start them with the
255
+ `VERBOSE` env variable set, they will print basic debugging
256
+ information. You can also set the `VVERBOSE` (very verbose) env
257
+ variable.
258
+
259
+ $ VVERBOSE=1 QUEUE=file_serve rake environment resque:work
260
+
261
+ ### Process IDs (PIDs)
262
+
263
+ There are scenarios where it's helpful to record the PID of a resque
264
+ worker process. Use the PIDFILE option for easy access to the PID:
265
+
266
+ $ PIDFILE=./resque.pid QUEUE=file_serve rake environment resque:work
267
+
268
+
269
+ ### Priorities and Queue Lists
270
+
271
+ Resque doesn't support numeric priorities but instead uses the order
272
+ of queues you give it. We call this list of queues the "queue list."
273
+
274
+ Let's say we add a `warm_cache` queue in addition to our `file_serve`
275
+ queue. We'd now start a worker like so:
276
+
277
+ $ QUEUES=file_serve,warm_cache rake resque:work
278
+
279
+ When the worker looks for new jobs, it will first check
280
+ `file_serve`. If it finds a job, it'll process it then check
281
+ `file_serve` again. It will keep checking `file_serve` until no more
282
+ jobs are available. At that point, it will check `warm_cache`. If it
283
+ finds a job it'll process it then check `file_serve` (repeating the
284
+ whole process).
285
+
286
+ In this way you can prioritize certain queues. At GitHub we start our
287
+ workers with something like this:
288
+
289
+ $ QUEUES=critical,archive,high,low rake resque:work
290
+
291
+ Notice the `archive` queue - it is specialized and in our future
292
+ architecture will only be run from a single machine.
293
+
294
+ At that point we'll start workers on our generalized background
295
+ machines with this command:
296
+
297
+ $ QUEUES=critical,high,low rake resque:work
298
+
299
+ And workers on our specialized archive machine with this command:
300
+
301
+ $ QUEUE=archive rake resque:work
302
+
303
+
304
+ ### Running All Queues
305
+
306
+ If you want your workers to work off of every queue, including new
307
+ queues created on the fly, you can use a splat:
308
+
309
+ $ QUEUE=* rake resque:work
310
+
311
+ Queues will be processed in alphabetical order.
312
+
313
+
314
+ ### Running Multiple Workers
315
+
316
+ At GitHub we use god to start and stop multiple workers. A sample god
317
+ configuration file is included under `examples/god`. We recommend this
318
+ method.
319
+
320
+ If you'd like to run multiple workers in development mode, you can do
321
+ so using the `resque:workers` rake task:
322
+
323
+ $ COUNT=5 QUEUE=* rake resque:workers
324
+
325
+ This will spawn five Resque workers, each in its own thread. Hitting
326
+ ctrl-c should be sufficient to stop them all.
327
+
328
+
329
+ ### Forking
330
+
331
+ On certain platforms, when a Resque worker reserves a job it
332
+ immediately forks a child process. The child processes the job then
333
+ exits. When the child has exited successfully, the worker reserves
334
+ another job and repeats the process.
335
+
336
+ Why?
337
+
338
+ Because Resque assumes chaos.
339
+
340
+ Resque assumes your background workers will lock up, run too long, or
341
+ have unwanted memory growth.
342
+
343
+ If Resque workers processed jobs themselves, it'd be hard to whip them
344
+ into shape. Let's say one is using too much memory: you send it a
345
+ signal that says "shutdown after you finish processing the current
346
+ job," and it does so. It then starts up again - loading your entire
347
+ application environment. This adds useless CPU cycles and causes a
348
+ delay in queue processing.
349
+
350
+ Plus, what if it's using too much memory and has stopped responding to
351
+ signals?
352
+
353
+ Thanks to Resque's parent / child architecture, jobs that use too much memory
354
+ release that memory upon completion. No unwanted growth.
355
+
356
+ And what if a job is running too long? You'd need to `kill -9` it then
357
+ start the worker again. With Resque's parent / child architecture you
358
+ can tell the parent to forcefully kill the child then immediately
359
+ start processing more jobs. No startup delay or wasted cycles.
360
+
361
+ The parent / child architecture helps us keep tabs on what workers are
362
+ doing, too. By eliminating the need to `kill -9` workers we can have
363
+ parents remove themselves from the global listing of workers. If we
364
+ just ruthlessly killed workers, we'd need a separate watchdog process
365
+ to add and remove them to the global listing - which becomes
366
+ complicated.
367
+
368
+ Workers instead handle their own state.
369
+
370
+
371
+ ### Parents and Children
372
+
373
+ Here's a parent / child pair doing some work:
374
+
375
+ $ ps -e -o pid,command | grep [r]esque
376
+ 92099 resque: Forked 92102 at 1253142769
377
+ 92102 resque: Processing file_serve since 1253142769
378
+
379
+ You can clearly see that process 92099 forked 92102, which has been
380
+ working since 1253142769.
381
+
382
+ (By advertising the time they began processing you can easily use monit
383
+ or god to kill stale workers.)
384
+
385
+ When a parent process is idle, it lets you know what queues it is
386
+ waiting for work on:
387
+
388
+ $ ps -e -o pid,command | grep [r]esque
389
+ 92099 resque: Waiting for file_serve,warm_cache
390
+
391
+
392
+ ### Signals
393
+
394
+ Resque workers respond to a few different signals:
395
+
396
+ * `QUIT` - Wait for child to finish processing then exit
397
+ * `TERM` / `INT` - Immediately kill child then exit
398
+ * `USR1` - Immediately kill child but don't exit
399
+ * `USR2` - Don't start to process any new jobs
400
+ * `CONT` - Start to process new jobs again after a USR2
401
+
402
+ If you want to gracefully shutdown a Resque worker, use `QUIT`.
403
+
404
+ If you want to kill a stale or stuck child, use `USR1`. Processing
405
+ will continue as normal unless the child was not found. In that case
406
+ Resque assumes the parent process is in a bad state and shuts down.
407
+
408
+ If you want to kill a stale or stuck child and shutdown, use `TERM`
409
+
410
+ If you want to stop processing jobs, but want to leave the worker running
411
+ (for example, to temporarily alleviate load), use `USR2` to stop processing,
412
+ then `CONT` to start it again.
413
+
414
+ ### Mysql::Error: MySQL server has gone away
415
+
416
+ If your workers remain idle for too long they may lose their MySQL
417
+ connection. If that happens we recommend using [this
418
+ Gist](http://gist.github.com/238999).
419
+
420
+
421
+ The Front End
422
+ -------------
423
+
424
+ Resque comes with a Sinatra-based front end for seeing what's up with
425
+ your queue.
426
+
427
+ ![The Front End](https://img.skitch.com/20110528-pc67a8qsfapgjxf5gagxd92fcu.png)
428
+
429
+ ### Standalone
430
+
431
+ If you've installed Resque as a gem running the front end standalone is easy:
432
+
433
+ $ resque-web
434
+
435
+ It's a thin layer around `rackup` so it's configurable as well:
436
+
437
+ $ resque-web -p 8282
438
+
439
+ If you have a Resque config file you want evaluated just pass it to
440
+ the script as the final argument:
441
+
442
+ $ resque-web -p 8282 rails_root/config/initializers/resque.rb
443
+
444
+ You can also set the namespace directly using `resque-web`:
445
+
446
+ $ resque-web -p 8282 -N myapp
447
+
448
+ ### Passenger
449
+
450
+ Using Passenger? Resque ships with a `config.ru` you can use. See
451
+ Phusion's guide:
452
+
453
+ Apache: <http://www.modrails.com/documentation/Users%20guide%20Apache.html#_deploying_a_rack_based_ruby_application>
454
+ Nginx: <http://www.modrails.com/documentation/Users%20guide%20Nginx.html#deploying_a_rack_app>
455
+
456
+ ### Rack::URLMap
457
+
458
+ If you want to load Resque on a subpath, possibly alongside other
459
+ apps, it's easy to do with Rack's `URLMap`:
460
+
461
+ ``` ruby
462
+ require 'resque/server'
463
+
464
+ run Rack::URLMap.new \
465
+ "/" => Your::App.new,
466
+ "/resque" => Resque::Server.new
467
+ ```
468
+
469
+ Check `examples/demo/config.ru` for a functional example (including
470
+ HTTP basic auth).
471
+
472
+
473
+ Resque vs DelayedJob
474
+ --------------------
475
+
476
+ How does Resque compare to DelayedJob, and why would you choose one
477
+ over the other?
478
+
479
+ * Resque supports multiple queues
480
+ * DelayedJob supports finer grained priorities
481
+ * Resque workers are resilient to memory leaks / bloat
482
+ * DelayedJob workers are extremely simple and easy to modify
483
+ * Resque requires Redis
484
+ * DelayedJob requires ActiveRecord
485
+ * Resque can only place JSONable Ruby objects on a queue as arguments
486
+ * DelayedJob can place _any_ Ruby object on its queue as arguments
487
+ * Resque includes a Sinatra app for monitoring what's going on
488
+ * DelayedJob can be queried from within your Rails app if you want to
489
+ add an interface
490
+
491
+ If you're doing Rails development, you already have a database and
492
+ ActiveRecord. DelayedJob is super easy to setup and works great.
493
+ GitHub used it for many months to process almost 200 million jobs.
494
+
495
+ Choose Resque if:
496
+
497
+ * You need multiple queues
498
+ * You don't care / dislike numeric priorities
499
+ * You don't need to persist every Ruby object ever
500
+ * You have potentially huge queues
501
+ * You want to see what's going on
502
+ * You expect a lot of failure / chaos
503
+ * You can setup Redis
504
+ * You're not running short on RAM
505
+
506
+ Choose DelayedJob if:
507
+
508
+ * You like numeric priorities
509
+ * You're not doing a gigantic amount of jobs each day
510
+ * Your queue stays small and nimble
511
+ * There is not a lot failure / chaos
512
+ * You want to easily throw anything on the queue
513
+ * You don't want to setup Redis
514
+
515
+ In no way is Resque a "better" DelayedJob, so make sure you pick the
516
+ tool that's best for your app.
517
+
518
+
519
+ Installing Redis
520
+ ----------------
521
+
522
+ Resque requires Redis 0.900 or higher.
523
+
524
+ Resque uses Redis' lists for its queues. It also stores worker state
525
+ data in Redis.
526
+
527
+ #### Homebrew
528
+
529
+ If you're on OS X, Homebrew is the simplest way to install Redis:
530
+
531
+ $ brew install redis
532
+ $ redis-server /usr/local/etc/redis.conf
533
+
534
+ You now have a Redis daemon running on 6379.
535
+
536
+ #### Via Resque
537
+
538
+ Resque includes Rake tasks (thanks to Ezra's redis-rb) that will
539
+ install and run Redis for you:
540
+
541
+ $ git clone git://github.com/defunkt/resque.git
542
+ $ cd resque
543
+ $ rake redis:install dtach:install
544
+ $ rake redis:start
545
+
546
+ Or, if you don't have admin access on your machine:
547
+
548
+ $ git clone git://github.com/defunkt/resque.git
549
+ $ cd resque
550
+ $ PREFIX=<your_prefix> rake redis:install dtach:install
551
+ $ rake redis:start
552
+
553
+ You now have Redis running on 6379. Wait a second then hit ctrl-\ to
554
+ detach and keep it running in the background.
555
+
556
+ The demo is probably the best way to figure out how to put the parts
557
+ together. But, it's not that hard.
558
+
559
+
560
+ Resque Dependencies
561
+ -------------------
562
+
563
+ $ gem install bundler
564
+ $ bundle install
565
+
566
+
567
+ Installing Resque
568
+ -----------------
569
+
570
+ ### In a Rack app, as a gem
571
+
572
+ First install the gem.
573
+
574
+ $ gem install resque
575
+
576
+ Next include it in your application.
577
+
578
+ ``` ruby
579
+ require 'resque'
580
+ ```
581
+
582
+ Now start your application:
583
+
584
+ rackup config.ru
585
+
586
+ That's it! You can now create Resque jobs from within your app.
587
+
588
+ To start a worker, create a Rakefile in your app's root (or add this
589
+ to an existing Rakefile):
590
+
591
+ ``` ruby
592
+ require 'your/app'
593
+ require 'resque/tasks'
594
+ ```
595
+
596
+ Now:
597
+
598
+ $ QUEUE=* rake resque:work
599
+
600
+ Alternately you can define a `resque:setup` hook in your Rakefile if you
601
+ don't want to load your app every time rake runs.
602
+
603
+
604
+ ### In a Rails 2.x app, as a gem
605
+
606
+ First install the gem.
607
+
608
+ $ gem install resque
609
+
610
+ Next include it in your application.
611
+
612
+ $ cat config/initializers/load_resque.rb
613
+ require 'resque'
614
+
615
+ Now start your application:
616
+
617
+ $ ./script/server
618
+
619
+ That's it! You can now create Resque jobs from within your app.
620
+
621
+ To start a worker, add this to your Rakefile in `RAILS_ROOT`:
622
+
623
+ ``` ruby
624
+ require 'resque/tasks'
625
+ ```
626
+
627
+ Now:
628
+
629
+ $ QUEUE=* rake environment resque:work
630
+
631
+ Don't forget you can define a `resque:setup` hook in
632
+ `lib/tasks/whatever.rake` that loads the `environment` task every time.
633
+
634
+
635
+ ### In a Rails 2.x app, as a plugin
636
+
637
+ $ ./script/plugin install git://github.com/defunkt/resque
638
+
639
+ That's it! Resque will automatically be available when your Rails app
640
+ loads.
641
+
642
+ To start a worker:
643
+
644
+ $ QUEUE=* rake environment resque:work
645
+
646
+ Don't forget you can define a `resque:setup` hook in
647
+ `lib/tasks/whatever.rake` that loads the `environment` task every time.
648
+
649
+
650
+ ### In a Rails 3 app, as a gem
651
+
652
+ First include it in your Gemfile.
653
+
654
+ $ cat Gemfile
655
+ ...
656
+ gem 'resque'
657
+ ...
658
+
659
+ Next install it with Bundler.
660
+
661
+ $ bundle install
662
+
663
+ Now start your application:
664
+
665
+ $ rails server
666
+
667
+ That's it! You can now create Resque jobs from within your app.
668
+
669
+ To start a worker, add this to a file in `lib/tasks` (ex:
670
+ `lib/tasks/resque.rake`):
671
+
672
+ ``` ruby
673
+ require 'resque/tasks'
674
+ ```
675
+
676
+ Now:
677
+
678
+ $ QUEUE=* rake environment resque:work
679
+
680
+ Don't forget you can define a `resque:setup` hook in
681
+ `lib/tasks/whatever.rake` that loads the `environment` task every time.
682
+
683
+
684
+ Configuration
685
+ -------------
686
+
687
+ You may want to change the Redis host and port Resque connects to, or
688
+ set various other options at startup.
689
+
690
+ Resque has a `redis` setter which can be given a string or a Redis
691
+ object. This means if you're already using Redis in your app, Resque
692
+ can re-use the existing connection.
693
+
694
+ String: `Resque.redis = 'localhost:6379'`
695
+
696
+ Redis: `Resque.redis = $redis`
697
+
698
+ For our rails app we have a `config/initializers/resque.rb` file where
699
+ we load `config/resque.yml` by hand and set the Redis information
700
+ appropriately.
701
+
702
+ Here's our `config/resque.yml`:
703
+
704
+ development: localhost:6379
705
+ test: localhost:6379
706
+ staging: redis1.se.github.com:6379
707
+ fi: localhost:6379
708
+ production: redis1.ae.github.com:6379
709
+
710
+ And our initializer:
711
+
712
+ ``` ruby
713
+ rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..'
714
+ rails_env = ENV['RAILS_ENV'] || 'development'
715
+
716
+ resque_config = YAML.load_file(rails_root + '/config/resque.yml')
717
+ Resque.redis = resque_config[rails_env]
718
+ ```
719
+
720
+ Easy peasy! Why not just use `RAILS_ROOT` and `RAILS_ENV`? Because
721
+ this way we can tell our Sinatra app about the config file:
722
+
723
+ $ RAILS_ENV=production resque-web rails_root/config/initializers/resque.rb
724
+
725
+ Now everyone is on the same page.
726
+
727
+ Also, you could disable jobs queueing by setting 'inline' attribute.
728
+ For example, if you want to run all jobs in the same process for cucumber, try:
729
+
730
+ ``` ruby
731
+ Resque.inline = ENV['RAILS_ENV'] == "cucumber"
732
+ ```
733
+
734
+
735
+ Plugins and Hooks
736
+ -----------------
737
+
738
+ For a list of available plugins see
739
+ <http://wiki.github.com/defunkt/resque/plugins>.
740
+
741
+ If you'd like to write your own plugin, or want to customize Resque
742
+ using hooks (such as `Resque.after_fork`), see
743
+ [docs/HOOKS.md](http://github.com/defunkt/resque/blob/master/docs/HOOKS.md).
744
+
745
+
746
+ Namespaces
747
+ ----------
748
+
749
+ If you're running multiple, separate instances of Resque you may want
750
+ to namespace the keyspaces so they do not overlap. This is not unlike
751
+ the approach taken by many memcached clients.
752
+
753
+ This feature is provided by the [redis-namespace][rs] library, which
754
+ Resque uses by default to separate the keys it manages from other keys
755
+ in your Redis server.
756
+
757
+ Simply use the `Resque.redis.namespace` accessor:
758
+
759
+ ``` ruby
760
+ Resque.redis.namespace = "resque:GitHub"
761
+ ```
762
+
763
+ We recommend sticking this in your initializer somewhere after Redis
764
+ is configured.
765
+
766
+
767
+ Demo
768
+ ----
769
+
770
+ Resque ships with a demo Sinatra app for creating jobs that are later
771
+ processed in the background.
772
+
773
+ Try it out by looking at the README, found at `examples/demo/README.markdown`.
774
+
775
+
776
+ Monitoring
777
+ ----------
778
+
779
+ ### god
780
+
781
+ If you're using god to monitor Resque, we have provided example
782
+ configs in `examples/god/`. One is for starting / stopping workers,
783
+ the other is for killing workers that have been running too long.
784
+
785
+ ### monit
786
+
787
+ If you're using monit, `examples/monit/resque.monit` is provided free
788
+ of charge. This is **not** used by GitHub in production, so please
789
+ send patches for any tweaks or improvements you can make to it.
790
+
791
+
792
+ Questions
793
+ ---------
794
+
795
+ Please add them to the [FAQ](https://github.com/defunkt/resque/wiki/FAQ) or
796
+ ask on the Mailing List. The Mailing List is explained further below
797
+
798
+
799
+ Development
800
+ -----------
801
+
802
+ Want to hack on Resque?
803
+
804
+ First clone the repo and run the tests:
805
+
806
+ git clone git://github.com/defunkt/resque.git
807
+ cd resque
808
+ rake test
809
+
810
+ If the tests do not pass make sure you have Redis installed
811
+ correctly (though we make an effort to tell you if we feel this is the
812
+ case). The tests attempt to start an isolated instance of Redis to
813
+ run against.
814
+
815
+ Also make sure you've installed all the dependencies correctly. For
816
+ example, try loading the `redis-namespace` gem after you've installed
817
+ it:
818
+
819
+ $ irb
820
+ >> require 'rubygems'
821
+ => true
822
+ >> require 'redis/namespace'
823
+ => true
824
+
825
+ If you get an error requiring any of the dependencies, you may have
826
+ failed to install them or be seeing load path issues.
827
+
828
+ Feel free to ping the mailing list with your problem and we'll try to
829
+ sort it out.
830
+
831
+
832
+ Contributing
833
+ ------------
834
+
835
+ Once you've made your great commits:
836
+
837
+ 1. [Fork][1] Resque
838
+ 2. Create a topic branch - `git checkout -b my_branch`
839
+ 3. Push to your branch - `git push origin my_branch`
840
+ 4. Create a [Pull Request](http://help.github.com/pull-requests/) from your branch
841
+ 5. That's it!
842
+
843
+ You might want to checkout our [Contributing][cb] wiki page for information
844
+ on coding standards, new features, etc.
845
+
846
+
847
+ Mailing List
848
+ ------------
849
+
850
+ To join the list simply send an email to <resque@librelist.com>. This
851
+ will subscribe you and send you information about your subscription,
852
+ including unsubscribe information.
853
+
854
+ The archive can be found at <http://librelist.com/browser/resque/>.
855
+
856
+
857
+ Meta
858
+ ----
859
+
860
+ * Code: `git clone git://github.com/defunkt/resque.git`
861
+ * Home: <http://github.com/defunkt/resque>
862
+ * Docs: <http://defunkt.github.com/resque/>
863
+ * Bugs: <http://github.com/defunkt/resque/issues>
864
+ * List: <resque@librelist.com>
865
+ * Chat: <irc://irc.freenode.net/resque>
866
+ * Gems: <http://gemcutter.org/gems/resque>
867
+
868
+ This project uses [Semantic Versioning][sv].
869
+
870
+
871
+ Author
872
+ ------
873
+
874
+ Chris Wanstrath :: chris@ozmm.org :: @defunkt
875
+
876
+ [0]: http://github.com/blog/542-introducing-resque
877
+ [1]: http://help.github.com/forking/
878
+ [2]: http://github.com/defunkt/resque/issues
879
+ [sv]: http://semver.org/
880
+ [rs]: http://github.com/defunkt/redis-namespace
881
+ [cb]: http://wiki.github.com/defunkt/resque/contributing