steini-resque 1.18.5

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (54) hide show
  1. data/HISTORY.md +322 -0
  2. data/LICENSE +20 -0
  3. data/README.markdown +881 -0
  4. data/Rakefile +78 -0
  5. data/bin/resque +81 -0
  6. data/bin/resque-web +23 -0
  7. data/lib/resque.rb +352 -0
  8. data/lib/resque/errors.rb +10 -0
  9. data/lib/resque/failure.rb +70 -0
  10. data/lib/resque/failure/base.rb +64 -0
  11. data/lib/resque/failure/hoptoad.rb +48 -0
  12. data/lib/resque/failure/multiple.rb +54 -0
  13. data/lib/resque/failure/redis.rb +51 -0
  14. data/lib/resque/helpers.rb +63 -0
  15. data/lib/resque/job.rb +205 -0
  16. data/lib/resque/plugin.rb +56 -0
  17. data/lib/resque/server.rb +231 -0
  18. data/lib/resque/server/public/favicon.ico +0 -0
  19. data/lib/resque/server/public/idle.png +0 -0
  20. data/lib/resque/server/public/jquery-1.3.2.min.js +19 -0
  21. data/lib/resque/server/public/jquery.relatize_date.js +95 -0
  22. data/lib/resque/server/public/poll.png +0 -0
  23. data/lib/resque/server/public/ranger.js +73 -0
  24. data/lib/resque/server/public/reset.css +48 -0
  25. data/lib/resque/server/public/style.css +85 -0
  26. data/lib/resque/server/public/working.png +0 -0
  27. data/lib/resque/server/test_helper.rb +19 -0
  28. data/lib/resque/server/views/error.erb +1 -0
  29. data/lib/resque/server/views/failed.erb +64 -0
  30. data/lib/resque/server/views/key_sets.erb +19 -0
  31. data/lib/resque/server/views/key_string.erb +11 -0
  32. data/lib/resque/server/views/layout.erb +44 -0
  33. data/lib/resque/server/views/next_more.erb +10 -0
  34. data/lib/resque/server/views/overview.erb +4 -0
  35. data/lib/resque/server/views/queues.erb +49 -0
  36. data/lib/resque/server/views/stats.erb +62 -0
  37. data/lib/resque/server/views/workers.erb +109 -0
  38. data/lib/resque/server/views/working.erb +72 -0
  39. data/lib/resque/stat.rb +53 -0
  40. data/lib/resque/tasks.rb +51 -0
  41. data/lib/resque/version.rb +3 -0
  42. data/lib/resque/worker.rb +533 -0
  43. data/lib/tasks/redis.rake +161 -0
  44. data/lib/tasks/resque.rake +2 -0
  45. data/test/hoptoad_test.rb +25 -0
  46. data/test/job_hooks_test.rb +363 -0
  47. data/test/job_plugins_test.rb +230 -0
  48. data/test/plugin_test.rb +116 -0
  49. data/test/redis-test.conf +115 -0
  50. data/test/resque-web_test.rb +53 -0
  51. data/test/resque_test.rb +259 -0
  52. data/test/test_helper.rb +148 -0
  53. data/test/worker_test.rb +332 -0
  54. metadata +183 -0
@@ -0,0 +1,322 @@
1
+ ## 1.18.5 (2011-08-24)
2
+
3
+ * Added support for Travis CI
4
+ * Bugfix: preload only happens in production Rails environment
5
+
6
+ ## 1.18.4 (2011-08-23)
7
+
8
+ * Bugfix: preload task depends on setup
9
+
10
+ ## 1.18.3 (2011-08-23)
11
+
12
+ * Bugfix: Fix preloading on Rails 3.x.
13
+
14
+ ## 1.18.2 (2011-08-19)
15
+
16
+ * Fix RAILS_ROOT deprecation warning
17
+
18
+ ## 1.18.1 (2011-08-19)
19
+
20
+ * Bugfix: Use RAILS_ROOT in preload task
21
+
22
+ ## 1.18.0 (2011-08-18)
23
+
24
+ * Added before_enqueue hook.
25
+ * Resque workers now preload files under app/ in Rails
26
+ * Switch to MultiJSON
27
+ * Bugfix: Finding worker pids on Solaris
28
+ * Web UI: Fix NaN days ago for worker screens
29
+ * Web UI: Add Cache-Control header to prevent proxy caching
30
+ * Web UI: Update Resque.redis_id so it can be used in a distributed ring.
31
+
32
+ ## 1.17.1 (2011-05-27)
33
+
34
+ * Reverted `exit` change. Back to `exit!`.
35
+
36
+ ## 1.17.0 (2011-05-26)
37
+
38
+ * Workers exit with `exit` instead of `exit!`. This means you
39
+ can now use `at_exit` hooks inside workers.
40
+ * More monit typo fixes.
41
+ * Fixed bug in Hoptoad backend.
42
+ * Web UI: Wrap preformatted arguments.
43
+
44
+ ## 1.16.1 (2011-05-17)
45
+
46
+ * Bugfix: Resque::Failure::Hoptoad.configure works again
47
+ * Bugfix: Loading rake tasks
48
+
49
+ ## 1.16.0 (2011-05-16)
50
+
51
+ * Optional Hoptoad backend extracted into hoptoad_notifier. Install the gem to use it.
52
+ * Added `Worker#paused?` method
53
+ * Bugfix: Properly reseed random number generator after forking.
54
+ * Bugfix: Resque.redis=(<a Redis::Namespace>)
55
+ * Bugfix: Monit example stdout/stderr redirection
56
+ * Bugfix: Removing single failure now works with multiple failure backends
57
+ * Web: 'Remove Queue' now requires confirmation
58
+ * Web: Favicon!
59
+ * Web Bugfix: Dates display in Safari
60
+ * Web Bugfix: Dates display timezone
61
+ * Web Bugfix: Race condition querying working workers
62
+ * Web Bugfix: Fix polling /workers/all in resque-web
63
+
64
+ ## 1.15.0 (2011-03-18)
65
+
66
+ * Fallback to Redis.connect. Makes ENV variables and whatnot work.
67
+ * Fixed Sinatra 1.2 compatibility
68
+
69
+ ## 1.14.0 (2011-03-17)
70
+
71
+ * Sleep interval can now be a float
72
+ * Added Resque.inline to allow in-process performing of jobs (for testing)
73
+ * Fixed tests for Ruby 1.9.2
74
+ * Added Resque.validate(klass) to validate a Job
75
+ * Decode errors are no longer ignored to help debugging
76
+ * Web: Sinatra 1.2 compatibility
77
+ * Fixed after_enqueue hook to actually run in `Resque.enqueue`
78
+ * Fixed very_verbose timestamps to use 24 hour time (AM/PM wasn't included)
79
+ * Fixed monit example
80
+ * Fixed Worker#pid
81
+
82
+ ## 1.13.0 (2011-02-07)
83
+
84
+ * Depend on redis-namespace >= 0.10
85
+ * README tweaks
86
+ * Use thread_safe option when setting redis url
87
+ * Bugfix: worker pruning
88
+
89
+ ## 1.12.0 (2011-02-03)
90
+
91
+ * Added pidfile writing from `rake resque:work`
92
+ * Added Worker#pid method
93
+ * Added configurable location for `rake install`
94
+ * Bugfix: Errors in failure backend are rescue'd
95
+ * Bugfix: Non-working workers no longer counted in "working" count
96
+ * Bugfix: Don't think resque-web is a worker
97
+
98
+ ## 1.11.0 (2010-08-23)
99
+
100
+ * Web UI: Group /workers page by hostnames
101
+
102
+ ## 1.10.0 (2010-08-23)
103
+
104
+ * Support redis:// string format in `Resque.redis=`
105
+ * Using new cross-platform JSON gem.
106
+ * Added `after_enqueue` plugin hook.
107
+ * Added `shutdown?` method which can be overridden.
108
+ * Added support for the "leftright" gem when running tests.
109
+ * Grammarfix: In the README
110
+
111
+ ## 1.9.10 (2010-08-06)
112
+
113
+ * Bugfix: before_fork should get passed the job
114
+
115
+ ## 1.9.9 (2010-07-26)
116
+
117
+ * Depend on redis-namespace 0.8.0
118
+ * Depend on json_pure instead of json (for JRuby compat)
119
+ * Bugfix: rails_env display in stats view
120
+
121
+ ## 1.9.8 (2010-07-20)
122
+
123
+ * Bugfix: Worker.all should never return nil
124
+ * monit example: Fixed Syntax Error and adding environment to the rake task
125
+ * redis rake task: Fixed typo in copy command
126
+
127
+ ## 1.9.7 (2010-07-09)
128
+
129
+ * Improved memory usage in Job.destroy
130
+ * redis-namespace 0.7.0 now required
131
+ * Bugfix: Reverted $0 changes
132
+ * Web Bugfix: Payload-less failures in the web ui work
133
+
134
+ ## 1.9.6 (2010-06-22)
135
+
136
+ * Bugfix: Rakefile logging works the same as all the other logging
137
+
138
+ ## 1.9.5 (2010-06-16)
139
+
140
+ * Web Bugfix: Display the configured namespace on the stats page
141
+ * Revert Bugfix: Make ps -o more cross platform friendly
142
+
143
+ ## 1.9.4 (2010-06-14)
144
+
145
+ * Bugfix: Multiple failure backend gets exception information when created
146
+
147
+ ## 1.9.3 (2010-06-14)
148
+
149
+ * Bugfix: Resque#queues always returns an array
150
+
151
+ ## 1.9.2 (2010-06-13)
152
+
153
+ * Bugfix: Worker.all returning nil fix
154
+ * Bugfix: Make ps -o more cross platform friendly
155
+
156
+ ## 1.9.1 (2010-06-04)
157
+
158
+ * Less strict JSON dependency
159
+ * Included HISTORY.md in gem
160
+
161
+ ## 1.9.0 (2010-06-04)
162
+
163
+ * Redis 2 support
164
+ * Depend on redis-namespace 0.5.0
165
+ * Added Resque::VERSION constant (alias of Resque::Version)
166
+ * Bugfix: Specify JSON dependency
167
+ * Bugfix: Hoptoad plugin now works on 1.9
168
+
169
+ ## 1.8.5 (2010-05-18)
170
+
171
+ * Bugfix: Be more liberal in which Redis clients we accept.
172
+
173
+ ## 1.8.4 (2010-05-18)
174
+
175
+ * Try to resolve redis-namespace dependency issue
176
+
177
+ ## 1.8.3 (2010-05-17)
178
+
179
+ * Depend on redis-rb ~> 1.0.7
180
+
181
+ ## 1.8.2 (2010-05-03)
182
+
183
+ * Bugfix: Include "tasks/" dir in RubyGem
184
+
185
+ ## 1.8.1 (2010-04-29)
186
+
187
+ * Bugfix: Multiple failure backend did not support requeue-ing failed jobs
188
+ * Bugfix: Fix /failed when error has no backtrace
189
+ * Bugfix: Add `Redis::DistRedis` as a valid client
190
+
191
+ ## 1.8.0 (2010-04-07)
192
+
193
+ * Jobs that never complete due to killed worker are now failed.
194
+ * Worker "working" state is now maintained by the parent, not the child.
195
+ * Stopped using deprecated redis.rb methods
196
+ * `Worker.working` race condition fixed
197
+ * `Worker#process` has been deprecated.
198
+ * Monit example fixed
199
+ * Redis::Client and Redis::Namespace can be passed to `Resque.redis=`
200
+
201
+ ## 1.7.1 (2010-04-02)
202
+
203
+ * Bugfix: Make job hook execution order consistent
204
+ * Bugfix: stdout buffering in child process
205
+
206
+ ## 1.7.0 (2010-03-31)
207
+
208
+ * Job hooks API. See docs/HOOKS.md.
209
+ * web: Hovering over dates shows a timestamp
210
+ * web: AJAXify retry action for failed jobs
211
+ * web bugfix: Fix pagination bug
212
+
213
+ ## 1.6.1 (2010-03-25)
214
+
215
+ * Bugfix: Workers may not be clearing their state correctly on
216
+ shutdown
217
+ * Added example monit config.
218
+ * Exception class is now recorded when an error is raised in a
219
+ worker.
220
+ * web: Unit tests
221
+ * web: Show namespace in header and footer
222
+ * web: Remove a queue
223
+ * web: Retry failed jobs
224
+
225
+ ## 1.6.0 (2010-03-09)
226
+
227
+ * Added `before_first_fork`, `before_fork`, and `after_fork` hooks.
228
+ * Hoptoad: Added server_environment config setting
229
+ * Hoptoad bugfix: Don't depend on RAILS_ROOT
230
+ * 1.8.6 compat fixes
231
+
232
+ ## 1.5.2 (2010-03-03)
233
+
234
+ * Bugfix: JSON check was crazy.
235
+
236
+ ## 1.5.1 (2010-03-03)
237
+
238
+ * `Job.destroy` and `Resque.dequeue` return the # of destroyed jobs.
239
+ * Hoptoad notifier improvements
240
+ * Specify the namespace with `resque-web` by passing `-N namespace`
241
+ * Bugfix: Don't crash when trying to parse invalid JSON.
242
+ * Bugfix: Non-standard namespace support
243
+ * Web: Red backgound for queue "failed" only shown if there are failed jobs.
244
+ * Web bugfix: Tabs highlight properly now
245
+ * Web bugfix: ZSET partial support in stats
246
+ * Web bugfix: Deleting failed jobs works again
247
+ * Web bugfix: Sets (or zsets, lists, etc) now paginate.
248
+
249
+ ## 1.5.0 (2010-02-17)
250
+
251
+ * Version now included in procline, e.g. `resque-1.5.0: Message`
252
+ * Web bugfix: Ignore idle works in the "working" page
253
+ * Added `Resque::Job.destroy(queue, klass, *args)`
254
+ * Added `Resque.dequeue(klass, *args)`
255
+
256
+ ## 1.4.0 (2010-02-11)
257
+
258
+ * Fallback when unable to bind QUIT and USR1 for Windows and JRuby.
259
+ * Fallback when no `Kernel.fork` is provided (for IronRuby).
260
+ * Web: Rounded corners in Firefox
261
+ * Cut down system calls in `Worker#prune_dead_workers`
262
+ * Enable switching DB in a Redis server from config
263
+ * Support USR2 and CONT to stop and start job processing.
264
+ * Web: Add example failing job
265
+ * Bugfix: `Worker#unregister_worker` shouldn't call `done_working`
266
+ * Bugfix: Example god config now restarts Resque properly.
267
+ * Multiple failure backends now permitted.
268
+ * Hoptoad failure backend updated to new API
269
+
270
+ ## 1.3.1 (2010-01-11)
271
+
272
+ * Vegas bugfix: Don't error without a config
273
+
274
+ ## 1.3.0 (2010-01-11)
275
+
276
+ * Use Vegas for resque-web
277
+ * Web Bugfix: Show proper date/time value for failed_at on Failures
278
+ * Web Bugfix: Make the / route more flexible
279
+ * Add Resque::Server.tabs array (so plugins can add their own tabs)
280
+ * Start using [Semantic Versioning](http://semver.org/)
281
+
282
+ ## 1.2.4 (2009-12-15)
283
+
284
+ * Web Bugfix: fix key links on stat page
285
+
286
+ ## 1.2.3 (2009-12-15)
287
+
288
+ * Bugfix: Fixed `rand` seeding in child processes.
289
+ * Bugfix: Better JSON encoding/decoding without Yajl.
290
+ * Bugfix: Avoid `ps` flag error on Linux
291
+ * Add `PREFIX` observance to `rake` install tasks.
292
+
293
+ ## 1.2.2 (2009-12-08)
294
+
295
+ * Bugfix: Job equality was not properly implemented.
296
+
297
+ ## 1.2.1 (2009-12-07)
298
+
299
+ * Added `rake resque:workers` task for starting multiple workers.
300
+ * 1.9.x compatibility
301
+ * Bugfix: Yajl decoder doesn't care about valid UTF-8
302
+ * config.ru loads RESQUECONFIG if the ENV variable is set.
303
+ * `resque-web` now sets RESQUECONFIG
304
+ * Job objects know if they are equal.
305
+ * Jobs can be re-queued using `Job#recreate`
306
+
307
+ ## 1.2.0 (2009-11-25)
308
+
309
+ * If USR1 is sent and no child is found, shutdown.
310
+ * Raise when a job class does not respond to `perform`.
311
+ * Added `Resque.remove_queue` for deleting a queue
312
+
313
+ ## 1.1.0 (2009-11-04)
314
+
315
+ * Bugfix: Broken ERB tag in failure UI
316
+ * Bugfix: Save the worker's ID, not the worker itself, in the failure module
317
+ * Redesigned the sinatra web interface
318
+ * Added option to clear failed jobs
319
+
320
+ ## 1.0.0 (2009-11-03)
321
+
322
+ * First release.
data/LICENSE ADDED
@@ -0,0 +1,20 @@
1
+ Copyright (c) Chris Wanstrath
2
+
3
+ Permission is hereby granted, free of charge, to any person obtaining
4
+ a copy of this software and associated documentation files (the
5
+ "Software"), to deal in the Software without restriction, including
6
+ without limitation the rights to use, copy, modify, merge, publish,
7
+ distribute, sublicense, and/or sell copies of the Software, and to
8
+ permit persons to whom the Software is furnished to do so, subject to
9
+ the following conditions:
10
+
11
+ The above copyright notice and this permission notice shall be
12
+ included in all copies or substantial portions of the Software.
13
+
14
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
15
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
16
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
17
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
18
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
19
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
20
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@@ -0,0 +1,881 @@
1
+ Resque
2
+ ======
3
+
4
+ Resque (pronounced like "rescue") is a Redis-backed library for creating
5
+ background jobs, placing those jobs on multiple queues, and processing
6
+ them later.
7
+
8
+ Background jobs can be any Ruby class or module that responds to
9
+ `perform`. Your existing classes can easily be converted to background
10
+ jobs or you can create new classes specifically to do work. Or, you
11
+ can do both.
12
+
13
+ Resque is heavily inspired by DelayedJob (which rocks) and comprises
14
+ three parts:
15
+
16
+ 1. A Ruby library for creating, querying, and processing jobs
17
+ 2. A Rake task for starting a worker which processes jobs
18
+ 3. A Sinatra app for monitoring queues, jobs, and workers.
19
+
20
+ Resque workers can be distributed between multiple machines,
21
+ support priorities, are resilient to memory bloat / "leaks," are
22
+ optimized for REE (but work on MRI and JRuby), tell you what they're
23
+ doing, and expect failure.
24
+
25
+ Resque queues are persistent; support constant time, atomic push and
26
+ pop (thanks to Redis); provide visibility into their contents; and
27
+ store jobs as simple JSON packages.
28
+
29
+ The Resque frontend tells you what workers are doing, what workers are
30
+ not doing, what queues you're using, what's in those queues, provides
31
+ general usage stats, and helps you track failures.
32
+
33
+
34
+ The Blog Post
35
+ -------------
36
+
37
+ For the backstory, philosophy, and history of Resque's beginnings,
38
+ please see [the blog post][0].
39
+
40
+
41
+ Overview
42
+ --------
43
+
44
+ Resque allows you to create jobs and place them on a queue, then,
45
+ later, pull those jobs off the queue and process them.
46
+
47
+ Resque jobs are Ruby classes (or modules) which respond to the
48
+ `perform` method. Here's an example:
49
+
50
+
51
+ ``` ruby
52
+ class Archive
53
+ @queue = :file_serve
54
+
55
+ def self.perform(repo_id, branch = 'master')
56
+ repo = Repository.find(repo_id)
57
+ repo.create_archive(branch)
58
+ end
59
+ end
60
+ ```
61
+
62
+ The `@queue` class instance variable determines which queue `Archive`
63
+ jobs will be placed in. Queues are arbitrary and created on the fly -
64
+ you can name them whatever you want and have as many as you want.
65
+
66
+ To place an `Archive` job on the `file_serve` queue, we might add this
67
+ to our application's pre-existing `Repository` class:
68
+
69
+ ``` ruby
70
+ class Repository
71
+ def async_create_archive(branch)
72
+ Resque.enqueue(Archive, self.id, branch)
73
+ end
74
+ end
75
+ ```
76
+
77
+ Now when we call `repo.async_create_archive('masterbrew')` in our
78
+ application, a job will be created and placed on the `file_serve`
79
+ queue.
80
+
81
+ Later, a worker will run something like this code to process the job:
82
+
83
+ ``` ruby
84
+ klass, args = Resque.reserve(:file_serve)
85
+ klass.perform(*args) if klass.respond_to? :perform
86
+ ```
87
+
88
+ Which translates to:
89
+
90
+ ``` ruby
91
+ Archive.perform(44, 'masterbrew')
92
+ ```
93
+
94
+ Let's start a worker to run `file_serve` jobs:
95
+
96
+ $ cd app_root
97
+ $ QUEUE=file_serve rake resque:work
98
+
99
+ This starts one Resque worker and tells it to work off the
100
+ `file_serve` queue. As soon as it's ready it'll try to run the
101
+ `Resque.reserve` code snippet above and process jobs until it can't
102
+ find any more, at which point it will sleep for a small period and
103
+ repeatedly poll the queue for more jobs.
104
+
105
+ Workers can be given multiple queues (a "queue list") and run on
106
+ multiple machines. In fact they can be run anywhere with network
107
+ access to the Redis server.
108
+
109
+
110
+ Jobs
111
+ ----
112
+
113
+ What should you run in the background? Anything that takes any time at
114
+ all. Slow INSERT statements, disk manipulating, data processing, etc.
115
+
116
+ At GitHub we use Resque to process the following types of jobs:
117
+
118
+ * Warming caches
119
+ * Counting disk usage
120
+ * Building tarballs
121
+ * Building Rubygems
122
+ * Firing off web hooks
123
+ * Creating events in the db and pre-caching them
124
+ * Building graphs
125
+ * Deleting users
126
+ * Updating our search index
127
+
128
+ As of writing we have about 35 different types of background jobs.
129
+
130
+ Keep in mind that you don't need a web app to use Resque - we just
131
+ mention "foreground" and "background" because they make conceptual
132
+ sense. You could easily be spidering sites and sticking data which
133
+ needs to be crunched later into a queue.
134
+
135
+
136
+ ### Persistence
137
+
138
+ Jobs are persisted to queues as JSON objects. Let's take our `Archive`
139
+ example from above. We'll run the following code to create a job:
140
+
141
+ ``` ruby
142
+ repo = Repository.find(44)
143
+ repo.async_create_archive('masterbrew')
144
+ ```
145
+
146
+ The following JSON will be stored in the `file_serve` queue:
147
+
148
+ ``` javascript
149
+ {
150
+ 'class': 'Archive',
151
+ 'args': [ 44, 'masterbrew' ]
152
+ }
153
+ ```
154
+
155
+ Because of this your jobs must only accept arguments that can be JSON encoded.
156
+
157
+ So instead of doing this:
158
+
159
+ ``` ruby
160
+ Resque.enqueue(Archive, self, branch)
161
+ ```
162
+
163
+ do this:
164
+
165
+ ``` ruby
166
+ Resque.enqueue(Archive, self.id, branch)
167
+ ```
168
+
169
+ This is why our above example (and all the examples in `examples/`)
170
+ uses object IDs instead of passing around the objects.
171
+
172
+ While this is less convenient than just sticking a marshaled object
173
+ in the database, it gives you a slight advantage: your jobs will be
174
+ run against the most recent version of an object because they need to
175
+ pull from the DB or cache.
176
+
177
+ If your jobs were run against marshaled objects, they could
178
+ potentially be operating on a stale record with out-of-date information.
179
+
180
+
181
+ ### send_later / async
182
+
183
+ Want something like DelayedJob's `send_later` or the ability to use
184
+ instance methods instead of just methods for jobs? See the `examples/`
185
+ directory for goodies.
186
+
187
+ We plan to provide first class `async` support in a future release.
188
+
189
+
190
+ ### Failure
191
+
192
+ If a job raises an exception, it is logged and handed off to the
193
+ `Resque::Failure` module. Failures are logged either locally in Redis
194
+ or using some different backend.
195
+
196
+ For example, Resque ships with Hoptoad support.
197
+
198
+ Keep this in mind when writing your jobs: you may want to throw
199
+ exceptions you would not normally throw in order to assist debugging.
200
+
201
+
202
+ Workers
203
+ -------
204
+
205
+ Resque workers are rake tasks that run forever. They basically do this:
206
+
207
+ ``` ruby
208
+ start
209
+ loop do
210
+ if job = reserve
211
+ job.process
212
+ else
213
+ sleep 5
214
+ end
215
+ end
216
+ shutdown
217
+ ```
218
+
219
+ Starting a worker is simple. Here's our example from earlier:
220
+
221
+ $ QUEUE=file_serve rake resque:work
222
+
223
+ By default Resque won't know about your application's
224
+ environment. That is, it won't be able to find and run your jobs - it
225
+ needs to load your application into memory.
226
+
227
+ If we've installed Resque as a Rails plugin, we might run this command
228
+ from our RAILS_ROOT:
229
+
230
+ $ QUEUE=file_serve rake environment resque:work
231
+
232
+ This will load the environment before starting a worker. Alternately
233
+ we can define a `resque:setup` task with a dependency on the
234
+ `environment` rake task:
235
+
236
+ ``` ruby
237
+ task "resque:setup" => :environment
238
+ ```
239
+
240
+ GitHub's setup task looks like this:
241
+
242
+ ``` ruby
243
+ task "resque:setup" => :environment do
244
+ Grit::Git.git_timeout = 10.minutes
245
+ end
246
+ ```
247
+
248
+ We don't want the `git_timeout` as high as 10 minutes in our web app,
249
+ but in the Resque workers it's fine.
250
+
251
+
252
+ ### Logging
253
+
254
+ Workers support basic logging to STDOUT. If you start them with the
255
+ `VERBOSE` env variable set, they will print basic debugging
256
+ information. You can also set the `VVERBOSE` (very verbose) env
257
+ variable.
258
+
259
+ $ VVERBOSE=1 QUEUE=file_serve rake environment resque:work
260
+
261
+ ### Process IDs (PIDs)
262
+
263
+ There are scenarios where it's helpful to record the PID of a resque
264
+ worker process. Use the PIDFILE option for easy access to the PID:
265
+
266
+ $ PIDFILE=./resque.pid QUEUE=file_serve rake environment resque:work
267
+
268
+
269
+ ### Priorities and Queue Lists
270
+
271
+ Resque doesn't support numeric priorities but instead uses the order
272
+ of queues you give it. We call this list of queues the "queue list."
273
+
274
+ Let's say we add a `warm_cache` queue in addition to our `file_serve`
275
+ queue. We'd now start a worker like so:
276
+
277
+ $ QUEUES=file_serve,warm_cache rake resque:work
278
+
279
+ When the worker looks for new jobs, it will first check
280
+ `file_serve`. If it finds a job, it'll process it then check
281
+ `file_serve` again. It will keep checking `file_serve` until no more
282
+ jobs are available. At that point, it will check `warm_cache`. If it
283
+ finds a job it'll process it then check `file_serve` (repeating the
284
+ whole process).
285
+
286
+ In this way you can prioritize certain queues. At GitHub we start our
287
+ workers with something like this:
288
+
289
+ $ QUEUES=critical,archive,high,low rake resque:work
290
+
291
+ Notice the `archive` queue - it is specialized and in our future
292
+ architecture will only be run from a single machine.
293
+
294
+ At that point we'll start workers on our generalized background
295
+ machines with this command:
296
+
297
+ $ QUEUES=critical,high,low rake resque:work
298
+
299
+ And workers on our specialized archive machine with this command:
300
+
301
+ $ QUEUE=archive rake resque:work
302
+
303
+
304
+ ### Running All Queues
305
+
306
+ If you want your workers to work off of every queue, including new
307
+ queues created on the fly, you can use a splat:
308
+
309
+ $ QUEUE=* rake resque:work
310
+
311
+ Queues will be processed in alphabetical order.
312
+
313
+
314
+ ### Running Multiple Workers
315
+
316
+ At GitHub we use god to start and stop multiple workers. A sample god
317
+ configuration file is included under `examples/god`. We recommend this
318
+ method.
319
+
320
+ If you'd like to run multiple workers in development mode, you can do
321
+ so using the `resque:workers` rake task:
322
+
323
+ $ COUNT=5 QUEUE=* rake resque:workers
324
+
325
+ This will spawn five Resque workers, each in its own thread. Hitting
326
+ ctrl-c should be sufficient to stop them all.
327
+
328
+
329
+ ### Forking
330
+
331
+ On certain platforms, when a Resque worker reserves a job it
332
+ immediately forks a child process. The child processes the job then
333
+ exits. When the child has exited successfully, the worker reserves
334
+ another job and repeats the process.
335
+
336
+ Why?
337
+
338
+ Because Resque assumes chaos.
339
+
340
+ Resque assumes your background workers will lock up, run too long, or
341
+ have unwanted memory growth.
342
+
343
+ If Resque workers processed jobs themselves, it'd be hard to whip them
344
+ into shape. Let's say one is using too much memory: you send it a
345
+ signal that says "shutdown after you finish processing the current
346
+ job," and it does so. It then starts up again - loading your entire
347
+ application environment. This adds useless CPU cycles and causes a
348
+ delay in queue processing.
349
+
350
+ Plus, what if it's using too much memory and has stopped responding to
351
+ signals?
352
+
353
+ Thanks to Resque's parent / child architecture, jobs that use too much memory
354
+ release that memory upon completion. No unwanted growth.
355
+
356
+ And what if a job is running too long? You'd need to `kill -9` it then
357
+ start the worker again. With Resque's parent / child architecture you
358
+ can tell the parent to forcefully kill the child then immediately
359
+ start processing more jobs. No startup delay or wasted cycles.
360
+
361
+ The parent / child architecture helps us keep tabs on what workers are
362
+ doing, too. By eliminating the need to `kill -9` workers we can have
363
+ parents remove themselves from the global listing of workers. If we
364
+ just ruthlessly killed workers, we'd need a separate watchdog process
365
+ to add and remove them to the global listing - which becomes
366
+ complicated.
367
+
368
+ Workers instead handle their own state.
369
+
370
+
371
+ ### Parents and Children
372
+
373
+ Here's a parent / child pair doing some work:
374
+
375
+ $ ps -e -o pid,command | grep [r]esque
376
+ 92099 resque: Forked 92102 at 1253142769
377
+ 92102 resque: Processing file_serve since 1253142769
378
+
379
+ You can clearly see that process 92099 forked 92102, which has been
380
+ working since 1253142769.
381
+
382
+ (By advertising the time they began processing you can easily use monit
383
+ or god to kill stale workers.)
384
+
385
+ When a parent process is idle, it lets you know what queues it is
386
+ waiting for work on:
387
+
388
+ $ ps -e -o pid,command | grep [r]esque
389
+ 92099 resque: Waiting for file_serve,warm_cache
390
+
391
+
392
+ ### Signals
393
+
394
+ Resque workers respond to a few different signals:
395
+
396
+ * `QUIT` - Wait for child to finish processing then exit
397
+ * `TERM` / `INT` - Immediately kill child then exit
398
+ * `USR1` - Immediately kill child but don't exit
399
+ * `USR2` - Don't start to process any new jobs
400
+ * `CONT` - Start to process new jobs again after a USR2
401
+
402
+ If you want to gracefully shutdown a Resque worker, use `QUIT`.
403
+
404
+ If you want to kill a stale or stuck child, use `USR1`. Processing
405
+ will continue as normal unless the child was not found. In that case
406
+ Resque assumes the parent process is in a bad state and shuts down.
407
+
408
+ If you want to kill a stale or stuck child and shutdown, use `TERM`
409
+
410
+ If you want to stop processing jobs, but want to leave the worker running
411
+ (for example, to temporarily alleviate load), use `USR2` to stop processing,
412
+ then `CONT` to start it again.
413
+
414
+ ### Mysql::Error: MySQL server has gone away
415
+
416
+ If your workers remain idle for too long they may lose their MySQL
417
+ connection. If that happens we recommend using [this
418
+ Gist](http://gist.github.com/238999).
419
+
420
+
421
+ The Front End
422
+ -------------
423
+
424
+ Resque comes with a Sinatra-based front end for seeing what's up with
425
+ your queue.
426
+
427
+ ![The Front End](https://img.skitch.com/20110528-pc67a8qsfapgjxf5gagxd92fcu.png)
428
+
429
+ ### Standalone
430
+
431
+ If you've installed Resque as a gem running the front end standalone is easy:
432
+
433
+ $ resque-web
434
+
435
+ It's a thin layer around `rackup` so it's configurable as well:
436
+
437
+ $ resque-web -p 8282
438
+
439
+ If you have a Resque config file you want evaluated just pass it to
440
+ the script as the final argument:
441
+
442
+ $ resque-web -p 8282 rails_root/config/initializers/resque.rb
443
+
444
+ You can also set the namespace directly using `resque-web`:
445
+
446
+ $ resque-web -p 8282 -N myapp
447
+
448
+ ### Passenger
449
+
450
+ Using Passenger? Resque ships with a `config.ru` you can use. See
451
+ Phusion's guide:
452
+
453
+ Apache: <http://www.modrails.com/documentation/Users%20guide%20Apache.html#_deploying_a_rack_based_ruby_application>
454
+ Nginx: <http://www.modrails.com/documentation/Users%20guide%20Nginx.html#deploying_a_rack_app>
455
+
456
+ ### Rack::URLMap
457
+
458
+ If you want to load Resque on a subpath, possibly alongside other
459
+ apps, it's easy to do with Rack's `URLMap`:
460
+
461
+ ``` ruby
462
+ require 'resque/server'
463
+
464
+ run Rack::URLMap.new \
465
+ "/" => Your::App.new,
466
+ "/resque" => Resque::Server.new
467
+ ```
468
+
469
+ Check `examples/demo/config.ru` for a functional example (including
470
+ HTTP basic auth).
471
+
472
+
473
+ Resque vs DelayedJob
474
+ --------------------
475
+
476
+ How does Resque compare to DelayedJob, and why would you choose one
477
+ over the other?
478
+
479
+ * Resque supports multiple queues
480
+ * DelayedJob supports finer grained priorities
481
+ * Resque workers are resilient to memory leaks / bloat
482
+ * DelayedJob workers are extremely simple and easy to modify
483
+ * Resque requires Redis
484
+ * DelayedJob requires ActiveRecord
485
+ * Resque can only place JSONable Ruby objects on a queue as arguments
486
+ * DelayedJob can place _any_ Ruby object on its queue as arguments
487
+ * Resque includes a Sinatra app for monitoring what's going on
488
+ * DelayedJob can be queried from within your Rails app if you want to
489
+ add an interface
490
+
491
+ If you're doing Rails development, you already have a database and
492
+ ActiveRecord. DelayedJob is super easy to setup and works great.
493
+ GitHub used it for many months to process almost 200 million jobs.
494
+
495
+ Choose Resque if:
496
+
497
+ * You need multiple queues
498
+ * You don't care / dislike numeric priorities
499
+ * You don't need to persist every Ruby object ever
500
+ * You have potentially huge queues
501
+ * You want to see what's going on
502
+ * You expect a lot of failure / chaos
503
+ * You can setup Redis
504
+ * You're not running short on RAM
505
+
506
+ Choose DelayedJob if:
507
+
508
+ * You like numeric priorities
509
+ * You're not doing a gigantic amount of jobs each day
510
+ * Your queue stays small and nimble
511
+ * There is not a lot failure / chaos
512
+ * You want to easily throw anything on the queue
513
+ * You don't want to setup Redis
514
+
515
+ In no way is Resque a "better" DelayedJob, so make sure you pick the
516
+ tool that's best for your app.
517
+
518
+
519
+ Installing Redis
520
+ ----------------
521
+
522
+ Resque requires Redis 0.900 or higher.
523
+
524
+ Resque uses Redis' lists for its queues. It also stores worker state
525
+ data in Redis.
526
+
527
+ #### Homebrew
528
+
529
+ If you're on OS X, Homebrew is the simplest way to install Redis:
530
+
531
+ $ brew install redis
532
+ $ redis-server /usr/local/etc/redis.conf
533
+
534
+ You now have a Redis daemon running on 6379.
535
+
536
+ #### Via Resque
537
+
538
+ Resque includes Rake tasks (thanks to Ezra's redis-rb) that will
539
+ install and run Redis for you:
540
+
541
+ $ git clone git://github.com/defunkt/resque.git
542
+ $ cd resque
543
+ $ rake redis:install dtach:install
544
+ $ rake redis:start
545
+
546
+ Or, if you don't have admin access on your machine:
547
+
548
+ $ git clone git://github.com/defunkt/resque.git
549
+ $ cd resque
550
+ $ PREFIX=<your_prefix> rake redis:install dtach:install
551
+ $ rake redis:start
552
+
553
+ You now have Redis running on 6379. Wait a second then hit ctrl-\ to
554
+ detach and keep it running in the background.
555
+
556
+ The demo is probably the best way to figure out how to put the parts
557
+ together. But, it's not that hard.
558
+
559
+
560
+ Resque Dependencies
561
+ -------------------
562
+
563
+ $ gem install bundler
564
+ $ bundle install
565
+
566
+
567
+ Installing Resque
568
+ -----------------
569
+
570
+ ### In a Rack app, as a gem
571
+
572
+ First install the gem.
573
+
574
+ $ gem install resque
575
+
576
+ Next include it in your application.
577
+
578
+ ``` ruby
579
+ require 'resque'
580
+ ```
581
+
582
+ Now start your application:
583
+
584
+ rackup config.ru
585
+
586
+ That's it! You can now create Resque jobs from within your app.
587
+
588
+ To start a worker, create a Rakefile in your app's root (or add this
589
+ to an existing Rakefile):
590
+
591
+ ``` ruby
592
+ require 'your/app'
593
+ require 'resque/tasks'
594
+ ```
595
+
596
+ Now:
597
+
598
+ $ QUEUE=* rake resque:work
599
+
600
+ Alternately you can define a `resque:setup` hook in your Rakefile if you
601
+ don't want to load your app every time rake runs.
602
+
603
+
604
+ ### In a Rails 2.x app, as a gem
605
+
606
+ First install the gem.
607
+
608
+ $ gem install resque
609
+
610
+ Next include it in your application.
611
+
612
+ $ cat config/initializers/load_resque.rb
613
+ require 'resque'
614
+
615
+ Now start your application:
616
+
617
+ $ ./script/server
618
+
619
+ That's it! You can now create Resque jobs from within your app.
620
+
621
+ To start a worker, add this to your Rakefile in `RAILS_ROOT`:
622
+
623
+ ``` ruby
624
+ require 'resque/tasks'
625
+ ```
626
+
627
+ Now:
628
+
629
+ $ QUEUE=* rake environment resque:work
630
+
631
+ Don't forget you can define a `resque:setup` hook in
632
+ `lib/tasks/whatever.rake` that loads the `environment` task every time.
633
+
634
+
635
+ ### In a Rails 2.x app, as a plugin
636
+
637
+ $ ./script/plugin install git://github.com/defunkt/resque
638
+
639
+ That's it! Resque will automatically be available when your Rails app
640
+ loads.
641
+
642
+ To start a worker:
643
+
644
+ $ QUEUE=* rake environment resque:work
645
+
646
+ Don't forget you can define a `resque:setup` hook in
647
+ `lib/tasks/whatever.rake` that loads the `environment` task every time.
648
+
649
+
650
+ ### In a Rails 3 app, as a gem
651
+
652
+ First include it in your Gemfile.
653
+
654
+ $ cat Gemfile
655
+ ...
656
+ gem 'resque'
657
+ ...
658
+
659
+ Next install it with Bundler.
660
+
661
+ $ bundle install
662
+
663
+ Now start your application:
664
+
665
+ $ rails server
666
+
667
+ That's it! You can now create Resque jobs from within your app.
668
+
669
+ To start a worker, add this to a file in `lib/tasks` (ex:
670
+ `lib/tasks/resque.rake`):
671
+
672
+ ``` ruby
673
+ require 'resque/tasks'
674
+ ```
675
+
676
+ Now:
677
+
678
+ $ QUEUE=* rake environment resque:work
679
+
680
+ Don't forget you can define a `resque:setup` hook in
681
+ `lib/tasks/whatever.rake` that loads the `environment` task every time.
682
+
683
+
684
+ Configuration
685
+ -------------
686
+
687
+ You may want to change the Redis host and port Resque connects to, or
688
+ set various other options at startup.
689
+
690
+ Resque has a `redis` setter which can be given a string or a Redis
691
+ object. This means if you're already using Redis in your app, Resque
692
+ can re-use the existing connection.
693
+
694
+ String: `Resque.redis = 'localhost:6379'`
695
+
696
+ Redis: `Resque.redis = $redis`
697
+
698
+ For our rails app we have a `config/initializers/resque.rb` file where
699
+ we load `config/resque.yml` by hand and set the Redis information
700
+ appropriately.
701
+
702
+ Here's our `config/resque.yml`:
703
+
704
+ development: localhost:6379
705
+ test: localhost:6379
706
+ staging: redis1.se.github.com:6379
707
+ fi: localhost:6379
708
+ production: redis1.ae.github.com:6379
709
+
710
+ And our initializer:
711
+
712
+ ``` ruby
713
+ rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..'
714
+ rails_env = ENV['RAILS_ENV'] || 'development'
715
+
716
+ resque_config = YAML.load_file(rails_root + '/config/resque.yml')
717
+ Resque.redis = resque_config[rails_env]
718
+ ```
719
+
720
+ Easy peasy! Why not just use `RAILS_ROOT` and `RAILS_ENV`? Because
721
+ this way we can tell our Sinatra app about the config file:
722
+
723
+ $ RAILS_ENV=production resque-web rails_root/config/initializers/resque.rb
724
+
725
+ Now everyone is on the same page.
726
+
727
+ Also, you could disable jobs queueing by setting 'inline' attribute.
728
+ For example, if you want to run all jobs in the same process for cucumber, try:
729
+
730
+ ``` ruby
731
+ Resque.inline = ENV['RAILS_ENV'] == "cucumber"
732
+ ```
733
+
734
+
735
+ Plugins and Hooks
736
+ -----------------
737
+
738
+ For a list of available plugins see
739
+ <http://wiki.github.com/defunkt/resque/plugins>.
740
+
741
+ If you'd like to write your own plugin, or want to customize Resque
742
+ using hooks (such as `Resque.after_fork`), see
743
+ [docs/HOOKS.md](http://github.com/defunkt/resque/blob/master/docs/HOOKS.md).
744
+
745
+
746
+ Namespaces
747
+ ----------
748
+
749
+ If you're running multiple, separate instances of Resque you may want
750
+ to namespace the keyspaces so they do not overlap. This is not unlike
751
+ the approach taken by many memcached clients.
752
+
753
+ This feature is provided by the [redis-namespace][rs] library, which
754
+ Resque uses by default to separate the keys it manages from other keys
755
+ in your Redis server.
756
+
757
+ Simply use the `Resque.redis.namespace` accessor:
758
+
759
+ ``` ruby
760
+ Resque.redis.namespace = "resque:GitHub"
761
+ ```
762
+
763
+ We recommend sticking this in your initializer somewhere after Redis
764
+ is configured.
765
+
766
+
767
+ Demo
768
+ ----
769
+
770
+ Resque ships with a demo Sinatra app for creating jobs that are later
771
+ processed in the background.
772
+
773
+ Try it out by looking at the README, found at `examples/demo/README.markdown`.
774
+
775
+
776
+ Monitoring
777
+ ----------
778
+
779
+ ### god
780
+
781
+ If you're using god to monitor Resque, we have provided example
782
+ configs in `examples/god/`. One is for starting / stopping workers,
783
+ the other is for killing workers that have been running too long.
784
+
785
+ ### monit
786
+
787
+ If you're using monit, `examples/monit/resque.monit` is provided free
788
+ of charge. This is **not** used by GitHub in production, so please
789
+ send patches for any tweaks or improvements you can make to it.
790
+
791
+
792
+ Questions
793
+ ---------
794
+
795
+ Please add them to the [FAQ](https://github.com/defunkt/resque/wiki/FAQ) or
796
+ ask on the Mailing List. The Mailing List is explained further below
797
+
798
+
799
+ Development
800
+ -----------
801
+
802
+ Want to hack on Resque?
803
+
804
+ First clone the repo and run the tests:
805
+
806
+ git clone git://github.com/defunkt/resque.git
807
+ cd resque
808
+ rake test
809
+
810
+ If the tests do not pass make sure you have Redis installed
811
+ correctly (though we make an effort to tell you if we feel this is the
812
+ case). The tests attempt to start an isolated instance of Redis to
813
+ run against.
814
+
815
+ Also make sure you've installed all the dependencies correctly. For
816
+ example, try loading the `redis-namespace` gem after you've installed
817
+ it:
818
+
819
+ $ irb
820
+ >> require 'rubygems'
821
+ => true
822
+ >> require 'redis/namespace'
823
+ => true
824
+
825
+ If you get an error requiring any of the dependencies, you may have
826
+ failed to install them or be seeing load path issues.
827
+
828
+ Feel free to ping the mailing list with your problem and we'll try to
829
+ sort it out.
830
+
831
+
832
+ Contributing
833
+ ------------
834
+
835
+ Once you've made your great commits:
836
+
837
+ 1. [Fork][1] Resque
838
+ 2. Create a topic branch - `git checkout -b my_branch`
839
+ 3. Push to your branch - `git push origin my_branch`
840
+ 4. Create a [Pull Request](http://help.github.com/pull-requests/) from your branch
841
+ 5. That's it!
842
+
843
+ You might want to checkout our [Contributing][cb] wiki page for information
844
+ on coding standards, new features, etc.
845
+
846
+
847
+ Mailing List
848
+ ------------
849
+
850
+ To join the list simply send an email to <resque@librelist.com>. This
851
+ will subscribe you and send you information about your subscription,
852
+ including unsubscribe information.
853
+
854
+ The archive can be found at <http://librelist.com/browser/resque/>.
855
+
856
+
857
+ Meta
858
+ ----
859
+
860
+ * Code: `git clone git://github.com/defunkt/resque.git`
861
+ * Home: <http://github.com/defunkt/resque>
862
+ * Docs: <http://defunkt.github.com/resque/>
863
+ * Bugs: <http://github.com/defunkt/resque/issues>
864
+ * List: <resque@librelist.com>
865
+ * Chat: <irc://irc.freenode.net/resque>
866
+ * Gems: <http://gemcutter.org/gems/resque>
867
+
868
+ This project uses [Semantic Versioning][sv].
869
+
870
+
871
+ Author
872
+ ------
873
+
874
+ Chris Wanstrath :: chris@ozmm.org :: @defunkt
875
+
876
+ [0]: http://github.com/blog/542-introducing-resque
877
+ [1]: http://help.github.com/forking/
878
+ [2]: http://github.com/defunkt/resque/issues
879
+ [sv]: http://semver.org/
880
+ [rs]: http://github.com/defunkt/redis-namespace
881
+ [cb]: http://wiki.github.com/defunkt/resque/contributing