resque-igo 1.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (50) hide show
  1. data/HISTORY.md +225 -0
  2. data/LICENSE +20 -0
  3. data/README.markdown +855 -0
  4. data/Rakefile +70 -0
  5. data/bin/resque +57 -0
  6. data/bin/resque-web +23 -0
  7. data/lib/resque.rb +380 -0
  8. data/lib/resque/errors.rb +10 -0
  9. data/lib/resque/failure.rb +66 -0
  10. data/lib/resque/failure/base.rb +61 -0
  11. data/lib/resque/failure/hoptoad.rb +132 -0
  12. data/lib/resque/failure/multiple.rb +50 -0
  13. data/lib/resque/failure/redis.rb +40 -0
  14. data/lib/resque/helpers.rb +71 -0
  15. data/lib/resque/job.rb +209 -0
  16. data/lib/resque/plugin.rb +51 -0
  17. data/lib/resque/server.rb +247 -0
  18. data/lib/resque/server/public/idle.png +0 -0
  19. data/lib/resque/server/public/jquery-1.3.2.min.js +19 -0
  20. data/lib/resque/server/public/jquery.relatize_date.js +95 -0
  21. data/lib/resque/server/public/poll.png +0 -0
  22. data/lib/resque/server/public/ranger.js +67 -0
  23. data/lib/resque/server/public/reset.css +48 -0
  24. data/lib/resque/server/public/style.css +86 -0
  25. data/lib/resque/server/public/working.png +0 -0
  26. data/lib/resque/server/test_helper.rb +19 -0
  27. data/lib/resque/server/views/error.erb +1 -0
  28. data/lib/resque/server/views/failed.erb +53 -0
  29. data/lib/resque/server/views/key_sets.erb +20 -0
  30. data/lib/resque/server/views/key_string.erb +11 -0
  31. data/lib/resque/server/views/layout.erb +42 -0
  32. data/lib/resque/server/views/next_more.erb +10 -0
  33. data/lib/resque/server/views/overview.erb +4 -0
  34. data/lib/resque/server/views/queues.erb +65 -0
  35. data/lib/resque/server/views/stats.erb +73 -0
  36. data/lib/resque/server/views/workers.erb +109 -0
  37. data/lib/resque/server/views/working.erb +68 -0
  38. data/lib/resque/stat.rb +54 -0
  39. data/lib/resque/tasks.rb +39 -0
  40. data/lib/resque/version.rb +3 -0
  41. data/lib/resque/worker.rb +478 -0
  42. data/tasks/resque.rake +2 -0
  43. data/test/job_hooks_test.rb +323 -0
  44. data/test/job_plugins_test.rb +230 -0
  45. data/test/plugin_test.rb +116 -0
  46. data/test/resque-web_test.rb +54 -0
  47. data/test/resque_test.rb +351 -0
  48. data/test/test_helper.rb +166 -0
  49. data/test/worker_test.rb +302 -0
  50. metadata +180 -0
data/HISTORY.md ADDED
@@ -0,0 +1,225 @@
1
+ ## 1.11.0 (2010-08-23)
2
+
3
+ * Web UI: Group /workers page by hostnames
4
+
5
+ ## 1.10.0 (2010-08-23)
6
+
7
+ * Support redis:// string format in `Resque.redis=`
8
+ * Using new cross-platform JSON gem.
9
+ * Added `after_enqueue` plugin hook.
10
+ * Added `shutdown?` method which can be overridden.
11
+ * Added support for the "leftright" gem when running tests.
12
+ * Grammarfix: In the README
13
+
14
+ ## 1.9.10 (2010-08-06)
15
+
16
+ * Bugfix: before_fork should get passed the job
17
+
18
+ ## 1.9.9 (2010-07-26)
19
+
20
+ * Depend on redis-namespace 0.8.0
21
+ * Depend on json_pure instead of json (for JRuby compat)
22
+ * Bugfix: rails_env display in stats view
23
+
24
+ ## 1.9.8 (2010-07-20)
25
+
26
+ * Bugfix: Worker.all should never return nil
27
+ * monit example: Fixed Syntax Error and adding environment to the rake task
28
+ * redis rake task: Fixed typo in copy command
29
+
30
+ ## 1.9.7 (2010-07-09)
31
+
32
+ * Improved memory usage in Job.destroy
33
+ * redis-namespace 0.7.0 now required
34
+ * Bugfix: Reverted $0 changes
35
+ * Web Bugfix: Payload-less failures in the web ui work
36
+
37
+ ## 1.9.6 (2010-06-22)
38
+
39
+ * Bugfix: Rakefile logging works the same as all the other logging
40
+
41
+ ## 1.9.5 (2010-06-16)
42
+
43
+ * Web Bugfix: Display the configured namespace on the stats page
44
+ * Revert Bugfix: Make ps -o more cross platform friendly
45
+
46
+ ## 1.9.4 (2010-06-14)
47
+
48
+ * Bugfix: Multiple failure backend gets exception information when created
49
+
50
+ ## 1.9.3 (2010-06-14)
51
+
52
+ * Bugfix: Resque#queues always returns an array
53
+
54
+ ## 1.9.2 (2010-06-13)
55
+
56
+ * Bugfix: Worker.all returning nil fix
57
+ * Bugfix: Make ps -o more cross platform friendly
58
+
59
+ ## 1.9.1 (2010-06-04)
60
+
61
+ * Less strict JSON dependency
62
+ * Included HISTORY.md in gem
63
+
64
+ ## 1.9.0 (2010-06-04)
65
+
66
+ * Redis 2 support
67
+ * Depend on redis-namespace 0.5.0
68
+ * Added Resque::VERSION constant (alias of Resque::Version)
69
+ * Bugfix: Specify JSON dependency
70
+ * Bugfix: Hoptoad plugin now works on 1.9
71
+
72
+ ## 1.8.5 (2010-05-18)
73
+
74
+ * Bugfix: Be more liberal in which Redis clients we accept.
75
+
76
+ ## 1.8.4 (2010-05-18)
77
+
78
+ * Try to resolve redis-namespace dependency issue
79
+
80
+ ## 1.8.3 (2010-05-17)
81
+
82
+ * Depend on redis-rb ~> 1.0.7
83
+
84
+ ## 1.8.2 (2010-05-03)
85
+
86
+ * Bugfix: Include "tasks/" dir in RubyGem
87
+
88
+ ## 1.8.1 (2010-04-29)
89
+
90
+ * Bugfix: Multiple failure backend did not support requeue-ing failed jobs
91
+ * Bugfix: Fix /failed when error has no backtrace
92
+ * Bugfix: Add `Redis::DistRedis` as a valid client
93
+
94
+ ## 1.8.0 (2010-04-07)
95
+
96
+ * Jobs that never complete due to killed worker are now failed.
97
+ * Worker "working" state is now maintained by the parent, not the child.
98
+ * Stopped using deprecated redis.rb methods
99
+ * `Worker.working` race condition fixed
100
+ * `Worker#process` has been deprecated.
101
+ * Monit example fixed
102
+ * Redis::Client and Redis::Namespace can be passed to `Resque.redis=`
103
+
104
+ ## 1.7.1 (2010-04-02)
105
+
106
+ * Bugfix: Make job hook execution order consistent
107
+ * Bugfix: stdout buffering in child process
108
+
109
+ ## 1.7.0 (2010-03-31)
110
+
111
+ * Job hooks API. See docs/HOOKS.md.
112
+ * web: Hovering over dates shows a timestamp
113
+ * web: AJAXify retry action for failed jobs
114
+ * web bugfix: Fix pagination bug
115
+
116
+ ## 1.6.1 (2010-03-25)
117
+
118
+ * Bugfix: Workers may not be clearing their state correctly on
119
+ shutdown
120
+ * Added example monit config.
121
+ * Exception class is now recorded when an error is raised in a
122
+ worker.
123
+ * web: Unit tests
124
+ * web: Show namespace in header and footer
125
+ * web: Remove a queue
126
+ * web: Retry failed jobs
127
+
128
+ ## 1.6.0 (2010-03-09)
129
+
130
+ * Added `before_first_fork`, `before_fork`, and `after_fork` hooks.
131
+ * Hoptoad: Added server_environment config setting
132
+ * Hoptoad bugfix: Don't depend on RAILS_ROOT
133
+ * 1.8.6 compat fixes
134
+
135
+ ## 1.5.2 (2010-03-03)
136
+
137
+ * Bugfix: JSON check was crazy.
138
+
139
+ ## 1.5.1 (2010-03-03)
140
+
141
+ * `Job.destroy` and `Resque.dequeue` return the # of destroyed jobs.
142
+ * Hoptoad notifier improvements
143
+ * Specify the namespace with `resque-web` by passing `-N namespace`
144
+ * Bugfix: Don't crash when trying to parse invalid JSON.
145
+ * Bugfix: Non-standard namespace support
146
+ * Web: Red backgound for queue "failed" only shown if there are failed jobs.
147
+ * Web bugfix: Tabs highlight properly now
148
+ * Web bugfix: ZSET partial support in stats
149
+ * Web bugfix: Deleting failed jobs works again
150
+ * Web bugfix: Sets (or zsets, lists, etc) now paginate.
151
+
152
+ ## 1.5.0 (2010-02-17)
153
+
154
+ * Version now included in procline, e.g. `resque-1.5.0: Message`
155
+ * Web bugfix: Ignore idle works in the "working" page
156
+ * Added `Resque::Job.destroy(queue, klass, *args)`
157
+ * Added `Resque.dequeue(klass, *args)`
158
+
159
+ ## 1.4.0 (2010-02-11)
160
+
161
+ * Fallback when unable to bind QUIT and USR1 for Windows and JRuby.
162
+ * Fallback when no `Kernel.fork` is provided (for IronRuby).
163
+ * Web: Rounded corners in Firefox
164
+ * Cut down system calls in `Worker#prune_dead_workers`
165
+ * Enable switching DB in a Redis server from config
166
+ * Support USR2 and CONT to stop and start job processing.
167
+ * Web: Add example failing job
168
+ * Bugfix: `Worker#unregister_worker` shouldn't call `done_working`
169
+ * Bugfix: Example god config now restarts Resque properly.
170
+ * Multiple failure backends now permitted.
171
+ * Hoptoad failure backend updated to new API
172
+
173
+ ## 1.3.1 (2010-01-11)
174
+
175
+ * Vegas bugfix: Don't error without a config
176
+
177
+ ## 1.3.0 (2010-01-11)
178
+
179
+ * Use Vegas for resque-web
180
+ * Web Bugfix: Show proper date/time value for failed_at on Failures
181
+ * Web Bugfix: Make the / route more flexible
182
+ * Add Resque::Server.tabs array (so plugins can add their own tabs)
183
+ * Start using [Semantic Versioning](http://semver.org/)
184
+
185
+ ## 1.2.4 (2009-12-15)
186
+
187
+ * Web Bugfix: fix key links on stat page
188
+
189
+ ## 1.2.3 (2009-12-15)
190
+
191
+ * Bugfix: Fixed `rand` seeding in child processes.
192
+ * Bugfix: Better JSON encoding/decoding without Yajl.
193
+ * Bugfix: Avoid `ps` flag error on Linux
194
+ * Add `PREFIX` observance to `rake` install tasks.
195
+
196
+ ## 1.2.2 (2009-12-08)
197
+
198
+ * Bugfix: Job equality was not properly implemented.
199
+
200
+ ## 1.2.1 (2009-12-07)
201
+
202
+ * Added `rake resque:workers` task for starting multiple workers.
203
+ * 1.9.x compatibility
204
+ * Bugfix: Yajl decoder doesn't care about valid UTF-8
205
+ * config.ru loads RESQUECONFIG if the ENV variable is set.
206
+ * `resque-web` now sets RESQUECONFIG
207
+ * Job objects know if they are equal.
208
+ * Jobs can be re-queued using `Job#recreate`
209
+
210
+ ## 1.2.0 (2009-11-25)
211
+
212
+ * If USR1 is sent and no child is found, shutdown.
213
+ * Raise when a job class does not respond to `perform`.
214
+ * Added `Resque.remove_queue` for deleting a queue
215
+
216
+ ## 1.1.0 (2009-11-04)
217
+
218
+ * Bugfix: Broken ERB tag in failure UI
219
+ * Bugfix: Save the worker's ID, not the worker itself, in the failure module
220
+ * Redesigned the sinatra web interface
221
+ * Added option to clear failed jobs
222
+
223
+ ## 1.0.0 (2009-11-03)
224
+
225
+ * First release.
data/LICENSE ADDED
@@ -0,0 +1,20 @@
1
+ Copyright (c) 2009 Chris Wanstrath
2
+
3
+ Permission is hereby granted, free of charge, to any person obtaining
4
+ a copy of this software and associated documentation files (the
5
+ "Software"), to deal in the Software without restriction, including
6
+ without limitation the rights to use, copy, modify, merge, publish,
7
+ distribute, sublicense, and/or sell copies of the Software, and to
8
+ permit persons to whom the Software is furnished to do so, subject to
9
+ the following conditions:
10
+
11
+ The above copyright notice and this permission notice shall be
12
+ included in all copies or substantial portions of the Software.
13
+
14
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
15
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
16
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
17
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
18
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
19
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
20
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
data/README.markdown ADDED
@@ -0,0 +1,855 @@
1
+ Resque
2
+ ======
3
+
4
+ Resque is a Redis-backed library for creating background jobs, placing
5
+ those jobs on multiple queues, and processing them later.
6
+
7
+ Background jobs can be any Ruby class or module that responds to
8
+ `perform`. Your existing classes can easily be converted to background
9
+ jobs or you can create new classes specifically to do work. Or, you
10
+ can do both.
11
+
12
+ Resque is heavily inspired by DelayedJob (which rocks) and comprises
13
+ three parts:
14
+
15
+ 1. A Ruby library for creating, querying, and processing jobs
16
+ 2. A Rake task for starting a worker which processes jobs
17
+ 3. A Sinatra app for monitoring queues, jobs, and workers.
18
+
19
+ Resque workers can be distributed between multiple machines,
20
+ support priorities, are resilient to memory bloat / "leaks," are
21
+ optimized for REE (but work on MRI and JRuby), tell you what they're
22
+ doing, and expect failure.
23
+
24
+ Resque queues are persistent; support constant time, atomic push and
25
+ pop (thanks to Redis); provide visibility into their contents; and
26
+ store jobs as simple JSON packages.
27
+
28
+ The Resque frontend tells you what workers are doing, what workers are
29
+ not doing, what queues you're using, what's in those queues, provides
30
+ general usage stats, and helps you track failures.
31
+
32
+
33
+ What did you guys do to Resque?
34
+ ===============================
35
+
36
+ This version of Resque uses mongo as the back end. It also has some neat
37
+ new features that you may enjoy
38
+
39
+ Features
40
+ --------
41
+
42
+ It stores each queue in its own collection.
43
+
44
+ For testing purposes. Resque.bypass_queues will call your job's
45
+ perform method AT ONCE, and not touch the queue at all. This can be
46
+ useful when writing unit tests against things that would normally
47
+ happen partially in your request and partially in a worker.
48
+
49
+ If your job class has a variable called @unique_jobs = true, you
50
+ can queue unique jobs - jobs that must only be processed once. In
51
+ order to take advantage of this feature, the arguments to your enqueue
52
+ call must include a hash containing a key called _id in the first
53
+ position.
54
+
55
+ If your job class indicates that @delayed_jobs = true, you can queue
56
+ delayed jobs. These jobs will not be popped off the queue until the
57
+ Time indicated in arg[0][:delay_until] has come. Note that you must
58
+ call Resque.enable_delay(:queue) before enququing any delayed jobs, to
59
+ ensure that the performance impact on other queues is minimal.
60
+
61
+ Stern Warnings
62
+ --------------
63
+
64
+ Sometimes, Resque-Mongo will drop a queue collection, or create some
65
+ indexes, or otherwise manipulate its database. For this reason, it is
66
+ STRONGLY recommended that you give it its own database in mongo.
67
+
68
+ All jobs should be queued via Resque.enqueue. All arguments passed to
69
+ this method must be BSON-encodable. Resque-Mongo does not serialize
70
+ your objects for you. Arrays, Hashes, Strings, Numbers, and Times
71
+ are all ok, so don't worry.
72
+
73
+ Many of the new queue-level features require the first argument of
74
+ your perform method to be an options hash. In fact, if you just start
75
+ making all your perform()s take one param, that is an options hash,
76
+ you'll probably save yourself some pain.
77
+
78
+ Resque-Mongo will not create any indexes on your queues, only on its
79
+ meta-data. You will need to create any indexes you want. Normally,
80
+ This is not a problem, because you aren't querying by keys, but you may
81
+ want to create indexes on the class key in some circumstances. If you
82
+ use the unique or delay features, you may want some additional indexes,
83
+ depending on the nature of your workload. Paranoid? Test enqueuing and
84
+ processing all your jobs, and run with --notablescans. Learn the profiler,
85
+ and use it often.
86
+
87
+ Specifically, a queue with many long-delayed jobs will result in slower queue pops
88
+ for all jobs using that queue. Index delay_until in the case of
89
+ thousands of delayed jobs.
90
+
91
+ Back to the original README
92
+ ===========================
93
+
94
+ The Blog Post
95
+ -------------
96
+
97
+ For the backstory, philosophy, and history of Resque's beginnings,
98
+ please see [the blog post][0].
99
+
100
+
101
+ Overview
102
+ --------
103
+
104
+ Resque allows you to create jobs and place them on a queue, then,
105
+ later, pull those jobs off the queue and process them.
106
+
107
+ Resque jobs are Ruby classes (or modules) which respond to the
108
+ `perform` method. Here's an example:
109
+
110
+ class Archive
111
+ @queue = :file_serve
112
+
113
+ def self.perform(repo_id, branch = 'master')
114
+ repo = Repository.find(repo_id)
115
+ repo.create_archive(branch)
116
+ end
117
+ end
118
+
119
+ The `@queue` class instance variable determines which queue `Archive`
120
+ jobs will be placed in. Queues are arbitrary and created on the fly -
121
+ you can name them whatever you want and have as many as you want.
122
+
123
+ To place an `Archive` job on the `file_serve` queue, we might add this
124
+ to our application's pre-existing `Repository` class:
125
+
126
+ class Repository
127
+ def async_create_archive(branch)
128
+ Resque.enqueue(Archive, self.id, branch)
129
+ end
130
+ end
131
+
132
+ Now when we call `repo.async_create_archive('masterbrew')` in our
133
+ application, a job will be created and placed on the `file_serve`
134
+ queue.
135
+
136
+ Later, a worker will run something like this code to process the job:
137
+
138
+ klass, args = Resque.reserve(:file_serve)
139
+ klass.perform(*args) if klass.respond_to? :perform
140
+
141
+ Which translates to:
142
+
143
+ Archive.perform(44, 'masterbrew')
144
+
145
+ Let's start a worker to run `file_serve` jobs:
146
+
147
+ $ cd app_root
148
+ $ QUEUE=file_serve rake resque:work
149
+
150
+ This starts one Resque worker and tells it to work off the
151
+ `file_serve` queue. As soon as it's ready it'll try to run the
152
+ `Resque.reserve` code snippet above and process jobs until it can't
153
+ find any more, at which point it will sleep for a small period and
154
+ repeatedly poll the queue for more jobs.
155
+
156
+ Workers can be given multiple queues (a "queue list") and run on
157
+ multiple machines. In fact they can be run anywhere with network
158
+ access to the Redis server.
159
+
160
+
161
+ Jobs
162
+ ----
163
+
164
+ What should you run in the background? Anything that takes any time at
165
+ all. Slow INSERT statements, disk manipulating, data processing, etc.
166
+
167
+ At GitHub we use Resque to process the following types of jobs:
168
+
169
+ * Warming caches
170
+ * Counting disk usage
171
+ * Building tarballs
172
+ * Building Rubygems
173
+ * Firing off web hooks
174
+ * Creating events in the db and pre-caching them
175
+ * Building graphs
176
+ * Deleting users
177
+ * Updating our search index
178
+
179
+ As of writing we have about 35 different types of background jobs.
180
+
181
+ Keep in mind that you don't need a web app to use Resque - we just
182
+ mention "foreground" and "background" because they make conceptual
183
+ sense. You could easily be spidering sites and sticking data which
184
+ needs to be crunched later into a queue.
185
+
186
+
187
+ ### Persistence
188
+
189
+ Jobs are persisted to queues as JSON objects. Let's take our `Archive`
190
+ example from above. We'll run the following code to create a job:
191
+
192
+ repo = Repository.find(44)
193
+ repo.async_create_archive('masterbrew')
194
+
195
+ The following JSON will be stored in the `file_serve` queue:
196
+
197
+ {
198
+ 'class': 'Archive',
199
+ 'args': [ 44, 'masterbrew' ]
200
+ }
201
+
202
+ Because of this your jobs must only accept arguments that can be JSON encoded.
203
+
204
+ So instead of doing this:
205
+
206
+ Resque.enqueue(Archive, self, branch)
207
+
208
+ do this:
209
+
210
+ Resque.enqueue(Archive, self.id, branch)
211
+
212
+ This is why our above example (and all the examples in `examples/`)
213
+ uses object IDs instead of passing around the objects.
214
+
215
+ While this is less convenient than just sticking a marshaled object
216
+ in the database, it gives you a slight advantage: your jobs will be
217
+ run against the most recent version of an object because they need to
218
+ pull from the DB or cache.
219
+
220
+ If your jobs were run against marshaled objects, they could
221
+ potentially be operating on a stale record with out-of-date information.
222
+
223
+
224
+ ### send_later / async
225
+
226
+ Want something like DelayedJob's `send_later` or the ability to use
227
+ instance methods instead of just methods for jobs? See the `examples/`
228
+ directory for goodies.
229
+
230
+ We plan to provide first class `async` support in a future release.
231
+
232
+
233
+ ### Failure
234
+
235
+ If a job raises an exception, it is logged and handed off to the
236
+ `Resque::Failure` module. Failures are logged either locally in Redis
237
+ or using some different backend.
238
+
239
+ For example, Resque ships with Hoptoad support.
240
+
241
+ Keep this in mind when writing your jobs: you may want to throw
242
+ exceptions you would not normally throw in order to assist debugging.
243
+
244
+
245
+ Workers
246
+ -------
247
+
248
+ Resque workers are rake tasks that run forever. They basically do this:
249
+
250
+ start
251
+ loop do
252
+ if job = reserve
253
+ job.process
254
+ else
255
+ sleep 5
256
+ end
257
+ end
258
+ shutdown
259
+
260
+ Starting a worker is simple. Here's our example from earlier:
261
+
262
+ $ QUEUE=file_serve rake resque:work
263
+
264
+ By default Resque won't know about your application's
265
+ environment. That is, it won't be able to find and run your jobs - it
266
+ needs to load your application into memory.
267
+
268
+ If we've installed Resque as a Rails plugin, we might run this command
269
+ from our RAILS_ROOT:
270
+
271
+ $ QUEUE=file_serve rake environment resque:work
272
+
273
+ This will load the environment before starting a worker. Alternately
274
+ we can define a `resque:setup` task with a dependency on the
275
+ `environment` rake task:
276
+
277
+ task "resque:setup" => :environment
278
+
279
+ GitHub's setup task looks like this:
280
+
281
+ task "resque:setup" => :environment do
282
+ Grit::Git.git_timeout = 10.minutes
283
+ end
284
+
285
+ We don't want the `git_timeout` as high as 10 minutes in our web app,
286
+ but in the Resque workers it's fine.
287
+
288
+
289
+ ### Logging
290
+
291
+ Workers support basic logging to STDOUT. If you start them with the
292
+ `VERBOSE` env variable set, they will print basic debugging
293
+ information. You can also set the `VVERBOSE` (very verbose) env
294
+ variable.
295
+
296
+ $ VVERBOSE=1 QUEUE=file_serve rake environment resque:work
297
+
298
+
299
+ ### Priorities and Queue Lists
300
+
301
+ Resque doesn't support numeric priorities but instead uses the order
302
+ of queues you give it. We call this list of queues the "queue list."
303
+
304
+ Let's say we add a `warm_cache` queue in addition to our `file_serve`
305
+ queue. We'd now start a worker like so:
306
+
307
+ $ QUEUES=file_serve,warm_cache rake resque:work
308
+
309
+ When the worker looks for new jobs, it will first check
310
+ `file_serve`. If it finds a job, it'll process it then check
311
+ `file_serve` again. It will keep checking `file_serve` until no more
312
+ jobs are available. At that point, it will check `warm_cache`. If it
313
+ finds a job it'll process it then check `file_serve` (repeating the
314
+ whole process).
315
+
316
+ In this way you can prioritize certain queues. At GitHub we start our
317
+ workers with something like this:
318
+
319
+ $ QUEUES=critical,archive,high,low rake resque:work
320
+
321
+ Notice the `archive` queue - it is specialized and in our future
322
+ architecture will only be run from a single machine.
323
+
324
+ At that point we'll start workers on our generalized background
325
+ machines with this command:
326
+
327
+ $ QUEUES=critical,high,low rake resque:work
328
+
329
+ And workers on our specialized archive machine with this command:
330
+
331
+ $ QUEUE=archive rake resque:work
332
+
333
+
334
+ ### Running All Queues
335
+
336
+ If you want your workers to work off of every queue, including new
337
+ queues created on the fly, you can use a splat:
338
+
339
+ $ QUEUE=* rake resque:work
340
+
341
+ Queues will be processed in alphabetical order.
342
+
343
+
344
+ ### Running Multiple Workers
345
+
346
+ At GitHub we use god to start and stop multiple workers. A sample god
347
+ configuration file is included under `examples/god`. We recommend this
348
+ method.
349
+
350
+ If you'd like to run multiple workers in development mode, you can do
351
+ so using the `resque:workers` rake task:
352
+
353
+ $ COUNT=5 QUEUE=* rake resque:workers
354
+
355
+ This will spawn five Resque workers, each in its own thread. Hitting
356
+ ctrl-c should be sufficient to stop them all.
357
+
358
+
359
+ ### Forking
360
+
361
+ On certain platforms, when a Resque worker reserves a job it
362
+ immediately forks a child process. The child processes the job then
363
+ exits. When the child has exited successfully, the worker reserves
364
+ another job and repeats the process.
365
+
366
+ Why?
367
+
368
+ Because Resque assumes chaos.
369
+
370
+ Resque assumes your background workers will lock up, run too long, or
371
+ have unwanted memory growth.
372
+
373
+ If Resque workers processed jobs themselves, it'd be hard to whip them
374
+ into shape. Let's say one is using too much memory: you send it a
375
+ signal that says "shutdown after you finish processing the current
376
+ job," and it does so. It then starts up again - loading your entire
377
+ application environment. This adds useless CPU cycles and causes a
378
+ delay in queue processing.
379
+
380
+ Plus, what if it's using too much memory and has stopped responding to
381
+ signals?
382
+
383
+ Thanks to Resque's parent / child architecture, jobs that use too much memory
384
+ release that memory upon completion. No unwanted growth.
385
+
386
+ And what if a job is running too long? You'd need to `kill -9` it then
387
+ start the worker again. With Resque's parent / child architecture you
388
+ can tell the parent to forcefully kill the child then immediately
389
+ start processing more jobs. No startup delay or wasted cycles.
390
+
391
+ The parent / child architecture helps us keep tabs on what workers are
392
+ doing, too. By eliminating the need to `kill -9` workers we can have
393
+ parents remove themselves from the global listing of workers. If we
394
+ just ruthlessly killed workers, we'd need a separate watchdog process
395
+ to add and remove them to the global listing - which becomes
396
+ complicated.
397
+
398
+ Workers instead handle their own state.
399
+
400
+
401
+ ### Parents and Children
402
+
403
+ Here's a parent / child pair doing some work:
404
+
405
+ $ ps -e -o pid,command | grep [r]esque
406
+ 92099 resque: Forked 92102 at 1253142769
407
+ 92102 resque: Processing file_serve since 1253142769
408
+
409
+ You can clearly see that process 92099 forked 92102, which has been
410
+ working since 1253142769.
411
+
412
+ (By advertising the time they began processing you can easily use monit
413
+ or god to kill stale workers.)
414
+
415
+ When a parent process is idle, it lets you know what queues it is
416
+ waiting for work on:
417
+
418
+ $ ps -e -o pid,command | grep [r]esque
419
+ 92099 resque: Waiting for file_serve,warm_cache
420
+
421
+
422
+ ### Signals
423
+
424
+ Resque workers respond to a few different signals:
425
+
426
+ * `QUIT` - Wait for child to finish processing then exit
427
+ * `TERM` / `INT` - Immediately kill child then exit
428
+ * `USR1` - Immediately kill child but don't exit
429
+ * `USR2` - Don't start to process any new jobs
430
+ * `CONT` - Start to process new jobs again after a USR2
431
+
432
+ If you want to gracefully shutdown a Resque worker, use `QUIT`.
433
+
434
+ If you want to kill a stale or stuck child, use `USR1`. Processing
435
+ will continue as normal unless the child was not found. In that case
436
+ Resque assumes the parent process is in a bad state and shuts down.
437
+
438
+ If you want to kill a stale or stuck child and shutdown, use `TERM`
439
+
440
+ If you want to stop processing jobs, but want to leave the worker running
441
+ (for example, to temporarily alleviate load), use `USR2` to stop processing,
442
+ then `CONT` to start it again.
443
+
444
+ ### Mysql::Error: MySQL server has gone away
445
+
446
+ If your workers remain idle for too long they may lose their MySQL
447
+ connection. If that happens we recommend using [this
448
+ Gist](http://gist.github.com/238999).
449
+
450
+
451
+ The Front End
452
+ -------------
453
+
454
+ Resque comes with a Sinatra-based front end for seeing what's up with
455
+ your queue.
456
+
457
+ ![The Front End](http://img.skitch.com/20091104-tqh5pgkwgbskjbk7qbtmpesnyw.jpg)
458
+
459
+ ### Standalone
460
+
461
+ If you've installed Resque as a gem running the front end standalone is easy:
462
+
463
+ $ resque-web
464
+
465
+ It's a thin layer around `rackup` so it's configurable as well:
466
+
467
+ $ resque-web -p 8282
468
+
469
+ If you have a Resque config file you want evaluated just pass it to
470
+ the script as the final argument:
471
+
472
+ $ resque-web -p 8282 rails_root/config/initializers/resque.rb
473
+
474
+ You can also set the namespace directly using `resque-web`:
475
+
476
+ $ resque-web -p 8282 -N myapp
477
+
478
+ ### Passenger
479
+
480
+ Using Passenger? Resque ships with a `config.ru` you can use. See
481
+ Phusion's guide:
482
+
483
+ <http://www.modrails.com/documentation/Users%20guide.html#_deploying_a_rack_based_ruby_application>
484
+
485
+ ### Rack::URLMap
486
+
487
+ If you want to load Resque on a subpath, possibly alongside other
488
+ apps, it's easy to do with Rack's `URLMap`:
489
+
490
+ require 'resque/server'
491
+
492
+ run Rack::URLMap.new \
493
+ "/" => Your::App.new,
494
+ "/resque" => Resque::Server.new
495
+
496
+ Check `examples/demo/config.ru` for a functional example (including
497
+ HTTP basic auth).
498
+
499
+
500
+ Resque vs DelayedJob
501
+ --------------------
502
+
503
+ How does Resque compare to DelayedJob, and why would you choose one
504
+ over the other?
505
+
506
+ * Resque supports multiple queues
507
+ * DelayedJob supports finer grained priorities
508
+ * Resque workers are resilient to memory leaks / bloat
509
+ * DelayedJob workers are extremely simple and easy to modify
510
+ * Resque requires Redis
511
+ * DelayedJob requires ActiveRecord
512
+ * Resque can only place JSONable Ruby objects on a queue as arguments
513
+ * DelayedJob can place _any_ Ruby object on its queue as arguments
514
+ * Resque includes a Sinatra app for monitoring what's going on
515
+ * DelayedJob can be queried from within your Rails app if you want to
516
+ add an interface
517
+
518
+ If you're doing Rails development, you already have a database and
519
+ ActiveRecord. DelayedJob is super easy to setup and works great.
520
+ GitHub used it for many months to process almost 200 million jobs.
521
+
522
+ Choose Resque if:
523
+
524
+ * You need multiple queues
525
+ * You don't care / dislike numeric priorities
526
+ * You don't need to persist every Ruby object ever
527
+ * You have potentially huge queues
528
+ * You want to see what's going on
529
+ * You expect a lot of failure / chaos
530
+ * You can setup Redis
531
+ * You're not running short on RAM
532
+
533
+ Choose DelayedJob if:
534
+
535
+ * You like numeric priorities
536
+ * You're not doing a gigantic amount of jobs each day
537
+ * Your queue stays small and nimble
538
+ * There is not a lot failure / chaos
539
+ * You want to easily throw anything on the queue
540
+ * You don't want to setup Redis
541
+
542
+ In no way is Resque a "better" DelayedJob, so make sure you pick the
543
+ tool that's best for your app.
544
+
545
+
546
+ Installing Redis
547
+ ----------------
548
+
549
+ Resque requires Redis 0.900 or higher.
550
+
551
+ Resque uses Redis' lists for its queues. It also stores worker state
552
+ data in Redis.
553
+
554
+ #### Homebrew
555
+
556
+ If you're on OS X, Homebrew is the simplest way to install Redis:
557
+
558
+ $ brew install redis
559
+ $ redis-server /usr/local/etc/redis.conf
560
+
561
+ You now have a Redis daemon running on 6379.
562
+
563
+ #### Via Resque
564
+
565
+ Resque includes Rake tasks (thanks to Ezra's redis-rb) that will
566
+ install and run Redis for you:
567
+
568
+ $ git clone git://github.com/defunkt/resque.git
569
+ $ cd resque
570
+ $ rake redis:install dtach:install
571
+ $ rake redis:start
572
+
573
+ Or, if you don't have admin access on your machine:
574
+
575
+ $ git clone git://github.com/defunkt/resque.git
576
+ $ cd resque
577
+ $ PREFIX=<your_prefix> rake redis:install dtach:install
578
+ $ rake redis:start
579
+
580
+ You now have Redis running on 6379. Wait a second then hit ctrl-\ to
581
+ detach and keep it running in the background.
582
+
583
+ The demo is probably the best way to figure out how to put the parts
584
+ together. But, it's not that hard.
585
+
586
+
587
+ Resque Dependencies
588
+ -------------------
589
+
590
+ gem install redis redis-namespace yajl-ruby
591
+
592
+ If you cannot install `yajl-ruby` (JRuby?), you can install the `json`
593
+ gem and Resque will use it instead.
594
+
595
+ When problems arise, make sure you have the newest versions of the
596
+ `redis` and `redis-namespace` gems.
597
+
598
+
599
+ Installing Resque
600
+ -----------------
601
+
602
+ ### In a Rack app, as a gem
603
+
604
+ First install the gem.
605
+
606
+ $ gem install resque
607
+
608
+ Next include it in your application.
609
+
610
+ require 'resque'
611
+
612
+ Now start your application:
613
+
614
+ rackup config.ru
615
+
616
+ That's it! You can now create Resque jobs from within your app.
617
+
618
+ To start a worker, create a Rakefile in your app's root (or add this
619
+ to an existing Rakefile):
620
+
621
+ require 'your/app'
622
+ require 'resque/tasks'
623
+
624
+ Now:
625
+
626
+ $ QUEUE=* rake resque:work
627
+
628
+ Alternately you can define a `resque:setup` hook in your Rakefile if you
629
+ don't want to load your app every time rake runs.
630
+
631
+
632
+ ### In a Rails app, as a gem
633
+
634
+ First install the gem.
635
+
636
+ $ gem install resque
637
+
638
+ Next include it in your application.
639
+
640
+ $ cat config/initializers/load_resque.rb
641
+ require 'resque'
642
+
643
+ Now start your application:
644
+
645
+ $ ./script/server
646
+
647
+ That's it! You can now create Resque jobs from within your app.
648
+
649
+ To start a worker, add this to your Rakefile in `RAILS_ROOT`:
650
+
651
+ require 'resque/tasks'
652
+
653
+ Now:
654
+
655
+ $ QUEUE=* rake environment resque:work
656
+
657
+ Don't forget you can define a `resque:setup` hook in
658
+ `lib/tasks/whatever.rake` that loads the `environment` task every time.
659
+
660
+
661
+ ### In a Rails app, as a plugin
662
+
663
+ $ ./script/plugin install git://github.com/defunkt/resque
664
+
665
+ That's it! Resque will automatically be available when your Rails app
666
+ loads.
667
+
668
+ To start a worker:
669
+
670
+ $ QUEUE=* rake environment resque:work
671
+
672
+ Don't forget you can define a `resque:setup` hook in
673
+ `lib/tasks/whatever.rake` that loads the `environment` task every time.
674
+
675
+
676
+ Configuration
677
+ -------------
678
+
679
+ You may want to change the Redis host and port Resque connects to, or
680
+ set various other options at startup.
681
+
682
+ Resque has a `redis` setter which can be given a string or a Redis
683
+ object. This means if you're already using Redis in your app, Resque
684
+ can re-use the existing connection.
685
+
686
+ String: `Resque.redis = 'localhost:6379'`
687
+
688
+ Redis: `Resque.redis = $redis`
689
+
690
+ For our rails app we have a `config/initializers/resque.rb` file where
691
+ we load `config/resque.yml` by hand and set the Redis information
692
+ appropriately.
693
+
694
+ Here's our `config/resque.yml`:
695
+
696
+ development: localhost:6379
697
+ test: localhost:6379
698
+ staging: redis1.se.github.com:6379
699
+ fi: localhost:6379
700
+ production: redis1.ae.github.com:6379
701
+
702
+ And our initializer:
703
+
704
+ rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..'
705
+ rails_env = ENV['RAILS_ENV'] || 'development'
706
+
707
+ resque_config = YAML.load_file(rails_root + '/config/resque.yml')
708
+ Resque.redis = resque_config[rails_env]
709
+
710
+ Easy peasy! Why not just use `RAILS_ROOT` and `RAILS_ENV`? Because
711
+ this way we can tell our Sinatra app about the config file:
712
+
713
+ $ RAILS_ENV=production resque-web rails_root/config/initializers/resque.rb
714
+
715
+ Now everyone is on the same page.
716
+
717
+
718
+ Plugins and Hooks
719
+ -----------------
720
+
721
+ For a list of available plugins see
722
+ <http://wiki.github.com/defunkt/resque/plugins>.
723
+
724
+ If you'd like to write your own plugin, or want to customize Resque
725
+ using hooks (such as `Resque.after_fork`), see
726
+ [docs/HOOKS.md](http://github.com/defunkt/resque/blob/master/docs/HOOKS.md).
727
+
728
+
729
+ Namespaces
730
+ ----------
731
+
732
+ If you're running multiple, separate instances of Resque you may want
733
+ to namespace the keyspaces so they do not overlap. This is not unlike
734
+ the approach taken by many memcached clients.
735
+
736
+ This feature is provided by the [redis-namespace][rs] library, which
737
+ Resque uses by default to separate the keys it manages from other keys
738
+ in your Redis server.
739
+
740
+ Simply use the `Resque.redis.namespace` accessor:
741
+
742
+ Resque.redis.namespace = "resque:GitHub"
743
+
744
+ We recommend sticking this in your initializer somewhere after Redis
745
+ is configured.
746
+
747
+
748
+ Demo
749
+ ----
750
+
751
+ Resque ships with a demo Sinatra app for creating jobs that are later
752
+ processed in the background.
753
+
754
+ Try it out by looking at the README, found at `examples/demo/README.markdown`.
755
+
756
+
757
+ Monitoring
758
+ ----------
759
+
760
+ ### god
761
+
762
+ If you're using god to monitor Resque, we have provided example
763
+ configs in `examples/god/`. One is for starting / stopping workers,
764
+ the other is for killing workers that have been running too long.
765
+
766
+ ### monit
767
+
768
+ If you're using monit, `examples/monit/resque.monit` is provided free
769
+ of charge. This is **not** used by GitHub in production, so please
770
+ send patches for any tweaks or improvements you can make to it.
771
+
772
+
773
+ Development
774
+ -----------
775
+
776
+ Want to hack on Resque?
777
+
778
+ First clone the repo and run the tests:
779
+
780
+ git clone git://github.com/defunkt/resque.git
781
+ cd resque
782
+ rake test
783
+
784
+ If the tests do not pass make sure you have Redis installed
785
+ correctly (though we make an effort to tell you if we feel this is the
786
+ case). The tests attempt to start an isolated instance of Redis to
787
+ run against.
788
+
789
+ Also make sure you've installed all the dependencies correctly. For
790
+ example, try loading the `redis-namespace` gem after you've installed
791
+ it:
792
+
793
+ $ irb
794
+ >> require 'rubygems'
795
+ => true
796
+ >> require 'redis/namespace'
797
+ => true
798
+
799
+ If you get an error requiring any of the dependencies, you may have
800
+ failed to install them or be seeing load path issues.
801
+
802
+ Feel free to ping the mailing list with your problem and we'll try to
803
+ sort it out.
804
+
805
+
806
+ Contributing
807
+ ------------
808
+
809
+ Once you've made your great commits:
810
+
811
+ 1. [Fork][1] Resque
812
+ 2. Create a topic branch - `git checkout -b my_branch`
813
+ 3. Push to your branch - `git push origin my_branch`
814
+ 4. Create an [Issue][2] with a link to your branch
815
+ 5. That's it!
816
+
817
+ You might want to checkout our [Contributing][cb] wiki page for information
818
+ on coding standards, new features, etc.
819
+
820
+
821
+ Mailing List
822
+ ------------
823
+
824
+ To join the list simply send an email to <resque@librelist.com>. This
825
+ will subscribe you and send you information about your subscription,
826
+ including unsubscribe information.
827
+
828
+ The archive can be found at <http://librelist.com/browser/resque/>.
829
+
830
+
831
+ Meta
832
+ ----
833
+
834
+ * Code: `git clone git://github.com/defunkt/resque.git`
835
+ * Home: <http://github.com/defunkt/resque>
836
+ * Docs: <http://defunkt.github.com/resque/>
837
+ * Bugs: <http://github.com/defunkt/resque/issues>
838
+ * List: <resque@librelist.com>
839
+ * Chat: <irc://irc.freenode.net/resque>
840
+ * Gems: <http://gemcutter.org/gems/resque>
841
+
842
+ This project uses [Semantic Versioning][sv].
843
+
844
+
845
+ Author
846
+ ------
847
+
848
+ Chris Wanstrath :: chris@ozmm.org :: @defunkt
849
+
850
+ [0]: http://github.com/blog/542-introducing-resque
851
+ [1]: http://help.github.com/forking/
852
+ [2]: http://github.com/defunkt/resque/issues
853
+ [sv]: http://semver.org/
854
+ [rs]: http://github.com/defunkt/redis-namespace
855
+ [cb]: http://wiki.github.com/defunkt/resque/contributing