resque-igo 1.1

Sign up to get free protection for your applications and to get access to all the features.
Files changed (50) hide show
  1. data/HISTORY.md +225 -0
  2. data/LICENSE +20 -0
  3. data/README.markdown +855 -0
  4. data/Rakefile +70 -0
  5. data/bin/resque +57 -0
  6. data/bin/resque-web +23 -0
  7. data/lib/resque.rb +380 -0
  8. data/lib/resque/errors.rb +10 -0
  9. data/lib/resque/failure.rb +66 -0
  10. data/lib/resque/failure/base.rb +61 -0
  11. data/lib/resque/failure/hoptoad.rb +132 -0
  12. data/lib/resque/failure/multiple.rb +50 -0
  13. data/lib/resque/failure/redis.rb +40 -0
  14. data/lib/resque/helpers.rb +71 -0
  15. data/lib/resque/job.rb +209 -0
  16. data/lib/resque/plugin.rb +51 -0
  17. data/lib/resque/server.rb +247 -0
  18. data/lib/resque/server/public/idle.png +0 -0
  19. data/lib/resque/server/public/jquery-1.3.2.min.js +19 -0
  20. data/lib/resque/server/public/jquery.relatize_date.js +95 -0
  21. data/lib/resque/server/public/poll.png +0 -0
  22. data/lib/resque/server/public/ranger.js +67 -0
  23. data/lib/resque/server/public/reset.css +48 -0
  24. data/lib/resque/server/public/style.css +86 -0
  25. data/lib/resque/server/public/working.png +0 -0
  26. data/lib/resque/server/test_helper.rb +19 -0
  27. data/lib/resque/server/views/error.erb +1 -0
  28. data/lib/resque/server/views/failed.erb +53 -0
  29. data/lib/resque/server/views/key_sets.erb +20 -0
  30. data/lib/resque/server/views/key_string.erb +11 -0
  31. data/lib/resque/server/views/layout.erb +42 -0
  32. data/lib/resque/server/views/next_more.erb +10 -0
  33. data/lib/resque/server/views/overview.erb +4 -0
  34. data/lib/resque/server/views/queues.erb +65 -0
  35. data/lib/resque/server/views/stats.erb +73 -0
  36. data/lib/resque/server/views/workers.erb +109 -0
  37. data/lib/resque/server/views/working.erb +68 -0
  38. data/lib/resque/stat.rb +54 -0
  39. data/lib/resque/tasks.rb +39 -0
  40. data/lib/resque/version.rb +3 -0
  41. data/lib/resque/worker.rb +478 -0
  42. data/tasks/resque.rake +2 -0
  43. data/test/job_hooks_test.rb +323 -0
  44. data/test/job_plugins_test.rb +230 -0
  45. data/test/plugin_test.rb +116 -0
  46. data/test/resque-web_test.rb +54 -0
  47. data/test/resque_test.rb +351 -0
  48. data/test/test_helper.rb +166 -0
  49. data/test/worker_test.rb +302 -0
  50. metadata +180 -0
data/HISTORY.md ADDED
@@ -0,0 +1,225 @@
1
+ ## 1.11.0 (2010-08-23)
2
+
3
+ * Web UI: Group /workers page by hostnames
4
+
5
+ ## 1.10.0 (2010-08-23)
6
+
7
+ * Support redis:// string format in `Resque.redis=`
8
+ * Using new cross-platform JSON gem.
9
+ * Added `after_enqueue` plugin hook.
10
+ * Added `shutdown?` method which can be overridden.
11
+ * Added support for the "leftright" gem when running tests.
12
+ * Grammarfix: In the README
13
+
14
+ ## 1.9.10 (2010-08-06)
15
+
16
+ * Bugfix: before_fork should get passed the job
17
+
18
+ ## 1.9.9 (2010-07-26)
19
+
20
+ * Depend on redis-namespace 0.8.0
21
+ * Depend on json_pure instead of json (for JRuby compat)
22
+ * Bugfix: rails_env display in stats view
23
+
24
+ ## 1.9.8 (2010-07-20)
25
+
26
+ * Bugfix: Worker.all should never return nil
27
+ * monit example: Fixed Syntax Error and adding environment to the rake task
28
+ * redis rake task: Fixed typo in copy command
29
+
30
+ ## 1.9.7 (2010-07-09)
31
+
32
+ * Improved memory usage in Job.destroy
33
+ * redis-namespace 0.7.0 now required
34
+ * Bugfix: Reverted $0 changes
35
+ * Web Bugfix: Payload-less failures in the web ui work
36
+
37
+ ## 1.9.6 (2010-06-22)
38
+
39
+ * Bugfix: Rakefile logging works the same as all the other logging
40
+
41
+ ## 1.9.5 (2010-06-16)
42
+
43
+ * Web Bugfix: Display the configured namespace on the stats page
44
+ * Revert Bugfix: Make ps -o more cross platform friendly
45
+
46
+ ## 1.9.4 (2010-06-14)
47
+
48
+ * Bugfix: Multiple failure backend gets exception information when created
49
+
50
+ ## 1.9.3 (2010-06-14)
51
+
52
+ * Bugfix: Resque#queues always returns an array
53
+
54
+ ## 1.9.2 (2010-06-13)
55
+
56
+ * Bugfix: Worker.all returning nil fix
57
+ * Bugfix: Make ps -o more cross platform friendly
58
+
59
+ ## 1.9.1 (2010-06-04)
60
+
61
+ * Less strict JSON dependency
62
+ * Included HISTORY.md in gem
63
+
64
+ ## 1.9.0 (2010-06-04)
65
+
66
+ * Redis 2 support
67
+ * Depend on redis-namespace 0.5.0
68
+ * Added Resque::VERSION constant (alias of Resque::Version)
69
+ * Bugfix: Specify JSON dependency
70
+ * Bugfix: Hoptoad plugin now works on 1.9
71
+
72
+ ## 1.8.5 (2010-05-18)
73
+
74
+ * Bugfix: Be more liberal in which Redis clients we accept.
75
+
76
+ ## 1.8.4 (2010-05-18)
77
+
78
+ * Try to resolve redis-namespace dependency issue
79
+
80
+ ## 1.8.3 (2010-05-17)
81
+
82
+ * Depend on redis-rb ~> 1.0.7
83
+
84
+ ## 1.8.2 (2010-05-03)
85
+
86
+ * Bugfix: Include "tasks/" dir in RubyGem
87
+
88
+ ## 1.8.1 (2010-04-29)
89
+
90
+ * Bugfix: Multiple failure backend did not support requeue-ing failed jobs
91
+ * Bugfix: Fix /failed when error has no backtrace
92
+ * Bugfix: Add `Redis::DistRedis` as a valid client
93
+
94
+ ## 1.8.0 (2010-04-07)
95
+
96
+ * Jobs that never complete due to killed worker are now failed.
97
+ * Worker "working" state is now maintained by the parent, not the child.
98
+ * Stopped using deprecated redis.rb methods
99
+ * `Worker.working` race condition fixed
100
+ * `Worker#process` has been deprecated.
101
+ * Monit example fixed
102
+ * Redis::Client and Redis::Namespace can be passed to `Resque.redis=`
103
+
104
+ ## 1.7.1 (2010-04-02)
105
+
106
+ * Bugfix: Make job hook execution order consistent
107
+ * Bugfix: stdout buffering in child process
108
+
109
+ ## 1.7.0 (2010-03-31)
110
+
111
+ * Job hooks API. See docs/HOOKS.md.
112
+ * web: Hovering over dates shows a timestamp
113
+ * web: AJAXify retry action for failed jobs
114
+ * web bugfix: Fix pagination bug
115
+
116
+ ## 1.6.1 (2010-03-25)
117
+
118
+ * Bugfix: Workers may not be clearing their state correctly on
119
+ shutdown
120
+ * Added example monit config.
121
+ * Exception class is now recorded when an error is raised in a
122
+ worker.
123
+ * web: Unit tests
124
+ * web: Show namespace in header and footer
125
+ * web: Remove a queue
126
+ * web: Retry failed jobs
127
+
128
+ ## 1.6.0 (2010-03-09)
129
+
130
+ * Added `before_first_fork`, `before_fork`, and `after_fork` hooks.
131
+ * Hoptoad: Added server_environment config setting
132
+ * Hoptoad bugfix: Don't depend on RAILS_ROOT
133
+ * 1.8.6 compat fixes
134
+
135
+ ## 1.5.2 (2010-03-03)
136
+
137
+ * Bugfix: JSON check was crazy.
138
+
139
+ ## 1.5.1 (2010-03-03)
140
+
141
+ * `Job.destroy` and `Resque.dequeue` return the # of destroyed jobs.
142
+ * Hoptoad notifier improvements
143
+ * Specify the namespace with `resque-web` by passing `-N namespace`
144
+ * Bugfix: Don't crash when trying to parse invalid JSON.
145
+ * Bugfix: Non-standard namespace support
146
+ * Web: Red backgound for queue "failed" only shown if there are failed jobs.
147
+ * Web bugfix: Tabs highlight properly now
148
+ * Web bugfix: ZSET partial support in stats
149
+ * Web bugfix: Deleting failed jobs works again
150
+ * Web bugfix: Sets (or zsets, lists, etc) now paginate.
151
+
152
+ ## 1.5.0 (2010-02-17)
153
+
154
+ * Version now included in procline, e.g. `resque-1.5.0: Message`
155
+ * Web bugfix: Ignore idle works in the "working" page
156
+ * Added `Resque::Job.destroy(queue, klass, *args)`
157
+ * Added `Resque.dequeue(klass, *args)`
158
+
159
+ ## 1.4.0 (2010-02-11)
160
+
161
+ * Fallback when unable to bind QUIT and USR1 for Windows and JRuby.
162
+ * Fallback when no `Kernel.fork` is provided (for IronRuby).
163
+ * Web: Rounded corners in Firefox
164
+ * Cut down system calls in `Worker#prune_dead_workers`
165
+ * Enable switching DB in a Redis server from config
166
+ * Support USR2 and CONT to stop and start job processing.
167
+ * Web: Add example failing job
168
+ * Bugfix: `Worker#unregister_worker` shouldn't call `done_working`
169
+ * Bugfix: Example god config now restarts Resque properly.
170
+ * Multiple failure backends now permitted.
171
+ * Hoptoad failure backend updated to new API
172
+
173
+ ## 1.3.1 (2010-01-11)
174
+
175
+ * Vegas bugfix: Don't error without a config
176
+
177
+ ## 1.3.0 (2010-01-11)
178
+
179
+ * Use Vegas for resque-web
180
+ * Web Bugfix: Show proper date/time value for failed_at on Failures
181
+ * Web Bugfix: Make the / route more flexible
182
+ * Add Resque::Server.tabs array (so plugins can add their own tabs)
183
+ * Start using [Semantic Versioning](http://semver.org/)
184
+
185
+ ## 1.2.4 (2009-12-15)
186
+
187
+ * Web Bugfix: fix key links on stat page
188
+
189
+ ## 1.2.3 (2009-12-15)
190
+
191
+ * Bugfix: Fixed `rand` seeding in child processes.
192
+ * Bugfix: Better JSON encoding/decoding without Yajl.
193
+ * Bugfix: Avoid `ps` flag error on Linux
194
+ * Add `PREFIX` observance to `rake` install tasks.
195
+
196
+ ## 1.2.2 (2009-12-08)
197
+
198
+ * Bugfix: Job equality was not properly implemented.
199
+
200
+ ## 1.2.1 (2009-12-07)
201
+
202
+ * Added `rake resque:workers` task for starting multiple workers.
203
+ * 1.9.x compatibility
204
+ * Bugfix: Yajl decoder doesn't care about valid UTF-8
205
+ * config.ru loads RESQUECONFIG if the ENV variable is set.
206
+ * `resque-web` now sets RESQUECONFIG
207
+ * Job objects know if they are equal.
208
+ * Jobs can be re-queued using `Job#recreate`
209
+
210
+ ## 1.2.0 (2009-11-25)
211
+
212
+ * If USR1 is sent and no child is found, shutdown.
213
+ * Raise when a job class does not respond to `perform`.
214
+ * Added `Resque.remove_queue` for deleting a queue
215
+
216
+ ## 1.1.0 (2009-11-04)
217
+
218
+ * Bugfix: Broken ERB tag in failure UI
219
+ * Bugfix: Save the worker's ID, not the worker itself, in the failure module
220
+ * Redesigned the sinatra web interface
221
+ * Added option to clear failed jobs
222
+
223
+ ## 1.0.0 (2009-11-03)
224
+
225
+ * First release.
data/LICENSE ADDED
@@ -0,0 +1,20 @@
1
+ Copyright (c) 2009 Chris Wanstrath
2
+
3
+ Permission is hereby granted, free of charge, to any person obtaining
4
+ a copy of this software and associated documentation files (the
5
+ "Software"), to deal in the Software without restriction, including
6
+ without limitation the rights to use, copy, modify, merge, publish,
7
+ distribute, sublicense, and/or sell copies of the Software, and to
8
+ permit persons to whom the Software is furnished to do so, subject to
9
+ the following conditions:
10
+
11
+ The above copyright notice and this permission notice shall be
12
+ included in all copies or substantial portions of the Software.
13
+
14
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
15
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
16
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
17
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
18
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
19
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
20
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
data/README.markdown ADDED
@@ -0,0 +1,855 @@
1
+ Resque
2
+ ======
3
+
4
+ Resque is a Redis-backed library for creating background jobs, placing
5
+ those jobs on multiple queues, and processing them later.
6
+
7
+ Background jobs can be any Ruby class or module that responds to
8
+ `perform`. Your existing classes can easily be converted to background
9
+ jobs or you can create new classes specifically to do work. Or, you
10
+ can do both.
11
+
12
+ Resque is heavily inspired by DelayedJob (which rocks) and comprises
13
+ three parts:
14
+
15
+ 1. A Ruby library for creating, querying, and processing jobs
16
+ 2. A Rake task for starting a worker which processes jobs
17
+ 3. A Sinatra app for monitoring queues, jobs, and workers.
18
+
19
+ Resque workers can be distributed between multiple machines,
20
+ support priorities, are resilient to memory bloat / "leaks," are
21
+ optimized for REE (but work on MRI and JRuby), tell you what they're
22
+ doing, and expect failure.
23
+
24
+ Resque queues are persistent; support constant time, atomic push and
25
+ pop (thanks to Redis); provide visibility into their contents; and
26
+ store jobs as simple JSON packages.
27
+
28
+ The Resque frontend tells you what workers are doing, what workers are
29
+ not doing, what queues you're using, what's in those queues, provides
30
+ general usage stats, and helps you track failures.
31
+
32
+
33
+ What did you guys do to Resque?
34
+ ===============================
35
+
36
+ This version of Resque uses mongo as the back end. It also has some neat
37
+ new features that you may enjoy
38
+
39
+ Features
40
+ --------
41
+
42
+ It stores each queue in its own collection.
43
+
44
+ For testing purposes. Resque.bypass_queues will call your job's
45
+ perform method AT ONCE, and not touch the queue at all. This can be
46
+ useful when writing unit tests against things that would normally
47
+ happen partially in your request and partially in a worker.
48
+
49
+ If your job class has a variable called @unique_jobs = true, you
50
+ can queue unique jobs - jobs that must only be processed once. In
51
+ order to take advantage of this feature, the arguments to your enqueue
52
+ call must include a hash containing a key called _id in the first
53
+ position.
54
+
55
+ If your job class indicates that @delayed_jobs = true, you can queue
56
+ delayed jobs. These jobs will not be popped off the queue until the
57
+ Time indicated in arg[0][:delay_until] has come. Note that you must
58
+ call Resque.enable_delay(:queue) before enququing any delayed jobs, to
59
+ ensure that the performance impact on other queues is minimal.
60
+
61
+ Stern Warnings
62
+ --------------
63
+
64
+ Sometimes, Resque-Mongo will drop a queue collection, or create some
65
+ indexes, or otherwise manipulate its database. For this reason, it is
66
+ STRONGLY recommended that you give it its own database in mongo.
67
+
68
+ All jobs should be queued via Resque.enqueue. All arguments passed to
69
+ this method must be BSON-encodable. Resque-Mongo does not serialize
70
+ your objects for you. Arrays, Hashes, Strings, Numbers, and Times
71
+ are all ok, so don't worry.
72
+
73
+ Many of the new queue-level features require the first argument of
74
+ your perform method to be an options hash. In fact, if you just start
75
+ making all your perform()s take one param, that is an options hash,
76
+ you'll probably save yourself some pain.
77
+
78
+ Resque-Mongo will not create any indexes on your queues, only on its
79
+ meta-data. You will need to create any indexes you want. Normally,
80
+ This is not a problem, because you aren't querying by keys, but you may
81
+ want to create indexes on the class key in some circumstances. If you
82
+ use the unique or delay features, you may want some additional indexes,
83
+ depending on the nature of your workload. Paranoid? Test enqueuing and
84
+ processing all your jobs, and run with --notablescans. Learn the profiler,
85
+ and use it often.
86
+
87
+ Specifically, a queue with many long-delayed jobs will result in slower queue pops
88
+ for all jobs using that queue. Index delay_until in the case of
89
+ thousands of delayed jobs.
90
+
91
+ Back to the original README
92
+ ===========================
93
+
94
+ The Blog Post
95
+ -------------
96
+
97
+ For the backstory, philosophy, and history of Resque's beginnings,
98
+ please see [the blog post][0].
99
+
100
+
101
+ Overview
102
+ --------
103
+
104
+ Resque allows you to create jobs and place them on a queue, then,
105
+ later, pull those jobs off the queue and process them.
106
+
107
+ Resque jobs are Ruby classes (or modules) which respond to the
108
+ `perform` method. Here's an example:
109
+
110
+ class Archive
111
+ @queue = :file_serve
112
+
113
+ def self.perform(repo_id, branch = 'master')
114
+ repo = Repository.find(repo_id)
115
+ repo.create_archive(branch)
116
+ end
117
+ end
118
+
119
+ The `@queue` class instance variable determines which queue `Archive`
120
+ jobs will be placed in. Queues are arbitrary and created on the fly -
121
+ you can name them whatever you want and have as many as you want.
122
+
123
+ To place an `Archive` job on the `file_serve` queue, we might add this
124
+ to our application's pre-existing `Repository` class:
125
+
126
+ class Repository
127
+ def async_create_archive(branch)
128
+ Resque.enqueue(Archive, self.id, branch)
129
+ end
130
+ end
131
+
132
+ Now when we call `repo.async_create_archive('masterbrew')` in our
133
+ application, a job will be created and placed on the `file_serve`
134
+ queue.
135
+
136
+ Later, a worker will run something like this code to process the job:
137
+
138
+ klass, args = Resque.reserve(:file_serve)
139
+ klass.perform(*args) if klass.respond_to? :perform
140
+
141
+ Which translates to:
142
+
143
+ Archive.perform(44, 'masterbrew')
144
+
145
+ Let's start a worker to run `file_serve` jobs:
146
+
147
+ $ cd app_root
148
+ $ QUEUE=file_serve rake resque:work
149
+
150
+ This starts one Resque worker and tells it to work off the
151
+ `file_serve` queue. As soon as it's ready it'll try to run the
152
+ `Resque.reserve` code snippet above and process jobs until it can't
153
+ find any more, at which point it will sleep for a small period and
154
+ repeatedly poll the queue for more jobs.
155
+
156
+ Workers can be given multiple queues (a "queue list") and run on
157
+ multiple machines. In fact they can be run anywhere with network
158
+ access to the Redis server.
159
+
160
+
161
+ Jobs
162
+ ----
163
+
164
+ What should you run in the background? Anything that takes any time at
165
+ all. Slow INSERT statements, disk manipulating, data processing, etc.
166
+
167
+ At GitHub we use Resque to process the following types of jobs:
168
+
169
+ * Warming caches
170
+ * Counting disk usage
171
+ * Building tarballs
172
+ * Building Rubygems
173
+ * Firing off web hooks
174
+ * Creating events in the db and pre-caching them
175
+ * Building graphs
176
+ * Deleting users
177
+ * Updating our search index
178
+
179
+ As of writing we have about 35 different types of background jobs.
180
+
181
+ Keep in mind that you don't need a web app to use Resque - we just
182
+ mention "foreground" and "background" because they make conceptual
183
+ sense. You could easily be spidering sites and sticking data which
184
+ needs to be crunched later into a queue.
185
+
186
+
187
+ ### Persistence
188
+
189
+ Jobs are persisted to queues as JSON objects. Let's take our `Archive`
190
+ example from above. We'll run the following code to create a job:
191
+
192
+ repo = Repository.find(44)
193
+ repo.async_create_archive('masterbrew')
194
+
195
+ The following JSON will be stored in the `file_serve` queue:
196
+
197
+ {
198
+ 'class': 'Archive',
199
+ 'args': [ 44, 'masterbrew' ]
200
+ }
201
+
202
+ Because of this your jobs must only accept arguments that can be JSON encoded.
203
+
204
+ So instead of doing this:
205
+
206
+ Resque.enqueue(Archive, self, branch)
207
+
208
+ do this:
209
+
210
+ Resque.enqueue(Archive, self.id, branch)
211
+
212
+ This is why our above example (and all the examples in `examples/`)
213
+ uses object IDs instead of passing around the objects.
214
+
215
+ While this is less convenient than just sticking a marshaled object
216
+ in the database, it gives you a slight advantage: your jobs will be
217
+ run against the most recent version of an object because they need to
218
+ pull from the DB or cache.
219
+
220
+ If your jobs were run against marshaled objects, they could
221
+ potentially be operating on a stale record with out-of-date information.
222
+
223
+
224
+ ### send_later / async
225
+
226
+ Want something like DelayedJob's `send_later` or the ability to use
227
+ instance methods instead of just methods for jobs? See the `examples/`
228
+ directory for goodies.
229
+
230
+ We plan to provide first class `async` support in a future release.
231
+
232
+
233
+ ### Failure
234
+
235
+ If a job raises an exception, it is logged and handed off to the
236
+ `Resque::Failure` module. Failures are logged either locally in Redis
237
+ or using some different backend.
238
+
239
+ For example, Resque ships with Hoptoad support.
240
+
241
+ Keep this in mind when writing your jobs: you may want to throw
242
+ exceptions you would not normally throw in order to assist debugging.
243
+
244
+
245
+ Workers
246
+ -------
247
+
248
+ Resque workers are rake tasks that run forever. They basically do this:
249
+
250
+ start
251
+ loop do
252
+ if job = reserve
253
+ job.process
254
+ else
255
+ sleep 5
256
+ end
257
+ end
258
+ shutdown
259
+
260
+ Starting a worker is simple. Here's our example from earlier:
261
+
262
+ $ QUEUE=file_serve rake resque:work
263
+
264
+ By default Resque won't know about your application's
265
+ environment. That is, it won't be able to find and run your jobs - it
266
+ needs to load your application into memory.
267
+
268
+ If we've installed Resque as a Rails plugin, we might run this command
269
+ from our RAILS_ROOT:
270
+
271
+ $ QUEUE=file_serve rake environment resque:work
272
+
273
+ This will load the environment before starting a worker. Alternately
274
+ we can define a `resque:setup` task with a dependency on the
275
+ `environment` rake task:
276
+
277
+ task "resque:setup" => :environment
278
+
279
+ GitHub's setup task looks like this:
280
+
281
+ task "resque:setup" => :environment do
282
+ Grit::Git.git_timeout = 10.minutes
283
+ end
284
+
285
+ We don't want the `git_timeout` as high as 10 minutes in our web app,
286
+ but in the Resque workers it's fine.
287
+
288
+
289
+ ### Logging
290
+
291
+ Workers support basic logging to STDOUT. If you start them with the
292
+ `VERBOSE` env variable set, they will print basic debugging
293
+ information. You can also set the `VVERBOSE` (very verbose) env
294
+ variable.
295
+
296
+ $ VVERBOSE=1 QUEUE=file_serve rake environment resque:work
297
+
298
+
299
+ ### Priorities and Queue Lists
300
+
301
+ Resque doesn't support numeric priorities but instead uses the order
302
+ of queues you give it. We call this list of queues the "queue list."
303
+
304
+ Let's say we add a `warm_cache` queue in addition to our `file_serve`
305
+ queue. We'd now start a worker like so:
306
+
307
+ $ QUEUES=file_serve,warm_cache rake resque:work
308
+
309
+ When the worker looks for new jobs, it will first check
310
+ `file_serve`. If it finds a job, it'll process it then check
311
+ `file_serve` again. It will keep checking `file_serve` until no more
312
+ jobs are available. At that point, it will check `warm_cache`. If it
313
+ finds a job it'll process it then check `file_serve` (repeating the
314
+ whole process).
315
+
316
+ In this way you can prioritize certain queues. At GitHub we start our
317
+ workers with something like this:
318
+
319
+ $ QUEUES=critical,archive,high,low rake resque:work
320
+
321
+ Notice the `archive` queue - it is specialized and in our future
322
+ architecture will only be run from a single machine.
323
+
324
+ At that point we'll start workers on our generalized background
325
+ machines with this command:
326
+
327
+ $ QUEUES=critical,high,low rake resque:work
328
+
329
+ And workers on our specialized archive machine with this command:
330
+
331
+ $ QUEUE=archive rake resque:work
332
+
333
+
334
+ ### Running All Queues
335
+
336
+ If you want your workers to work off of every queue, including new
337
+ queues created on the fly, you can use a splat:
338
+
339
+ $ QUEUE=* rake resque:work
340
+
341
+ Queues will be processed in alphabetical order.
342
+
343
+
344
+ ### Running Multiple Workers
345
+
346
+ At GitHub we use god to start and stop multiple workers. A sample god
347
+ configuration file is included under `examples/god`. We recommend this
348
+ method.
349
+
350
+ If you'd like to run multiple workers in development mode, you can do
351
+ so using the `resque:workers` rake task:
352
+
353
+ $ COUNT=5 QUEUE=* rake resque:workers
354
+
355
+ This will spawn five Resque workers, each in its own thread. Hitting
356
+ ctrl-c should be sufficient to stop them all.
357
+
358
+
359
+ ### Forking
360
+
361
+ On certain platforms, when a Resque worker reserves a job it
362
+ immediately forks a child process. The child processes the job then
363
+ exits. When the child has exited successfully, the worker reserves
364
+ another job and repeats the process.
365
+
366
+ Why?
367
+
368
+ Because Resque assumes chaos.
369
+
370
+ Resque assumes your background workers will lock up, run too long, or
371
+ have unwanted memory growth.
372
+
373
+ If Resque workers processed jobs themselves, it'd be hard to whip them
374
+ into shape. Let's say one is using too much memory: you send it a
375
+ signal that says "shutdown after you finish processing the current
376
+ job," and it does so. It then starts up again - loading your entire
377
+ application environment. This adds useless CPU cycles and causes a
378
+ delay in queue processing.
379
+
380
+ Plus, what if it's using too much memory and has stopped responding to
381
+ signals?
382
+
383
+ Thanks to Resque's parent / child architecture, jobs that use too much memory
384
+ release that memory upon completion. No unwanted growth.
385
+
386
+ And what if a job is running too long? You'd need to `kill -9` it then
387
+ start the worker again. With Resque's parent / child architecture you
388
+ can tell the parent to forcefully kill the child then immediately
389
+ start processing more jobs. No startup delay or wasted cycles.
390
+
391
+ The parent / child architecture helps us keep tabs on what workers are
392
+ doing, too. By eliminating the need to `kill -9` workers we can have
393
+ parents remove themselves from the global listing of workers. If we
394
+ just ruthlessly killed workers, we'd need a separate watchdog process
395
+ to add and remove them to the global listing - which becomes
396
+ complicated.
397
+
398
+ Workers instead handle their own state.
399
+
400
+
401
+ ### Parents and Children
402
+
403
+ Here's a parent / child pair doing some work:
404
+
405
+ $ ps -e -o pid,command | grep [r]esque
406
+ 92099 resque: Forked 92102 at 1253142769
407
+ 92102 resque: Processing file_serve since 1253142769
408
+
409
+ You can clearly see that process 92099 forked 92102, which has been
410
+ working since 1253142769.
411
+
412
+ (By advertising the time they began processing you can easily use monit
413
+ or god to kill stale workers.)
414
+
415
+ When a parent process is idle, it lets you know what queues it is
416
+ waiting for work on:
417
+
418
+ $ ps -e -o pid,command | grep [r]esque
419
+ 92099 resque: Waiting for file_serve,warm_cache
420
+
421
+
422
+ ### Signals
423
+
424
+ Resque workers respond to a few different signals:
425
+
426
+ * `QUIT` - Wait for child to finish processing then exit
427
+ * `TERM` / `INT` - Immediately kill child then exit
428
+ * `USR1` - Immediately kill child but don't exit
429
+ * `USR2` - Don't start to process any new jobs
430
+ * `CONT` - Start to process new jobs again after a USR2
431
+
432
+ If you want to gracefully shutdown a Resque worker, use `QUIT`.
433
+
434
+ If you want to kill a stale or stuck child, use `USR1`. Processing
435
+ will continue as normal unless the child was not found. In that case
436
+ Resque assumes the parent process is in a bad state and shuts down.
437
+
438
+ If you want to kill a stale or stuck child and shutdown, use `TERM`
439
+
440
+ If you want to stop processing jobs, but want to leave the worker running
441
+ (for example, to temporarily alleviate load), use `USR2` to stop processing,
442
+ then `CONT` to start it again.
443
+
444
+ ### Mysql::Error: MySQL server has gone away
445
+
446
+ If your workers remain idle for too long they may lose their MySQL
447
+ connection. If that happens we recommend using [this
448
+ Gist](http://gist.github.com/238999).
449
+
450
+
451
+ The Front End
452
+ -------------
453
+
454
+ Resque comes with a Sinatra-based front end for seeing what's up with
455
+ your queue.
456
+
457
+ ![The Front End](http://img.skitch.com/20091104-tqh5pgkwgbskjbk7qbtmpesnyw.jpg)
458
+
459
+ ### Standalone
460
+
461
+ If you've installed Resque as a gem running the front end standalone is easy:
462
+
463
+ $ resque-web
464
+
465
+ It's a thin layer around `rackup` so it's configurable as well:
466
+
467
+ $ resque-web -p 8282
468
+
469
+ If you have a Resque config file you want evaluated just pass it to
470
+ the script as the final argument:
471
+
472
+ $ resque-web -p 8282 rails_root/config/initializers/resque.rb
473
+
474
+ You can also set the namespace directly using `resque-web`:
475
+
476
+ $ resque-web -p 8282 -N myapp
477
+
478
+ ### Passenger
479
+
480
+ Using Passenger? Resque ships with a `config.ru` you can use. See
481
+ Phusion's guide:
482
+
483
+ <http://www.modrails.com/documentation/Users%20guide.html#_deploying_a_rack_based_ruby_application>
484
+
485
+ ### Rack::URLMap
486
+
487
+ If you want to load Resque on a subpath, possibly alongside other
488
+ apps, it's easy to do with Rack's `URLMap`:
489
+
490
+ require 'resque/server'
491
+
492
+ run Rack::URLMap.new \
493
+ "/" => Your::App.new,
494
+ "/resque" => Resque::Server.new
495
+
496
+ Check `examples/demo/config.ru` for a functional example (including
497
+ HTTP basic auth).
498
+
499
+
500
+ Resque vs DelayedJob
501
+ --------------------
502
+
503
+ How does Resque compare to DelayedJob, and why would you choose one
504
+ over the other?
505
+
506
+ * Resque supports multiple queues
507
+ * DelayedJob supports finer grained priorities
508
+ * Resque workers are resilient to memory leaks / bloat
509
+ * DelayedJob workers are extremely simple and easy to modify
510
+ * Resque requires Redis
511
+ * DelayedJob requires ActiveRecord
512
+ * Resque can only place JSONable Ruby objects on a queue as arguments
513
+ * DelayedJob can place _any_ Ruby object on its queue as arguments
514
+ * Resque includes a Sinatra app for monitoring what's going on
515
+ * DelayedJob can be queried from within your Rails app if you want to
516
+ add an interface
517
+
518
+ If you're doing Rails development, you already have a database and
519
+ ActiveRecord. DelayedJob is super easy to setup and works great.
520
+ GitHub used it for many months to process almost 200 million jobs.
521
+
522
+ Choose Resque if:
523
+
524
+ * You need multiple queues
525
+ * You don't care / dislike numeric priorities
526
+ * You don't need to persist every Ruby object ever
527
+ * You have potentially huge queues
528
+ * You want to see what's going on
529
+ * You expect a lot of failure / chaos
530
+ * You can setup Redis
531
+ * You're not running short on RAM
532
+
533
+ Choose DelayedJob if:
534
+
535
+ * You like numeric priorities
536
+ * You're not doing a gigantic amount of jobs each day
537
+ * Your queue stays small and nimble
538
+ * There is not a lot failure / chaos
539
+ * You want to easily throw anything on the queue
540
+ * You don't want to setup Redis
541
+
542
+ In no way is Resque a "better" DelayedJob, so make sure you pick the
543
+ tool that's best for your app.
544
+
545
+
546
+ Installing Redis
547
+ ----------------
548
+
549
+ Resque requires Redis 0.900 or higher.
550
+
551
+ Resque uses Redis' lists for its queues. It also stores worker state
552
+ data in Redis.
553
+
554
+ #### Homebrew
555
+
556
+ If you're on OS X, Homebrew is the simplest way to install Redis:
557
+
558
+ $ brew install redis
559
+ $ redis-server /usr/local/etc/redis.conf
560
+
561
+ You now have a Redis daemon running on 6379.
562
+
563
+ #### Via Resque
564
+
565
+ Resque includes Rake tasks (thanks to Ezra's redis-rb) that will
566
+ install and run Redis for you:
567
+
568
+ $ git clone git://github.com/defunkt/resque.git
569
+ $ cd resque
570
+ $ rake redis:install dtach:install
571
+ $ rake redis:start
572
+
573
+ Or, if you don't have admin access on your machine:
574
+
575
+ $ git clone git://github.com/defunkt/resque.git
576
+ $ cd resque
577
+ $ PREFIX=<your_prefix> rake redis:install dtach:install
578
+ $ rake redis:start
579
+
580
+ You now have Redis running on 6379. Wait a second then hit ctrl-\ to
581
+ detach and keep it running in the background.
582
+
583
+ The demo is probably the best way to figure out how to put the parts
584
+ together. But, it's not that hard.
585
+
586
+
587
+ Resque Dependencies
588
+ -------------------
589
+
590
+ gem install redis redis-namespace yajl-ruby
591
+
592
+ If you cannot install `yajl-ruby` (JRuby?), you can install the `json`
593
+ gem and Resque will use it instead.
594
+
595
+ When problems arise, make sure you have the newest versions of the
596
+ `redis` and `redis-namespace` gems.
597
+
598
+
599
+ Installing Resque
600
+ -----------------
601
+
602
+ ### In a Rack app, as a gem
603
+
604
+ First install the gem.
605
+
606
+ $ gem install resque
607
+
608
+ Next include it in your application.
609
+
610
+ require 'resque'
611
+
612
+ Now start your application:
613
+
614
+ rackup config.ru
615
+
616
+ That's it! You can now create Resque jobs from within your app.
617
+
618
+ To start a worker, create a Rakefile in your app's root (or add this
619
+ to an existing Rakefile):
620
+
621
+ require 'your/app'
622
+ require 'resque/tasks'
623
+
624
+ Now:
625
+
626
+ $ QUEUE=* rake resque:work
627
+
628
+ Alternately you can define a `resque:setup` hook in your Rakefile if you
629
+ don't want to load your app every time rake runs.
630
+
631
+
632
+ ### In a Rails app, as a gem
633
+
634
+ First install the gem.
635
+
636
+ $ gem install resque
637
+
638
+ Next include it in your application.
639
+
640
+ $ cat config/initializers/load_resque.rb
641
+ require 'resque'
642
+
643
+ Now start your application:
644
+
645
+ $ ./script/server
646
+
647
+ That's it! You can now create Resque jobs from within your app.
648
+
649
+ To start a worker, add this to your Rakefile in `RAILS_ROOT`:
650
+
651
+ require 'resque/tasks'
652
+
653
+ Now:
654
+
655
+ $ QUEUE=* rake environment resque:work
656
+
657
+ Don't forget you can define a `resque:setup` hook in
658
+ `lib/tasks/whatever.rake` that loads the `environment` task every time.
659
+
660
+
661
+ ### In a Rails app, as a plugin
662
+
663
+ $ ./script/plugin install git://github.com/defunkt/resque
664
+
665
+ That's it! Resque will automatically be available when your Rails app
666
+ loads.
667
+
668
+ To start a worker:
669
+
670
+ $ QUEUE=* rake environment resque:work
671
+
672
+ Don't forget you can define a `resque:setup` hook in
673
+ `lib/tasks/whatever.rake` that loads the `environment` task every time.
674
+
675
+
676
+ Configuration
677
+ -------------
678
+
679
+ You may want to change the Redis host and port Resque connects to, or
680
+ set various other options at startup.
681
+
682
+ Resque has a `redis` setter which can be given a string or a Redis
683
+ object. This means if you're already using Redis in your app, Resque
684
+ can re-use the existing connection.
685
+
686
+ String: `Resque.redis = 'localhost:6379'`
687
+
688
+ Redis: `Resque.redis = $redis`
689
+
690
+ For our rails app we have a `config/initializers/resque.rb` file where
691
+ we load `config/resque.yml` by hand and set the Redis information
692
+ appropriately.
693
+
694
+ Here's our `config/resque.yml`:
695
+
696
+ development: localhost:6379
697
+ test: localhost:6379
698
+ staging: redis1.se.github.com:6379
699
+ fi: localhost:6379
700
+ production: redis1.ae.github.com:6379
701
+
702
+ And our initializer:
703
+
704
+ rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..'
705
+ rails_env = ENV['RAILS_ENV'] || 'development'
706
+
707
+ resque_config = YAML.load_file(rails_root + '/config/resque.yml')
708
+ Resque.redis = resque_config[rails_env]
709
+
710
+ Easy peasy! Why not just use `RAILS_ROOT` and `RAILS_ENV`? Because
711
+ this way we can tell our Sinatra app about the config file:
712
+
713
+ $ RAILS_ENV=production resque-web rails_root/config/initializers/resque.rb
714
+
715
+ Now everyone is on the same page.
716
+
717
+
718
+ Plugins and Hooks
719
+ -----------------
720
+
721
+ For a list of available plugins see
722
+ <http://wiki.github.com/defunkt/resque/plugins>.
723
+
724
+ If you'd like to write your own plugin, or want to customize Resque
725
+ using hooks (such as `Resque.after_fork`), see
726
+ [docs/HOOKS.md](http://github.com/defunkt/resque/blob/master/docs/HOOKS.md).
727
+
728
+
729
+ Namespaces
730
+ ----------
731
+
732
+ If you're running multiple, separate instances of Resque you may want
733
+ to namespace the keyspaces so they do not overlap. This is not unlike
734
+ the approach taken by many memcached clients.
735
+
736
+ This feature is provided by the [redis-namespace][rs] library, which
737
+ Resque uses by default to separate the keys it manages from other keys
738
+ in your Redis server.
739
+
740
+ Simply use the `Resque.redis.namespace` accessor:
741
+
742
+ Resque.redis.namespace = "resque:GitHub"
743
+
744
+ We recommend sticking this in your initializer somewhere after Redis
745
+ is configured.
746
+
747
+
748
+ Demo
749
+ ----
750
+
751
+ Resque ships with a demo Sinatra app for creating jobs that are later
752
+ processed in the background.
753
+
754
+ Try it out by looking at the README, found at `examples/demo/README.markdown`.
755
+
756
+
757
+ Monitoring
758
+ ----------
759
+
760
+ ### god
761
+
762
+ If you're using god to monitor Resque, we have provided example
763
+ configs in `examples/god/`. One is for starting / stopping workers,
764
+ the other is for killing workers that have been running too long.
765
+
766
+ ### monit
767
+
768
+ If you're using monit, `examples/monit/resque.monit` is provided free
769
+ of charge. This is **not** used by GitHub in production, so please
770
+ send patches for any tweaks or improvements you can make to it.
771
+
772
+
773
+ Development
774
+ -----------
775
+
776
+ Want to hack on Resque?
777
+
778
+ First clone the repo and run the tests:
779
+
780
+ git clone git://github.com/defunkt/resque.git
781
+ cd resque
782
+ rake test
783
+
784
+ If the tests do not pass make sure you have Redis installed
785
+ correctly (though we make an effort to tell you if we feel this is the
786
+ case). The tests attempt to start an isolated instance of Redis to
787
+ run against.
788
+
789
+ Also make sure you've installed all the dependencies correctly. For
790
+ example, try loading the `redis-namespace` gem after you've installed
791
+ it:
792
+
793
+ $ irb
794
+ >> require 'rubygems'
795
+ => true
796
+ >> require 'redis/namespace'
797
+ => true
798
+
799
+ If you get an error requiring any of the dependencies, you may have
800
+ failed to install them or be seeing load path issues.
801
+
802
+ Feel free to ping the mailing list with your problem and we'll try to
803
+ sort it out.
804
+
805
+
806
+ Contributing
807
+ ------------
808
+
809
+ Once you've made your great commits:
810
+
811
+ 1. [Fork][1] Resque
812
+ 2. Create a topic branch - `git checkout -b my_branch`
813
+ 3. Push to your branch - `git push origin my_branch`
814
+ 4. Create an [Issue][2] with a link to your branch
815
+ 5. That's it!
816
+
817
+ You might want to checkout our [Contributing][cb] wiki page for information
818
+ on coding standards, new features, etc.
819
+
820
+
821
+ Mailing List
822
+ ------------
823
+
824
+ To join the list simply send an email to <resque@librelist.com>. This
825
+ will subscribe you and send you information about your subscription,
826
+ including unsubscribe information.
827
+
828
+ The archive can be found at <http://librelist.com/browser/resque/>.
829
+
830
+
831
+ Meta
832
+ ----
833
+
834
+ * Code: `git clone git://github.com/defunkt/resque.git`
835
+ * Home: <http://github.com/defunkt/resque>
836
+ * Docs: <http://defunkt.github.com/resque/>
837
+ * Bugs: <http://github.com/defunkt/resque/issues>
838
+ * List: <resque@librelist.com>
839
+ * Chat: <irc://irc.freenode.net/resque>
840
+ * Gems: <http://gemcutter.org/gems/resque>
841
+
842
+ This project uses [Semantic Versioning][sv].
843
+
844
+
845
+ Author
846
+ ------
847
+
848
+ Chris Wanstrath :: chris@ozmm.org :: @defunkt
849
+
850
+ [0]: http://github.com/blog/542-introducing-resque
851
+ [1]: http://help.github.com/forking/
852
+ [2]: http://github.com/defunkt/resque/issues
853
+ [sv]: http://semver.org/
854
+ [rs]: http://github.com/defunkt/redis-namespace
855
+ [cb]: http://wiki.github.com/defunkt/resque/contributing