opengotham_resque 1.8.2
Sign up to get free protection for your applications and to get access to all the features.
- data/.gitignore +2 -0
- data/.kick +26 -0
- data/HISTORY.md +142 -0
- data/LICENSE +20 -0
- data/README.markdown +794 -0
- data/Rakefile +112 -0
- data/bin/resque +57 -0
- data/bin/resque-web +23 -0
- data/config.ru +14 -0
- data/deps.rip +7 -0
- data/docs/HOOKS.md +121 -0
- data/docs/PLUGINS.md +93 -0
- data/examples/async_helper.rb +31 -0
- data/examples/demo/README.markdown +71 -0
- data/examples/demo/Rakefile +8 -0
- data/examples/demo/app.rb +38 -0
- data/examples/demo/config.ru +19 -0
- data/examples/demo/job.rb +22 -0
- data/examples/god/resque.god +53 -0
- data/examples/god/stale.god +26 -0
- data/examples/instance.rb +11 -0
- data/examples/monit/resque.monit +6 -0
- data/examples/simple.rb +30 -0
- data/init.rb +1 -0
- data/lib/resque.rb +287 -0
- data/lib/resque/errors.rb +10 -0
- data/lib/resque/failure.rb +66 -0
- data/lib/resque/failure/base.rb +61 -0
- data/lib/resque/failure/hoptoad.rb +132 -0
- data/lib/resque/failure/multiple.rb +48 -0
- data/lib/resque/failure/redis.rb +40 -0
- data/lib/resque/helpers.rb +63 -0
- data/lib/resque/job.rb +207 -0
- data/lib/resque/plugin.rb +46 -0
- data/lib/resque/server.rb +201 -0
- data/lib/resque/server/public/idle.png +0 -0
- data/lib/resque/server/public/jquery-1.3.2.min.js +19 -0
- data/lib/resque/server/public/jquery.relatize_date.js +95 -0
- data/lib/resque/server/public/poll.png +0 -0
- data/lib/resque/server/public/ranger.js +67 -0
- data/lib/resque/server/public/reset.css +48 -0
- data/lib/resque/server/public/style.css +81 -0
- data/lib/resque/server/public/working.png +0 -0
- data/lib/resque/server/test_helper.rb +19 -0
- data/lib/resque/server/views/error.erb +1 -0
- data/lib/resque/server/views/failed.erb +53 -0
- data/lib/resque/server/views/key_sets.erb +20 -0
- data/lib/resque/server/views/key_string.erb +11 -0
- data/lib/resque/server/views/layout.erb +44 -0
- data/lib/resque/server/views/next_more.erb +10 -0
- data/lib/resque/server/views/overview.erb +4 -0
- data/lib/resque/server/views/queues.erb +49 -0
- data/lib/resque/server/views/stats.erb +62 -0
- data/lib/resque/server/views/workers.erb +78 -0
- data/lib/resque/server/views/working.erb +69 -0
- data/lib/resque/stat.rb +53 -0
- data/lib/resque/tasks.rb +39 -0
- data/lib/resque/version.rb +3 -0
- data/lib/resque/worker.rb +478 -0
- data/tasks/redis.rake +159 -0
- data/tasks/resque.rake +2 -0
- data/test/job_hooks_test.rb +302 -0
- data/test/job_plugins_test.rb +209 -0
- data/test/plugin_test.rb +116 -0
- data/test/redis-test.conf +132 -0
- data/test/resque-web_test.rb +54 -0
- data/test/resque_test.rb +225 -0
- data/test/test_helper.rb +111 -0
- data/test/worker_test.rb +302 -0
- metadata +199 -0
data/.gitignore
ADDED
data/.kick
ADDED
@@ -0,0 +1,26 @@
|
|
1
|
+
# take control of the growl notifications
|
2
|
+
module GrowlHacks
|
3
|
+
def growl(type, subject, body, *args, &block)
|
4
|
+
case type
|
5
|
+
when Kicker::GROWL_NOTIFICATIONS[:succeeded]
|
6
|
+
puts subject = "Success"
|
7
|
+
body = body.split("\n").last
|
8
|
+
when Kicker::GROWL_NOTIFICATIONS[:failed]
|
9
|
+
subject = "Failure"
|
10
|
+
puts body
|
11
|
+
body = body.split("\n").last
|
12
|
+
else
|
13
|
+
return nil
|
14
|
+
end
|
15
|
+
super(type, subject, body, *args, &block)
|
16
|
+
end
|
17
|
+
end
|
18
|
+
|
19
|
+
Kicker.send :extend, GrowlHacks
|
20
|
+
|
21
|
+
# no logging
|
22
|
+
Kicker::Utils.module_eval do
|
23
|
+
def log(message)
|
24
|
+
nil
|
25
|
+
end
|
26
|
+
end
|
data/HISTORY.md
ADDED
@@ -0,0 +1,142 @@
|
|
1
|
+
## 1.8.2 (2010-05-03)
|
2
|
+
|
3
|
+
* Bugfix: Include "tasks/" dir in RubyGem
|
4
|
+
|
5
|
+
## 1.8.1 (2010-04-29)
|
6
|
+
|
7
|
+
* Bugfix: Multiple failure backend did not support requeue-ing failed jobs
|
8
|
+
* Bugfix: Fix /failed when error has no backtrace
|
9
|
+
* Bugfix: Add `Redis::DistRedis` as a valid client
|
10
|
+
|
11
|
+
## 1.8.0 (2010-04-07)
|
12
|
+
|
13
|
+
* Jobs that never complete due to killed worker are now failed.
|
14
|
+
* Worker "working" state is now maintained by the parent, not the child.
|
15
|
+
* Stopped using deprecated redis.rb methods
|
16
|
+
* `Worker.working` race condition fixed
|
17
|
+
* `Worker#process` has been deprecated.
|
18
|
+
* Monit example fixed
|
19
|
+
* Redis::Client and Redis::Namespace can be passed to `Resque.redis=`
|
20
|
+
|
21
|
+
## 1.7.1 (2010-04-02)
|
22
|
+
|
23
|
+
* Bugfix: Make job hook execution order consistent
|
24
|
+
* Bugfix: stdout buffering in child process
|
25
|
+
|
26
|
+
## 1.7.0 (2010-03-31)
|
27
|
+
|
28
|
+
* Job hooks API. See docs/HOOKS.md.
|
29
|
+
* web: Hovering over dates shows a timestamp
|
30
|
+
* web: AJAXify retry action for failed jobs
|
31
|
+
* web bugfix: Fix pagination bug
|
32
|
+
|
33
|
+
## 1.6.1 (2010-03-25)
|
34
|
+
|
35
|
+
* Bugfix: Workers may not be clearing their state correctly on
|
36
|
+
shutdown
|
37
|
+
* Added example monit config.
|
38
|
+
* Exception class is now recorded when an error is raised in a
|
39
|
+
worker.
|
40
|
+
* web: Unit tests
|
41
|
+
* web: Show namespace in header and footer
|
42
|
+
* web: Remove a queue
|
43
|
+
* web: Retry failed jobs
|
44
|
+
|
45
|
+
## 1.6.0 (2010-03-09)
|
46
|
+
|
47
|
+
* Added `before_first_fork`, `before_fork`, and `after_fork` hooks.
|
48
|
+
* Hoptoad: Added server_environment config setting
|
49
|
+
* Hoptoad bugfix: Don't depend on RAILS_ROOT
|
50
|
+
* 1.8.6 compat fixes
|
51
|
+
|
52
|
+
## 1.5.2 (2010-03-03)
|
53
|
+
|
54
|
+
* Bugfix: JSON check was crazy.
|
55
|
+
|
56
|
+
## 1.5.1 (2010-03-03)
|
57
|
+
|
58
|
+
* `Job.destroy` and `Resque.dequeue` return the # of destroyed jobs.
|
59
|
+
* Hoptoad notifier improvements
|
60
|
+
* Specify the namespace with `resque-web` by passing `-N namespace`
|
61
|
+
* Bugfix: Don't crash when trying to parse invalid JSON.
|
62
|
+
* Bugfix: Non-standard namespace support
|
63
|
+
* Web: Red backgound for queue "failed" only shown if there are failed jobs.
|
64
|
+
* Web bugfix: Tabs highlight properly now
|
65
|
+
* Web bugfix: ZSET partial support in stats
|
66
|
+
* Web bugfix: Deleting failed jobs works again
|
67
|
+
* Web bugfix: Sets (or zsets, lists, etc) now paginate.
|
68
|
+
|
69
|
+
## 1.5.0 (2010-02-17)
|
70
|
+
|
71
|
+
* Version now included in procline, e.g. `resque-1.5.0: Message`
|
72
|
+
* Web bugfix: Ignore idle works in the "working" page
|
73
|
+
* Added `Resque::Job.destroy(queue, klass, *args)`
|
74
|
+
* Added `Resque.dequeue(klass, *args)`
|
75
|
+
|
76
|
+
## 1.4.0 (2010-02-11)
|
77
|
+
|
78
|
+
* Fallback when unable to bind QUIT and USR1 for Windows and JRuby.
|
79
|
+
* Fallback when no `Kernel.fork` is provided (for IronRuby).
|
80
|
+
* Web: Rounded corners in Firefox
|
81
|
+
* Cut down system calls in `Worker#prune_dead_workers`
|
82
|
+
* Enable switching DB in a Redis server from config
|
83
|
+
* Support USR2 and CONT to stop and start job processing.
|
84
|
+
* Web: Add example failing job
|
85
|
+
* Bugfix: `Worker#unregister_worker` shouldn't call `done_working`
|
86
|
+
* Bugfix: Example god config now restarts Resque properly.
|
87
|
+
* Multiple failure backends now permitted.
|
88
|
+
* Hoptoad failure backend updated to new API
|
89
|
+
|
90
|
+
## 1.3.1 (2010-01-11)
|
91
|
+
|
92
|
+
* Vegas bugfix: Don't error without a config
|
93
|
+
|
94
|
+
## 1.3.0 (2010-01-11)
|
95
|
+
|
96
|
+
* Use Vegas for resque-web
|
97
|
+
* Web Bugfix: Show proper date/time value for failed_at on Failures
|
98
|
+
* Web Bugfix: Make the / route more flexible
|
99
|
+
* Add Resque::Server.tabs array (so plugins can add their own tabs)
|
100
|
+
* Start using [Semantic Versioning](http://semver.org/)
|
101
|
+
|
102
|
+
## 1.2.4 (2009-12-15)
|
103
|
+
|
104
|
+
* Web Bugfix: fix key links on stat page
|
105
|
+
|
106
|
+
## 1.2.3 (2009-12-15)
|
107
|
+
|
108
|
+
* Bugfix: Fixed `rand` seeding in child processes.
|
109
|
+
* Bugfix: Better JSON encoding/decoding without Yajl.
|
110
|
+
* Bugfix: Avoid `ps` flag error on Linux
|
111
|
+
* Add `PREFIX` observance to `rake` install tasks.
|
112
|
+
|
113
|
+
## 1.2.2 (2009-12-08)
|
114
|
+
|
115
|
+
* Bugfix: Job equality was not properly implemented.
|
116
|
+
|
117
|
+
## 1.2.1 (2009-12-07)
|
118
|
+
|
119
|
+
* Added `rake resque:workers` task for starting multiple workers.
|
120
|
+
* 1.9.x compatibility
|
121
|
+
* Bugfix: Yajl decoder doesn't care about valid UTF-8
|
122
|
+
* config.ru loads RESQUECONFIG if the ENV variable is set.
|
123
|
+
* `resque-web` now sets RESQUECONFIG
|
124
|
+
* Job objects know if they are equal.
|
125
|
+
* Jobs can be re-queued using `Job#recreate`
|
126
|
+
|
127
|
+
## 1.2.0 (2009-11-25)
|
128
|
+
|
129
|
+
* If USR1 is sent and no child is found, shutdown.
|
130
|
+
* Raise when a job class does not respond to `perform`.
|
131
|
+
* Added `Resque.remove_queue` for deleting a queue
|
132
|
+
|
133
|
+
## 1.1.0 (2009-11-04)
|
134
|
+
|
135
|
+
* Bugfix: Broken ERB tag in failure UI
|
136
|
+
* Bugfix: Save the worker's ID, not the worker itself, in the failure module
|
137
|
+
* Redesigned the sinatra web interface
|
138
|
+
* Added option to clear failed jobs
|
139
|
+
|
140
|
+
## 1.0.0 (2009-11-03)
|
141
|
+
|
142
|
+
* First release.
|
data/LICENSE
ADDED
@@ -0,0 +1,20 @@
|
|
1
|
+
Copyright (c) 2009 Chris Wanstrath
|
2
|
+
|
3
|
+
Permission is hereby granted, free of charge, to any person obtaining
|
4
|
+
a copy of this software and associated documentation files (the
|
5
|
+
"Software"), to deal in the Software without restriction, including
|
6
|
+
without limitation the rights to use, copy, modify, merge, publish,
|
7
|
+
distribute, sublicense, and/or sell copies of the Software, and to
|
8
|
+
permit persons to whom the Software is furnished to do so, subject to
|
9
|
+
the following conditions:
|
10
|
+
|
11
|
+
The above copyright notice and this permission notice shall be
|
12
|
+
included in all copies or substantial portions of the Software.
|
13
|
+
|
14
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
15
|
+
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
16
|
+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
17
|
+
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
18
|
+
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
19
|
+
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
20
|
+
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
data/README.markdown
ADDED
@@ -0,0 +1,794 @@
|
|
1
|
+
Resque
|
2
|
+
======
|
3
|
+
|
4
|
+
Resque is a Redis-backed library for creating background jobs, placing
|
5
|
+
those jobs on multiple queues, and processing them later.
|
6
|
+
|
7
|
+
Background jobs can be any Ruby class or module that responds to
|
8
|
+
`perform`. Your existing classes can easily be converted to background
|
9
|
+
jobs or you can create new classes specifically to do work. Or, you
|
10
|
+
can do both.
|
11
|
+
|
12
|
+
Resque is heavily inspired by DelayedJob (which rocks) and is
|
13
|
+
comprised of three parts:
|
14
|
+
|
15
|
+
1. A Ruby library for creating, querying, and processing jobs
|
16
|
+
2. A Rake task for starting a worker which processes jobs
|
17
|
+
3. A Sinatra app for monitoring queues, jobs, and workers.
|
18
|
+
|
19
|
+
Resque workers can be distributed between multiple machines,
|
20
|
+
support priorities, are resilient to memory bloat / "leaks," are
|
21
|
+
optimized for REE (but work on MRI and JRuby), tell you what they're
|
22
|
+
doing, and expect failure.
|
23
|
+
|
24
|
+
Resque queues are persistent; support constant time, atomic push and
|
25
|
+
pop (thanks to Redis); provide visibility into their contents; and
|
26
|
+
store jobs as simple JSON packages.
|
27
|
+
|
28
|
+
The Resque frontend tells you what workers are doing, what workers are
|
29
|
+
not doing, what queues you're using, what's in those queues, provides
|
30
|
+
general usage stats, and helps you track failures.
|
31
|
+
|
32
|
+
|
33
|
+
The Blog Post
|
34
|
+
-------------
|
35
|
+
|
36
|
+
For the backstory, philosophy, and history of Resque's beginnings,
|
37
|
+
please see [the blog post][0].
|
38
|
+
|
39
|
+
|
40
|
+
Overview
|
41
|
+
--------
|
42
|
+
|
43
|
+
Resque allows you to create jobs and place them on a queue, then,
|
44
|
+
later, pull those jobs off the queue and process them.
|
45
|
+
|
46
|
+
Resque jobs are Ruby classes (or modules) which respond to the
|
47
|
+
`perform` method. Here's an example:
|
48
|
+
|
49
|
+
class Archive
|
50
|
+
@queue = :file_serve
|
51
|
+
|
52
|
+
def self.perform(repo_id, branch = 'master')
|
53
|
+
repo = Repository.find(repo_id)
|
54
|
+
repo.create_archive(branch)
|
55
|
+
end
|
56
|
+
end
|
57
|
+
|
58
|
+
The `@queue` class instance variable determines which queue `Archive`
|
59
|
+
jobs will be placed in. Queues are arbitrary and created on the fly -
|
60
|
+
you can name them whatever you want and have as many as you want.
|
61
|
+
|
62
|
+
To place an `Archive` job on the `file_serve` queue, we might add this
|
63
|
+
to our application's pre-existing `Repository` class:
|
64
|
+
|
65
|
+
class Repository
|
66
|
+
def async_create_archive(branch)
|
67
|
+
Resque.enqueue(Archive, self.id, branch)
|
68
|
+
end
|
69
|
+
end
|
70
|
+
|
71
|
+
Now when we call `repo.async_create_archive('masterbrew')` in our
|
72
|
+
application, a job will be created and placed on the `file_serve`
|
73
|
+
queue.
|
74
|
+
|
75
|
+
Later, a worker will run something like this code to process the job:
|
76
|
+
|
77
|
+
klass, args = Resque.reserve(:file_serve)
|
78
|
+
klass.perform(*args) if klass.respond_to? :perform
|
79
|
+
|
80
|
+
Which translates to:
|
81
|
+
|
82
|
+
Archive.perform(44, 'masterbrew')
|
83
|
+
|
84
|
+
Let's start a worker to run `file_serve` jobs:
|
85
|
+
|
86
|
+
$ cd app_root
|
87
|
+
$ QUEUE=file_serve rake resque:work
|
88
|
+
|
89
|
+
This starts one Resque worker and tells it to work off the
|
90
|
+
`file_serve` queue. As soon as it's ready it'll try to run the
|
91
|
+
`Resque.reserve` code snippet above and process jobs until it can't
|
92
|
+
find any more, at which point it will sleep for a small period and
|
93
|
+
repeatedly poll the queue for more jobs.
|
94
|
+
|
95
|
+
Workers can be given multiple queues (a "queue list") and run on
|
96
|
+
multiple machines. In fact they can be run anywhere with network
|
97
|
+
access to the Redis server.
|
98
|
+
|
99
|
+
|
100
|
+
Jobs
|
101
|
+
----
|
102
|
+
|
103
|
+
What should you run in the background? Anything that takes any time at
|
104
|
+
all. Slow INSERT statements, disk manipulating, data processing, etc.
|
105
|
+
|
106
|
+
At GitHub we use Resque to process the following types of jobs:
|
107
|
+
|
108
|
+
* Warming caches
|
109
|
+
* Counting disk usage
|
110
|
+
* Building tarballs
|
111
|
+
* Building Rubygems
|
112
|
+
* Firing off web hooks
|
113
|
+
* Creating events in the db and pre-caching them
|
114
|
+
* Building graphs
|
115
|
+
* Deleting users
|
116
|
+
* Updating our search index
|
117
|
+
|
118
|
+
As of writing we have about 35 different types of background jobs.
|
119
|
+
|
120
|
+
Keep in mind that you don't need a web app to use Resque - we just
|
121
|
+
mention "foreground" and "background" because they make conceptual
|
122
|
+
sense. You could easily be spidering sites and sticking data which
|
123
|
+
needs to be crunched later into a queue.
|
124
|
+
|
125
|
+
|
126
|
+
### Persistence
|
127
|
+
|
128
|
+
Jobs are persisted to queues as JSON objects. Let's take our `Archive`
|
129
|
+
example from above. We'll run the following code to create a job:
|
130
|
+
|
131
|
+
repo = Repository.find(44)
|
132
|
+
repo.async_create_archive('masterbrew')
|
133
|
+
|
134
|
+
The following JSON will be stored in the `file_serve` queue:
|
135
|
+
|
136
|
+
{
|
137
|
+
'class': 'Archive',
|
138
|
+
'args': [ 44, 'masterbrew' ]
|
139
|
+
}
|
140
|
+
|
141
|
+
Because of this your jobs must only accept arguments that can be JSON encoded.
|
142
|
+
|
143
|
+
So instead of doing this:
|
144
|
+
|
145
|
+
Resque.enqueue(Archive, self, branch)
|
146
|
+
|
147
|
+
do this:
|
148
|
+
|
149
|
+
Resque.enqueue(Archive, self.id, branch)
|
150
|
+
|
151
|
+
This is why our above example (and all the examples in `examples/`)
|
152
|
+
uses object IDs instead of passing around the objects.
|
153
|
+
|
154
|
+
While this is less convenient than just sticking a marshaled object
|
155
|
+
in the database, it gives you a slight advantage: your jobs will be
|
156
|
+
run against the most recent version of an object because they need to
|
157
|
+
pull from the DB or cache.
|
158
|
+
|
159
|
+
If your jobs were run against marshaled objects, they could
|
160
|
+
potentially be operating on a stale record with out-of-date information.
|
161
|
+
|
162
|
+
|
163
|
+
### send_later / async
|
164
|
+
|
165
|
+
Want something like DelayedJob's `send_later` or the ability to use
|
166
|
+
instance methods instead of just methods for jobs? See the `examples/`
|
167
|
+
directory for goodies.
|
168
|
+
|
169
|
+
We plan to provide first class `async` support in a future release.
|
170
|
+
|
171
|
+
|
172
|
+
### Failure
|
173
|
+
|
174
|
+
If a job raises an exception, it is logged and handed off to the
|
175
|
+
`Resque::Failure` module. Failures are logged either locally in Redis
|
176
|
+
or using some different backend.
|
177
|
+
|
178
|
+
For example, Resque ships with Hoptoad support.
|
179
|
+
|
180
|
+
Keep this in mind when writing your jobs: you may want to throw
|
181
|
+
exceptions you would not normally throw in order to assist debugging.
|
182
|
+
|
183
|
+
|
184
|
+
Workers
|
185
|
+
-------
|
186
|
+
|
187
|
+
Resque workers are rake tasks that run forever. They basically do this:
|
188
|
+
|
189
|
+
start
|
190
|
+
loop do
|
191
|
+
if job = reserve
|
192
|
+
job.process
|
193
|
+
else
|
194
|
+
sleep 5
|
195
|
+
end
|
196
|
+
end
|
197
|
+
shutdown
|
198
|
+
|
199
|
+
Starting a worker is simple. Here's our example from earlier:
|
200
|
+
|
201
|
+
$ QUEUE=file_serve rake resque:work
|
202
|
+
|
203
|
+
By default Resque won't know about your application's
|
204
|
+
environment. That is, it won't be able to find and run your jobs - it
|
205
|
+
needs to load your application into memory.
|
206
|
+
|
207
|
+
If we've installed Resque as a Rails plugin, we might run this command
|
208
|
+
from our RAILS_ROOT:
|
209
|
+
|
210
|
+
$ QUEUE=file_serve rake environment resque:work
|
211
|
+
|
212
|
+
This will load the environment before starting a worker. Alternately
|
213
|
+
we can define a `resque:setup` task with a dependency on the
|
214
|
+
`environment` rake task:
|
215
|
+
|
216
|
+
task "resque:setup" => :environment
|
217
|
+
|
218
|
+
GitHub's setup task looks like this:
|
219
|
+
|
220
|
+
task "resque:setup" => :environment do
|
221
|
+
Grit::Git.git_timeout = 10.minutes
|
222
|
+
end
|
223
|
+
|
224
|
+
We don't want the `git_timeout` as high as 10 minutes in our web app,
|
225
|
+
but in the Resque workers it's fine.
|
226
|
+
|
227
|
+
|
228
|
+
### Logging
|
229
|
+
|
230
|
+
Workers support basic logging to STDOUT. If you start them with the
|
231
|
+
`VERBOSE` env variable set, they will print basic debugging
|
232
|
+
information. You can also set the `VVERBOSE` (very verbose) env
|
233
|
+
variable.
|
234
|
+
|
235
|
+
$ VVERBOSE=1 QUEUE=file_serve rake environment resque:work
|
236
|
+
|
237
|
+
|
238
|
+
### Priorities and Queue Lists
|
239
|
+
|
240
|
+
Resque doesn't support numeric priorities but instead uses the order
|
241
|
+
of queues you give it. We call this list of queues the "queue list."
|
242
|
+
|
243
|
+
Let's say we add a `warm_cache` queue in addition to our `file_serve`
|
244
|
+
queue. We'd now start a worker like so:
|
245
|
+
|
246
|
+
$ QUEUES=file_serve,warm_cache rake resque:work
|
247
|
+
|
248
|
+
When the worker looks for new jobs, it will first check
|
249
|
+
`file_serve`. If it finds a job, it'll process it then check
|
250
|
+
`file_serve` again. It will keep checking `file_serve` until no more
|
251
|
+
jobs are available. At that point, it will check `warm_cache`. If it
|
252
|
+
finds a job it'll process it then check `file_serve` (repeating the
|
253
|
+
whole process).
|
254
|
+
|
255
|
+
In this way you can prioritize certain queues. At GitHub we start our
|
256
|
+
workers with something like this:
|
257
|
+
|
258
|
+
$ QUEUES=critical,archive,high,low rake resque:work
|
259
|
+
|
260
|
+
Notice the `archive` queue - it is specialized and in our future
|
261
|
+
architecture will only be run from a single machine.
|
262
|
+
|
263
|
+
At that point we'll start workers on our generalized background
|
264
|
+
machines with this command:
|
265
|
+
|
266
|
+
$ QUEUES=critical,high,low rake resque:work
|
267
|
+
|
268
|
+
And workers on our specialized archive machine with this command:
|
269
|
+
|
270
|
+
$ QUEUE=archive rake resque:work
|
271
|
+
|
272
|
+
|
273
|
+
### Running All Queues
|
274
|
+
|
275
|
+
If you want your workers to work off of every queue, including new
|
276
|
+
queues created on the fly, you can use a splat:
|
277
|
+
|
278
|
+
$ QUEUE=* rake resque:work
|
279
|
+
|
280
|
+
Queues will be processed in alphabetical order.
|
281
|
+
|
282
|
+
|
283
|
+
### Running Multiple Workers
|
284
|
+
|
285
|
+
At GitHub we use god to start and stop multiple workers. A sample god
|
286
|
+
configuration file is included under `examples/god`. We recommend this
|
287
|
+
method.
|
288
|
+
|
289
|
+
If you'd like to run multiple workers in development mode, you can do
|
290
|
+
so using the `resque:workers` rake task:
|
291
|
+
|
292
|
+
$ COUNT=5 QUEUE=* rake resque:workers
|
293
|
+
|
294
|
+
This will spawn five Resque workers, each in its own thread. Hitting
|
295
|
+
ctrl-c should be sufficient to stop them all.
|
296
|
+
|
297
|
+
|
298
|
+
### Forking
|
299
|
+
|
300
|
+
On certain platforms, when a Resque worker reserves a job it
|
301
|
+
immediately forks a child process. The child processes the job then
|
302
|
+
exits. When the child has exited successfully, the worker reserves
|
303
|
+
another job and repeats the process.
|
304
|
+
|
305
|
+
Why?
|
306
|
+
|
307
|
+
Because Resque assumes chaos.
|
308
|
+
|
309
|
+
Resque assumes your background workers will lock up, run too long, or
|
310
|
+
have unwanted memory growth.
|
311
|
+
|
312
|
+
If Resque workers processed jobs themselves, it'd be hard to whip them
|
313
|
+
into shape. Let's say one is using too much memory: you send it a
|
314
|
+
signal that says "shutdown after you finish processing the current
|
315
|
+
job," and it does so. It then starts up again - loading your entire
|
316
|
+
application environment. This adds useless CPU cycles and causes a
|
317
|
+
delay in queue processing.
|
318
|
+
|
319
|
+
Plus, what if it's using too much memory and has stopped responding to
|
320
|
+
signals?
|
321
|
+
|
322
|
+
Thanks to Resque's parent / child architecture, jobs that use too much memory
|
323
|
+
release that memory upon completion. No unwanted growth.
|
324
|
+
|
325
|
+
And what if a job is running too long? You'd need to `kill -9` it then
|
326
|
+
start the worker again. With Resque's parent / child architecture you
|
327
|
+
can tell the parent to forcefully kill the child then immediately
|
328
|
+
start processing more jobs. No startup delay or wasted cycles.
|
329
|
+
|
330
|
+
The parent / child architecture helps us keep tabs on what workers are
|
331
|
+
doing, too. By eliminating the need to `kill -9` workers we can have
|
332
|
+
parents remove themselves from the global listing of workers. If we
|
333
|
+
just ruthlessly killed workers, we'd need a separate watchdog process
|
334
|
+
to add and remove them to the global listing - which becomes
|
335
|
+
complicated.
|
336
|
+
|
337
|
+
Workers instead handle their own state.
|
338
|
+
|
339
|
+
|
340
|
+
### Parents and Children
|
341
|
+
|
342
|
+
Here's a parent / child pair doing some work:
|
343
|
+
|
344
|
+
$ ps -e -o pid,command | grep [r]esque
|
345
|
+
92099 resque: Forked 92102 at 1253142769
|
346
|
+
92102 resque: Processing file_serve since 1253142769
|
347
|
+
|
348
|
+
You can clearly see that process 92099 forked 92102, which has been
|
349
|
+
working since 1253142769.
|
350
|
+
|
351
|
+
(By advertising the time they began processing you can easily use monit
|
352
|
+
or god to kill stale workers.)
|
353
|
+
|
354
|
+
When a parent process is idle, it lets you know what queues it is
|
355
|
+
waiting for work on:
|
356
|
+
|
357
|
+
$ ps -e -o pid,command | grep [r]esque
|
358
|
+
92099 resque: Waiting for file_serve,warm_cache
|
359
|
+
|
360
|
+
|
361
|
+
### Signals
|
362
|
+
|
363
|
+
Resque workers respond to a few different signals:
|
364
|
+
|
365
|
+
* `QUIT` - Wait for child to finish processing then exit
|
366
|
+
* `TERM` / `INT` - Immediately kill child then exit
|
367
|
+
* `USR1` - Immediately kill child but don't exit
|
368
|
+
* `USR2` - Don't start to process any new jobs
|
369
|
+
* `CONT` - Start to process new jobs again after a USR2
|
370
|
+
|
371
|
+
If you want to gracefully shutdown a Resque worker, use `QUIT`.
|
372
|
+
|
373
|
+
If you want to kill a stale or stuck child, use `USR1`. Processing
|
374
|
+
will continue as normal unless the child was not found. In that case
|
375
|
+
Resque assumes the parent process is in a bad state and shuts down.
|
376
|
+
|
377
|
+
If you want to kill a stale or stuck child and shutdown, use `TERM`
|
378
|
+
|
379
|
+
If you want to stop processing jobs, but want to leave the worker running
|
380
|
+
(for example, to temporarily alleviate load), use `USR2` to stop processing,
|
381
|
+
then `CONT` to start it again.
|
382
|
+
|
383
|
+
### Mysql::Error: MySQL server has gone away
|
384
|
+
|
385
|
+
If your workers remain idle for too long they may lose their MySQL
|
386
|
+
connection. If that happens we recommend using [this
|
387
|
+
Gist](http://gist.github.com/238999).
|
388
|
+
|
389
|
+
|
390
|
+
The Front End
|
391
|
+
-------------
|
392
|
+
|
393
|
+
Resque comes with a Sinatra-based front end for seeing what's up with
|
394
|
+
your queue.
|
395
|
+
|
396
|
+
![The Front End](http://img.skitch.com/20091104-tqh5pgkwgbskjbk7qbtmpesnyw.jpg)
|
397
|
+
|
398
|
+
### Standalone
|
399
|
+
|
400
|
+
If you've installed Resque as a gem running the front end standalone is easy:
|
401
|
+
|
402
|
+
$ resque-web
|
403
|
+
|
404
|
+
It's a thin layer around `rackup` so it's configurable as well:
|
405
|
+
|
406
|
+
$ resque-web -p 8282
|
407
|
+
|
408
|
+
If you have a Resque config file you want evaluated just pass it to
|
409
|
+
the script as the final argument:
|
410
|
+
|
411
|
+
$ resque-web -p 8282 rails_root/config/initializers/resque.rb
|
412
|
+
|
413
|
+
You can also set the namespace directly using `resque-web`:
|
414
|
+
|
415
|
+
$ resque-web -p 8282 -N myapp
|
416
|
+
|
417
|
+
### Passenger
|
418
|
+
|
419
|
+
Using Passenger? Resque ships with a `config.ru` you can use. See
|
420
|
+
Phusion's guide:
|
421
|
+
|
422
|
+
<http://www.modrails.com/documentation/Users%20guide.html#_deploying_a_rack_based_ruby_application>
|
423
|
+
|
424
|
+
### Rack::URLMap
|
425
|
+
|
426
|
+
If you want to load Resque on a subpath, possibly alongside other
|
427
|
+
apps, it's easy to do with Rack's `URLMap`:
|
428
|
+
|
429
|
+
require 'resque/server'
|
430
|
+
|
431
|
+
run Rack::URLMap.new \
|
432
|
+
"/" => Your::App.new,
|
433
|
+
"/resque" => Resque::Server.new
|
434
|
+
|
435
|
+
Check `examples/demo/config.ru` for a functional example (including
|
436
|
+
HTTP basic auth).
|
437
|
+
|
438
|
+
|
439
|
+
Resque vs DelayedJob
|
440
|
+
--------------------
|
441
|
+
|
442
|
+
How does Resque compare to DelayedJob, and why would you choose one
|
443
|
+
over the other?
|
444
|
+
|
445
|
+
* Resque supports multiple queues
|
446
|
+
* DelayedJob supports finer grained priorities
|
447
|
+
* Resque workers are resilient to memory leaks / bloat
|
448
|
+
* DelayedJob workers are extremely simple and easy to modify
|
449
|
+
* Resque requires Redis
|
450
|
+
* DelayedJob requires ActiveRecord
|
451
|
+
* Resque can only place JSONable Ruby objects on a queue as arguments
|
452
|
+
* DelayedJob can place _any_ Ruby object on its queue as arguments
|
453
|
+
* Resque includes a Sinatra app for monitoring what's going on
|
454
|
+
* DelayedJob can be queried from within your Rails app if you want to
|
455
|
+
add an interface
|
456
|
+
|
457
|
+
If you're doing Rails development, you already have a database and
|
458
|
+
ActiveRecord. DelayedJob is super easy to setup and works great.
|
459
|
+
GitHub used it for many months to process almost 200 million jobs.
|
460
|
+
|
461
|
+
Choose Resque if:
|
462
|
+
|
463
|
+
* You need multiple queues
|
464
|
+
* You don't care / dislike numeric priorities
|
465
|
+
* You don't need to persist every Ruby object ever
|
466
|
+
* You have potentially huge queues
|
467
|
+
* You want to see what's going on
|
468
|
+
* You expect a lot of failure / chaos
|
469
|
+
* You can setup Redis
|
470
|
+
* You're not running short on RAM
|
471
|
+
|
472
|
+
Choose DelayedJob if:
|
473
|
+
|
474
|
+
* You like numeric priorities
|
475
|
+
* You're not doing a gigantic amount of jobs each day
|
476
|
+
* Your queue stays small and nimble
|
477
|
+
* There is not a lot failure / chaos
|
478
|
+
* You want to easily throw anything on the queue
|
479
|
+
* You don't want to setup Redis
|
480
|
+
|
481
|
+
In no way is Resque a "better" DelayedJob, so make sure you pick the
|
482
|
+
tool that's best for your app.
|
483
|
+
|
484
|
+
|
485
|
+
Installing Redis
|
486
|
+
----------------
|
487
|
+
|
488
|
+
Resque requires Redis 0.900 or higher.
|
489
|
+
|
490
|
+
Resque uses Redis' lists for its queues. It also stores worker state
|
491
|
+
data in Redis.
|
492
|
+
|
493
|
+
#### Homebrew
|
494
|
+
|
495
|
+
If you're on OS X, Homebrew is the simplest way to install Redis:
|
496
|
+
|
497
|
+
$ brew install redis
|
498
|
+
$ redis-server /usr/local/etc/redis.conf
|
499
|
+
|
500
|
+
You now have a Redis daemon running on 6379.
|
501
|
+
|
502
|
+
#### Via Resque
|
503
|
+
|
504
|
+
Resque includes Rake tasks (thanks to Ezra's redis-rb) that will
|
505
|
+
install and run Redis for you:
|
506
|
+
|
507
|
+
$ git clone git://github.com/defunkt/resque.git
|
508
|
+
$ cd resque
|
509
|
+
$ rake redis:install dtach:install
|
510
|
+
$ rake redis:start
|
511
|
+
|
512
|
+
Or, if you don't have admin access on your machine:
|
513
|
+
|
514
|
+
$ git clone git://github.com/defunkt/resque.git
|
515
|
+
$ cd resque
|
516
|
+
$ PREFIX=<your_prefix> rake redis:install dtach:install
|
517
|
+
$ rake redis:start
|
518
|
+
|
519
|
+
You now have Redis running on 6379. Wait a second then hit ctrl-\ to
|
520
|
+
detach and keep it running in the background.
|
521
|
+
|
522
|
+
The demo is probably the best way to figure out how to put the parts
|
523
|
+
together. But, it's not that hard.
|
524
|
+
|
525
|
+
|
526
|
+
Resque Dependencies
|
527
|
+
-------------------
|
528
|
+
|
529
|
+
gem install redis redis-namespace yajl-ruby
|
530
|
+
|
531
|
+
If you cannot install `yajl-ruby` (JRuby?), you can install the `json`
|
532
|
+
gem and Resque will use it instead.
|
533
|
+
|
534
|
+
When problems arise, make sure you have the newest versions of the
|
535
|
+
`redis` and `redis-namespace` gems.
|
536
|
+
|
537
|
+
|
538
|
+
Installing Resque
|
539
|
+
-----------------
|
540
|
+
|
541
|
+
### In a Rack app, as a gem
|
542
|
+
|
543
|
+
First install the gem.
|
544
|
+
|
545
|
+
$ gem install resque
|
546
|
+
|
547
|
+
Next include it in your application.
|
548
|
+
|
549
|
+
require 'resque'
|
550
|
+
|
551
|
+
Now start your application:
|
552
|
+
|
553
|
+
rackup config.ru
|
554
|
+
|
555
|
+
That's it! You can now create Resque jobs from within your app.
|
556
|
+
|
557
|
+
To start a worker, create a Rakefile in your app's root (or add this
|
558
|
+
to an existing Rakefile):
|
559
|
+
|
560
|
+
require 'your/app'
|
561
|
+
require 'resque/tasks'
|
562
|
+
|
563
|
+
Now:
|
564
|
+
|
565
|
+
$ QUEUE=* rake resque:work
|
566
|
+
|
567
|
+
Alternately you can define a `resque:setup` hook in your Rakefile if you
|
568
|
+
don't want to load your app every time rake runs.
|
569
|
+
|
570
|
+
|
571
|
+
### In a Rails app, as a gem
|
572
|
+
|
573
|
+
First install the gem.
|
574
|
+
|
575
|
+
$ gem install resque
|
576
|
+
|
577
|
+
Next include it in your application.
|
578
|
+
|
579
|
+
$ cat config/initializers/load_resque.rb
|
580
|
+
require 'resque'
|
581
|
+
|
582
|
+
Now start your application:
|
583
|
+
|
584
|
+
$ ./script/server
|
585
|
+
|
586
|
+
That's it! You can now create Resque jobs from within your app.
|
587
|
+
|
588
|
+
To start a worker, add this to your Rakefile in `RAILS_ROOT`:
|
589
|
+
|
590
|
+
require 'resque/tasks'
|
591
|
+
|
592
|
+
Now:
|
593
|
+
|
594
|
+
$ QUEUE=* rake environment resque:work
|
595
|
+
|
596
|
+
Don't forget you can define a `resque:setup` hook in
|
597
|
+
`lib/tasks/whatever.rake` that loads the `environment` task every time.
|
598
|
+
|
599
|
+
|
600
|
+
### In a Rails app, as a plugin
|
601
|
+
|
602
|
+
$ ./script/plugin install git://github.com/defunkt/resque
|
603
|
+
|
604
|
+
That's it! Resque will automatically be available when your Rails app
|
605
|
+
loads.
|
606
|
+
|
607
|
+
To start a worker:
|
608
|
+
|
609
|
+
$ QUEUE=* rake environment resque:work
|
610
|
+
|
611
|
+
Don't forget you can define a `resque:setup` hook in
|
612
|
+
`lib/tasks/whatever.rake` that loads the `environment` task every time.
|
613
|
+
|
614
|
+
|
615
|
+
Configuration
|
616
|
+
-------------
|
617
|
+
|
618
|
+
You may want to change the Redis host and port Resque connects to, or
|
619
|
+
set various other options at startup.
|
620
|
+
|
621
|
+
Resque has a `redis` setter which can be given a string or a Redis
|
622
|
+
object. This means if you're already using Redis in your app, Resque
|
623
|
+
can re-use the existing connection.
|
624
|
+
|
625
|
+
String: `Resque.redis = 'localhost:6379'`
|
626
|
+
|
627
|
+
Redis: `Resque.redis = $redis`
|
628
|
+
|
629
|
+
For our rails app we have a `config/initializers/resque.rb` file where
|
630
|
+
we load `config/resque.yml` by hand and set the Redis information
|
631
|
+
appropriately.
|
632
|
+
|
633
|
+
Here's our `config/resque.yml`:
|
634
|
+
|
635
|
+
development: localhost:6379
|
636
|
+
test: localhost:6379
|
637
|
+
staging: redis1.se.github.com:6379
|
638
|
+
fi: localhost:6379
|
639
|
+
production: redis1.ae.github.com:6379
|
640
|
+
|
641
|
+
And our initializer:
|
642
|
+
|
643
|
+
rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..'
|
644
|
+
rails_env = ENV['RAILS_ENV'] || 'development'
|
645
|
+
|
646
|
+
resque_config = YAML.load_file(rails_root + '/config/resque.yml')
|
647
|
+
Resque.redis = resque_config[rails_env]
|
648
|
+
|
649
|
+
Easy peasy! Why not just use `RAILS_ROOT` and `RAILS_ENV`? Because
|
650
|
+
this way we can tell our Sinatra app about the config file:
|
651
|
+
|
652
|
+
$ RAILS_ENV=production resque-web rails_root/config/initializers/resque.rb
|
653
|
+
|
654
|
+
Now everyone is on the same page.
|
655
|
+
|
656
|
+
|
657
|
+
Plugins and Hooks
|
658
|
+
-----------------
|
659
|
+
|
660
|
+
For a list of available plugins see
|
661
|
+
<http://wiki.github.com/defunkt/resque/plugins>.
|
662
|
+
|
663
|
+
If you'd like to write your own plugin, or want to customize Resque
|
664
|
+
using hooks (such as `Resque.after_fork`), see
|
665
|
+
[docs/HOOKS.md](http://github.com/defunkt/resque/blob/master/HOOKS.md).
|
666
|
+
|
667
|
+
|
668
|
+
Namespaces
|
669
|
+
----------
|
670
|
+
|
671
|
+
If you're running multiple, separate instances of Resque you may want
|
672
|
+
to namespace the keyspaces so they do not overlap. This is not unlike
|
673
|
+
the approach taken by many memcached clients.
|
674
|
+
|
675
|
+
This feature is provided by the [redis-namespace][rs] library, which
|
676
|
+
Resque uses by default to separate the keys it manages from other keys
|
677
|
+
in your Redis server.
|
678
|
+
|
679
|
+
Simply use the `Resque.redis.namespace` accessor:
|
680
|
+
|
681
|
+
Resque.redis.namespace = "resque:GitHub"
|
682
|
+
|
683
|
+
We recommend sticking this in your initializer somewhere after Redis
|
684
|
+
is configured.
|
685
|
+
|
686
|
+
|
687
|
+
Demo
|
688
|
+
----
|
689
|
+
|
690
|
+
Resque ships with a demo Sinatra app for creating jobs that are later
|
691
|
+
processed in the background.
|
692
|
+
|
693
|
+
Try it out by looking at the README, found at `examples/demo/README.markdown`.
|
694
|
+
|
695
|
+
|
696
|
+
Monitoring
|
697
|
+
----------
|
698
|
+
|
699
|
+
### god
|
700
|
+
|
701
|
+
If you're using god to monitor Resque, we have provided example
|
702
|
+
configs in `examples/god/`. One is for starting / stopping workers,
|
703
|
+
the other is for killing workers that have been running too long.
|
704
|
+
|
705
|
+
### monit
|
706
|
+
|
707
|
+
If you're using monit, `examples/monit/resque.monit` is provided free
|
708
|
+
of charge. This is **not** used by GitHub in production, so please
|
709
|
+
send patches for any tweaks or improvements you can make to it.
|
710
|
+
|
711
|
+
|
712
|
+
Development
|
713
|
+
-----------
|
714
|
+
|
715
|
+
Want to hack on Resque?
|
716
|
+
|
717
|
+
First clone the repo and run the tests:
|
718
|
+
|
719
|
+
git clone git://github.com/defunkt/resque.git
|
720
|
+
cd resque
|
721
|
+
rake test
|
722
|
+
|
723
|
+
If the tests do not pass make sure you have Redis installed
|
724
|
+
correctly (though we make an effort to tell you if we feel this is the
|
725
|
+
case). The tests attempt to start an isolated instance of Redis to
|
726
|
+
run against.
|
727
|
+
|
728
|
+
Also make sure you've installed all the dependencies correctly. For
|
729
|
+
example, try loading the `redis-namespace` gem after you've installed
|
730
|
+
it:
|
731
|
+
|
732
|
+
$ irb
|
733
|
+
>> require 'rubygems'
|
734
|
+
=> true
|
735
|
+
>> require 'redis/namespace'
|
736
|
+
=> true
|
737
|
+
|
738
|
+
If you get an error requiring any of the dependencies, you may have
|
739
|
+
failed to install them or be seeing load path issues.
|
740
|
+
|
741
|
+
Feel free to ping the mailing list with your problem and we'll try to
|
742
|
+
sort it out.
|
743
|
+
|
744
|
+
|
745
|
+
Contributing
|
746
|
+
------------
|
747
|
+
|
748
|
+
Once you've made your great commits:
|
749
|
+
|
750
|
+
1. [Fork][1] Resque
|
751
|
+
2. Create a topic branch - `git checkout -b my_branch`
|
752
|
+
3. Push to your branch - `git push origin my_branch`
|
753
|
+
4. Create an [Issue][2] with a link to your branch
|
754
|
+
5. That's it!
|
755
|
+
|
756
|
+
You might want to checkout our [Contributing][cb] wiki page for information
|
757
|
+
on coding standards, new features, etc.
|
758
|
+
|
759
|
+
|
760
|
+
Mailing List
|
761
|
+
------------
|
762
|
+
|
763
|
+
To join the list simply send an email to <resque@librelist.com>. This
|
764
|
+
will subscribe you and send you information about your subscription,
|
765
|
+
including unsubscribe information.
|
766
|
+
|
767
|
+
The archive can be found at <http://librelist.com/browser/>.
|
768
|
+
|
769
|
+
|
770
|
+
Meta
|
771
|
+
----
|
772
|
+
|
773
|
+
* Code: `git clone git://github.com/defunkt/resque.git`
|
774
|
+
* Home: <http://github.com/defunkt/resque>
|
775
|
+
* Docs: <http://defunkt.github.com/resque/>
|
776
|
+
* Bugs: <http://github.com/defunkt/resque/issues>
|
777
|
+
* List: <resque@librelist.com>
|
778
|
+
* Chat: <irc://irc.freenode.net/resque>
|
779
|
+
* Gems: <http://gemcutter.org/gems/resque>
|
780
|
+
|
781
|
+
This project uses [Semantic Versioning][sv].
|
782
|
+
|
783
|
+
|
784
|
+
Author
|
785
|
+
------
|
786
|
+
|
787
|
+
Chris Wanstrath :: chris@ozmm.org :: @defunkt
|
788
|
+
|
789
|
+
[0]: http://github.com/blog/542-introducing-resque
|
790
|
+
[1]: http://help.github.com/forking/
|
791
|
+
[2]: http://github.com/defunkt/resque/issues
|
792
|
+
[sv]: http://semver.org/
|
793
|
+
[rs]: http://github.com/defunkt/redis-namespace
|
794
|
+
[cb]: http://wiki.github.com/defunkt/resque/contributing
|