opengotham_resque 1.8.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- data/.gitignore +2 -0
- data/.kick +26 -0
- data/HISTORY.md +142 -0
- data/LICENSE +20 -0
- data/README.markdown +794 -0
- data/Rakefile +112 -0
- data/bin/resque +57 -0
- data/bin/resque-web +23 -0
- data/config.ru +14 -0
- data/deps.rip +7 -0
- data/docs/HOOKS.md +121 -0
- data/docs/PLUGINS.md +93 -0
- data/examples/async_helper.rb +31 -0
- data/examples/demo/README.markdown +71 -0
- data/examples/demo/Rakefile +8 -0
- data/examples/demo/app.rb +38 -0
- data/examples/demo/config.ru +19 -0
- data/examples/demo/job.rb +22 -0
- data/examples/god/resque.god +53 -0
- data/examples/god/stale.god +26 -0
- data/examples/instance.rb +11 -0
- data/examples/monit/resque.monit +6 -0
- data/examples/simple.rb +30 -0
- data/init.rb +1 -0
- data/lib/resque.rb +287 -0
- data/lib/resque/errors.rb +10 -0
- data/lib/resque/failure.rb +66 -0
- data/lib/resque/failure/base.rb +61 -0
- data/lib/resque/failure/hoptoad.rb +132 -0
- data/lib/resque/failure/multiple.rb +48 -0
- data/lib/resque/failure/redis.rb +40 -0
- data/lib/resque/helpers.rb +63 -0
- data/lib/resque/job.rb +207 -0
- data/lib/resque/plugin.rb +46 -0
- data/lib/resque/server.rb +201 -0
- data/lib/resque/server/public/idle.png +0 -0
- data/lib/resque/server/public/jquery-1.3.2.min.js +19 -0
- data/lib/resque/server/public/jquery.relatize_date.js +95 -0
- data/lib/resque/server/public/poll.png +0 -0
- data/lib/resque/server/public/ranger.js +67 -0
- data/lib/resque/server/public/reset.css +48 -0
- data/lib/resque/server/public/style.css +81 -0
- data/lib/resque/server/public/working.png +0 -0
- data/lib/resque/server/test_helper.rb +19 -0
- data/lib/resque/server/views/error.erb +1 -0
- data/lib/resque/server/views/failed.erb +53 -0
- data/lib/resque/server/views/key_sets.erb +20 -0
- data/lib/resque/server/views/key_string.erb +11 -0
- data/lib/resque/server/views/layout.erb +44 -0
- data/lib/resque/server/views/next_more.erb +10 -0
- data/lib/resque/server/views/overview.erb +4 -0
- data/lib/resque/server/views/queues.erb +49 -0
- data/lib/resque/server/views/stats.erb +62 -0
- data/lib/resque/server/views/workers.erb +78 -0
- data/lib/resque/server/views/working.erb +69 -0
- data/lib/resque/stat.rb +53 -0
- data/lib/resque/tasks.rb +39 -0
- data/lib/resque/version.rb +3 -0
- data/lib/resque/worker.rb +478 -0
- data/tasks/redis.rake +159 -0
- data/tasks/resque.rake +2 -0
- data/test/job_hooks_test.rb +302 -0
- data/test/job_plugins_test.rb +209 -0
- data/test/plugin_test.rb +116 -0
- data/test/redis-test.conf +132 -0
- data/test/resque-web_test.rb +54 -0
- data/test/resque_test.rb +225 -0
- data/test/test_helper.rb +111 -0
- data/test/worker_test.rb +302 -0
- metadata +199 -0
data/.gitignore
ADDED
data/.kick
ADDED
@@ -0,0 +1,26 @@
|
|
1
|
+
# take control of the growl notifications
|
2
|
+
module GrowlHacks
|
3
|
+
def growl(type, subject, body, *args, &block)
|
4
|
+
case type
|
5
|
+
when Kicker::GROWL_NOTIFICATIONS[:succeeded]
|
6
|
+
puts subject = "Success"
|
7
|
+
body = body.split("\n").last
|
8
|
+
when Kicker::GROWL_NOTIFICATIONS[:failed]
|
9
|
+
subject = "Failure"
|
10
|
+
puts body
|
11
|
+
body = body.split("\n").last
|
12
|
+
else
|
13
|
+
return nil
|
14
|
+
end
|
15
|
+
super(type, subject, body, *args, &block)
|
16
|
+
end
|
17
|
+
end
|
18
|
+
|
19
|
+
Kicker.send :extend, GrowlHacks
|
20
|
+
|
21
|
+
# no logging
|
22
|
+
Kicker::Utils.module_eval do
|
23
|
+
def log(message)
|
24
|
+
nil
|
25
|
+
end
|
26
|
+
end
|
data/HISTORY.md
ADDED
@@ -0,0 +1,142 @@
|
|
1
|
+
## 1.8.2 (2010-05-03)
|
2
|
+
|
3
|
+
* Bugfix: Include "tasks/" dir in RubyGem
|
4
|
+
|
5
|
+
## 1.8.1 (2010-04-29)
|
6
|
+
|
7
|
+
* Bugfix: Multiple failure backend did not support requeue-ing failed jobs
|
8
|
+
* Bugfix: Fix /failed when error has no backtrace
|
9
|
+
* Bugfix: Add `Redis::DistRedis` as a valid client
|
10
|
+
|
11
|
+
## 1.8.0 (2010-04-07)
|
12
|
+
|
13
|
+
* Jobs that never complete due to killed worker are now failed.
|
14
|
+
* Worker "working" state is now maintained by the parent, not the child.
|
15
|
+
* Stopped using deprecated redis.rb methods
|
16
|
+
* `Worker.working` race condition fixed
|
17
|
+
* `Worker#process` has been deprecated.
|
18
|
+
* Monit example fixed
|
19
|
+
* Redis::Client and Redis::Namespace can be passed to `Resque.redis=`
|
20
|
+
|
21
|
+
## 1.7.1 (2010-04-02)
|
22
|
+
|
23
|
+
* Bugfix: Make job hook execution order consistent
|
24
|
+
* Bugfix: stdout buffering in child process
|
25
|
+
|
26
|
+
## 1.7.0 (2010-03-31)
|
27
|
+
|
28
|
+
* Job hooks API. See docs/HOOKS.md.
|
29
|
+
* web: Hovering over dates shows a timestamp
|
30
|
+
* web: AJAXify retry action for failed jobs
|
31
|
+
* web bugfix: Fix pagination bug
|
32
|
+
|
33
|
+
## 1.6.1 (2010-03-25)
|
34
|
+
|
35
|
+
* Bugfix: Workers may not be clearing their state correctly on
|
36
|
+
shutdown
|
37
|
+
* Added example monit config.
|
38
|
+
* Exception class is now recorded when an error is raised in a
|
39
|
+
worker.
|
40
|
+
* web: Unit tests
|
41
|
+
* web: Show namespace in header and footer
|
42
|
+
* web: Remove a queue
|
43
|
+
* web: Retry failed jobs
|
44
|
+
|
45
|
+
## 1.6.0 (2010-03-09)
|
46
|
+
|
47
|
+
* Added `before_first_fork`, `before_fork`, and `after_fork` hooks.
|
48
|
+
* Hoptoad: Added server_environment config setting
|
49
|
+
* Hoptoad bugfix: Don't depend on RAILS_ROOT
|
50
|
+
* 1.8.6 compat fixes
|
51
|
+
|
52
|
+
## 1.5.2 (2010-03-03)
|
53
|
+
|
54
|
+
* Bugfix: JSON check was crazy.
|
55
|
+
|
56
|
+
## 1.5.1 (2010-03-03)
|
57
|
+
|
58
|
+
* `Job.destroy` and `Resque.dequeue` return the # of destroyed jobs.
|
59
|
+
* Hoptoad notifier improvements
|
60
|
+
* Specify the namespace with `resque-web` by passing `-N namespace`
|
61
|
+
* Bugfix: Don't crash when trying to parse invalid JSON.
|
62
|
+
* Bugfix: Non-standard namespace support
|
63
|
+
* Web: Red backgound for queue "failed" only shown if there are failed jobs.
|
64
|
+
* Web bugfix: Tabs highlight properly now
|
65
|
+
* Web bugfix: ZSET partial support in stats
|
66
|
+
* Web bugfix: Deleting failed jobs works again
|
67
|
+
* Web bugfix: Sets (or zsets, lists, etc) now paginate.
|
68
|
+
|
69
|
+
## 1.5.0 (2010-02-17)
|
70
|
+
|
71
|
+
* Version now included in procline, e.g. `resque-1.5.0: Message`
|
72
|
+
* Web bugfix: Ignore idle works in the "working" page
|
73
|
+
* Added `Resque::Job.destroy(queue, klass, *args)`
|
74
|
+
* Added `Resque.dequeue(klass, *args)`
|
75
|
+
|
76
|
+
## 1.4.0 (2010-02-11)
|
77
|
+
|
78
|
+
* Fallback when unable to bind QUIT and USR1 for Windows and JRuby.
|
79
|
+
* Fallback when no `Kernel.fork` is provided (for IronRuby).
|
80
|
+
* Web: Rounded corners in Firefox
|
81
|
+
* Cut down system calls in `Worker#prune_dead_workers`
|
82
|
+
* Enable switching DB in a Redis server from config
|
83
|
+
* Support USR2 and CONT to stop and start job processing.
|
84
|
+
* Web: Add example failing job
|
85
|
+
* Bugfix: `Worker#unregister_worker` shouldn't call `done_working`
|
86
|
+
* Bugfix: Example god config now restarts Resque properly.
|
87
|
+
* Multiple failure backends now permitted.
|
88
|
+
* Hoptoad failure backend updated to new API
|
89
|
+
|
90
|
+
## 1.3.1 (2010-01-11)
|
91
|
+
|
92
|
+
* Vegas bugfix: Don't error without a config
|
93
|
+
|
94
|
+
## 1.3.0 (2010-01-11)
|
95
|
+
|
96
|
+
* Use Vegas for resque-web
|
97
|
+
* Web Bugfix: Show proper date/time value for failed_at on Failures
|
98
|
+
* Web Bugfix: Make the / route more flexible
|
99
|
+
* Add Resque::Server.tabs array (so plugins can add their own tabs)
|
100
|
+
* Start using [Semantic Versioning](http://semver.org/)
|
101
|
+
|
102
|
+
## 1.2.4 (2009-12-15)
|
103
|
+
|
104
|
+
* Web Bugfix: fix key links on stat page
|
105
|
+
|
106
|
+
## 1.2.3 (2009-12-15)
|
107
|
+
|
108
|
+
* Bugfix: Fixed `rand` seeding in child processes.
|
109
|
+
* Bugfix: Better JSON encoding/decoding without Yajl.
|
110
|
+
* Bugfix: Avoid `ps` flag error on Linux
|
111
|
+
* Add `PREFIX` observance to `rake` install tasks.
|
112
|
+
|
113
|
+
## 1.2.2 (2009-12-08)
|
114
|
+
|
115
|
+
* Bugfix: Job equality was not properly implemented.
|
116
|
+
|
117
|
+
## 1.2.1 (2009-12-07)
|
118
|
+
|
119
|
+
* Added `rake resque:workers` task for starting multiple workers.
|
120
|
+
* 1.9.x compatibility
|
121
|
+
* Bugfix: Yajl decoder doesn't care about valid UTF-8
|
122
|
+
* config.ru loads RESQUECONFIG if the ENV variable is set.
|
123
|
+
* `resque-web` now sets RESQUECONFIG
|
124
|
+
* Job objects know if they are equal.
|
125
|
+
* Jobs can be re-queued using `Job#recreate`
|
126
|
+
|
127
|
+
## 1.2.0 (2009-11-25)
|
128
|
+
|
129
|
+
* If USR1 is sent and no child is found, shutdown.
|
130
|
+
* Raise when a job class does not respond to `perform`.
|
131
|
+
* Added `Resque.remove_queue` for deleting a queue
|
132
|
+
|
133
|
+
## 1.1.0 (2009-11-04)
|
134
|
+
|
135
|
+
* Bugfix: Broken ERB tag in failure UI
|
136
|
+
* Bugfix: Save the worker's ID, not the worker itself, in the failure module
|
137
|
+
* Redesigned the sinatra web interface
|
138
|
+
* Added option to clear failed jobs
|
139
|
+
|
140
|
+
## 1.0.0 (2009-11-03)
|
141
|
+
|
142
|
+
* First release.
|
data/LICENSE
ADDED
@@ -0,0 +1,20 @@
|
|
1
|
+
Copyright (c) 2009 Chris Wanstrath
|
2
|
+
|
3
|
+
Permission is hereby granted, free of charge, to any person obtaining
|
4
|
+
a copy of this software and associated documentation files (the
|
5
|
+
"Software"), to deal in the Software without restriction, including
|
6
|
+
without limitation the rights to use, copy, modify, merge, publish,
|
7
|
+
distribute, sublicense, and/or sell copies of the Software, and to
|
8
|
+
permit persons to whom the Software is furnished to do so, subject to
|
9
|
+
the following conditions:
|
10
|
+
|
11
|
+
The above copyright notice and this permission notice shall be
|
12
|
+
included in all copies or substantial portions of the Software.
|
13
|
+
|
14
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
15
|
+
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
16
|
+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
17
|
+
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
18
|
+
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
19
|
+
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
20
|
+
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
data/README.markdown
ADDED
@@ -0,0 +1,794 @@
|
|
1
|
+
Resque
|
2
|
+
======
|
3
|
+
|
4
|
+
Resque is a Redis-backed library for creating background jobs, placing
|
5
|
+
those jobs on multiple queues, and processing them later.
|
6
|
+
|
7
|
+
Background jobs can be any Ruby class or module that responds to
|
8
|
+
`perform`. Your existing classes can easily be converted to background
|
9
|
+
jobs or you can create new classes specifically to do work. Or, you
|
10
|
+
can do both.
|
11
|
+
|
12
|
+
Resque is heavily inspired by DelayedJob (which rocks) and is
|
13
|
+
comprised of three parts:
|
14
|
+
|
15
|
+
1. A Ruby library for creating, querying, and processing jobs
|
16
|
+
2. A Rake task for starting a worker which processes jobs
|
17
|
+
3. A Sinatra app for monitoring queues, jobs, and workers.
|
18
|
+
|
19
|
+
Resque workers can be distributed between multiple machines,
|
20
|
+
support priorities, are resilient to memory bloat / "leaks," are
|
21
|
+
optimized for REE (but work on MRI and JRuby), tell you what they're
|
22
|
+
doing, and expect failure.
|
23
|
+
|
24
|
+
Resque queues are persistent; support constant time, atomic push and
|
25
|
+
pop (thanks to Redis); provide visibility into their contents; and
|
26
|
+
store jobs as simple JSON packages.
|
27
|
+
|
28
|
+
The Resque frontend tells you what workers are doing, what workers are
|
29
|
+
not doing, what queues you're using, what's in those queues, provides
|
30
|
+
general usage stats, and helps you track failures.
|
31
|
+
|
32
|
+
|
33
|
+
The Blog Post
|
34
|
+
-------------
|
35
|
+
|
36
|
+
For the backstory, philosophy, and history of Resque's beginnings,
|
37
|
+
please see [the blog post][0].
|
38
|
+
|
39
|
+
|
40
|
+
Overview
|
41
|
+
--------
|
42
|
+
|
43
|
+
Resque allows you to create jobs and place them on a queue, then,
|
44
|
+
later, pull those jobs off the queue and process them.
|
45
|
+
|
46
|
+
Resque jobs are Ruby classes (or modules) which respond to the
|
47
|
+
`perform` method. Here's an example:
|
48
|
+
|
49
|
+
class Archive
|
50
|
+
@queue = :file_serve
|
51
|
+
|
52
|
+
def self.perform(repo_id, branch = 'master')
|
53
|
+
repo = Repository.find(repo_id)
|
54
|
+
repo.create_archive(branch)
|
55
|
+
end
|
56
|
+
end
|
57
|
+
|
58
|
+
The `@queue` class instance variable determines which queue `Archive`
|
59
|
+
jobs will be placed in. Queues are arbitrary and created on the fly -
|
60
|
+
you can name them whatever you want and have as many as you want.
|
61
|
+
|
62
|
+
To place an `Archive` job on the `file_serve` queue, we might add this
|
63
|
+
to our application's pre-existing `Repository` class:
|
64
|
+
|
65
|
+
class Repository
|
66
|
+
def async_create_archive(branch)
|
67
|
+
Resque.enqueue(Archive, self.id, branch)
|
68
|
+
end
|
69
|
+
end
|
70
|
+
|
71
|
+
Now when we call `repo.async_create_archive('masterbrew')` in our
|
72
|
+
application, a job will be created and placed on the `file_serve`
|
73
|
+
queue.
|
74
|
+
|
75
|
+
Later, a worker will run something like this code to process the job:
|
76
|
+
|
77
|
+
klass, args = Resque.reserve(:file_serve)
|
78
|
+
klass.perform(*args) if klass.respond_to? :perform
|
79
|
+
|
80
|
+
Which translates to:
|
81
|
+
|
82
|
+
Archive.perform(44, 'masterbrew')
|
83
|
+
|
84
|
+
Let's start a worker to run `file_serve` jobs:
|
85
|
+
|
86
|
+
$ cd app_root
|
87
|
+
$ QUEUE=file_serve rake resque:work
|
88
|
+
|
89
|
+
This starts one Resque worker and tells it to work off the
|
90
|
+
`file_serve` queue. As soon as it's ready it'll try to run the
|
91
|
+
`Resque.reserve` code snippet above and process jobs until it can't
|
92
|
+
find any more, at which point it will sleep for a small period and
|
93
|
+
repeatedly poll the queue for more jobs.
|
94
|
+
|
95
|
+
Workers can be given multiple queues (a "queue list") and run on
|
96
|
+
multiple machines. In fact they can be run anywhere with network
|
97
|
+
access to the Redis server.
|
98
|
+
|
99
|
+
|
100
|
+
Jobs
|
101
|
+
----
|
102
|
+
|
103
|
+
What should you run in the background? Anything that takes any time at
|
104
|
+
all. Slow INSERT statements, disk manipulating, data processing, etc.
|
105
|
+
|
106
|
+
At GitHub we use Resque to process the following types of jobs:
|
107
|
+
|
108
|
+
* Warming caches
|
109
|
+
* Counting disk usage
|
110
|
+
* Building tarballs
|
111
|
+
* Building Rubygems
|
112
|
+
* Firing off web hooks
|
113
|
+
* Creating events in the db and pre-caching them
|
114
|
+
* Building graphs
|
115
|
+
* Deleting users
|
116
|
+
* Updating our search index
|
117
|
+
|
118
|
+
As of writing we have about 35 different types of background jobs.
|
119
|
+
|
120
|
+
Keep in mind that you don't need a web app to use Resque - we just
|
121
|
+
mention "foreground" and "background" because they make conceptual
|
122
|
+
sense. You could easily be spidering sites and sticking data which
|
123
|
+
needs to be crunched later into a queue.
|
124
|
+
|
125
|
+
|
126
|
+
### Persistence
|
127
|
+
|
128
|
+
Jobs are persisted to queues as JSON objects. Let's take our `Archive`
|
129
|
+
example from above. We'll run the following code to create a job:
|
130
|
+
|
131
|
+
repo = Repository.find(44)
|
132
|
+
repo.async_create_archive('masterbrew')
|
133
|
+
|
134
|
+
The following JSON will be stored in the `file_serve` queue:
|
135
|
+
|
136
|
+
{
|
137
|
+
'class': 'Archive',
|
138
|
+
'args': [ 44, 'masterbrew' ]
|
139
|
+
}
|
140
|
+
|
141
|
+
Because of this your jobs must only accept arguments that can be JSON encoded.
|
142
|
+
|
143
|
+
So instead of doing this:
|
144
|
+
|
145
|
+
Resque.enqueue(Archive, self, branch)
|
146
|
+
|
147
|
+
do this:
|
148
|
+
|
149
|
+
Resque.enqueue(Archive, self.id, branch)
|
150
|
+
|
151
|
+
This is why our above example (and all the examples in `examples/`)
|
152
|
+
uses object IDs instead of passing around the objects.
|
153
|
+
|
154
|
+
While this is less convenient than just sticking a marshaled object
|
155
|
+
in the database, it gives you a slight advantage: your jobs will be
|
156
|
+
run against the most recent version of an object because they need to
|
157
|
+
pull from the DB or cache.
|
158
|
+
|
159
|
+
If your jobs were run against marshaled objects, they could
|
160
|
+
potentially be operating on a stale record with out-of-date information.
|
161
|
+
|
162
|
+
|
163
|
+
### send_later / async
|
164
|
+
|
165
|
+
Want something like DelayedJob's `send_later` or the ability to use
|
166
|
+
instance methods instead of just methods for jobs? See the `examples/`
|
167
|
+
directory for goodies.
|
168
|
+
|
169
|
+
We plan to provide first class `async` support in a future release.
|
170
|
+
|
171
|
+
|
172
|
+
### Failure
|
173
|
+
|
174
|
+
If a job raises an exception, it is logged and handed off to the
|
175
|
+
`Resque::Failure` module. Failures are logged either locally in Redis
|
176
|
+
or using some different backend.
|
177
|
+
|
178
|
+
For example, Resque ships with Hoptoad support.
|
179
|
+
|
180
|
+
Keep this in mind when writing your jobs: you may want to throw
|
181
|
+
exceptions you would not normally throw in order to assist debugging.
|
182
|
+
|
183
|
+
|
184
|
+
Workers
|
185
|
+
-------
|
186
|
+
|
187
|
+
Resque workers are rake tasks that run forever. They basically do this:
|
188
|
+
|
189
|
+
start
|
190
|
+
loop do
|
191
|
+
if job = reserve
|
192
|
+
job.process
|
193
|
+
else
|
194
|
+
sleep 5
|
195
|
+
end
|
196
|
+
end
|
197
|
+
shutdown
|
198
|
+
|
199
|
+
Starting a worker is simple. Here's our example from earlier:
|
200
|
+
|
201
|
+
$ QUEUE=file_serve rake resque:work
|
202
|
+
|
203
|
+
By default Resque won't know about your application's
|
204
|
+
environment. That is, it won't be able to find and run your jobs - it
|
205
|
+
needs to load your application into memory.
|
206
|
+
|
207
|
+
If we've installed Resque as a Rails plugin, we might run this command
|
208
|
+
from our RAILS_ROOT:
|
209
|
+
|
210
|
+
$ QUEUE=file_serve rake environment resque:work
|
211
|
+
|
212
|
+
This will load the environment before starting a worker. Alternately
|
213
|
+
we can define a `resque:setup` task with a dependency on the
|
214
|
+
`environment` rake task:
|
215
|
+
|
216
|
+
task "resque:setup" => :environment
|
217
|
+
|
218
|
+
GitHub's setup task looks like this:
|
219
|
+
|
220
|
+
task "resque:setup" => :environment do
|
221
|
+
Grit::Git.git_timeout = 10.minutes
|
222
|
+
end
|
223
|
+
|
224
|
+
We don't want the `git_timeout` as high as 10 minutes in our web app,
|
225
|
+
but in the Resque workers it's fine.
|
226
|
+
|
227
|
+
|
228
|
+
### Logging
|
229
|
+
|
230
|
+
Workers support basic logging to STDOUT. If you start them with the
|
231
|
+
`VERBOSE` env variable set, they will print basic debugging
|
232
|
+
information. You can also set the `VVERBOSE` (very verbose) env
|
233
|
+
variable.
|
234
|
+
|
235
|
+
$ VVERBOSE=1 QUEUE=file_serve rake environment resque:work
|
236
|
+
|
237
|
+
|
238
|
+
### Priorities and Queue Lists
|
239
|
+
|
240
|
+
Resque doesn't support numeric priorities but instead uses the order
|
241
|
+
of queues you give it. We call this list of queues the "queue list."
|
242
|
+
|
243
|
+
Let's say we add a `warm_cache` queue in addition to our `file_serve`
|
244
|
+
queue. We'd now start a worker like so:
|
245
|
+
|
246
|
+
$ QUEUES=file_serve,warm_cache rake resque:work
|
247
|
+
|
248
|
+
When the worker looks for new jobs, it will first check
|
249
|
+
`file_serve`. If it finds a job, it'll process it then check
|
250
|
+
`file_serve` again. It will keep checking `file_serve` until no more
|
251
|
+
jobs are available. At that point, it will check `warm_cache`. If it
|
252
|
+
finds a job it'll process it then check `file_serve` (repeating the
|
253
|
+
whole process).
|
254
|
+
|
255
|
+
In this way you can prioritize certain queues. At GitHub we start our
|
256
|
+
workers with something like this:
|
257
|
+
|
258
|
+
$ QUEUES=critical,archive,high,low rake resque:work
|
259
|
+
|
260
|
+
Notice the `archive` queue - it is specialized and in our future
|
261
|
+
architecture will only be run from a single machine.
|
262
|
+
|
263
|
+
At that point we'll start workers on our generalized background
|
264
|
+
machines with this command:
|
265
|
+
|
266
|
+
$ QUEUES=critical,high,low rake resque:work
|
267
|
+
|
268
|
+
And workers on our specialized archive machine with this command:
|
269
|
+
|
270
|
+
$ QUEUE=archive rake resque:work
|
271
|
+
|
272
|
+
|
273
|
+
### Running All Queues
|
274
|
+
|
275
|
+
If you want your workers to work off of every queue, including new
|
276
|
+
queues created on the fly, you can use a splat:
|
277
|
+
|
278
|
+
$ QUEUE=* rake resque:work
|
279
|
+
|
280
|
+
Queues will be processed in alphabetical order.
|
281
|
+
|
282
|
+
|
283
|
+
### Running Multiple Workers
|
284
|
+
|
285
|
+
At GitHub we use god to start and stop multiple workers. A sample god
|
286
|
+
configuration file is included under `examples/god`. We recommend this
|
287
|
+
method.
|
288
|
+
|
289
|
+
If you'd like to run multiple workers in development mode, you can do
|
290
|
+
so using the `resque:workers` rake task:
|
291
|
+
|
292
|
+
$ COUNT=5 QUEUE=* rake resque:workers
|
293
|
+
|
294
|
+
This will spawn five Resque workers, each in its own thread. Hitting
|
295
|
+
ctrl-c should be sufficient to stop them all.
|
296
|
+
|
297
|
+
|
298
|
+
### Forking
|
299
|
+
|
300
|
+
On certain platforms, when a Resque worker reserves a job it
|
301
|
+
immediately forks a child process. The child processes the job then
|
302
|
+
exits. When the child has exited successfully, the worker reserves
|
303
|
+
another job and repeats the process.
|
304
|
+
|
305
|
+
Why?
|
306
|
+
|
307
|
+
Because Resque assumes chaos.
|
308
|
+
|
309
|
+
Resque assumes your background workers will lock up, run too long, or
|
310
|
+
have unwanted memory growth.
|
311
|
+
|
312
|
+
If Resque workers processed jobs themselves, it'd be hard to whip them
|
313
|
+
into shape. Let's say one is using too much memory: you send it a
|
314
|
+
signal that says "shutdown after you finish processing the current
|
315
|
+
job," and it does so. It then starts up again - loading your entire
|
316
|
+
application environment. This adds useless CPU cycles and causes a
|
317
|
+
delay in queue processing.
|
318
|
+
|
319
|
+
Plus, what if it's using too much memory and has stopped responding to
|
320
|
+
signals?
|
321
|
+
|
322
|
+
Thanks to Resque's parent / child architecture, jobs that use too much memory
|
323
|
+
release that memory upon completion. No unwanted growth.
|
324
|
+
|
325
|
+
And what if a job is running too long? You'd need to `kill -9` it then
|
326
|
+
start the worker again. With Resque's parent / child architecture you
|
327
|
+
can tell the parent to forcefully kill the child then immediately
|
328
|
+
start processing more jobs. No startup delay or wasted cycles.
|
329
|
+
|
330
|
+
The parent / child architecture helps us keep tabs on what workers are
|
331
|
+
doing, too. By eliminating the need to `kill -9` workers we can have
|
332
|
+
parents remove themselves from the global listing of workers. If we
|
333
|
+
just ruthlessly killed workers, we'd need a separate watchdog process
|
334
|
+
to add and remove them to the global listing - which becomes
|
335
|
+
complicated.
|
336
|
+
|
337
|
+
Workers instead handle their own state.
|
338
|
+
|
339
|
+
|
340
|
+
### Parents and Children
|
341
|
+
|
342
|
+
Here's a parent / child pair doing some work:
|
343
|
+
|
344
|
+
$ ps -e -o pid,command | grep [r]esque
|
345
|
+
92099 resque: Forked 92102 at 1253142769
|
346
|
+
92102 resque: Processing file_serve since 1253142769
|
347
|
+
|
348
|
+
You can clearly see that process 92099 forked 92102, which has been
|
349
|
+
working since 1253142769.
|
350
|
+
|
351
|
+
(By advertising the time they began processing you can easily use monit
|
352
|
+
or god to kill stale workers.)
|
353
|
+
|
354
|
+
When a parent process is idle, it lets you know what queues it is
|
355
|
+
waiting for work on:
|
356
|
+
|
357
|
+
$ ps -e -o pid,command | grep [r]esque
|
358
|
+
92099 resque: Waiting for file_serve,warm_cache
|
359
|
+
|
360
|
+
|
361
|
+
### Signals
|
362
|
+
|
363
|
+
Resque workers respond to a few different signals:
|
364
|
+
|
365
|
+
* `QUIT` - Wait for child to finish processing then exit
|
366
|
+
* `TERM` / `INT` - Immediately kill child then exit
|
367
|
+
* `USR1` - Immediately kill child but don't exit
|
368
|
+
* `USR2` - Don't start to process any new jobs
|
369
|
+
* `CONT` - Start to process new jobs again after a USR2
|
370
|
+
|
371
|
+
If you want to gracefully shutdown a Resque worker, use `QUIT`.
|
372
|
+
|
373
|
+
If you want to kill a stale or stuck child, use `USR1`. Processing
|
374
|
+
will continue as normal unless the child was not found. In that case
|
375
|
+
Resque assumes the parent process is in a bad state and shuts down.
|
376
|
+
|
377
|
+
If you want to kill a stale or stuck child and shutdown, use `TERM`
|
378
|
+
|
379
|
+
If you want to stop processing jobs, but want to leave the worker running
|
380
|
+
(for example, to temporarily alleviate load), use `USR2` to stop processing,
|
381
|
+
then `CONT` to start it again.
|
382
|
+
|
383
|
+
### Mysql::Error: MySQL server has gone away
|
384
|
+
|
385
|
+
If your workers remain idle for too long they may lose their MySQL
|
386
|
+
connection. If that happens we recommend using [this
|
387
|
+
Gist](http://gist.github.com/238999).
|
388
|
+
|
389
|
+
|
390
|
+
The Front End
|
391
|
+
-------------
|
392
|
+
|
393
|
+
Resque comes with a Sinatra-based front end for seeing what's up with
|
394
|
+
your queue.
|
395
|
+
|
396
|
+

|
397
|
+
|
398
|
+
### Standalone
|
399
|
+
|
400
|
+
If you've installed Resque as a gem running the front end standalone is easy:
|
401
|
+
|
402
|
+
$ resque-web
|
403
|
+
|
404
|
+
It's a thin layer around `rackup` so it's configurable as well:
|
405
|
+
|
406
|
+
$ resque-web -p 8282
|
407
|
+
|
408
|
+
If you have a Resque config file you want evaluated just pass it to
|
409
|
+
the script as the final argument:
|
410
|
+
|
411
|
+
$ resque-web -p 8282 rails_root/config/initializers/resque.rb
|
412
|
+
|
413
|
+
You can also set the namespace directly using `resque-web`:
|
414
|
+
|
415
|
+
$ resque-web -p 8282 -N myapp
|
416
|
+
|
417
|
+
### Passenger
|
418
|
+
|
419
|
+
Using Passenger? Resque ships with a `config.ru` you can use. See
|
420
|
+
Phusion's guide:
|
421
|
+
|
422
|
+
<http://www.modrails.com/documentation/Users%20guide.html#_deploying_a_rack_based_ruby_application>
|
423
|
+
|
424
|
+
### Rack::URLMap
|
425
|
+
|
426
|
+
If you want to load Resque on a subpath, possibly alongside other
|
427
|
+
apps, it's easy to do with Rack's `URLMap`:
|
428
|
+
|
429
|
+
require 'resque/server'
|
430
|
+
|
431
|
+
run Rack::URLMap.new \
|
432
|
+
"/" => Your::App.new,
|
433
|
+
"/resque" => Resque::Server.new
|
434
|
+
|
435
|
+
Check `examples/demo/config.ru` for a functional example (including
|
436
|
+
HTTP basic auth).
|
437
|
+
|
438
|
+
|
439
|
+
Resque vs DelayedJob
|
440
|
+
--------------------
|
441
|
+
|
442
|
+
How does Resque compare to DelayedJob, and why would you choose one
|
443
|
+
over the other?
|
444
|
+
|
445
|
+
* Resque supports multiple queues
|
446
|
+
* DelayedJob supports finer grained priorities
|
447
|
+
* Resque workers are resilient to memory leaks / bloat
|
448
|
+
* DelayedJob workers are extremely simple and easy to modify
|
449
|
+
* Resque requires Redis
|
450
|
+
* DelayedJob requires ActiveRecord
|
451
|
+
* Resque can only place JSONable Ruby objects on a queue as arguments
|
452
|
+
* DelayedJob can place _any_ Ruby object on its queue as arguments
|
453
|
+
* Resque includes a Sinatra app for monitoring what's going on
|
454
|
+
* DelayedJob can be queried from within your Rails app if you want to
|
455
|
+
add an interface
|
456
|
+
|
457
|
+
If you're doing Rails development, you already have a database and
|
458
|
+
ActiveRecord. DelayedJob is super easy to setup and works great.
|
459
|
+
GitHub used it for many months to process almost 200 million jobs.
|
460
|
+
|
461
|
+
Choose Resque if:
|
462
|
+
|
463
|
+
* You need multiple queues
|
464
|
+
* You don't care / dislike numeric priorities
|
465
|
+
* You don't need to persist every Ruby object ever
|
466
|
+
* You have potentially huge queues
|
467
|
+
* You want to see what's going on
|
468
|
+
* You expect a lot of failure / chaos
|
469
|
+
* You can setup Redis
|
470
|
+
* You're not running short on RAM
|
471
|
+
|
472
|
+
Choose DelayedJob if:
|
473
|
+
|
474
|
+
* You like numeric priorities
|
475
|
+
* You're not doing a gigantic amount of jobs each day
|
476
|
+
* Your queue stays small and nimble
|
477
|
+
* There is not a lot failure / chaos
|
478
|
+
* You want to easily throw anything on the queue
|
479
|
+
* You don't want to setup Redis
|
480
|
+
|
481
|
+
In no way is Resque a "better" DelayedJob, so make sure you pick the
|
482
|
+
tool that's best for your app.
|
483
|
+
|
484
|
+
|
485
|
+
Installing Redis
|
486
|
+
----------------
|
487
|
+
|
488
|
+
Resque requires Redis 0.900 or higher.
|
489
|
+
|
490
|
+
Resque uses Redis' lists for its queues. It also stores worker state
|
491
|
+
data in Redis.
|
492
|
+
|
493
|
+
#### Homebrew
|
494
|
+
|
495
|
+
If you're on OS X, Homebrew is the simplest way to install Redis:
|
496
|
+
|
497
|
+
$ brew install redis
|
498
|
+
$ redis-server /usr/local/etc/redis.conf
|
499
|
+
|
500
|
+
You now have a Redis daemon running on 6379.
|
501
|
+
|
502
|
+
#### Via Resque
|
503
|
+
|
504
|
+
Resque includes Rake tasks (thanks to Ezra's redis-rb) that will
|
505
|
+
install and run Redis for you:
|
506
|
+
|
507
|
+
$ git clone git://github.com/defunkt/resque.git
|
508
|
+
$ cd resque
|
509
|
+
$ rake redis:install dtach:install
|
510
|
+
$ rake redis:start
|
511
|
+
|
512
|
+
Or, if you don't have admin access on your machine:
|
513
|
+
|
514
|
+
$ git clone git://github.com/defunkt/resque.git
|
515
|
+
$ cd resque
|
516
|
+
$ PREFIX=<your_prefix> rake redis:install dtach:install
|
517
|
+
$ rake redis:start
|
518
|
+
|
519
|
+
You now have Redis running on 6379. Wait a second then hit ctrl-\ to
|
520
|
+
detach and keep it running in the background.
|
521
|
+
|
522
|
+
The demo is probably the best way to figure out how to put the parts
|
523
|
+
together. But, it's not that hard.
|
524
|
+
|
525
|
+
|
526
|
+
Resque Dependencies
|
527
|
+
-------------------
|
528
|
+
|
529
|
+
gem install redis redis-namespace yajl-ruby
|
530
|
+
|
531
|
+
If you cannot install `yajl-ruby` (JRuby?), you can install the `json`
|
532
|
+
gem and Resque will use it instead.
|
533
|
+
|
534
|
+
When problems arise, make sure you have the newest versions of the
|
535
|
+
`redis` and `redis-namespace` gems.
|
536
|
+
|
537
|
+
|
538
|
+
Installing Resque
|
539
|
+
-----------------
|
540
|
+
|
541
|
+
### In a Rack app, as a gem
|
542
|
+
|
543
|
+
First install the gem.
|
544
|
+
|
545
|
+
$ gem install resque
|
546
|
+
|
547
|
+
Next include it in your application.
|
548
|
+
|
549
|
+
require 'resque'
|
550
|
+
|
551
|
+
Now start your application:
|
552
|
+
|
553
|
+
rackup config.ru
|
554
|
+
|
555
|
+
That's it! You can now create Resque jobs from within your app.
|
556
|
+
|
557
|
+
To start a worker, create a Rakefile in your app's root (or add this
|
558
|
+
to an existing Rakefile):
|
559
|
+
|
560
|
+
require 'your/app'
|
561
|
+
require 'resque/tasks'
|
562
|
+
|
563
|
+
Now:
|
564
|
+
|
565
|
+
$ QUEUE=* rake resque:work
|
566
|
+
|
567
|
+
Alternately you can define a `resque:setup` hook in your Rakefile if you
|
568
|
+
don't want to load your app every time rake runs.
|
569
|
+
|
570
|
+
|
571
|
+
### In a Rails app, as a gem
|
572
|
+
|
573
|
+
First install the gem.
|
574
|
+
|
575
|
+
$ gem install resque
|
576
|
+
|
577
|
+
Next include it in your application.
|
578
|
+
|
579
|
+
$ cat config/initializers/load_resque.rb
|
580
|
+
require 'resque'
|
581
|
+
|
582
|
+
Now start your application:
|
583
|
+
|
584
|
+
$ ./script/server
|
585
|
+
|
586
|
+
That's it! You can now create Resque jobs from within your app.
|
587
|
+
|
588
|
+
To start a worker, add this to your Rakefile in `RAILS_ROOT`:
|
589
|
+
|
590
|
+
require 'resque/tasks'
|
591
|
+
|
592
|
+
Now:
|
593
|
+
|
594
|
+
$ QUEUE=* rake environment resque:work
|
595
|
+
|
596
|
+
Don't forget you can define a `resque:setup` hook in
|
597
|
+
`lib/tasks/whatever.rake` that loads the `environment` task every time.
|
598
|
+
|
599
|
+
|
600
|
+
### In a Rails app, as a plugin
|
601
|
+
|
602
|
+
$ ./script/plugin install git://github.com/defunkt/resque
|
603
|
+
|
604
|
+
That's it! Resque will automatically be available when your Rails app
|
605
|
+
loads.
|
606
|
+
|
607
|
+
To start a worker:
|
608
|
+
|
609
|
+
$ QUEUE=* rake environment resque:work
|
610
|
+
|
611
|
+
Don't forget you can define a `resque:setup` hook in
|
612
|
+
`lib/tasks/whatever.rake` that loads the `environment` task every time.
|
613
|
+
|
614
|
+
|
615
|
+
Configuration
|
616
|
+
-------------
|
617
|
+
|
618
|
+
You may want to change the Redis host and port Resque connects to, or
|
619
|
+
set various other options at startup.
|
620
|
+
|
621
|
+
Resque has a `redis` setter which can be given a string or a Redis
|
622
|
+
object. This means if you're already using Redis in your app, Resque
|
623
|
+
can re-use the existing connection.
|
624
|
+
|
625
|
+
String: `Resque.redis = 'localhost:6379'`
|
626
|
+
|
627
|
+
Redis: `Resque.redis = $redis`
|
628
|
+
|
629
|
+
For our rails app we have a `config/initializers/resque.rb` file where
|
630
|
+
we load `config/resque.yml` by hand and set the Redis information
|
631
|
+
appropriately.
|
632
|
+
|
633
|
+
Here's our `config/resque.yml`:
|
634
|
+
|
635
|
+
development: localhost:6379
|
636
|
+
test: localhost:6379
|
637
|
+
staging: redis1.se.github.com:6379
|
638
|
+
fi: localhost:6379
|
639
|
+
production: redis1.ae.github.com:6379
|
640
|
+
|
641
|
+
And our initializer:
|
642
|
+
|
643
|
+
rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..'
|
644
|
+
rails_env = ENV['RAILS_ENV'] || 'development'
|
645
|
+
|
646
|
+
resque_config = YAML.load_file(rails_root + '/config/resque.yml')
|
647
|
+
Resque.redis = resque_config[rails_env]
|
648
|
+
|
649
|
+
Easy peasy! Why not just use `RAILS_ROOT` and `RAILS_ENV`? Because
|
650
|
+
this way we can tell our Sinatra app about the config file:
|
651
|
+
|
652
|
+
$ RAILS_ENV=production resque-web rails_root/config/initializers/resque.rb
|
653
|
+
|
654
|
+
Now everyone is on the same page.
|
655
|
+
|
656
|
+
|
657
|
+
Plugins and Hooks
|
658
|
+
-----------------
|
659
|
+
|
660
|
+
For a list of available plugins see
|
661
|
+
<http://wiki.github.com/defunkt/resque/plugins>.
|
662
|
+
|
663
|
+
If you'd like to write your own plugin, or want to customize Resque
|
664
|
+
using hooks (such as `Resque.after_fork`), see
|
665
|
+
[docs/HOOKS.md](http://github.com/defunkt/resque/blob/master/HOOKS.md).
|
666
|
+
|
667
|
+
|
668
|
+
Namespaces
|
669
|
+
----------
|
670
|
+
|
671
|
+
If you're running multiple, separate instances of Resque you may want
|
672
|
+
to namespace the keyspaces so they do not overlap. This is not unlike
|
673
|
+
the approach taken by many memcached clients.
|
674
|
+
|
675
|
+
This feature is provided by the [redis-namespace][rs] library, which
|
676
|
+
Resque uses by default to separate the keys it manages from other keys
|
677
|
+
in your Redis server.
|
678
|
+
|
679
|
+
Simply use the `Resque.redis.namespace` accessor:
|
680
|
+
|
681
|
+
Resque.redis.namespace = "resque:GitHub"
|
682
|
+
|
683
|
+
We recommend sticking this in your initializer somewhere after Redis
|
684
|
+
is configured.
|
685
|
+
|
686
|
+
|
687
|
+
Demo
|
688
|
+
----
|
689
|
+
|
690
|
+
Resque ships with a demo Sinatra app for creating jobs that are later
|
691
|
+
processed in the background.
|
692
|
+
|
693
|
+
Try it out by looking at the README, found at `examples/demo/README.markdown`.
|
694
|
+
|
695
|
+
|
696
|
+
Monitoring
|
697
|
+
----------
|
698
|
+
|
699
|
+
### god
|
700
|
+
|
701
|
+
If you're using god to monitor Resque, we have provided example
|
702
|
+
configs in `examples/god/`. One is for starting / stopping workers,
|
703
|
+
the other is for killing workers that have been running too long.
|
704
|
+
|
705
|
+
### monit
|
706
|
+
|
707
|
+
If you're using monit, `examples/monit/resque.monit` is provided free
|
708
|
+
of charge. This is **not** used by GitHub in production, so please
|
709
|
+
send patches for any tweaks or improvements you can make to it.
|
710
|
+
|
711
|
+
|
712
|
+
Development
|
713
|
+
-----------
|
714
|
+
|
715
|
+
Want to hack on Resque?
|
716
|
+
|
717
|
+
First clone the repo and run the tests:
|
718
|
+
|
719
|
+
git clone git://github.com/defunkt/resque.git
|
720
|
+
cd resque
|
721
|
+
rake test
|
722
|
+
|
723
|
+
If the tests do not pass make sure you have Redis installed
|
724
|
+
correctly (though we make an effort to tell you if we feel this is the
|
725
|
+
case). The tests attempt to start an isolated instance of Redis to
|
726
|
+
run against.
|
727
|
+
|
728
|
+
Also make sure you've installed all the dependencies correctly. For
|
729
|
+
example, try loading the `redis-namespace` gem after you've installed
|
730
|
+
it:
|
731
|
+
|
732
|
+
$ irb
|
733
|
+
>> require 'rubygems'
|
734
|
+
=> true
|
735
|
+
>> require 'redis/namespace'
|
736
|
+
=> true
|
737
|
+
|
738
|
+
If you get an error requiring any of the dependencies, you may have
|
739
|
+
failed to install them or be seeing load path issues.
|
740
|
+
|
741
|
+
Feel free to ping the mailing list with your problem and we'll try to
|
742
|
+
sort it out.
|
743
|
+
|
744
|
+
|
745
|
+
Contributing
|
746
|
+
------------
|
747
|
+
|
748
|
+
Once you've made your great commits:
|
749
|
+
|
750
|
+
1. [Fork][1] Resque
|
751
|
+
2. Create a topic branch - `git checkout -b my_branch`
|
752
|
+
3. Push to your branch - `git push origin my_branch`
|
753
|
+
4. Create an [Issue][2] with a link to your branch
|
754
|
+
5. That's it!
|
755
|
+
|
756
|
+
You might want to checkout our [Contributing][cb] wiki page for information
|
757
|
+
on coding standards, new features, etc.
|
758
|
+
|
759
|
+
|
760
|
+
Mailing List
|
761
|
+
------------
|
762
|
+
|
763
|
+
To join the list simply send an email to <resque@librelist.com>. This
|
764
|
+
will subscribe you and send you information about your subscription,
|
765
|
+
including unsubscribe information.
|
766
|
+
|
767
|
+
The archive can be found at <http://librelist.com/browser/>.
|
768
|
+
|
769
|
+
|
770
|
+
Meta
|
771
|
+
----
|
772
|
+
|
773
|
+
* Code: `git clone git://github.com/defunkt/resque.git`
|
774
|
+
* Home: <http://github.com/defunkt/resque>
|
775
|
+
* Docs: <http://defunkt.github.com/resque/>
|
776
|
+
* Bugs: <http://github.com/defunkt/resque/issues>
|
777
|
+
* List: <resque@librelist.com>
|
778
|
+
* Chat: <irc://irc.freenode.net/resque>
|
779
|
+
* Gems: <http://gemcutter.org/gems/resque>
|
780
|
+
|
781
|
+
This project uses [Semantic Versioning][sv].
|
782
|
+
|
783
|
+
|
784
|
+
Author
|
785
|
+
------
|
786
|
+
|
787
|
+
Chris Wanstrath :: chris@ozmm.org :: @defunkt
|
788
|
+
|
789
|
+
[0]: http://github.com/blog/542-introducing-resque
|
790
|
+
[1]: http://help.github.com/forking/
|
791
|
+
[2]: http://github.com/defunkt/resque/issues
|
792
|
+
[sv]: http://semver.org/
|
793
|
+
[rs]: http://github.com/defunkt/redis-namespace
|
794
|
+
[cb]: http://wiki.github.com/defunkt/resque/contributing
|