onyx-resque-retry 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- data/HISTORY.md +33 -0
- data/LICENSE +21 -0
- data/README.md +289 -0
- data/Rakefile +25 -0
- data/lib/resque-retry.rb +6 -0
- data/lib/resque-retry/server.rb +51 -0
- data/lib/resque-retry/server/views/retry.erb +48 -0
- data/lib/resque-retry/server/views/retry_timestamp.erb +59 -0
- data/lib/resque/failure/multiple_with_retry_suppression.rb +93 -0
- data/lib/resque/plugins/exponential_backoff.rb +64 -0
- data/lib/resque/plugins/retry.rb +221 -0
- data/test/exponential_backoff_test.rb +62 -0
- data/test/multiple_failure_test.rb +86 -0
- data/test/redis-test.conf +132 -0
- data/test/resque_test.rb +18 -0
- data/test/retry_criteria_test.rb +75 -0
- data/test/retry_inheriting_checks_test.rb +33 -0
- data/test/retry_test.rb +173 -0
- data/test/test_helper.rb +78 -0
- data/test/test_jobs.rb +280 -0
- metadata +189 -0
data/HISTORY.md
ADDED
@@ -0,0 +1,33 @@
|
|
1
|
+
## 0.1.0 (2010-08-29)
|
2
|
+
|
3
|
+
* Feature: Multiple failure backend with retry suppression.
|
4
|
+
* Feature: resque-web tab showing retry information.
|
5
|
+
* Improved README documentation, added a 'Quick Start' section.
|
6
|
+
|
7
|
+
## 0.0.6 (2010-07-12)
|
8
|
+
|
9
|
+
* Feature: Added support for custom retry criteria check callbacks.
|
10
|
+
|
11
|
+
## 0.0.5 (2010-06-27)
|
12
|
+
|
13
|
+
* Handle our own dependancies.
|
14
|
+
|
15
|
+
## 0.0.4 (2010-06-16)
|
16
|
+
|
17
|
+
* Relax gemspec dependancies.
|
18
|
+
|
19
|
+
## 0.0.3 (2010-06-02)
|
20
|
+
|
21
|
+
* Bugfix: Make sure that `redis_retry_key` has no whitespace.
|
22
|
+
|
23
|
+
## 0.0.2 (2010-05-06)
|
24
|
+
|
25
|
+
* Bugfix: Were calling non-existent method to delete redis key.
|
26
|
+
* Delay no-longer falls back to `sleep`. resque-scheduler is a required
|
27
|
+
dependancy.
|
28
|
+
* Redis key doesn't include ending colon `:` if no args were passed
|
29
|
+
to the job.
|
30
|
+
|
31
|
+
## 0.0.1 (2010-04-27)
|
32
|
+
|
33
|
+
* First release.
|
data/LICENSE
ADDED
@@ -0,0 +1,21 @@
|
|
1
|
+
Copyright (c) 2010 Luke Antins
|
2
|
+
Copyright (c) 2010 Ryan Carver
|
3
|
+
|
4
|
+
Permission is hereby granted, free of charge, to any person obtaining
|
5
|
+
a copy of this software and associated documentation files (the
|
6
|
+
Software), to deal in the Software without restriction, including
|
7
|
+
without limitation the rights to use, copy, modify, merge, publish,
|
8
|
+
distribute, sublicense, and/or sell copies of the Software, and to
|
9
|
+
permit persons to whom the Software is furnished to do so, subject to
|
10
|
+
the following conditions:
|
11
|
+
|
12
|
+
The above copyright notice and this permission notice shall be
|
13
|
+
included in all copies or substantial portions of the Software.
|
14
|
+
|
15
|
+
THE SOFTWARE IS PROVIDED AS IS, WITHOUT WARRANTY OF ANY KIND,
|
16
|
+
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
17
|
+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
18
|
+
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
19
|
+
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
20
|
+
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
21
|
+
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
data/README.md
ADDED
@@ -0,0 +1,289 @@
|
|
1
|
+
resque-retry
|
2
|
+
============
|
3
|
+
|
4
|
+
A [Resque][rq] plugin. Requires Resque 1.8.0 & [resque-scheduler][rqs].
|
5
|
+
|
6
|
+
resque-retry provides retry, delay and exponential backoff support for
|
7
|
+
resque jobs.
|
8
|
+
|
9
|
+
* Redis backed retry count/limit.
|
10
|
+
* Retry on all or specific exceptions.
|
11
|
+
* Exponential backoff (varying the delay between retrys).
|
12
|
+
* Multiple failure backend with retry suppression & resque-web tab.
|
13
|
+
* Small & Extendable - plenty of places to override retry logic/settings.
|
14
|
+
|
15
|
+
Install & Quick Start
|
16
|
+
---------------------
|
17
|
+
|
18
|
+
To install:
|
19
|
+
|
20
|
+
$ gem install resque-retry
|
21
|
+
|
22
|
+
Add this to your `Rakefile`:
|
23
|
+
|
24
|
+
require 'resque/tasks'
|
25
|
+
require 'resque_scheduler/tasks'
|
26
|
+
|
27
|
+
The delay between retry attempts is provided by [resque-scheduler][rqs].
|
28
|
+
You'll want to run the scheduler process, otherwise delayed retry attempts
|
29
|
+
will never perform:
|
30
|
+
|
31
|
+
$ rake resque:scheduler
|
32
|
+
|
33
|
+
Use the plugin:
|
34
|
+
|
35
|
+
require 'resque-retry'
|
36
|
+
|
37
|
+
class ExampleRetryJob
|
38
|
+
extend Resque::Plugins::Retry
|
39
|
+
@queue = :example_queue
|
40
|
+
|
41
|
+
@retry_limit = 3
|
42
|
+
@retry_delay = 60
|
43
|
+
|
44
|
+
def self.perform(*args)
|
45
|
+
# your magic/heavy lifting goes here.
|
46
|
+
end
|
47
|
+
end
|
48
|
+
|
49
|
+
Then start up a resque worker as normal:
|
50
|
+
|
51
|
+
$ QUEUE=* rake resque:work
|
52
|
+
|
53
|
+
Now if you ExampleRetryJob fails, it will be retried 3 times, with a 60 second
|
54
|
+
delay between attempts.
|
55
|
+
|
56
|
+
For more explanation and examples, please see the remaining documentation.
|
57
|
+
|
58
|
+
Failure Backend & Resque Web Additions
|
59
|
+
--------------------------------------
|
60
|
+
|
61
|
+
Lets say your using the Redis failure backend of resque (the default).
|
62
|
+
Every time a job fails, the failure queue is populated with the job and
|
63
|
+
exception details.
|
64
|
+
|
65
|
+
Normally this is useful, but if your jobs retry... it can cause a bit of a mess.
|
66
|
+
|
67
|
+
For example: given a job that retried 4 times before completing successful.
|
68
|
+
You'll have a lot of failures for the same job and you wont be sure if it
|
69
|
+
actually completed successfully just by just using the resque-web interface.
|
70
|
+
|
71
|
+
### Failure Backend
|
72
|
+
|
73
|
+
`MultipleWithRetrySuppression` is a multiple failure backend, with retry suppression.
|
74
|
+
|
75
|
+
Here's an example, using the Redis failure backend:
|
76
|
+
|
77
|
+
require 'resque-retry'
|
78
|
+
require 'resque/failure/redis'
|
79
|
+
|
80
|
+
# require your jobs & application code.
|
81
|
+
|
82
|
+
Resque::Failure::MultipleWithRetrySuppression.classes = [Resque::Failure::Redis]
|
83
|
+
Resque::Failure.backend = Resque::Failure::MultipleWithRetrySuppression
|
84
|
+
|
85
|
+
If a job fails, but **can and will** retry, the failure details wont be
|
86
|
+
logged in the Redis failed queue *(visible via resque-web)*.
|
87
|
+
|
88
|
+
If the job fails, but **can't or won't** retry, the failure will be logged in
|
89
|
+
the Redis failed queue, like a normal failure *(without retry)* would.
|
90
|
+
|
91
|
+
### Resque Web Additions
|
92
|
+
|
93
|
+
If your using the `MultipleWithRetrySuppression` failure backend, you should
|
94
|
+
also checkout the resque-web additions!
|
95
|
+
|
96
|
+
The new Retry tab displays delayed jobs with retry information; the number of
|
97
|
+
attempts and the exception details from the last failure.
|
98
|
+
|
99
|
+
Make sure you include this in your `config.ru` or similar file:
|
100
|
+
|
101
|
+
require 'resque-retry'
|
102
|
+
require 'resque-retry/server'
|
103
|
+
|
104
|
+
# require your jobs & application code.
|
105
|
+
|
106
|
+
run Resque::Server.new
|
107
|
+
|
108
|
+
Retry Options & Logic
|
109
|
+
---------------------
|
110
|
+
|
111
|
+
Please take a look at the yardoc/code for more details on methods you may
|
112
|
+
wish to override.
|
113
|
+
|
114
|
+
Customisation is pretty easy, the below examples should give you
|
115
|
+
some ideas =), adapt for your own usage and feel free to pick and mix!
|
116
|
+
|
117
|
+
### Retry Defaults
|
118
|
+
|
119
|
+
Retry the job **once** on failure, with zero delay.
|
120
|
+
|
121
|
+
require 'resque-retry'
|
122
|
+
|
123
|
+
class DeliverWebHook
|
124
|
+
extend Resque::Plugins::Retry
|
125
|
+
@queue = :web_hooks
|
126
|
+
|
127
|
+
def self.perform(url, hook_id, hmac_key)
|
128
|
+
heavy_lifting
|
129
|
+
end
|
130
|
+
end
|
131
|
+
|
132
|
+
When a job runs, the number of retry attempts is checked and incremented
|
133
|
+
in Redis. If your job fails, the number of retry attempts is used to
|
134
|
+
determine if we can requeue the job for another go.
|
135
|
+
|
136
|
+
### Custom Retry
|
137
|
+
|
138
|
+
class DeliverWebHook
|
139
|
+
extend Resque::Plugins::Retry
|
140
|
+
@queue = :web_hooks
|
141
|
+
|
142
|
+
@retry_limit = 10
|
143
|
+
@retry_delay = 120
|
144
|
+
|
145
|
+
def self.perform(url, hook_id, hmac_key)
|
146
|
+
heavy_lifting
|
147
|
+
end
|
148
|
+
end
|
149
|
+
|
150
|
+
The above modification will allow your job to retry upto 10 times, with
|
151
|
+
a delay of 120 seconds, or 2 minutes between retry attempts.
|
152
|
+
|
153
|
+
Alternatively you could override the `retry_delay` method to do something
|
154
|
+
more special.
|
155
|
+
|
156
|
+
### Exponential Backoff
|
157
|
+
|
158
|
+
Use this if you wish to vary the delay between retry attempts:
|
159
|
+
|
160
|
+
class DeliverSMS
|
161
|
+
extend Resque::Plugins::ExponentialBackoff
|
162
|
+
@queue = :mt_messages
|
163
|
+
|
164
|
+
def self.perform(mt_id, mobile_number, message)
|
165
|
+
heavy_lifting
|
166
|
+
end
|
167
|
+
end
|
168
|
+
|
169
|
+
**Default Settings**
|
170
|
+
|
171
|
+
key: m = minutes, h = hours
|
172
|
+
|
173
|
+
no delay, 1m, 10m, 1h, 3h, 6h
|
174
|
+
@backoff_strategy = [0, 60, 600, 3600, 10800, 21600]
|
175
|
+
|
176
|
+
The first delay will be 0 seconds, the 2nd will be 60 seconds, etc...
|
177
|
+
Again, tweak to your own needs.
|
178
|
+
|
179
|
+
The number if retrys is equal to the size of the `backoff_strategy`
|
180
|
+
array, unless you set `retry_limit` yourself.
|
181
|
+
|
182
|
+
### Retry Specific Exceptions
|
183
|
+
|
184
|
+
The default will allow a retry for any type of exception. You may change
|
185
|
+
it so only specific exceptions are retried using `retry_exceptions`:
|
186
|
+
|
187
|
+
class DeliverSMS
|
188
|
+
extend Resque::Plugins::Retry
|
189
|
+
@queue = :mt_messages
|
190
|
+
|
191
|
+
@retry_exceptions = [NetworkError]
|
192
|
+
|
193
|
+
def self.perform(mt_id, mobile_number, message)
|
194
|
+
heavy_lifting
|
195
|
+
end
|
196
|
+
end
|
197
|
+
|
198
|
+
The above modification will **only** retry if a `NetworkError` (or subclass)
|
199
|
+
exception is thrown.
|
200
|
+
|
201
|
+
### Custom Retry Criteria Check Callbacks
|
202
|
+
|
203
|
+
You may define custom retry criteria callbacks:
|
204
|
+
|
205
|
+
class TurkWorker
|
206
|
+
extend Resque::Plugins::Retry
|
207
|
+
@queue = :turk_job_processor
|
208
|
+
|
209
|
+
@retry_exceptions = [NetworkError]
|
210
|
+
|
211
|
+
retry_criteria_check do |exception, *args|
|
212
|
+
if exception.message =~ /InvalidJobId/
|
213
|
+
false # don't retry if we got passed a invalid job id.
|
214
|
+
else
|
215
|
+
true # its okay for a retry attempt to continue.
|
216
|
+
end
|
217
|
+
end
|
218
|
+
|
219
|
+
def self.perform(job_id)
|
220
|
+
heavy_lifting
|
221
|
+
end
|
222
|
+
end
|
223
|
+
|
224
|
+
Similar to the previous example, this job will retry if either a
|
225
|
+
`NetworkError` (or subclass) exception is thrown **or** any of the callbacks
|
226
|
+
return true.
|
227
|
+
|
228
|
+
Use `@retry_exceptions = []` to **only** use callbacks, to determine if the
|
229
|
+
job should retry.
|
230
|
+
|
231
|
+
### Retry Arguments
|
232
|
+
|
233
|
+
You may override `args_for_retry`, which is passed the current
|
234
|
+
job arguments, to modify the arguments for the next retry attempt.
|
235
|
+
|
236
|
+
class DeliverViaSMSC
|
237
|
+
extend Resque::Plugins::Retry
|
238
|
+
@queue = :mt_smsc_messages
|
239
|
+
|
240
|
+
# retry using the emergency SMSC.
|
241
|
+
def self.args_for_retry(smsc_id, mt_message)
|
242
|
+
[999, mt_message]
|
243
|
+
end
|
244
|
+
|
245
|
+
self.perform(smsc_id, mt_message)
|
246
|
+
heavy_lifting
|
247
|
+
end
|
248
|
+
end
|
249
|
+
|
250
|
+
### Job Identifier/Key
|
251
|
+
|
252
|
+
The retry attempt is incremented and stored in a Redis key. The key is
|
253
|
+
built using the `identifier`. If you have a lot of arguments or really long
|
254
|
+
ones, you should consider overriding `identifier` to define a more precise
|
255
|
+
or loose custom identifier.
|
256
|
+
|
257
|
+
The default identifier is just your job arguments joined with a dash `-`.
|
258
|
+
|
259
|
+
By default the key uses this format:
|
260
|
+
`resque-retry:<job class name>:<identifier>`.
|
261
|
+
|
262
|
+
Or you can define the entire key by overriding `redis_retry_key`.
|
263
|
+
|
264
|
+
class DeliverSMS
|
265
|
+
extend Resque::Plugins::Retry
|
266
|
+
@queue = :mt_messages
|
267
|
+
|
268
|
+
def self.identifier(mt_id, mobile_number, message)
|
269
|
+
"#{mobile_number}:#{mt_id}"
|
270
|
+
end
|
271
|
+
|
272
|
+
self.perform(mt_id, mobile_number, message)
|
273
|
+
heavy_lifting
|
274
|
+
end
|
275
|
+
end
|
276
|
+
|
277
|
+
Contributing/Pull Requests
|
278
|
+
--------------------------
|
279
|
+
|
280
|
+
* Yes please!
|
281
|
+
* Fork the project.
|
282
|
+
* Make your feature addition or bug fix.
|
283
|
+
* Add tests for it.
|
284
|
+
* Commit.
|
285
|
+
* Send me a pull request. Bonus points for topic branches.
|
286
|
+
* If you edit the gemspec/version etc, do it in another commit please.
|
287
|
+
|
288
|
+
[rq]: http://github.com/defunkt/resque
|
289
|
+
[rqs]: http://github.com/bvandenbos/resque-scheduler
|
data/Rakefile
ADDED
@@ -0,0 +1,25 @@
|
|
1
|
+
$LOAD_PATH.unshift 'lib'
|
2
|
+
|
3
|
+
require 'rake/testtask'
|
4
|
+
require 'fileutils'
|
5
|
+
require 'yard'
|
6
|
+
require 'yard/rake/yardoc_task'
|
7
|
+
|
8
|
+
task :default => :test
|
9
|
+
|
10
|
+
##
|
11
|
+
# Test task.
|
12
|
+
Rake::TestTask.new(:test) do |task|
|
13
|
+
task.test_files = FileList['test/*_test.rb']
|
14
|
+
task.verbose = true
|
15
|
+
end
|
16
|
+
|
17
|
+
##
|
18
|
+
# docs task.
|
19
|
+
YARD::Rake::YardocTask.new :yardoc do |t|
|
20
|
+
t.files = ['lib/**/*.rb']
|
21
|
+
t.options = ['--output-dir', "doc/",
|
22
|
+
'--files', 'LICENSE',
|
23
|
+
'--readme', 'README.md',
|
24
|
+
'--title', 'resque-retry documentation']
|
25
|
+
end
|
data/lib/resque-retry.rb
ADDED
@@ -0,0 +1,51 @@
|
|
1
|
+
# Extend Resque::Server to add tabs.
|
2
|
+
module ResqueRetry
|
3
|
+
module Server
|
4
|
+
|
5
|
+
def self.included(base)
|
6
|
+
base.class_eval {
|
7
|
+
helpers do
|
8
|
+
# builds a retry key for the specified job.
|
9
|
+
def retry_key_for_job(job)
|
10
|
+
klass = Resque.constantize(job['class'])
|
11
|
+
if klass.respond_to?(:redis_retry_key)
|
12
|
+
klass.redis_retry_key(job['args'])
|
13
|
+
else
|
14
|
+
nil
|
15
|
+
end
|
16
|
+
end
|
17
|
+
|
18
|
+
# gets the number of retry attempts for a job.
|
19
|
+
def retry_attempts_for_job(job)
|
20
|
+
Resque.redis.get(retry_key_for_job(job))
|
21
|
+
end
|
22
|
+
|
23
|
+
# gets the failure details hash for a job.
|
24
|
+
def retry_failure_details(retry_key)
|
25
|
+
Resque.decode(Resque.redis["failure_#{retry_key}"])
|
26
|
+
end
|
27
|
+
|
28
|
+
# reads a 'local' template file.
|
29
|
+
def local_template(path)
|
30
|
+
# Is there a better way to specify alternate template locations with sinatra?
|
31
|
+
File.read(File.join(File.dirname(__FILE__), "server/views/#{path}"))
|
32
|
+
end
|
33
|
+
end
|
34
|
+
|
35
|
+
get '/retry' do
|
36
|
+
erb local_template('retry.erb')
|
37
|
+
end
|
38
|
+
|
39
|
+
get '/retry/:timestamp' do
|
40
|
+
erb local_template('retry_timestamp.erb')
|
41
|
+
end
|
42
|
+
}
|
43
|
+
end
|
44
|
+
|
45
|
+
end
|
46
|
+
end
|
47
|
+
|
48
|
+
Resque::Server.tabs << 'Retry'
|
49
|
+
Resque::Server.class_eval do
|
50
|
+
include ResqueRetry::Server
|
51
|
+
end
|
@@ -0,0 +1,48 @@
|
|
1
|
+
<h1>Delayed Jobs with Retry Information</h1>
|
2
|
+
|
3
|
+
<p class="intro">
|
4
|
+
This list below contains the timestamps for scheduled delayed jobs with
|
5
|
+
retry information.
|
6
|
+
</p>
|
7
|
+
|
8
|
+
<p class="sub">
|
9
|
+
Showing <%= start = params[:start].to_i %> to <%= start + 20 %> of
|
10
|
+
<b><%= size = resque.delayed_queue_schedule_size %></b> timestamps
|
11
|
+
</p>
|
12
|
+
|
13
|
+
<table>
|
14
|
+
<tr>
|
15
|
+
<th></th>
|
16
|
+
<th>Timestamp</th>
|
17
|
+
<th>Job count</th>
|
18
|
+
<th>Class</th>
|
19
|
+
<th>Args</th>
|
20
|
+
<th>Retry Attempts</th>
|
21
|
+
</tr>
|
22
|
+
<% timestamps = resque.delayed_queue_peek(start, start+20) %>
|
23
|
+
<% timestamps.each do |timestamp| %>
|
24
|
+
<% job = resque.delayed_timestamp_peek(timestamp, 0, 1).first %>
|
25
|
+
<% retry_key = retry_key_for_job(job) %>
|
26
|
+
<tr>
|
27
|
+
<td>
|
28
|
+
<form action="<%= url "/delayed/queue_now" %>" method="post">
|
29
|
+
<input type="hidden" name="timestamp" value="<%= timestamp.to_i %>">
|
30
|
+
<input type="submit" value="Queue now">
|
31
|
+
</form>
|
32
|
+
</td>
|
33
|
+
<td><a href="<%= url "retry/#{timestamp}" %>"><%= format_time(Time.at(timestamp)) %></a></td>
|
34
|
+
<td><%= delayed_timestamp_size = resque.delayed_timestamp_size(timestamp) %></td>
|
35
|
+
<% if job && delayed_timestamp_size == 1 %>
|
36
|
+
<td><%= h job['class'] %></td>
|
37
|
+
<td><%= h job['args'].inspect %></td>
|
38
|
+
<td><%= retry_attempts_for_job(job) || '<i>n/a</i>' %></td>
|
39
|
+
<% else %>
|
40
|
+
<td><a href="<%= url "retry/#{timestamp}" %>">see details</a></td>
|
41
|
+
<td></td>
|
42
|
+
<td></td>
|
43
|
+
<% end %>
|
44
|
+
</tr>
|
45
|
+
<% end %>
|
46
|
+
</table>
|
47
|
+
|
48
|
+
<%= partial :next_more, :start => start, :size => size %>
|