resque-retry 0.0.6 → 0.1.0
Sign up to get free protection for your applications and to get access to all the features.
- data/HISTORY.md +7 -1
- data/README.md +129 -38
- data/lib/resque-retry.rb +2 -1
- data/lib/resque-retry/server.rb +51 -0
- data/lib/resque-retry/server/views/retry.erb +48 -0
- data/lib/resque-retry/server/views/retry_timestamp.erb +59 -0
- data/lib/resque/failure/multiple_with_retry_suppression.rb +93 -0
- data/lib/resque/plugins/retry.rb +1 -1
- data/test/exponential_backoff_test.rb +9 -6
- data/test/multiple_failure_test.rb +86 -0
- data/test/test_helper.rb +19 -1
- data/test/test_jobs.rb +43 -0
- metadata +55 -7
data/HISTORY.md
CHANGED
@@ -1,4 +1,10 @@
|
|
1
|
-
## 0.0
|
1
|
+
## 0.1.0 (2010-08-29)
|
2
|
+
|
3
|
+
* Feature: Multiple failure backend with retry suppression.
|
4
|
+
* Feature: resque-web tab showing retry information.
|
5
|
+
* Improved README documentation, added a 'Quick Start' section.
|
6
|
+
|
7
|
+
## 0.0.6 (2010-07-12)
|
2
8
|
|
3
9
|
* Feature: Added support for custom retry criteria check callbacks.
|
4
10
|
|
data/README.md
CHANGED
@@ -1,27 +1,120 @@
|
|
1
1
|
resque-retry
|
2
2
|
============
|
3
3
|
|
4
|
-
A [Resque][rq] plugin. Requires Resque 1.8.0 & [resque-scheduler][rqs]
|
4
|
+
A [Resque][rq] plugin. Requires Resque 1.8.0 & [resque-scheduler][rqs].
|
5
5
|
|
6
6
|
resque-retry provides retry, delay and exponential backoff support for
|
7
7
|
resque jobs.
|
8
8
|
|
9
|
-
|
9
|
+
* Redis backed retry count/limit.
|
10
|
+
* Retry on all or specific exceptions.
|
11
|
+
* Exponential backoff (varying the delay between retrys).
|
12
|
+
* Multiple failure backend with retry suppression & resque-web tab.
|
13
|
+
* Small & Extendable - plenty of places to override retry logic/settings.
|
10
14
|
|
11
|
-
|
12
|
-
|
13
|
-
- Exponential backoff (varying the delay between retrys).
|
14
|
-
- Small & Extendable - plenty of places to override retry logic/settings.
|
15
|
+
Install & Quick Start
|
16
|
+
---------------------
|
15
17
|
|
16
|
-
|
17
|
-
----------------
|
18
|
+
To install:
|
18
19
|
|
19
|
-
|
20
|
+
$ gem install resque-retry
|
21
|
+
|
22
|
+
You'll want add this to your `Rakefile`:
|
23
|
+
|
24
|
+
require 'resque/tasks'
|
25
|
+
require 'resque_scheduler/tasks'
|
26
|
+
|
27
|
+
The delay between retry attempts is provided by [resque-scheduler][rqs].
|
28
|
+
You'll want to run the scheduler process, otherwise delayed retry attempts
|
29
|
+
will never perform:
|
30
|
+
|
31
|
+
$ rake resque:scheduler
|
32
|
+
|
33
|
+
Use the plugin:
|
34
|
+
|
35
|
+
require 'resque-retry'
|
36
|
+
|
37
|
+
class ExampleRetryJob
|
38
|
+
extend Resque::Plugins::Retry
|
39
|
+
@queue = :example_queue
|
40
|
+
|
41
|
+
@retry_limit = 3
|
42
|
+
@retry_delay = 60
|
43
|
+
|
44
|
+
def self.perform(*args)
|
45
|
+
# your magic/heavy lifting goes here.
|
46
|
+
end
|
47
|
+
end
|
48
|
+
|
49
|
+
Then start up a resque worker as normal:
|
50
|
+
|
51
|
+
$ QUEUE=* rake resque:work
|
52
|
+
|
53
|
+
Now if you ExampleRetryJob fails, it will be retried 3 times, with a 60 second
|
54
|
+
delay between attempts.
|
55
|
+
|
56
|
+
For more explanation and examples, please see the remaining documentation.
|
57
|
+
|
58
|
+
Failure Backend & Resque Web Additions
|
59
|
+
--------------------------------------
|
60
|
+
|
61
|
+
Lets say your using the Redis failure backend of resque (the default).
|
62
|
+
Every time a job fails, the failure queue is populated with the job and
|
63
|
+
exception details.
|
64
|
+
|
65
|
+
Normally this is useful, but if your jobs retry... it can cause a bit of a mess.
|
66
|
+
|
67
|
+
For example: given a job that retried 4 times before completing successful.
|
68
|
+
You'll have a lot of failures for the same job and you wont be sure if it
|
69
|
+
actually completed successfully just by just using the resque-web interface.
|
70
|
+
|
71
|
+
### Failure Backend
|
72
|
+
|
73
|
+
`MultipleWithRetrySuppression` is a multiple failure backend, with retry suppression.
|
74
|
+
|
75
|
+
Here's an example, using the Redis failure backend:
|
76
|
+
|
77
|
+
require 'resque-retry'
|
78
|
+
require 'resque/failure/redis'
|
79
|
+
|
80
|
+
# require your jobs & application code.
|
81
|
+
|
82
|
+
Resque::Failure::MultipleWithRetrySuppression.classes = [Resque::Failure::Redis]
|
83
|
+
Resque::Failure.backend = Resque::Failure::MultipleWithRetrySuppression
|
84
|
+
|
85
|
+
If a job fails, but **can and will** retry, the failure details wont be
|
86
|
+
logged in the Redis failed queue *(visible via resque-web)*.
|
87
|
+
|
88
|
+
If the job fails, but **can't or won't** retry, the failure will be logged in
|
89
|
+
the Redis failed queue, like a normal failure *(without retry)* would.
|
90
|
+
|
91
|
+
### Resque Web Additions
|
92
|
+
|
93
|
+
If your using the `MultipleWithRetrySuppression` failure backend, you should
|
94
|
+
also checkout the resque-web additions!
|
95
|
+
|
96
|
+
The new Retry tab displays delayed jobs with retry information; the number of
|
97
|
+
attempts and the exception details from the last failure.
|
98
|
+
|
99
|
+
Make sure you include this in your `config.ru` or similar file:
|
100
|
+
|
101
|
+
require 'resque-retry'
|
102
|
+
require 'resque-retry/server'
|
103
|
+
|
104
|
+
# require your jobs & application code.
|
105
|
+
|
106
|
+
run Resque::Server.new
|
107
|
+
|
108
|
+
Retry Options & Logic
|
109
|
+
---------------------
|
110
|
+
|
111
|
+
Please take a look at the yardoc/code for more details on methods you may
|
112
|
+
wish to override.
|
20
113
|
|
21
114
|
Customisation is pretty easy, the below examples should give you
|
22
115
|
some ideas =), adapt for your own usage and feel free to pick and mix!
|
23
116
|
|
24
|
-
### Retry
|
117
|
+
### Retry Defaults
|
25
118
|
|
26
119
|
Retry the job **once** on failure, with zero delay.
|
27
120
|
|
@@ -135,13 +228,24 @@ return true.
|
|
135
228
|
Use `@retry_exceptions = []` to **only** use callbacks, to determine if the
|
136
229
|
job should retry.
|
137
230
|
|
138
|
-
|
139
|
-
------------------
|
231
|
+
### Retry Arguments
|
140
232
|
|
141
|
-
|
142
|
-
|
233
|
+
You may override `args_for_retry`, which is passed the current
|
234
|
+
job arguments, to modify the arguments for the next retry attempt.
|
235
|
+
|
236
|
+
class DeliverViaSMSC
|
237
|
+
extend Resque::Plugins::Retry
|
238
|
+
@queue = :mt_smsc_messages
|
143
239
|
|
144
|
-
|
240
|
+
# retry using the emergency SMSC.
|
241
|
+
def self.args_for_retry(smsc_id, mt_message)
|
242
|
+
[999, mt_message]
|
243
|
+
end
|
244
|
+
|
245
|
+
self.perform(smsc_id, mt_message)
|
246
|
+
heavy_lifting
|
247
|
+
end
|
248
|
+
end
|
145
249
|
|
146
250
|
### Job Identifier/Key
|
147
251
|
|
@@ -170,29 +274,16 @@ Or you can define the entire key by overriding `redis_retry_key`.
|
|
170
274
|
end
|
171
275
|
end
|
172
276
|
|
173
|
-
|
277
|
+
Contributing/Pull Requests
|
278
|
+
--------------------------
|
174
279
|
|
175
|
-
|
176
|
-
|
177
|
-
|
178
|
-
|
179
|
-
|
180
|
-
|
181
|
-
|
182
|
-
# retry using the emergency SMSC.
|
183
|
-
def self.args_for_retry(smsc_id, mt_message)
|
184
|
-
[999, mt_message]
|
185
|
-
end
|
186
|
-
|
187
|
-
self.perform(smsc_id, mt_message)
|
188
|
-
heavy_lifting
|
189
|
-
end
|
190
|
-
end
|
191
|
-
|
192
|
-
Install
|
193
|
-
-------
|
194
|
-
|
195
|
-
$ gem install resque-retry
|
280
|
+
* Yes please!
|
281
|
+
* Fork the project.
|
282
|
+
* Make your feature addition or bug fix.
|
283
|
+
* Add tests for it.
|
284
|
+
* Commit.
|
285
|
+
* Send me a pull request. Bonus points for topic branches.
|
286
|
+
* If you edit the gemspec/version etc, do it in another commit please.
|
196
287
|
|
197
288
|
[rq]: http://github.com/defunkt/resque
|
198
|
-
[rqs]: http://github.com/bvandenbos/resque-scheduler
|
289
|
+
[rqs]: http://github.com/bvandenbos/resque-scheduler
|
data/lib/resque-retry.rb
CHANGED
@@ -0,0 +1,51 @@
|
|
1
|
+
# Extend Resque::Server to add tabs.
|
2
|
+
module ResqueRetry
|
3
|
+
module Server
|
4
|
+
|
5
|
+
def self.included(base)
|
6
|
+
base.class_eval {
|
7
|
+
helpers do
|
8
|
+
# builds a retry key for the specified job.
|
9
|
+
def retry_key_for_job(job)
|
10
|
+
klass = Resque.constantize(job['class'])
|
11
|
+
if klass.respond_to?(:redis_retry_key)
|
12
|
+
klass.redis_retry_key(job['args'])
|
13
|
+
else
|
14
|
+
nil
|
15
|
+
end
|
16
|
+
end
|
17
|
+
|
18
|
+
# gets the number of retry attempts for a job.
|
19
|
+
def retry_attempts_for_job(job)
|
20
|
+
Resque.redis.get(retry_key_for_job(job))
|
21
|
+
end
|
22
|
+
|
23
|
+
# gets the failure details hash for a job.
|
24
|
+
def retry_failure_details(retry_key)
|
25
|
+
Resque.decode(Resque.redis["failure_#{retry_key}"])
|
26
|
+
end
|
27
|
+
|
28
|
+
# reads a 'local' template file.
|
29
|
+
def local_template(path)
|
30
|
+
# Is there a better way to specify alternate template locations with sinatra?
|
31
|
+
File.read(File.join(File.dirname(__FILE__), "server/views/#{path}"))
|
32
|
+
end
|
33
|
+
end
|
34
|
+
|
35
|
+
get '/retry' do
|
36
|
+
erb local_template('retry.erb')
|
37
|
+
end
|
38
|
+
|
39
|
+
get '/retry/:timestamp' do
|
40
|
+
erb local_template('retry_timestamp.erb')
|
41
|
+
end
|
42
|
+
}
|
43
|
+
end
|
44
|
+
|
45
|
+
end
|
46
|
+
end
|
47
|
+
|
48
|
+
Resque::Server.tabs << 'Retry'
|
49
|
+
Resque::Server.class_eval do
|
50
|
+
include ResqueRetry::Server
|
51
|
+
end
|
@@ -0,0 +1,48 @@
|
|
1
|
+
<h1>Delayed Jobs with Retry Information</h1>
|
2
|
+
|
3
|
+
<p class="intro">
|
4
|
+
This list below contains the timestamps for scheduled delayed jobs with
|
5
|
+
retry information.
|
6
|
+
</p>
|
7
|
+
|
8
|
+
<p class="sub">
|
9
|
+
Showing <%= start = params[:start].to_i %> to <%= start + 20 %> of
|
10
|
+
<b><%= size = resque.delayed_queue_schedule_size %></b> timestamps
|
11
|
+
</p>
|
12
|
+
|
13
|
+
<table>
|
14
|
+
<tr>
|
15
|
+
<th></th>
|
16
|
+
<th>Timestamp</th>
|
17
|
+
<th>Job count</th>
|
18
|
+
<th>Class</th>
|
19
|
+
<th>Args</th>
|
20
|
+
<th>Retry Attempts</th>
|
21
|
+
</tr>
|
22
|
+
<% timestamps = resque.delayed_queue_peek(start, start+20) %>
|
23
|
+
<% timestamps.each do |timestamp| %>
|
24
|
+
<% job = resque.delayed_timestamp_peek(timestamp, 0, 1).first %>
|
25
|
+
<% retry_key = retry_key_for_job(job) %>
|
26
|
+
<tr>
|
27
|
+
<td>
|
28
|
+
<form action="<%= url "/delayed/queue_now" %>" method="post">
|
29
|
+
<input type="hidden" name="timestamp" value="<%= timestamp.to_i %>">
|
30
|
+
<input type="submit" value="Queue now">
|
31
|
+
</form>
|
32
|
+
</td>
|
33
|
+
<td><a href="<%= url "retry/#{timestamp}" %>"><%= format_time(Time.at(timestamp)) %></a></td>
|
34
|
+
<td><%= delayed_timestamp_size = resque.delayed_timestamp_size(timestamp) %></td>
|
35
|
+
<% if job && delayed_timestamp_size == 1 %>
|
36
|
+
<td><%= h job['class'] %></td>
|
37
|
+
<td><%= h job['args'].inspect %></td>
|
38
|
+
<td><%= retry_attempts_for_job(job) || '<i>n/a</i>' %></td>
|
39
|
+
<% else %>
|
40
|
+
<td><a href="<%= url "retry/#{timestamp}" %>">see details</a></td>
|
41
|
+
<td></td>
|
42
|
+
<td></td>
|
43
|
+
<% end %>
|
44
|
+
</tr>
|
45
|
+
<% end %>
|
46
|
+
</table>
|
47
|
+
|
48
|
+
<%= partial :next_more, :start => start, :size => size %>
|
@@ -0,0 +1,59 @@
|
|
1
|
+
<% timestamp = params[:timestamp].to_i %>
|
2
|
+
|
3
|
+
<h1>
|
4
|
+
Delayed Jobs scheduled for <%= format_time(Time.at(timestamp)) %>
|
5
|
+
(with Retry Information)
|
6
|
+
</h1>
|
7
|
+
|
8
|
+
<p class="intro">
|
9
|
+
This list below contains the delayed jobs scheduled for the current
|
10
|
+
timestamp, with retry information.
|
11
|
+
</p>
|
12
|
+
|
13
|
+
<p class="sub">
|
14
|
+
Showing <%= start = params[:start].to_i %> to <%= start + 20 %> of
|
15
|
+
<b><%= size = resque.delayed_timestamp_size(timestamp) %></b> jobs
|
16
|
+
</p>
|
17
|
+
|
18
|
+
<table class="jobs">
|
19
|
+
<tr>
|
20
|
+
<th>Class</th>
|
21
|
+
<th>Args</th>
|
22
|
+
<th>Retry Attempts</th>
|
23
|
+
<th>Exception</th>
|
24
|
+
<th>Backtrace</th>
|
25
|
+
</tr>
|
26
|
+
<% jobs = resque.delayed_timestamp_peek(timestamp, start, 20) %>
|
27
|
+
<% jobs.each do |job| %>
|
28
|
+
<% retry_key = retry_key_for_job(job) %>
|
29
|
+
<% retry_attempts = retry_attempts_for_job(job) %>
|
30
|
+
<tr>
|
31
|
+
<td class="class"><%= h job['class'] %></td>
|
32
|
+
<td class="args"><%= h job['args'].inspect %></td>
|
33
|
+
<% if retry_attempts.nil? %>
|
34
|
+
<td colspan="3"><i>n/a - normal delayed job</i></td>
|
35
|
+
<% else %>
|
36
|
+
<td><%= retry_attempts %></td>
|
37
|
+
<% failure = retry_failure_details(retry_key) %>
|
38
|
+
<td><code><%= failure['exception'] %></code></td>
|
39
|
+
<td class="error">
|
40
|
+
<% if failure['backtrace'] %>
|
41
|
+
<a href="#" class="backtrace"><%= h(failure['error']) %></a>
|
42
|
+
<pre style="display:none"><%= h failure['backtrace'].join("\n") %></pre>
|
43
|
+
<% else %>
|
44
|
+
<%= h failure['error'] %>
|
45
|
+
<% end %>
|
46
|
+
</td>
|
47
|
+
<% end %>
|
48
|
+
</tr>
|
49
|
+
<% end %>
|
50
|
+
<% if jobs.empty? %>
|
51
|
+
<tr>
|
52
|
+
<td class="no-data" colspan="5">
|
53
|
+
There are no pending jobs scheduled for this time.
|
54
|
+
</td>
|
55
|
+
</tr>
|
56
|
+
<% end %>
|
57
|
+
</table>
|
58
|
+
|
59
|
+
<%= partial :next_more, :start => start, :size => size %>
|
@@ -0,0 +1,93 @@
|
|
1
|
+
require 'resque/failure/multiple'
|
2
|
+
|
3
|
+
module Resque
|
4
|
+
module Failure
|
5
|
+
|
6
|
+
# A multiple failure backend, with retry suppression.
|
7
|
+
#
|
8
|
+
# For example: if you had a job that could retry 5 times, your failure
|
9
|
+
# backends are not notified unless the _final_ retry attempt also fails.
|
10
|
+
#
|
11
|
+
# Example:
|
12
|
+
#
|
13
|
+
# require 'resque-retry'
|
14
|
+
# require 'resque/failure/redis'
|
15
|
+
#
|
16
|
+
# Resque::Failure::MultipleWithRetrySuppression.classes = [Resque::Failure::Redis]
|
17
|
+
# Resque::Failure.backend = Resque::Failure::MultipleWithRetrySuppression
|
18
|
+
#
|
19
|
+
class MultipleWithRetrySuppression < Multiple
|
20
|
+
include Resque::Helpers
|
21
|
+
|
22
|
+
module CleanupHooks
|
23
|
+
# Resque after_perform hook.
|
24
|
+
#
|
25
|
+
# Deletes retry failure information from Redis.
|
26
|
+
def after_perform_retry_failure_cleanup(*args)
|
27
|
+
retry_key = redis_retry_key(*args)
|
28
|
+
failure_key = Resque::Failure::MultipleWithRetrySuppression.failure_key(retry_key)
|
29
|
+
Resque.redis.del(failure_key)
|
30
|
+
end
|
31
|
+
end
|
32
|
+
|
33
|
+
# Called when the job fails.
|
34
|
+
#
|
35
|
+
# If the job will retry, suppress the failure from the other backends.
|
36
|
+
# Store the lastest failure information in redis, used by the web
|
37
|
+
# interface.
|
38
|
+
def save
|
39
|
+
unless retryable? && retrying?
|
40
|
+
cleanup_retry_failure_log!
|
41
|
+
super
|
42
|
+
else
|
43
|
+
data = {
|
44
|
+
:failed_at => Time.now.strftime("%Y/%m/%d %H:%M:%S"),
|
45
|
+
:payload => payload,
|
46
|
+
:exception => exception.class.to_s,
|
47
|
+
:error => exception.to_s,
|
48
|
+
:backtrace => Array(exception.backtrace),
|
49
|
+
:worker => worker.to_s,
|
50
|
+
:queue => queue
|
51
|
+
}
|
52
|
+
|
53
|
+
# Register cleanup hooks.
|
54
|
+
unless klass.respond_to?(:after_perform_retry_failure_cleanup)
|
55
|
+
klass.send(:extend, CleanupHooks)
|
56
|
+
end
|
57
|
+
|
58
|
+
redis[failure_key] = Resque.encode(data)
|
59
|
+
end
|
60
|
+
end
|
61
|
+
|
62
|
+
# Expose this for the hook's use.
|
63
|
+
def self.failure_key(retry_key)
|
64
|
+
'failure_' + retry_key
|
65
|
+
end
|
66
|
+
|
67
|
+
protected
|
68
|
+
def klass
|
69
|
+
constantize(payload['class'])
|
70
|
+
end
|
71
|
+
|
72
|
+
def retry_key
|
73
|
+
klass.redis_retry_key(payload['args'])
|
74
|
+
end
|
75
|
+
|
76
|
+
def failure_key
|
77
|
+
self.class.failure_key(retry_key)
|
78
|
+
end
|
79
|
+
|
80
|
+
def retryable?
|
81
|
+
klass.respond_to?(:redis_retry_key)
|
82
|
+
end
|
83
|
+
|
84
|
+
def retrying?
|
85
|
+
redis.exists(retry_key)
|
86
|
+
end
|
87
|
+
|
88
|
+
def cleanup_retry_failure_log!
|
89
|
+
redis.del(failure_key) if retryable?
|
90
|
+
end
|
91
|
+
end
|
92
|
+
end
|
93
|
+
end
|
data/lib/resque/plugins/retry.rb
CHANGED
@@ -16,13 +16,16 @@ class ExponentialBackoffTest < Test::Unit::TestCase
|
|
16
16
|
def test_default_backoff_strategy
|
17
17
|
now = Time.now
|
18
18
|
Resque.enqueue(ExponentialBackoffJob)
|
19
|
-
2.times do
|
20
|
-
perform_next_job @worker
|
21
|
-
end
|
22
19
|
|
23
|
-
|
24
|
-
assert_equal
|
25
|
-
assert_equal
|
20
|
+
perform_next_job @worker
|
21
|
+
assert_equal 1, Resque.info[:processed], '1 processed job'
|
22
|
+
assert_equal 1, Resque.info[:failed], 'first ever run, and it should of failed, but never retried'
|
23
|
+
assert_equal 1, Resque.info[:pending], '1 pending job, because it never hits the scheduler'
|
24
|
+
|
25
|
+
perform_next_job @worker
|
26
|
+
assert_equal 2, Resque.info[:processed], '2nd run, but first retry'
|
27
|
+
assert_equal 2, Resque.info[:failed], 'should of failed again, this is the first retry attempt'
|
28
|
+
assert_equal 0, Resque.info[:pending], '0 pending jobs, it should be in the delayed queue'
|
26
29
|
|
27
30
|
delayed = Resque.delayed_queue_peek(0, 1)
|
28
31
|
assert_equal now.to_i + 60, delayed[0], '2nd delay' # the first had a zero delay.
|
@@ -0,0 +1,86 @@
|
|
1
|
+
require File.dirname(__FILE__) + '/test_helper'
|
2
|
+
|
3
|
+
class MultipleFailureTest < Test::Unit::TestCase
|
4
|
+
def setup
|
5
|
+
Resque.redis.flushall
|
6
|
+
@worker = Resque::Worker.new(:testing)
|
7
|
+
@worker.register_worker
|
8
|
+
|
9
|
+
@old_failure_backend = Resque::Failure.backend
|
10
|
+
MockFailureBackend.errors = []
|
11
|
+
Resque::Failure::MultipleWithRetrySuppression.classes = [MockFailureBackend]
|
12
|
+
Resque::Failure.backend = Resque::Failure::MultipleWithRetrySuppression
|
13
|
+
end
|
14
|
+
|
15
|
+
def failure_key_for(klass)
|
16
|
+
args = []
|
17
|
+
key = "failure_" + klass.redis_retry_key(args)
|
18
|
+
end
|
19
|
+
|
20
|
+
def test_last_failure_is_saved_in_redis
|
21
|
+
Resque.enqueue(LimitThreeJob)
|
22
|
+
perform_next_job(@worker)
|
23
|
+
|
24
|
+
# I don't like this, but...
|
25
|
+
key = failure_key_for(LimitThreeJob)
|
26
|
+
assert Resque.redis.exists(key)
|
27
|
+
end
|
28
|
+
|
29
|
+
def test_last_failure_removed_from_redis_after_error_limit
|
30
|
+
Resque.enqueue(LimitThreeJob)
|
31
|
+
3.times do
|
32
|
+
perform_next_job(@worker)
|
33
|
+
end
|
34
|
+
|
35
|
+
key = failure_key_for(LimitThreeJob)
|
36
|
+
assert Resque.redis.exists(key)
|
37
|
+
|
38
|
+
perform_next_job(@worker)
|
39
|
+
assert !Resque.redis.exists(key)
|
40
|
+
end
|
41
|
+
|
42
|
+
def test_on_success_failure_log_removed_from_redis
|
43
|
+
SwitchToSuccessJob.successful_after = 1
|
44
|
+
Resque.enqueue(SwitchToSuccessJob)
|
45
|
+
perform_next_job(@worker)
|
46
|
+
|
47
|
+
key = failure_key_for(SwitchToSuccessJob)
|
48
|
+
assert Resque.redis.exists(key)
|
49
|
+
|
50
|
+
perform_next_job(@worker)
|
51
|
+
assert !Resque.redis.exists(key), 'key removed on success'
|
52
|
+
ensure
|
53
|
+
SwitchToSuccessJob.reset_defaults
|
54
|
+
end
|
55
|
+
|
56
|
+
def test_errors_are_suppressed_up_to_retry_limit
|
57
|
+
Resque.enqueue(LimitThreeJob)
|
58
|
+
3.times do
|
59
|
+
perform_next_job(@worker)
|
60
|
+
end
|
61
|
+
|
62
|
+
assert_equal 0, MockFailureBackend.errors.size
|
63
|
+
end
|
64
|
+
|
65
|
+
def test_errors_are_logged_after_retry_limit
|
66
|
+
Resque.enqueue(LimitThreeJob)
|
67
|
+
4.times do
|
68
|
+
perform_next_job(@worker)
|
69
|
+
end
|
70
|
+
|
71
|
+
assert_equal 1, MockFailureBackend.errors.size
|
72
|
+
end
|
73
|
+
|
74
|
+
def test_jobs_without_retry_log_errors
|
75
|
+
5.times do
|
76
|
+
Resque.enqueue(NoRetryJob)
|
77
|
+
perform_next_job(@worker)
|
78
|
+
end
|
79
|
+
|
80
|
+
assert_equal 5, MockFailureBackend.errors.size
|
81
|
+
end
|
82
|
+
|
83
|
+
def teardown
|
84
|
+
Resque::Failure.backend = @old_failure_backend
|
85
|
+
end
|
86
|
+
end
|
data/test/test_helper.rb
CHANGED
@@ -5,6 +5,11 @@ $TESTING = true
|
|
5
5
|
require 'test/unit'
|
6
6
|
require 'rubygems'
|
7
7
|
require 'turn'
|
8
|
+
require 'simplecov-html'
|
9
|
+
|
10
|
+
SimpleCov.start do
|
11
|
+
add_filter "/test/"
|
12
|
+
end
|
8
13
|
|
9
14
|
require 'resque-retry'
|
10
15
|
require dir + '/test_jobs'
|
@@ -40,6 +45,19 @@ puts "Starting redis for testing at localhost:9736..."
|
|
40
45
|
`redis-server #{dir}/redis-test.conf`
|
41
46
|
Resque.redis = '127.0.0.1:9736'
|
42
47
|
|
48
|
+
# Mock failure backend for testing MultipleWithRetrySuppression
|
49
|
+
class MockFailureBackend < Resque::Failure::Base
|
50
|
+
class << self
|
51
|
+
attr_accessor :errors
|
52
|
+
end
|
53
|
+
|
54
|
+
def save
|
55
|
+
self.class.errors << exception.to_s
|
56
|
+
end
|
57
|
+
|
58
|
+
self.errors = []
|
59
|
+
end
|
60
|
+
|
43
61
|
# Test helpers
|
44
62
|
class Test::Unit::TestCase
|
45
63
|
def perform_next_job(worker, &block)
|
@@ -57,4 +75,4 @@ class Test::Unit::TestCase
|
|
57
75
|
worker.perform(job)
|
58
76
|
worker.done_working
|
59
77
|
end
|
60
|
-
end
|
78
|
+
end
|
data/test/test_jobs.rb
CHANGED
@@ -2,6 +2,14 @@ CustomException = Class.new(StandardError)
|
|
2
2
|
HierarchyCustomException = Class.new(CustomException)
|
3
3
|
AnotherCustomException = Class.new(StandardError)
|
4
4
|
|
5
|
+
class NoRetryJob
|
6
|
+
@queue = :testing
|
7
|
+
|
8
|
+
def self.perform(*args)
|
9
|
+
raise "error"
|
10
|
+
end
|
11
|
+
end
|
12
|
+
|
5
13
|
class GoodJob
|
6
14
|
extend Resque::Plugins::Retry
|
7
15
|
@queue = :testing
|
@@ -50,6 +58,15 @@ class NeverGiveUpJob < RetryDefaultsJob
|
|
50
58
|
@retry_limit = 0
|
51
59
|
end
|
52
60
|
|
61
|
+
class LimitThreeJob < RetryDefaultsJob
|
62
|
+
@queue = :testing
|
63
|
+
@retry_limit = 3
|
64
|
+
|
65
|
+
def self.perform(*args)
|
66
|
+
raise ArgumentError, "custom message"
|
67
|
+
end
|
68
|
+
end
|
69
|
+
|
53
70
|
class FailFiveTimesJob < RetryDefaultsJob
|
54
71
|
@queue = :testing
|
55
72
|
@retry_limit = 6
|
@@ -187,6 +204,32 @@ class InheritOrderingJobExtendFirst
|
|
187
204
|
end
|
188
205
|
end
|
189
206
|
|
207
|
+
# This job switches to successful after
|
208
|
+
# n +tries+.
|
209
|
+
class SwitchToSuccessJob < GoodJob
|
210
|
+
@queue = :testing
|
211
|
+
@max_retries = 3
|
212
|
+
|
213
|
+
class << self
|
214
|
+
attr_accessor :successful_after
|
215
|
+
attr_accessor :tries
|
216
|
+
|
217
|
+
def reset_defaults
|
218
|
+
self.tries = 0
|
219
|
+
self.successful_after = 2
|
220
|
+
end
|
221
|
+
end
|
222
|
+
|
223
|
+
reset_defaults
|
224
|
+
|
225
|
+
def self.perform(*args)
|
226
|
+
if self.tries < self.successful_after
|
227
|
+
self.tries += 1
|
228
|
+
raise "error"
|
229
|
+
end
|
230
|
+
end
|
231
|
+
end
|
232
|
+
|
190
233
|
class InheritOrderingJobExtendLast
|
191
234
|
class << self
|
192
235
|
attr_accessor :test_value
|
metadata
CHANGED
@@ -1,12 +1,13 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: resque-retry
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
+
hash: 27
|
4
5
|
prerelease: false
|
5
6
|
segments:
|
6
7
|
- 0
|
8
|
+
- 1
|
7
9
|
- 0
|
8
|
-
|
9
|
-
version: 0.0.6
|
10
|
+
version: 0.1.0
|
10
11
|
platform: ruby
|
11
12
|
authors:
|
12
13
|
- Luke Antins
|
@@ -15,16 +16,18 @@ autorequire:
|
|
15
16
|
bindir: bin
|
16
17
|
cert_chain: []
|
17
18
|
|
18
|
-
date: 2010-
|
19
|
+
date: 2010-08-29 00:00:00 +01:00
|
19
20
|
default_executable:
|
20
21
|
dependencies:
|
21
22
|
- !ruby/object:Gem::Dependency
|
22
23
|
name: resque
|
23
24
|
prerelease: false
|
24
25
|
requirement: &id001 !ruby/object:Gem::Requirement
|
26
|
+
none: false
|
25
27
|
requirements:
|
26
28
|
- - ">="
|
27
29
|
- !ruby/object:Gem::Version
|
30
|
+
hash: 55
|
28
31
|
segments:
|
29
32
|
- 1
|
30
33
|
- 8
|
@@ -36,9 +39,11 @@ dependencies:
|
|
36
39
|
name: resque-scheduler
|
37
40
|
prerelease: false
|
38
41
|
requirement: &id002 !ruby/object:Gem::Requirement
|
42
|
+
none: false
|
39
43
|
requirements:
|
40
44
|
- - ">="
|
41
45
|
- !ruby/object:Gem::Version
|
46
|
+
hash: 55
|
42
47
|
segments:
|
43
48
|
- 1
|
44
49
|
- 8
|
@@ -47,30 +52,64 @@ dependencies:
|
|
47
52
|
type: :runtime
|
48
53
|
version_requirements: *id002
|
49
54
|
- !ruby/object:Gem::Dependency
|
50
|
-
name:
|
55
|
+
name: test-unit
|
51
56
|
prerelease: false
|
52
57
|
requirement: &id003 !ruby/object:Gem::Requirement
|
58
|
+
none: false
|
53
59
|
requirements:
|
54
60
|
- - ">="
|
55
61
|
- !ruby/object:Gem::Version
|
62
|
+
hash: 3
|
56
63
|
segments:
|
57
64
|
- 0
|
58
65
|
version: "0"
|
59
66
|
type: :development
|
60
67
|
version_requirements: *id003
|
61
68
|
- !ruby/object:Gem::Dependency
|
62
|
-
name:
|
69
|
+
name: turn
|
63
70
|
prerelease: false
|
64
71
|
requirement: &id004 !ruby/object:Gem::Requirement
|
72
|
+
none: false
|
65
73
|
requirements:
|
66
74
|
- - ">="
|
67
75
|
- !ruby/object:Gem::Version
|
76
|
+
hash: 3
|
68
77
|
segments:
|
69
78
|
- 0
|
70
79
|
version: "0"
|
71
80
|
type: :development
|
72
81
|
version_requirements: *id004
|
73
|
-
|
82
|
+
- !ruby/object:Gem::Dependency
|
83
|
+
name: yard
|
84
|
+
prerelease: false
|
85
|
+
requirement: &id005 !ruby/object:Gem::Requirement
|
86
|
+
none: false
|
87
|
+
requirements:
|
88
|
+
- - ">="
|
89
|
+
- !ruby/object:Gem::Version
|
90
|
+
hash: 3
|
91
|
+
segments:
|
92
|
+
- 0
|
93
|
+
version: "0"
|
94
|
+
type: :development
|
95
|
+
version_requirements: *id005
|
96
|
+
- !ruby/object:Gem::Dependency
|
97
|
+
name: simplecov-html
|
98
|
+
prerelease: false
|
99
|
+
requirement: &id006 !ruby/object:Gem::Requirement
|
100
|
+
none: false
|
101
|
+
requirements:
|
102
|
+
- - ">="
|
103
|
+
- !ruby/object:Gem::Version
|
104
|
+
hash: 19
|
105
|
+
segments:
|
106
|
+
- 0
|
107
|
+
- 3
|
108
|
+
- 0
|
109
|
+
version: 0.3.0
|
110
|
+
type: :development
|
111
|
+
version_requirements: *id006
|
112
|
+
description: " resque-retry provides retry, delay and exponential backoff support for\n resque jobs.\n\n Features:\n\n * Redis backed retry count/limit.\n * Retry on all or specific exceptions.\n * Exponential backoff (varying the delay between retrys).\n * Multiple failure backend with retry suppression & resque-web tab.\n * Small & Extendable - plenty of places to override retry logic/settings.\n"
|
74
113
|
email: luke@lividpenguin.com
|
75
114
|
executables: []
|
76
115
|
|
@@ -84,6 +123,7 @@ files:
|
|
84
123
|
- README.md
|
85
124
|
- HISTORY.md
|
86
125
|
- test/exponential_backoff_test.rb
|
126
|
+
- test/multiple_failure_test.rb
|
87
127
|
- test/redis-test.conf
|
88
128
|
- test/resque_test.rb
|
89
129
|
- test/retry_criteria_test.rb
|
@@ -91,8 +131,12 @@ files:
|
|
91
131
|
- test/retry_test.rb
|
92
132
|
- test/test_helper.rb
|
93
133
|
- test/test_jobs.rb
|
134
|
+
- lib/resque/failure/multiple_with_retry_suppression.rb
|
94
135
|
- lib/resque/plugins/exponential_backoff.rb
|
95
136
|
- lib/resque/plugins/retry.rb
|
137
|
+
- lib/resque-retry/server/views/retry.erb
|
138
|
+
- lib/resque-retry/server/views/retry_timestamp.erb
|
139
|
+
- lib/resque-retry/server.rb
|
96
140
|
- lib/resque-retry.rb
|
97
141
|
has_rdoc: false
|
98
142
|
homepage: http://github.com/lantins/resque-retry
|
@@ -104,23 +148,27 @@ rdoc_options: []
|
|
104
148
|
require_paths:
|
105
149
|
- lib
|
106
150
|
required_ruby_version: !ruby/object:Gem::Requirement
|
151
|
+
none: false
|
107
152
|
requirements:
|
108
153
|
- - ">="
|
109
154
|
- !ruby/object:Gem::Version
|
155
|
+
hash: 3
|
110
156
|
segments:
|
111
157
|
- 0
|
112
158
|
version: "0"
|
113
159
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
160
|
+
none: false
|
114
161
|
requirements:
|
115
162
|
- - ">="
|
116
163
|
- !ruby/object:Gem::Version
|
164
|
+
hash: 3
|
117
165
|
segments:
|
118
166
|
- 0
|
119
167
|
version: "0"
|
120
168
|
requirements: []
|
121
169
|
|
122
170
|
rubyforge_project:
|
123
|
-
rubygems_version: 1.3.
|
171
|
+
rubygems_version: 1.3.7
|
124
172
|
signing_key:
|
125
173
|
specification_version: 3
|
126
174
|
summary: A resque plugin; provides retry, delay and exponential backoff support for resque jobs.
|