resque-retry 0.1.0 → 0.2.1
Sign up to get free protection for your applications and to get access to all the features.
- data/HISTORY.md +22 -1
- data/README.md +31 -5
- data/lib/resque-retry.rb +1 -1
- data/lib/resque-retry/server.rb +26 -1
- data/lib/resque-retry/server/views/retry.erb +6 -1
- data/lib/resque-retry/server/views/retry_timestamp.erb +21 -10
- data/lib/resque/failure/multiple_with_retry_suppression.rb +10 -20
- data/lib/resque/plugins/retry.rb +19 -5
- data/test/exponential_backoff_test.rb +4 -5
- data/test/multiple_failure_test.rb +24 -23
- data/test/redis-test.conf +1 -18
- data/test/resque_test.rb +1 -1
- data/test/retry_criteria_test.rb +1 -1
- data/test/retry_inheriting_checks_test.rb +1 -1
- data/test/retry_test.rb +34 -8
- data/test/server_test.rb +25 -0
- data/test/test_helper.rb +5 -5
- data/test/test_jobs.rb +46 -26
- metadata +95 -111
data/HISTORY.md
CHANGED
@@ -1,3 +1,24 @@
|
|
1
|
+
## 0.2.1 (2011-11-23)
|
2
|
+
|
3
|
+
* Bugfix: Fixed error when we tried to parse a number/string as JSON on the
|
4
|
+
reque-retry web interface.
|
5
|
+
|
6
|
+
## 0.2.0 (2011-11-22)
|
7
|
+
|
8
|
+
**INCLUDES NON-BACKWARDS COMPATIBLE CHANGES**
|
9
|
+
|
10
|
+
* IMPORTANT: `retry_limit` behaviour has changed. (Nicolas Fouché)
|
11
|
+
PREVIOUSLY: 0 == infinite retries.
|
12
|
+
NOW: -1 == infinite retries; 0 == means never retry.
|
13
|
+
|
14
|
+
* Bugfix: `#redis_retry_key` incorrectly built key when custom identifier
|
15
|
+
was used. (Bogdan Gusiev)
|
16
|
+
* Feature: Ability to sleep worker after re-queuing a job, may be used to bias
|
17
|
+
against the same worker from picking up the job again. (Michael Keirnan)
|
18
|
+
* Feature: Ability to remove retry jobs using resque-web. (Thiago Morello)
|
19
|
+
* Added example demo application.
|
20
|
+
* Added Bundler `Gemfile`.
|
21
|
+
|
1
22
|
## 0.1.0 (2010-08-29)
|
2
23
|
|
3
24
|
* Feature: Multiple failure backend with retry suppression.
|
@@ -30,4 +51,4 @@
|
|
30
51
|
|
31
52
|
## 0.0.1 (2010-04-27)
|
32
53
|
|
33
|
-
* First release.
|
54
|
+
* First release.
|
data/README.md
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
resque-retry
|
2
2
|
============
|
3
3
|
|
4
|
-
A [Resque][rq] plugin. Requires Resque 1.8.0 & [resque-scheduler][rqs].
|
4
|
+
A [Resque][rq] plugin. Requires Resque >= 1.8.0 & [resque-scheduler][rqs].
|
5
5
|
|
6
6
|
resque-retry provides retry, delay and exponential backoff support for
|
7
7
|
resque jobs.
|
@@ -19,7 +19,7 @@ To install:
|
|
19
19
|
|
20
20
|
$ gem install resque-retry
|
21
21
|
|
22
|
-
|
22
|
+
Add this to your `Rakefile`:
|
23
23
|
|
24
24
|
require 'resque/tasks'
|
25
25
|
require 'resque_scheduler/tasks'
|
@@ -118,7 +118,7 @@ some ideas =), adapt for your own usage and feel free to pick and mix!
|
|
118
118
|
|
119
119
|
Retry the job **once** on failure, with zero delay.
|
120
120
|
|
121
|
-
require '
|
121
|
+
require 'resque-retry'
|
122
122
|
|
123
123
|
class DeliverWebHook
|
124
124
|
extend Resque::Plugins::Retry
|
@@ -147,12 +147,37 @@ determine if we can requeue the job for another go.
|
|
147
147
|
end
|
148
148
|
end
|
149
149
|
|
150
|
-
The above modification will allow your job to retry
|
150
|
+
The above modification will allow your job to retry up to 10 times, with
|
151
151
|
a delay of 120 seconds, or 2 minutes between retry attempts.
|
152
152
|
|
153
153
|
Alternatively you could override the `retry_delay` method to do something
|
154
154
|
more special.
|
155
155
|
|
156
|
+
### Sleep After Requeuing
|
157
|
+
|
158
|
+
Sometimes it is useful to delay the worker that failed a job attempt, but still requeue the job for immediate processing
|
159
|
+
by other workers. This can be done with `@sleep_after_requeue`:
|
160
|
+
|
161
|
+
class DeliverWebHook
|
162
|
+
extend Resque::Plugins::Retry
|
163
|
+
@queue = :web_hooks
|
164
|
+
|
165
|
+
@sleep_after_requeue = 5
|
166
|
+
|
167
|
+
def self.perform(url, hook_id, hmac_key)
|
168
|
+
heavy_lifting
|
169
|
+
end
|
170
|
+
end
|
171
|
+
|
172
|
+
This retries the job once and causes the worker that failed to sleep for 5 seconds after requeuing the job. If there are
|
173
|
+
multiple workers in the system this allows the job to be retried immediately while the original worker heals itself. For
|
174
|
+
example failed jobs may cause other (non-worker) OS processes to die. A system monitor such as [god][god] can fix
|
175
|
+
the server while the job is being retried on a different worker.
|
176
|
+
|
177
|
+
`@sleep_after_requeue` is independent of `@retry_delay`. If you set both, they both take effect.
|
178
|
+
|
179
|
+
You can override the method `sleep_after_requeue` to set the sleep value dynamically.
|
180
|
+
|
156
181
|
### Exponential Backoff
|
157
182
|
|
158
183
|
Use this if you wish to vary the delay between retry attempts:
|
@@ -176,7 +201,7 @@ Use this if you wish to vary the delay between retry attempts:
|
|
176
201
|
The first delay will be 0 seconds, the 2nd will be 60 seconds, etc...
|
177
202
|
Again, tweak to your own needs.
|
178
203
|
|
179
|
-
The number
|
204
|
+
The number of retries is equal to the size of the `backoff_strategy`
|
180
205
|
array, unless you set `retry_limit` yourself.
|
181
206
|
|
182
207
|
### Retry Specific Exceptions
|
@@ -285,5 +310,6 @@ Contributing/Pull Requests
|
|
285
310
|
* Send me a pull request. Bonus points for topic branches.
|
286
311
|
* If you edit the gemspec/version etc, do it in another commit please.
|
287
312
|
|
313
|
+
[god]: http://github.com/mojombo/god
|
288
314
|
[rq]: http://github.com/defunkt/resque
|
289
315
|
[rqs]: http://github.com/bvandenbos/resque-scheduler
|
data/lib/resque-retry.rb
CHANGED
data/lib/resque-retry/server.rb
CHANGED
@@ -1,3 +1,5 @@
|
|
1
|
+
require 'cgi'
|
2
|
+
|
1
3
|
# Extend Resque::Server to add tabs.
|
2
4
|
module ResqueRetry
|
3
5
|
module Server
|
@@ -22,7 +24,7 @@ module ResqueRetry
|
|
22
24
|
|
23
25
|
# gets the failure details hash for a job.
|
24
26
|
def retry_failure_details(retry_key)
|
25
|
-
Resque.decode(Resque.redis
|
27
|
+
Resque.decode(Resque.redis.get("failure_#{retry_key}"))
|
26
28
|
end
|
27
29
|
|
28
30
|
# reads a 'local' template file.
|
@@ -30,6 +32,16 @@ module ResqueRetry
|
|
30
32
|
# Is there a better way to specify alternate template locations with sinatra?
|
31
33
|
File.read(File.join(File.dirname(__FILE__), "server/views/#{path}"))
|
32
34
|
end
|
35
|
+
|
36
|
+
# Cancels job retry
|
37
|
+
#
|
38
|
+
def cancel_retry(job)
|
39
|
+
klass = Resque.constantize(job['class'])
|
40
|
+
retry_key = retry_key_for_job(job)
|
41
|
+
Resque.remove_delayed(klass, *job['args'])
|
42
|
+
Resque.redis.del("failure_#{retry_key}")
|
43
|
+
Resque.redis.del(retry_key)
|
44
|
+
end
|
33
45
|
end
|
34
46
|
|
35
47
|
get '/retry' do
|
@@ -39,6 +51,19 @@ module ResqueRetry
|
|
39
51
|
get '/retry/:timestamp' do
|
40
52
|
erb local_template('retry_timestamp.erb')
|
41
53
|
end
|
54
|
+
|
55
|
+
post '/retry/:timestamp/remove' do
|
56
|
+
Resque.delayed_timestamp_peek(params[:timestamp], 0, 0).each do |job|
|
57
|
+
cancel_retry(job)
|
58
|
+
end
|
59
|
+
redirect u('retry')
|
60
|
+
end
|
61
|
+
|
62
|
+
post '/retry/:timestamp/jobs/:id/remove' do
|
63
|
+
job = Resque.decode(params[:id])
|
64
|
+
cancel_retry(job)
|
65
|
+
redirect u("retry/#{params[:timestamp]}")
|
66
|
+
end
|
42
67
|
}
|
43
68
|
end
|
44
69
|
|
@@ -22,9 +22,14 @@
|
|
22
22
|
<% timestamps = resque.delayed_queue_peek(start, start+20) %>
|
23
23
|
<% timestamps.each do |timestamp| %>
|
24
24
|
<% job = resque.delayed_timestamp_peek(timestamp, 0, 1).first %>
|
25
|
+
<% next unless job %>
|
25
26
|
<% retry_key = retry_key_for_job(job) %>
|
26
27
|
<tr>
|
27
28
|
<td>
|
29
|
+
<form action="<%= url "retry/#{timestamp}/remove" %>" method="post">
|
30
|
+
<input type="hidden" name="timestamp" value="<%= timestamp.to_i %>">
|
31
|
+
<input type="submit" value="Remove">
|
32
|
+
</form>
|
28
33
|
<form action="<%= url "/delayed/queue_now" %>" method="post">
|
29
34
|
<input type="hidden" name="timestamp" value="<%= timestamp.to_i %>">
|
30
35
|
<input type="submit" value="Queue now">
|
@@ -45,4 +50,4 @@
|
|
45
50
|
<% end %>
|
46
51
|
</table>
|
47
52
|
|
48
|
-
<%= partial :next_more, :start => start, :size => size %>
|
53
|
+
<%= partial :next_more, :start => start, :size => size %>
|
@@ -22,6 +22,7 @@
|
|
22
22
|
<th>Retry Attempts</th>
|
23
23
|
<th>Exception</th>
|
24
24
|
<th>Backtrace</th>
|
25
|
+
<th>Actions</th>
|
25
26
|
</tr>
|
26
27
|
<% jobs = resque.delayed_timestamp_peek(timestamp, start, 20) %>
|
27
28
|
<% jobs.each do |job| %>
|
@@ -35,21 +36,31 @@
|
|
35
36
|
<% else %>
|
36
37
|
<td><%= retry_attempts %></td>
|
37
38
|
<% failure = retry_failure_details(retry_key) %>
|
38
|
-
|
39
|
-
|
40
|
-
|
41
|
-
|
42
|
-
|
43
|
-
|
44
|
-
|
45
|
-
|
46
|
-
|
39
|
+
<% if failure.nil? %>
|
40
|
+
<td colspan="2"><i>n/a - not using resque-retry failure backend</i></td>
|
41
|
+
<% else %>
|
42
|
+
<td><code><%= failure['exception'] %></code></td>
|
43
|
+
<td class="error">
|
44
|
+
<% if failure['backtrace'] %>
|
45
|
+
<a href="#" class="backtrace"><%= h(failure['error']) %></a>
|
46
|
+
<pre style="display:none"><%= h failure['backtrace'].join("\n") %></pre>
|
47
|
+
<% else %>
|
48
|
+
<%= h failure['error'] %>
|
49
|
+
<% end %>
|
50
|
+
</td>
|
51
|
+
<% end %>
|
47
52
|
<% end %>
|
53
|
+
<td>
|
54
|
+
<form action="<%= url "retry/#{timestamp}/jobs/#{CGI.escape(Resque.encode(job))}/remove" %>" method="post">
|
55
|
+
<input type="hidden" name="timestamp" value="<%= timestamp.to_i %>">
|
56
|
+
<input type="submit" value="Remove">
|
57
|
+
</form>
|
58
|
+
</td>
|
48
59
|
</tr>
|
49
60
|
<% end %>
|
50
61
|
<% if jobs.empty? %>
|
51
62
|
<tr>
|
52
|
-
<td class="no-data" colspan="
|
63
|
+
<td class="no-data" colspan="6">
|
53
64
|
There are no pending jobs scheduled for this time.
|
54
65
|
</td>
|
55
66
|
</tr>
|
@@ -19,27 +19,16 @@ module Resque
|
|
19
19
|
class MultipleWithRetrySuppression < Multiple
|
20
20
|
include Resque::Helpers
|
21
21
|
|
22
|
-
module CleanupHooks
|
23
|
-
# Resque after_perform hook.
|
24
|
-
#
|
25
|
-
# Deletes retry failure information from Redis.
|
26
|
-
def after_perform_retry_failure_cleanup(*args)
|
27
|
-
retry_key = redis_retry_key(*args)
|
28
|
-
failure_key = Resque::Failure::MultipleWithRetrySuppression.failure_key(retry_key)
|
29
|
-
Resque.redis.del(failure_key)
|
30
|
-
end
|
31
|
-
end
|
32
|
-
|
33
22
|
# Called when the job fails.
|
34
23
|
#
|
35
24
|
# If the job will retry, suppress the failure from the other backends.
|
36
25
|
# Store the lastest failure information in redis, used by the web
|
37
26
|
# interface.
|
38
27
|
def save
|
39
|
-
|
28
|
+
if ! (retryable? && retrying?)
|
40
29
|
cleanup_retry_failure_log!
|
41
30
|
super
|
42
|
-
|
31
|
+
elsif retry_delay > 0
|
43
32
|
data = {
|
44
33
|
:failed_at => Time.now.strftime("%Y/%m/%d %H:%M:%S"),
|
45
34
|
:payload => payload,
|
@@ -50,12 +39,7 @@ module Resque
|
|
50
39
|
:queue => queue
|
51
40
|
}
|
52
41
|
|
53
|
-
|
54
|
-
unless klass.respond_to?(:after_perform_retry_failure_cleanup)
|
55
|
-
klass.send(:extend, CleanupHooks)
|
56
|
-
end
|
57
|
-
|
58
|
-
redis[failure_key] = Resque.encode(data)
|
42
|
+
redis.setex(failure_key, 2*retry_delay, Resque.encode(data))
|
59
43
|
end
|
60
44
|
end
|
61
45
|
|
@@ -69,8 +53,12 @@ module Resque
|
|
69
53
|
constantize(payload['class'])
|
70
54
|
end
|
71
55
|
|
56
|
+
def retry_delay
|
57
|
+
klass.retry_delay
|
58
|
+
end
|
59
|
+
|
72
60
|
def retry_key
|
73
|
-
klass.redis_retry_key(payload['args'])
|
61
|
+
klass.redis_retry_key(*payload['args'])
|
74
62
|
end
|
75
63
|
|
76
64
|
def failure_key
|
@@ -79,6 +67,8 @@ module Resque
|
|
79
67
|
|
80
68
|
def retryable?
|
81
69
|
klass.respond_to?(:redis_retry_key)
|
70
|
+
rescue NameError
|
71
|
+
false
|
82
72
|
end
|
83
73
|
|
84
74
|
def retrying?
|
data/lib/resque/plugins/retry.rb
CHANGED
@@ -63,7 +63,9 @@ module Resque
|
|
63
63
|
end
|
64
64
|
|
65
65
|
# Maximum number of retrys we can attempt to successfully perform the job.
|
66
|
-
#
|
66
|
+
#
|
67
|
+
# A retry limit of 0 will *never* retry.
|
68
|
+
# A retry limit of -1 or below will retry forever.
|
67
69
|
#
|
68
70
|
# @return [Fixnum]
|
69
71
|
def retry_limit
|
@@ -88,6 +90,14 @@ module Resque
|
|
88
90
|
@retry_delay ||= 0
|
89
91
|
end
|
90
92
|
|
93
|
+
# @abstract
|
94
|
+
# Number of seconds to sleep after job is requeued
|
95
|
+
#
|
96
|
+
# @return [Number] number of seconds to sleep
|
97
|
+
def sleep_after_requeue
|
98
|
+
@sleep_after_requeue ||= 0
|
99
|
+
end
|
100
|
+
|
91
101
|
# @abstract
|
92
102
|
# Modify the arguments used to retry the job. Use this to do something
|
93
103
|
# other than try the exact same job again.
|
@@ -147,10 +157,13 @@ module Resque
|
|
147
157
|
#
|
148
158
|
# @return [Boolean]
|
149
159
|
def retry_limit_reached?
|
150
|
-
if retry_limit
|
151
|
-
|
160
|
+
if retry_limit == 0
|
161
|
+
true
|
162
|
+
elsif retry_limit > 0
|
163
|
+
true if retry_attempt >= retry_limit
|
164
|
+
else
|
165
|
+
false
|
152
166
|
end
|
153
|
-
false
|
154
167
|
end
|
155
168
|
|
156
169
|
# Register a retry criteria check callback to be run before retrying
|
@@ -177,7 +190,7 @@ module Resque
|
|
177
190
|
retry_criteria_checks << block
|
178
191
|
end
|
179
192
|
|
180
|
-
#
|
193
|
+
# Retries the job.
|
181
194
|
def try_again(*args)
|
182
195
|
if retry_delay <= 0
|
183
196
|
# If the delay is 0, no point passing it through the scheduler
|
@@ -185,6 +198,7 @@ module Resque
|
|
185
198
|
else
|
186
199
|
Resque.enqueue_in(retry_delay, self, *args_for_retry(*args))
|
187
200
|
end
|
201
|
+
sleep(sleep_after_requeue) if sleep_after_requeue > 0
|
188
202
|
end
|
189
203
|
|
190
204
|
# Resque before_perform hook.
|
@@ -1,6 +1,6 @@
|
|
1
1
|
require File.dirname(__FILE__) + '/test_helper'
|
2
2
|
|
3
|
-
class ExponentialBackoffTest <
|
3
|
+
class ExponentialBackoffTest < MiniTest::Unit::TestCase
|
4
4
|
def setup
|
5
5
|
Resque.redis.flushall
|
6
6
|
@worker = Resque::Worker.new(:testing)
|
@@ -8,9 +8,8 @@ class ExponentialBackoffTest < Test::Unit::TestCase
|
|
8
8
|
end
|
9
9
|
|
10
10
|
def test_resque_plugin_lint
|
11
|
-
|
12
|
-
|
13
|
-
end
|
11
|
+
# will raise exception if were not a good plugin.
|
12
|
+
assert Resque::Plugin.lint(Resque::Plugins::ExponentialBackoff)
|
14
13
|
end
|
15
14
|
|
16
15
|
def test_default_backoff_strategy
|
@@ -19,7 +18,7 @@ class ExponentialBackoffTest < Test::Unit::TestCase
|
|
19
18
|
|
20
19
|
perform_next_job @worker
|
21
20
|
assert_equal 1, Resque.info[:processed], '1 processed job'
|
22
|
-
assert_equal 1, Resque.info[:failed], 'first ever run, and it should
|
21
|
+
assert_equal 1, Resque.info[:failed], 'first ever run, and it should have failed, but never retried'
|
23
22
|
assert_equal 1, Resque.info[:pending], '1 pending job, because it never hits the scheduler'
|
24
23
|
|
25
24
|
perform_next_job @worker
|
@@ -1,6 +1,6 @@
|
|
1
1
|
require File.dirname(__FILE__) + '/test_helper'
|
2
2
|
|
3
|
-
class MultipleFailureTest <
|
3
|
+
class MultipleFailureTest < MiniTest::Unit::TestCase
|
4
4
|
def setup
|
5
5
|
Resque.redis.flushall
|
6
6
|
@worker = Resque::Worker.new(:testing)
|
@@ -17,42 +17,35 @@ class MultipleFailureTest < Test::Unit::TestCase
|
|
17
17
|
key = "failure_" + klass.redis_retry_key(args)
|
18
18
|
end
|
19
19
|
|
20
|
-
def
|
21
|
-
Resque.enqueue(
|
20
|
+
def test_last_failure_is_saved_in_redis_if_delay
|
21
|
+
Resque.enqueue(LimitThreeJobDelay1Hour)
|
22
22
|
perform_next_job(@worker)
|
23
23
|
|
24
24
|
# I don't like this, but...
|
25
|
-
key = failure_key_for(
|
25
|
+
key = failure_key_for(LimitThreeJobDelay1Hour)
|
26
26
|
assert Resque.redis.exists(key)
|
27
27
|
end
|
28
28
|
|
29
|
-
def test_last_failure_removed_from_redis_after_error_limit
|
30
|
-
Resque.enqueue(LimitThreeJob)
|
31
|
-
3.times do
|
32
|
-
perform_next_job(@worker)
|
33
|
-
end
|
34
|
-
|
35
|
-
key = failure_key_for(LimitThreeJob)
|
36
|
-
assert Resque.redis.exists(key)
|
37
29
|
|
30
|
+
def test_last_failure_has_double_delay_redis_expiry_if_delay
|
31
|
+
Resque.enqueue(LimitThreeJobDelay1Hour)
|
38
32
|
perform_next_job(@worker)
|
39
|
-
|
33
|
+
|
34
|
+
# I don't like this, but...
|
35
|
+
key = failure_key_for(LimitThreeJobDelay1Hour)
|
36
|
+
assert_equal 7200, Resque.redis.ttl(key)
|
40
37
|
end
|
41
38
|
|
42
|
-
def
|
43
|
-
|
44
|
-
Resque.enqueue(SwitchToSuccessJob)
|
39
|
+
def test_last_failure_is_not_saved_in_redis_if_no_delay
|
40
|
+
Resque.enqueue(LimitThreeJob)
|
45
41
|
perform_next_job(@worker)
|
46
42
|
|
47
|
-
|
48
|
-
|
49
|
-
|
50
|
-
perform_next_job(@worker)
|
51
|
-
assert !Resque.redis.exists(key), 'key removed on success'
|
52
|
-
ensure
|
53
|
-
SwitchToSuccessJob.reset_defaults
|
43
|
+
# I don't like this, but...
|
44
|
+
key = failure_key_for(LimitThreeJob)
|
45
|
+
assert !Resque.redis.exists(key)
|
54
46
|
end
|
55
47
|
|
48
|
+
|
56
49
|
def test_errors_are_suppressed_up_to_retry_limit
|
57
50
|
Resque.enqueue(LimitThreeJob)
|
58
51
|
3.times do
|
@@ -80,6 +73,14 @@ class MultipleFailureTest < Test::Unit::TestCase
|
|
80
73
|
assert_equal 5, MockFailureBackend.errors.size
|
81
74
|
end
|
82
75
|
|
76
|
+
def test_custom_identifier_job
|
77
|
+
Resque.enqueue(CustomIdentifierFailingJob, 'qq', 2)
|
78
|
+
4.times do
|
79
|
+
perform_next_job(@worker)
|
80
|
+
end
|
81
|
+
assert_equal 1, MockFailureBackend.errors.size
|
82
|
+
end
|
83
|
+
|
83
84
|
def teardown
|
84
85
|
Resque::Failure.backend = @old_failure_backend
|
85
86
|
end
|
data/test/redis-test.conf
CHANGED
@@ -112,21 +112,4 @@ databases 16
|
|
112
112
|
# Glue small output buffers together in order to send small replies in a
|
113
113
|
# single TCP packet. Uses a bit more CPU but most of the times it is a win
|
114
114
|
# in terms of number of queries per second. Use 'yes' if unsure.
|
115
|
-
glueoutputbuf yes
|
116
|
-
|
117
|
-
# Use object sharing. Can save a lot of memory if you have many common
|
118
|
-
# string in your dataset, but performs lookups against the shared objects
|
119
|
-
# pool so it uses more CPU and can be a bit slower. Usually it's a good
|
120
|
-
# idea.
|
121
|
-
#
|
122
|
-
# When object sharing is enabled (shareobjects yes) you can use
|
123
|
-
# shareobjectspoolsize to control the size of the pool used in order to try
|
124
|
-
# object sharing. A bigger pool size will lead to better sharing capabilities.
|
125
|
-
# In general you want this value to be at least the double of the number of
|
126
|
-
# very common strings you have in your dataset.
|
127
|
-
#
|
128
|
-
# WARNING: object sharing is experimental, don't enable this feature
|
129
|
-
# in production before of Redis 1.0-stable. Still please try this feature in
|
130
|
-
# your development environment so that we can test it better.
|
131
|
-
# shareobjects no
|
132
|
-
# shareobjectspoolsize 1024
|
115
|
+
glueoutputbuf yes
|
data/test/resque_test.rb
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
require File.dirname(__FILE__) + '/test_helper'
|
2
2
|
|
3
3
|
# make sure the worlds not fallen from beneith us.
|
4
|
-
class ResqueTest <
|
4
|
+
class ResqueTest < MiniTest::Unit::TestCase
|
5
5
|
def test_resque_version
|
6
6
|
major, minor, patch = Resque::Version.split('.')
|
7
7
|
assert_equal 1, major.to_i, 'major version does not match'
|
data/test/retry_criteria_test.rb
CHANGED
data/test/retry_test.rb
CHANGED
@@ -1,6 +1,6 @@
|
|
1
1
|
require File.dirname(__FILE__) + '/test_helper'
|
2
2
|
|
3
|
-
class RetryTest <
|
3
|
+
class RetryTest < MiniTest::Unit::TestCase
|
4
4
|
def setup
|
5
5
|
Resque.redis.flushall
|
6
6
|
@worker = Resque::Worker.new(:testing)
|
@@ -8,16 +8,15 @@ class RetryTest < Test::Unit::TestCase
|
|
8
8
|
end
|
9
9
|
|
10
10
|
def test_resque_plugin_lint
|
11
|
-
|
12
|
-
|
13
|
-
end
|
11
|
+
# will raise exception if were not a good plugin.
|
12
|
+
assert Resque::Plugin.lint(Resque::Plugins::Retry)
|
14
13
|
end
|
15
14
|
|
16
15
|
def test_default_settings
|
17
|
-
assert_equal 1,
|
18
|
-
assert_equal 0,
|
19
|
-
assert_equal nil,
|
20
|
-
assert_equal 0,
|
16
|
+
assert_equal 1, RetryDefaultSettingsJob.retry_limit, 'default retry limit'
|
17
|
+
assert_equal 0, RetryDefaultSettingsJob.retry_attempt, 'default number of retry attempts'
|
18
|
+
assert_equal nil, RetryDefaultSettingsJob.retry_exceptions, 'default retry exceptions; nil = any'
|
19
|
+
assert_equal 0, RetryDefaultSettingsJob.retry_delay, 'default seconds until retry'
|
21
20
|
end
|
22
21
|
|
23
22
|
def test_retry_once_by_default
|
@@ -71,6 +70,17 @@ class RetryTest < Test::Unit::TestCase
|
|
71
70
|
assert_equal 10, Resque.info[:processed], 'processed job'
|
72
71
|
end
|
73
72
|
|
73
|
+
def test_retry_never_retry
|
74
|
+
Resque.enqueue(NeverRetryJob)
|
75
|
+
10.times do
|
76
|
+
perform_next_job(@worker)
|
77
|
+
end
|
78
|
+
|
79
|
+
assert_equal 0, Resque.info[:pending], 'pending jobs'
|
80
|
+
assert_equal 1, Resque.info[:failed], 'failed jobs'
|
81
|
+
assert_equal 1, Resque.info[:processed], 'processed job'
|
82
|
+
end
|
83
|
+
|
74
84
|
def test_fail_five_times_then_succeed
|
75
85
|
Resque.enqueue(FailFiveTimesJob)
|
76
86
|
7.times do
|
@@ -82,6 +92,22 @@ class RetryTest < Test::Unit::TestCase
|
|
82
92
|
assert_equal 0, Resque.info[:pending], 'pending jobs'
|
83
93
|
end
|
84
94
|
|
95
|
+
def test_retry_delay_sleep
|
96
|
+
assert_equal 0, Resque.info[:failed], 'failed jobs'
|
97
|
+
Resque.enqueue(SleepDelay1SecondJob)
|
98
|
+
before = Time.now
|
99
|
+
3.times do
|
100
|
+
perform_next_job(@worker)
|
101
|
+
end
|
102
|
+
actual_delay = Time.now - before
|
103
|
+
|
104
|
+
assert actual_delay >= 1, "did not sleep long enough: #{actual_delay} seconds"
|
105
|
+
assert actual_delay < 2, "slept too long: #{actual_delay} seconds"
|
106
|
+
assert_equal 1, Resque.info[:failed], 'failed jobs'
|
107
|
+
assert_equal 2, Resque.info[:processed], 'processed job'
|
108
|
+
assert_equal 0, Resque.info[:pending], 'pending jobs'
|
109
|
+
end
|
110
|
+
|
85
111
|
def test_can_determine_if_exception_may_be_retried
|
86
112
|
assert_equal true, RetryDefaultsJob.retry_exception?(StandardError), 'StandardError may retry'
|
87
113
|
assert_equal true, RetryDefaultsJob.retry_exception?(CustomException), 'CustomException may retry'
|
data/test/server_test.rb
ADDED
@@ -0,0 +1,25 @@
|
|
1
|
+
require File.dirname(__FILE__) + '/test_helper'
|
2
|
+
|
3
|
+
require 'resque-retry/server'
|
4
|
+
ENV['RACK_ENV'] = 'test'
|
5
|
+
|
6
|
+
# Testing the Resque web interface additions.
|
7
|
+
class ServerTest < MiniTest::Unit::TestCase
|
8
|
+
include Rack::Test::Methods
|
9
|
+
|
10
|
+
def app
|
11
|
+
Resque::Server
|
12
|
+
end
|
13
|
+
|
14
|
+
# make sure the world looks okay first =)
|
15
|
+
def test_were_able_to_access_normal_resque_web_overview
|
16
|
+
get '/overview'
|
17
|
+
assert_equal 200, last_response.status, 'HTTP status code should be 200'
|
18
|
+
end
|
19
|
+
|
20
|
+
def test_should_include_retry_tab
|
21
|
+
get '/overview'
|
22
|
+
assert last_response.body.include?('/retry')
|
23
|
+
end
|
24
|
+
|
25
|
+
end
|
data/test/test_helper.rb
CHANGED
@@ -2,10 +2,10 @@ dir = File.dirname(File.expand_path(__FILE__))
|
|
2
2
|
$LOAD_PATH.unshift dir + '/../lib'
|
3
3
|
$TESTING = true
|
4
4
|
|
5
|
-
require '
|
6
|
-
require '
|
7
|
-
require '
|
8
|
-
require 'simplecov
|
5
|
+
require 'minitest/unit'
|
6
|
+
require 'minitest/pride'
|
7
|
+
require 'rack/test'
|
8
|
+
require 'simplecov'
|
9
9
|
|
10
10
|
SimpleCov.start do
|
11
11
|
add_filter "/test/"
|
@@ -59,7 +59,7 @@ class MockFailureBackend < Resque::Failure::Base
|
|
59
59
|
end
|
60
60
|
|
61
61
|
# Test helpers
|
62
|
-
class
|
62
|
+
class MiniTest::Unit::TestCase
|
63
63
|
def perform_next_job(worker, &block)
|
64
64
|
return unless job = @worker.reserve
|
65
65
|
@worker.perform(job, &block)
|
data/test/test_jobs.rb
CHANGED
@@ -13,6 +13,15 @@ end
|
|
13
13
|
class GoodJob
|
14
14
|
extend Resque::Plugins::Retry
|
15
15
|
@queue = :testing
|
16
|
+
|
17
|
+
def self.perform(*args)
|
18
|
+
end
|
19
|
+
end
|
20
|
+
|
21
|
+
class RetryDefaultSettingsJob
|
22
|
+
extend Resque::Plugins::Retry
|
23
|
+
@queue = :testing
|
24
|
+
|
16
25
|
def self.perform(*args)
|
17
26
|
end
|
18
27
|
end
|
@@ -26,6 +35,15 @@ class RetryDefaultsJob
|
|
26
35
|
end
|
27
36
|
end
|
28
37
|
|
38
|
+
class SleepDelay1SecondJob < RetryDefaultsJob
|
39
|
+
@queue = :testing
|
40
|
+
@sleep_after_requeue = 1
|
41
|
+
|
42
|
+
def self.perform(*args)
|
43
|
+
raise if retry_attempt == 0
|
44
|
+
end
|
45
|
+
end
|
46
|
+
|
29
47
|
class InheritTestJob < RetryDefaultsJob
|
30
48
|
end
|
31
49
|
|
@@ -54,6 +72,11 @@ class RetryWithModifiedArgsJob < RetryDefaultsJob
|
|
54
72
|
end
|
55
73
|
|
56
74
|
class NeverGiveUpJob < RetryDefaultsJob
|
75
|
+
@queue = :testing
|
76
|
+
@retry_limit = -1
|
77
|
+
end
|
78
|
+
|
79
|
+
class NeverRetryJob < RetryDefaultsJob
|
57
80
|
@queue = :testing
|
58
81
|
@retry_limit = 0
|
59
82
|
end
|
@@ -67,6 +90,12 @@ class LimitThreeJob < RetryDefaultsJob
|
|
67
90
|
end
|
68
91
|
end
|
69
92
|
|
93
|
+
class LimitThreeJobDelay1Hour < LimitThreeJob
|
94
|
+
@queue = :testing
|
95
|
+
@retry_limit = 3
|
96
|
+
@retry_delay = 3600
|
97
|
+
end
|
98
|
+
|
70
99
|
class FailFiveTimesJob < RetryDefaultsJob
|
71
100
|
@queue = :testing
|
72
101
|
@retry_limit = 6
|
@@ -204,32 +233,6 @@ class InheritOrderingJobExtendFirst
|
|
204
233
|
end
|
205
234
|
end
|
206
235
|
|
207
|
-
# This job switches to successful after
|
208
|
-
# n +tries+.
|
209
|
-
class SwitchToSuccessJob < GoodJob
|
210
|
-
@queue = :testing
|
211
|
-
@max_retries = 3
|
212
|
-
|
213
|
-
class << self
|
214
|
-
attr_accessor :successful_after
|
215
|
-
attr_accessor :tries
|
216
|
-
|
217
|
-
def reset_defaults
|
218
|
-
self.tries = 0
|
219
|
-
self.successful_after = 2
|
220
|
-
end
|
221
|
-
end
|
222
|
-
|
223
|
-
reset_defaults
|
224
|
-
|
225
|
-
def self.perform(*args)
|
226
|
-
if self.tries < self.successful_after
|
227
|
-
self.tries += 1
|
228
|
-
raise "error"
|
229
|
-
end
|
230
|
-
end
|
231
|
-
end
|
232
|
-
|
233
236
|
class InheritOrderingJobExtendLast
|
234
237
|
class << self
|
235
238
|
attr_accessor :test_value
|
@@ -247,5 +250,22 @@ class InheritOrderingJobExtendLast
|
|
247
250
|
end
|
248
251
|
end
|
249
252
|
|
253
|
+
|
250
254
|
class InheritOrderingJobExtendFirstSubclass < InheritOrderingJobExtendFirst; end
|
251
255
|
class InheritOrderingJobExtendLastSubclass < InheritOrderingJobExtendLast; end
|
256
|
+
|
257
|
+
class CustomIdentifierFailingJob
|
258
|
+
extend Resque::Plugins::Retry
|
259
|
+
|
260
|
+
@queue = :testing
|
261
|
+
@retry_limit = 2
|
262
|
+
@retry_delay = 0
|
263
|
+
|
264
|
+
def self.identifier(*args)
|
265
|
+
args.first.to_s
|
266
|
+
end
|
267
|
+
|
268
|
+
def self.perform(*args)
|
269
|
+
raise 'failed'
|
270
|
+
end
|
271
|
+
end
|
metadata
CHANGED
@@ -1,123 +1,115 @@
|
|
1
|
-
--- !ruby/object:Gem::Specification
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
2
|
name: resque-retry
|
3
|
-
version: !ruby/object:Gem::Version
|
4
|
-
|
5
|
-
prerelease:
|
6
|
-
segments:
|
7
|
-
- 0
|
8
|
-
- 1
|
9
|
-
- 0
|
10
|
-
version: 0.1.0
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 0.2.1
|
5
|
+
prerelease:
|
11
6
|
platform: ruby
|
12
|
-
authors:
|
7
|
+
authors:
|
13
8
|
- Luke Antins
|
14
9
|
- Ryan Carver
|
15
10
|
autorequire:
|
16
11
|
bindir: bin
|
17
12
|
cert_chain: []
|
18
|
-
|
19
|
-
|
20
|
-
|
21
|
-
|
22
|
-
|
23
|
-
|
13
|
+
date: 2011-11-23 00:00:00.000000000Z
|
14
|
+
dependencies:
|
15
|
+
- !ruby/object:Gem::Dependency
|
16
|
+
name: rake
|
17
|
+
requirement: &2165851780 !ruby/object:Gem::Requirement
|
18
|
+
none: false
|
19
|
+
requirements:
|
20
|
+
- - ! '>='
|
21
|
+
- !ruby/object:Gem::Version
|
22
|
+
version: '0'
|
23
|
+
type: :runtime
|
24
24
|
prerelease: false
|
25
|
-
|
25
|
+
version_requirements: *2165851780
|
26
|
+
- !ruby/object:Gem::Dependency
|
27
|
+
name: resque
|
28
|
+
requirement: &2165851020 !ruby/object:Gem::Requirement
|
26
29
|
none: false
|
27
|
-
requirements:
|
28
|
-
- -
|
29
|
-
- !ruby/object:Gem::Version
|
30
|
-
hash: 55
|
31
|
-
segments:
|
32
|
-
- 1
|
33
|
-
- 8
|
34
|
-
- 0
|
30
|
+
requirements:
|
31
|
+
- - ! '>='
|
32
|
+
- !ruby/object:Gem::Version
|
35
33
|
version: 1.8.0
|
36
34
|
type: :runtime
|
37
|
-
version_requirements: *id001
|
38
|
-
- !ruby/object:Gem::Dependency
|
39
|
-
name: resque-scheduler
|
40
35
|
prerelease: false
|
41
|
-
|
36
|
+
version_requirements: *2165851020
|
37
|
+
- !ruby/object:Gem::Dependency
|
38
|
+
name: resque-scheduler
|
39
|
+
requirement: &2165835400 !ruby/object:Gem::Requirement
|
42
40
|
none: false
|
43
|
-
requirements:
|
44
|
-
- -
|
45
|
-
- !ruby/object:Gem::Version
|
46
|
-
hash: 55
|
47
|
-
segments:
|
48
|
-
- 1
|
49
|
-
- 8
|
50
|
-
- 0
|
41
|
+
requirements:
|
42
|
+
- - ! '>='
|
43
|
+
- !ruby/object:Gem::Version
|
51
44
|
version: 1.8.0
|
52
45
|
type: :runtime
|
53
|
-
version_requirements: *id002
|
54
|
-
- !ruby/object:Gem::Dependency
|
55
|
-
name: test-unit
|
56
46
|
prerelease: false
|
57
|
-
|
47
|
+
version_requirements: *2165835400
|
48
|
+
- !ruby/object:Gem::Dependency
|
49
|
+
name: minitest
|
50
|
+
requirement: &2165834800 !ruby/object:Gem::Requirement
|
58
51
|
none: false
|
59
|
-
requirements:
|
60
|
-
- -
|
61
|
-
- !ruby/object:Gem::Version
|
62
|
-
|
63
|
-
segments:
|
64
|
-
- 0
|
65
|
-
version: "0"
|
52
|
+
requirements:
|
53
|
+
- - ! '>='
|
54
|
+
- !ruby/object:Gem::Version
|
55
|
+
version: '0'
|
66
56
|
type: :development
|
67
|
-
version_requirements: *id003
|
68
|
-
- !ruby/object:Gem::Dependency
|
69
|
-
name: turn
|
70
57
|
prerelease: false
|
71
|
-
|
58
|
+
version_requirements: *2165834800
|
59
|
+
- !ruby/object:Gem::Dependency
|
60
|
+
name: rack-test
|
61
|
+
requirement: &2165834000 !ruby/object:Gem::Requirement
|
72
62
|
none: false
|
73
|
-
requirements:
|
74
|
-
- -
|
75
|
-
- !ruby/object:Gem::Version
|
76
|
-
|
77
|
-
segments:
|
78
|
-
- 0
|
79
|
-
version: "0"
|
63
|
+
requirements:
|
64
|
+
- - ! '>='
|
65
|
+
- !ruby/object:Gem::Version
|
66
|
+
version: '0'
|
80
67
|
type: :development
|
81
|
-
|
82
|
-
|
68
|
+
prerelease: false
|
69
|
+
version_requirements: *2165834000
|
70
|
+
- !ruby/object:Gem::Dependency
|
83
71
|
name: yard
|
72
|
+
requirement: &2165833040 !ruby/object:Gem::Requirement
|
73
|
+
none: false
|
74
|
+
requirements:
|
75
|
+
- - ! '>='
|
76
|
+
- !ruby/object:Gem::Version
|
77
|
+
version: '0'
|
78
|
+
type: :development
|
84
79
|
prerelease: false
|
85
|
-
|
80
|
+
version_requirements: *2165833040
|
81
|
+
- !ruby/object:Gem::Dependency
|
82
|
+
name: json
|
83
|
+
requirement: &2165832180 !ruby/object:Gem::Requirement
|
86
84
|
none: false
|
87
|
-
requirements:
|
88
|
-
- -
|
89
|
-
- !ruby/object:Gem::Version
|
90
|
-
|
91
|
-
segments:
|
92
|
-
- 0
|
93
|
-
version: "0"
|
85
|
+
requirements:
|
86
|
+
- - ! '>='
|
87
|
+
- !ruby/object:Gem::Version
|
88
|
+
version: '0'
|
94
89
|
type: :development
|
95
|
-
version_requirements: *id005
|
96
|
-
- !ruby/object:Gem::Dependency
|
97
|
-
name: simplecov-html
|
98
90
|
prerelease: false
|
99
|
-
|
91
|
+
version_requirements: *2165832180
|
92
|
+
- !ruby/object:Gem::Dependency
|
93
|
+
name: simplecov
|
94
|
+
requirement: &2165831320 !ruby/object:Gem::Requirement
|
100
95
|
none: false
|
101
|
-
requirements:
|
102
|
-
- -
|
103
|
-
- !ruby/object:Gem::Version
|
104
|
-
hash: 19
|
105
|
-
segments:
|
106
|
-
- 0
|
107
|
-
- 3
|
108
|
-
- 0
|
96
|
+
requirements:
|
97
|
+
- - ! '>='
|
98
|
+
- !ruby/object:Gem::Version
|
109
99
|
version: 0.3.0
|
110
100
|
type: :development
|
111
|
-
|
112
|
-
|
101
|
+
prerelease: false
|
102
|
+
version_requirements: *2165831320
|
103
|
+
description: ! " resque-retry provides retry, delay and exponential backoff support
|
104
|
+
for\n resque jobs.\n\n Features:\n\n * Redis backed retry count/limit.\n * Retry
|
105
|
+
on all or specific exceptions.\n * Exponential backoff (varying the delay between
|
106
|
+
retrys).\n * Multiple failure backend with retry suppression & resque-web tab.\n
|
107
|
+
\ * Small & Extendable - plenty of places to override retry logic/settings.\n"
|
113
108
|
email: luke@lividpenguin.com
|
114
109
|
executables: []
|
115
|
-
|
116
110
|
extensions: []
|
117
|
-
|
118
111
|
extra_rdoc_files: []
|
119
|
-
|
120
|
-
files:
|
112
|
+
files:
|
121
113
|
- LICENSE
|
122
114
|
- Rakefile
|
123
115
|
- README.md
|
@@ -129,6 +121,7 @@ files:
|
|
129
121
|
- test/retry_criteria_test.rb
|
130
122
|
- test/retry_inheriting_checks_test.rb
|
131
123
|
- test/retry_test.rb
|
124
|
+
- test/server_test.rb
|
132
125
|
- test/test_helper.rb
|
133
126
|
- test/test_jobs.rb
|
134
127
|
- lib/resque/failure/multiple_with_retry_suppression.rb
|
@@ -138,39 +131,30 @@ files:
|
|
138
131
|
- lib/resque-retry/server/views/retry_timestamp.erb
|
139
132
|
- lib/resque-retry/server.rb
|
140
133
|
- lib/resque-retry.rb
|
141
|
-
has_rdoc: false
|
142
134
|
homepage: http://github.com/lantins/resque-retry
|
143
135
|
licenses: []
|
144
|
-
|
145
136
|
post_install_message:
|
146
137
|
rdoc_options: []
|
147
|
-
|
148
|
-
require_paths:
|
138
|
+
require_paths:
|
149
139
|
- lib
|
150
|
-
required_ruby_version: !ruby/object:Gem::Requirement
|
140
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
151
141
|
none: false
|
152
|
-
requirements:
|
153
|
-
- -
|
154
|
-
- !ruby/object:Gem::Version
|
155
|
-
|
156
|
-
|
157
|
-
- 0
|
158
|
-
version: "0"
|
159
|
-
required_rubygems_version: !ruby/object:Gem::Requirement
|
142
|
+
requirements:
|
143
|
+
- - ! '>='
|
144
|
+
- !ruby/object:Gem::Version
|
145
|
+
version: '0'
|
146
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
160
147
|
none: false
|
161
|
-
requirements:
|
162
|
-
- -
|
163
|
-
- !ruby/object:Gem::Version
|
164
|
-
|
165
|
-
segments:
|
166
|
-
- 0
|
167
|
-
version: "0"
|
148
|
+
requirements:
|
149
|
+
- - ! '>='
|
150
|
+
- !ruby/object:Gem::Version
|
151
|
+
version: '0'
|
168
152
|
requirements: []
|
169
|
-
|
170
153
|
rubyforge_project:
|
171
|
-
rubygems_version: 1.
|
154
|
+
rubygems_version: 1.8.10
|
172
155
|
signing_key:
|
173
156
|
specification_version: 3
|
174
|
-
summary: A resque plugin; provides retry, delay and exponential backoff support for
|
157
|
+
summary: A resque plugin; provides retry, delay and exponential backoff support for
|
158
|
+
resque jobs.
|
175
159
|
test_files: []
|
176
|
-
|
160
|
+
has_rdoc: false
|