resque-retry 0.1.0 → 0.2.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- data/HISTORY.md +22 -1
- data/README.md +31 -5
- data/lib/resque-retry.rb +1 -1
- data/lib/resque-retry/server.rb +26 -1
- data/lib/resque-retry/server/views/retry.erb +6 -1
- data/lib/resque-retry/server/views/retry_timestamp.erb +21 -10
- data/lib/resque/failure/multiple_with_retry_suppression.rb +10 -20
- data/lib/resque/plugins/retry.rb +19 -5
- data/test/exponential_backoff_test.rb +4 -5
- data/test/multiple_failure_test.rb +24 -23
- data/test/redis-test.conf +1 -18
- data/test/resque_test.rb +1 -1
- data/test/retry_criteria_test.rb +1 -1
- data/test/retry_inheriting_checks_test.rb +1 -1
- data/test/retry_test.rb +34 -8
- data/test/server_test.rb +25 -0
- data/test/test_helper.rb +5 -5
- data/test/test_jobs.rb +46 -26
- metadata +95 -111
data/HISTORY.md
CHANGED
@@ -1,3 +1,24 @@
|
|
1
|
+
## 0.2.1 (2011-11-23)
|
2
|
+
|
3
|
+
* Bugfix: Fixed error when we tried to parse a number/string as JSON on the
|
4
|
+
reque-retry web interface.
|
5
|
+
|
6
|
+
## 0.2.0 (2011-11-22)
|
7
|
+
|
8
|
+
**INCLUDES NON-BACKWARDS COMPATIBLE CHANGES**
|
9
|
+
|
10
|
+
* IMPORTANT: `retry_limit` behaviour has changed. (Nicolas Fouché)
|
11
|
+
PREVIOUSLY: 0 == infinite retries.
|
12
|
+
NOW: -1 == infinite retries; 0 == means never retry.
|
13
|
+
|
14
|
+
* Bugfix: `#redis_retry_key` incorrectly built key when custom identifier
|
15
|
+
was used. (Bogdan Gusiev)
|
16
|
+
* Feature: Ability to sleep worker after re-queuing a job, may be used to bias
|
17
|
+
against the same worker from picking up the job again. (Michael Keirnan)
|
18
|
+
* Feature: Ability to remove retry jobs using resque-web. (Thiago Morello)
|
19
|
+
* Added example demo application.
|
20
|
+
* Added Bundler `Gemfile`.
|
21
|
+
|
1
22
|
## 0.1.0 (2010-08-29)
|
2
23
|
|
3
24
|
* Feature: Multiple failure backend with retry suppression.
|
@@ -30,4 +51,4 @@
|
|
30
51
|
|
31
52
|
## 0.0.1 (2010-04-27)
|
32
53
|
|
33
|
-
* First release.
|
54
|
+
* First release.
|
data/README.md
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
resque-retry
|
2
2
|
============
|
3
3
|
|
4
|
-
A [Resque][rq] plugin. Requires Resque 1.8.0 & [resque-scheduler][rqs].
|
4
|
+
A [Resque][rq] plugin. Requires Resque >= 1.8.0 & [resque-scheduler][rqs].
|
5
5
|
|
6
6
|
resque-retry provides retry, delay and exponential backoff support for
|
7
7
|
resque jobs.
|
@@ -19,7 +19,7 @@ To install:
|
|
19
19
|
|
20
20
|
$ gem install resque-retry
|
21
21
|
|
22
|
-
|
22
|
+
Add this to your `Rakefile`:
|
23
23
|
|
24
24
|
require 'resque/tasks'
|
25
25
|
require 'resque_scheduler/tasks'
|
@@ -118,7 +118,7 @@ some ideas =), adapt for your own usage and feel free to pick and mix!
|
|
118
118
|
|
119
119
|
Retry the job **once** on failure, with zero delay.
|
120
120
|
|
121
|
-
require '
|
121
|
+
require 'resque-retry'
|
122
122
|
|
123
123
|
class DeliverWebHook
|
124
124
|
extend Resque::Plugins::Retry
|
@@ -147,12 +147,37 @@ determine if we can requeue the job for another go.
|
|
147
147
|
end
|
148
148
|
end
|
149
149
|
|
150
|
-
The above modification will allow your job to retry
|
150
|
+
The above modification will allow your job to retry up to 10 times, with
|
151
151
|
a delay of 120 seconds, or 2 minutes between retry attempts.
|
152
152
|
|
153
153
|
Alternatively you could override the `retry_delay` method to do something
|
154
154
|
more special.
|
155
155
|
|
156
|
+
### Sleep After Requeuing
|
157
|
+
|
158
|
+
Sometimes it is useful to delay the worker that failed a job attempt, but still requeue the job for immediate processing
|
159
|
+
by other workers. This can be done with `@sleep_after_requeue`:
|
160
|
+
|
161
|
+
class DeliverWebHook
|
162
|
+
extend Resque::Plugins::Retry
|
163
|
+
@queue = :web_hooks
|
164
|
+
|
165
|
+
@sleep_after_requeue = 5
|
166
|
+
|
167
|
+
def self.perform(url, hook_id, hmac_key)
|
168
|
+
heavy_lifting
|
169
|
+
end
|
170
|
+
end
|
171
|
+
|
172
|
+
This retries the job once and causes the worker that failed to sleep for 5 seconds after requeuing the job. If there are
|
173
|
+
multiple workers in the system this allows the job to be retried immediately while the original worker heals itself. For
|
174
|
+
example failed jobs may cause other (non-worker) OS processes to die. A system monitor such as [god][god] can fix
|
175
|
+
the server while the job is being retried on a different worker.
|
176
|
+
|
177
|
+
`@sleep_after_requeue` is independent of `@retry_delay`. If you set both, they both take effect.
|
178
|
+
|
179
|
+
You can override the method `sleep_after_requeue` to set the sleep value dynamically.
|
180
|
+
|
156
181
|
### Exponential Backoff
|
157
182
|
|
158
183
|
Use this if you wish to vary the delay between retry attempts:
|
@@ -176,7 +201,7 @@ Use this if you wish to vary the delay between retry attempts:
|
|
176
201
|
The first delay will be 0 seconds, the 2nd will be 60 seconds, etc...
|
177
202
|
Again, tweak to your own needs.
|
178
203
|
|
179
|
-
The number
|
204
|
+
The number of retries is equal to the size of the `backoff_strategy`
|
180
205
|
array, unless you set `retry_limit` yourself.
|
181
206
|
|
182
207
|
### Retry Specific Exceptions
|
@@ -285,5 +310,6 @@ Contributing/Pull Requests
|
|
285
310
|
* Send me a pull request. Bonus points for topic branches.
|
286
311
|
* If you edit the gemspec/version etc, do it in another commit please.
|
287
312
|
|
313
|
+
[god]: http://github.com/mojombo/god
|
288
314
|
[rq]: http://github.com/defunkt/resque
|
289
315
|
[rqs]: http://github.com/bvandenbos/resque-scheduler
|
data/lib/resque-retry.rb
CHANGED
data/lib/resque-retry/server.rb
CHANGED
@@ -1,3 +1,5 @@
|
|
1
|
+
require 'cgi'
|
2
|
+
|
1
3
|
# Extend Resque::Server to add tabs.
|
2
4
|
module ResqueRetry
|
3
5
|
module Server
|
@@ -22,7 +24,7 @@ module ResqueRetry
|
|
22
24
|
|
23
25
|
# gets the failure details hash for a job.
|
24
26
|
def retry_failure_details(retry_key)
|
25
|
-
Resque.decode(Resque.redis
|
27
|
+
Resque.decode(Resque.redis.get("failure_#{retry_key}"))
|
26
28
|
end
|
27
29
|
|
28
30
|
# reads a 'local' template file.
|
@@ -30,6 +32,16 @@ module ResqueRetry
|
|
30
32
|
# Is there a better way to specify alternate template locations with sinatra?
|
31
33
|
File.read(File.join(File.dirname(__FILE__), "server/views/#{path}"))
|
32
34
|
end
|
35
|
+
|
36
|
+
# Cancels job retry
|
37
|
+
#
|
38
|
+
def cancel_retry(job)
|
39
|
+
klass = Resque.constantize(job['class'])
|
40
|
+
retry_key = retry_key_for_job(job)
|
41
|
+
Resque.remove_delayed(klass, *job['args'])
|
42
|
+
Resque.redis.del("failure_#{retry_key}")
|
43
|
+
Resque.redis.del(retry_key)
|
44
|
+
end
|
33
45
|
end
|
34
46
|
|
35
47
|
get '/retry' do
|
@@ -39,6 +51,19 @@ module ResqueRetry
|
|
39
51
|
get '/retry/:timestamp' do
|
40
52
|
erb local_template('retry_timestamp.erb')
|
41
53
|
end
|
54
|
+
|
55
|
+
post '/retry/:timestamp/remove' do
|
56
|
+
Resque.delayed_timestamp_peek(params[:timestamp], 0, 0).each do |job|
|
57
|
+
cancel_retry(job)
|
58
|
+
end
|
59
|
+
redirect u('retry')
|
60
|
+
end
|
61
|
+
|
62
|
+
post '/retry/:timestamp/jobs/:id/remove' do
|
63
|
+
job = Resque.decode(params[:id])
|
64
|
+
cancel_retry(job)
|
65
|
+
redirect u("retry/#{params[:timestamp]}")
|
66
|
+
end
|
42
67
|
}
|
43
68
|
end
|
44
69
|
|
@@ -22,9 +22,14 @@
|
|
22
22
|
<% timestamps = resque.delayed_queue_peek(start, start+20) %>
|
23
23
|
<% timestamps.each do |timestamp| %>
|
24
24
|
<% job = resque.delayed_timestamp_peek(timestamp, 0, 1).first %>
|
25
|
+
<% next unless job %>
|
25
26
|
<% retry_key = retry_key_for_job(job) %>
|
26
27
|
<tr>
|
27
28
|
<td>
|
29
|
+
<form action="<%= url "retry/#{timestamp}/remove" %>" method="post">
|
30
|
+
<input type="hidden" name="timestamp" value="<%= timestamp.to_i %>">
|
31
|
+
<input type="submit" value="Remove">
|
32
|
+
</form>
|
28
33
|
<form action="<%= url "/delayed/queue_now" %>" method="post">
|
29
34
|
<input type="hidden" name="timestamp" value="<%= timestamp.to_i %>">
|
30
35
|
<input type="submit" value="Queue now">
|
@@ -45,4 +50,4 @@
|
|
45
50
|
<% end %>
|
46
51
|
</table>
|
47
52
|
|
48
|
-
<%= partial :next_more, :start => start, :size => size %>
|
53
|
+
<%= partial :next_more, :start => start, :size => size %>
|
@@ -22,6 +22,7 @@
|
|
22
22
|
<th>Retry Attempts</th>
|
23
23
|
<th>Exception</th>
|
24
24
|
<th>Backtrace</th>
|
25
|
+
<th>Actions</th>
|
25
26
|
</tr>
|
26
27
|
<% jobs = resque.delayed_timestamp_peek(timestamp, start, 20) %>
|
27
28
|
<% jobs.each do |job| %>
|
@@ -35,21 +36,31 @@
|
|
35
36
|
<% else %>
|
36
37
|
<td><%= retry_attempts %></td>
|
37
38
|
<% failure = retry_failure_details(retry_key) %>
|
38
|
-
|
39
|
-
|
40
|
-
|
41
|
-
|
42
|
-
|
43
|
-
|
44
|
-
|
45
|
-
|
46
|
-
|
39
|
+
<% if failure.nil? %>
|
40
|
+
<td colspan="2"><i>n/a - not using resque-retry failure backend</i></td>
|
41
|
+
<% else %>
|
42
|
+
<td><code><%= failure['exception'] %></code></td>
|
43
|
+
<td class="error">
|
44
|
+
<% if failure['backtrace'] %>
|
45
|
+
<a href="#" class="backtrace"><%= h(failure['error']) %></a>
|
46
|
+
<pre style="display:none"><%= h failure['backtrace'].join("\n") %></pre>
|
47
|
+
<% else %>
|
48
|
+
<%= h failure['error'] %>
|
49
|
+
<% end %>
|
50
|
+
</td>
|
51
|
+
<% end %>
|
47
52
|
<% end %>
|
53
|
+
<td>
|
54
|
+
<form action="<%= url "retry/#{timestamp}/jobs/#{CGI.escape(Resque.encode(job))}/remove" %>" method="post">
|
55
|
+
<input type="hidden" name="timestamp" value="<%= timestamp.to_i %>">
|
56
|
+
<input type="submit" value="Remove">
|
57
|
+
</form>
|
58
|
+
</td>
|
48
59
|
</tr>
|
49
60
|
<% end %>
|
50
61
|
<% if jobs.empty? %>
|
51
62
|
<tr>
|
52
|
-
<td class="no-data" colspan="
|
63
|
+
<td class="no-data" colspan="6">
|
53
64
|
There are no pending jobs scheduled for this time.
|
54
65
|
</td>
|
55
66
|
</tr>
|
@@ -19,27 +19,16 @@ module Resque
|
|
19
19
|
class MultipleWithRetrySuppression < Multiple
|
20
20
|
include Resque::Helpers
|
21
21
|
|
22
|
-
module CleanupHooks
|
23
|
-
# Resque after_perform hook.
|
24
|
-
#
|
25
|
-
# Deletes retry failure information from Redis.
|
26
|
-
def after_perform_retry_failure_cleanup(*args)
|
27
|
-
retry_key = redis_retry_key(*args)
|
28
|
-
failure_key = Resque::Failure::MultipleWithRetrySuppression.failure_key(retry_key)
|
29
|
-
Resque.redis.del(failure_key)
|
30
|
-
end
|
31
|
-
end
|
32
|
-
|
33
22
|
# Called when the job fails.
|
34
23
|
#
|
35
24
|
# If the job will retry, suppress the failure from the other backends.
|
36
25
|
# Store the lastest failure information in redis, used by the web
|
37
26
|
# interface.
|
38
27
|
def save
|
39
|
-
|
28
|
+
if ! (retryable? && retrying?)
|
40
29
|
cleanup_retry_failure_log!
|
41
30
|
super
|
42
|
-
|
31
|
+
elsif retry_delay > 0
|
43
32
|
data = {
|
44
33
|
:failed_at => Time.now.strftime("%Y/%m/%d %H:%M:%S"),
|
45
34
|
:payload => payload,
|
@@ -50,12 +39,7 @@ module Resque
|
|
50
39
|
:queue => queue
|
51
40
|
}
|
52
41
|
|
53
|
-
|
54
|
-
unless klass.respond_to?(:after_perform_retry_failure_cleanup)
|
55
|
-
klass.send(:extend, CleanupHooks)
|
56
|
-
end
|
57
|
-
|
58
|
-
redis[failure_key] = Resque.encode(data)
|
42
|
+
redis.setex(failure_key, 2*retry_delay, Resque.encode(data))
|
59
43
|
end
|
60
44
|
end
|
61
45
|
|
@@ -69,8 +53,12 @@ module Resque
|
|
69
53
|
constantize(payload['class'])
|
70
54
|
end
|
71
55
|
|
56
|
+
def retry_delay
|
57
|
+
klass.retry_delay
|
58
|
+
end
|
59
|
+
|
72
60
|
def retry_key
|
73
|
-
klass.redis_retry_key(payload['args'])
|
61
|
+
klass.redis_retry_key(*payload['args'])
|
74
62
|
end
|
75
63
|
|
76
64
|
def failure_key
|
@@ -79,6 +67,8 @@ module Resque
|
|
79
67
|
|
80
68
|
def retryable?
|
81
69
|
klass.respond_to?(:redis_retry_key)
|
70
|
+
rescue NameError
|
71
|
+
false
|
82
72
|
end
|
83
73
|
|
84
74
|
def retrying?
|
data/lib/resque/plugins/retry.rb
CHANGED
@@ -63,7 +63,9 @@ module Resque
|
|
63
63
|
end
|
64
64
|
|
65
65
|
# Maximum number of retrys we can attempt to successfully perform the job.
|
66
|
-
#
|
66
|
+
#
|
67
|
+
# A retry limit of 0 will *never* retry.
|
68
|
+
# A retry limit of -1 or below will retry forever.
|
67
69
|
#
|
68
70
|
# @return [Fixnum]
|
69
71
|
def retry_limit
|
@@ -88,6 +90,14 @@ module Resque
|
|
88
90
|
@retry_delay ||= 0
|
89
91
|
end
|
90
92
|
|
93
|
+
# @abstract
|
94
|
+
# Number of seconds to sleep after job is requeued
|
95
|
+
#
|
96
|
+
# @return [Number] number of seconds to sleep
|
97
|
+
def sleep_after_requeue
|
98
|
+
@sleep_after_requeue ||= 0
|
99
|
+
end
|
100
|
+
|
91
101
|
# @abstract
|
92
102
|
# Modify the arguments used to retry the job. Use this to do something
|
93
103
|
# other than try the exact same job again.
|
@@ -147,10 +157,13 @@ module Resque
|
|
147
157
|
#
|
148
158
|
# @return [Boolean]
|
149
159
|
def retry_limit_reached?
|
150
|
-
if retry_limit
|
151
|
-
|
160
|
+
if retry_limit == 0
|
161
|
+
true
|
162
|
+
elsif retry_limit > 0
|
163
|
+
true if retry_attempt >= retry_limit
|
164
|
+
else
|
165
|
+
false
|
152
166
|
end
|
153
|
-
false
|
154
167
|
end
|
155
168
|
|
156
169
|
# Register a retry criteria check callback to be run before retrying
|
@@ -177,7 +190,7 @@ module Resque
|
|
177
190
|
retry_criteria_checks << block
|
178
191
|
end
|
179
192
|
|
180
|
-
#
|
193
|
+
# Retries the job.
|
181
194
|
def try_again(*args)
|
182
195
|
if retry_delay <= 0
|
183
196
|
# If the delay is 0, no point passing it through the scheduler
|
@@ -185,6 +198,7 @@ module Resque
|
|
185
198
|
else
|
186
199
|
Resque.enqueue_in(retry_delay, self, *args_for_retry(*args))
|
187
200
|
end
|
201
|
+
sleep(sleep_after_requeue) if sleep_after_requeue > 0
|
188
202
|
end
|
189
203
|
|
190
204
|
# Resque before_perform hook.
|
@@ -1,6 +1,6 @@
|
|
1
1
|
require File.dirname(__FILE__) + '/test_helper'
|
2
2
|
|
3
|
-
class ExponentialBackoffTest <
|
3
|
+
class ExponentialBackoffTest < MiniTest::Unit::TestCase
|
4
4
|
def setup
|
5
5
|
Resque.redis.flushall
|
6
6
|
@worker = Resque::Worker.new(:testing)
|
@@ -8,9 +8,8 @@ class ExponentialBackoffTest < Test::Unit::TestCase
|
|
8
8
|
end
|
9
9
|
|
10
10
|
def test_resque_plugin_lint
|
11
|
-
|
12
|
-
|
13
|
-
end
|
11
|
+
# will raise exception if were not a good plugin.
|
12
|
+
assert Resque::Plugin.lint(Resque::Plugins::ExponentialBackoff)
|
14
13
|
end
|
15
14
|
|
16
15
|
def test_default_backoff_strategy
|
@@ -19,7 +18,7 @@ class ExponentialBackoffTest < Test::Unit::TestCase
|
|
19
18
|
|
20
19
|
perform_next_job @worker
|
21
20
|
assert_equal 1, Resque.info[:processed], '1 processed job'
|
22
|
-
assert_equal 1, Resque.info[:failed], 'first ever run, and it should
|
21
|
+
assert_equal 1, Resque.info[:failed], 'first ever run, and it should have failed, but never retried'
|
23
22
|
assert_equal 1, Resque.info[:pending], '1 pending job, because it never hits the scheduler'
|
24
23
|
|
25
24
|
perform_next_job @worker
|
@@ -1,6 +1,6 @@
|
|
1
1
|
require File.dirname(__FILE__) + '/test_helper'
|
2
2
|
|
3
|
-
class MultipleFailureTest <
|
3
|
+
class MultipleFailureTest < MiniTest::Unit::TestCase
|
4
4
|
def setup
|
5
5
|
Resque.redis.flushall
|
6
6
|
@worker = Resque::Worker.new(:testing)
|
@@ -17,42 +17,35 @@ class MultipleFailureTest < Test::Unit::TestCase
|
|
17
17
|
key = "failure_" + klass.redis_retry_key(args)
|
18
18
|
end
|
19
19
|
|
20
|
-
def
|
21
|
-
Resque.enqueue(
|
20
|
+
def test_last_failure_is_saved_in_redis_if_delay
|
21
|
+
Resque.enqueue(LimitThreeJobDelay1Hour)
|
22
22
|
perform_next_job(@worker)
|
23
23
|
|
24
24
|
# I don't like this, but...
|
25
|
-
key = failure_key_for(
|
25
|
+
key = failure_key_for(LimitThreeJobDelay1Hour)
|
26
26
|
assert Resque.redis.exists(key)
|
27
27
|
end
|
28
28
|
|
29
|
-
def test_last_failure_removed_from_redis_after_error_limit
|
30
|
-
Resque.enqueue(LimitThreeJob)
|
31
|
-
3.times do
|
32
|
-
perform_next_job(@worker)
|
33
|
-
end
|
34
|
-
|
35
|
-
key = failure_key_for(LimitThreeJob)
|
36
|
-
assert Resque.redis.exists(key)
|
37
29
|
|
30
|
+
def test_last_failure_has_double_delay_redis_expiry_if_delay
|
31
|
+
Resque.enqueue(LimitThreeJobDelay1Hour)
|
38
32
|
perform_next_job(@worker)
|
39
|
-
|
33
|
+
|
34
|
+
# I don't like this, but...
|
35
|
+
key = failure_key_for(LimitThreeJobDelay1Hour)
|
36
|
+
assert_equal 7200, Resque.redis.ttl(key)
|
40
37
|
end
|
41
38
|
|
42
|
-
def
|
43
|
-
|
44
|
-
Resque.enqueue(SwitchToSuccessJob)
|
39
|
+
def test_last_failure_is_not_saved_in_redis_if_no_delay
|
40
|
+
Resque.enqueue(LimitThreeJob)
|
45
41
|
perform_next_job(@worker)
|
46
42
|
|
47
|
-
|
48
|
-
|
49
|
-
|
50
|
-
perform_next_job(@worker)
|
51
|
-
assert !Resque.redis.exists(key), 'key removed on success'
|
52
|
-
ensure
|
53
|
-
SwitchToSuccessJob.reset_defaults
|
43
|
+
# I don't like this, but...
|
44
|
+
key = failure_key_for(LimitThreeJob)
|
45
|
+
assert !Resque.redis.exists(key)
|
54
46
|
end
|
55
47
|
|
48
|
+
|
56
49
|
def test_errors_are_suppressed_up_to_retry_limit
|
57
50
|
Resque.enqueue(LimitThreeJob)
|
58
51
|
3.times do
|
@@ -80,6 +73,14 @@ class MultipleFailureTest < Test::Unit::TestCase
|
|
80
73
|
assert_equal 5, MockFailureBackend.errors.size
|
81
74
|
end
|
82
75
|
|
76
|
+
def test_custom_identifier_job
|
77
|
+
Resque.enqueue(CustomIdentifierFailingJob, 'qq', 2)
|
78
|
+
4.times do
|
79
|
+
perform_next_job(@worker)
|
80
|
+
end
|
81
|
+
assert_equal 1, MockFailureBackend.errors.size
|
82
|
+
end
|
83
|
+
|
83
84
|
def teardown
|
84
85
|
Resque::Failure.backend = @old_failure_backend
|
85
86
|
end
|
data/test/redis-test.conf
CHANGED
@@ -112,21 +112,4 @@ databases 16
|
|
112
112
|
# Glue small output buffers together in order to send small replies in a
|
113
113
|
# single TCP packet. Uses a bit more CPU but most of the times it is a win
|
114
114
|
# in terms of number of queries per second. Use 'yes' if unsure.
|
115
|
-
glueoutputbuf yes
|
116
|
-
|
117
|
-
# Use object sharing. Can save a lot of memory if you have many common
|
118
|
-
# string in your dataset, but performs lookups against the shared objects
|
119
|
-
# pool so it uses more CPU and can be a bit slower. Usually it's a good
|
120
|
-
# idea.
|
121
|
-
#
|
122
|
-
# When object sharing is enabled (shareobjects yes) you can use
|
123
|
-
# shareobjectspoolsize to control the size of the pool used in order to try
|
124
|
-
# object sharing. A bigger pool size will lead to better sharing capabilities.
|
125
|
-
# In general you want this value to be at least the double of the number of
|
126
|
-
# very common strings you have in your dataset.
|
127
|
-
#
|
128
|
-
# WARNING: object sharing is experimental, don't enable this feature
|
129
|
-
# in production before of Redis 1.0-stable. Still please try this feature in
|
130
|
-
# your development environment so that we can test it better.
|
131
|
-
# shareobjects no
|
132
|
-
# shareobjectspoolsize 1024
|
115
|
+
glueoutputbuf yes
|
data/test/resque_test.rb
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
require File.dirname(__FILE__) + '/test_helper'
|
2
2
|
|
3
3
|
# make sure the worlds not fallen from beneith us.
|
4
|
-
class ResqueTest <
|
4
|
+
class ResqueTest < MiniTest::Unit::TestCase
|
5
5
|
def test_resque_version
|
6
6
|
major, minor, patch = Resque::Version.split('.')
|
7
7
|
assert_equal 1, major.to_i, 'major version does not match'
|
data/test/retry_criteria_test.rb
CHANGED
data/test/retry_test.rb
CHANGED
@@ -1,6 +1,6 @@
|
|
1
1
|
require File.dirname(__FILE__) + '/test_helper'
|
2
2
|
|
3
|
-
class RetryTest <
|
3
|
+
class RetryTest < MiniTest::Unit::TestCase
|
4
4
|
def setup
|
5
5
|
Resque.redis.flushall
|
6
6
|
@worker = Resque::Worker.new(:testing)
|
@@ -8,16 +8,15 @@ class RetryTest < Test::Unit::TestCase
|
|
8
8
|
end
|
9
9
|
|
10
10
|
def test_resque_plugin_lint
|
11
|
-
|
12
|
-
|
13
|
-
end
|
11
|
+
# will raise exception if were not a good plugin.
|
12
|
+
assert Resque::Plugin.lint(Resque::Plugins::Retry)
|
14
13
|
end
|
15
14
|
|
16
15
|
def test_default_settings
|
17
|
-
assert_equal 1,
|
18
|
-
assert_equal 0,
|
19
|
-
assert_equal nil,
|
20
|
-
assert_equal 0,
|
16
|
+
assert_equal 1, RetryDefaultSettingsJob.retry_limit, 'default retry limit'
|
17
|
+
assert_equal 0, RetryDefaultSettingsJob.retry_attempt, 'default number of retry attempts'
|
18
|
+
assert_equal nil, RetryDefaultSettingsJob.retry_exceptions, 'default retry exceptions; nil = any'
|
19
|
+
assert_equal 0, RetryDefaultSettingsJob.retry_delay, 'default seconds until retry'
|
21
20
|
end
|
22
21
|
|
23
22
|
def test_retry_once_by_default
|
@@ -71,6 +70,17 @@ class RetryTest < Test::Unit::TestCase
|
|
71
70
|
assert_equal 10, Resque.info[:processed], 'processed job'
|
72
71
|
end
|
73
72
|
|
73
|
+
def test_retry_never_retry
|
74
|
+
Resque.enqueue(NeverRetryJob)
|
75
|
+
10.times do
|
76
|
+
perform_next_job(@worker)
|
77
|
+
end
|
78
|
+
|
79
|
+
assert_equal 0, Resque.info[:pending], 'pending jobs'
|
80
|
+
assert_equal 1, Resque.info[:failed], 'failed jobs'
|
81
|
+
assert_equal 1, Resque.info[:processed], 'processed job'
|
82
|
+
end
|
83
|
+
|
74
84
|
def test_fail_five_times_then_succeed
|
75
85
|
Resque.enqueue(FailFiveTimesJob)
|
76
86
|
7.times do
|
@@ -82,6 +92,22 @@ class RetryTest < Test::Unit::TestCase
|
|
82
92
|
assert_equal 0, Resque.info[:pending], 'pending jobs'
|
83
93
|
end
|
84
94
|
|
95
|
+
def test_retry_delay_sleep
|
96
|
+
assert_equal 0, Resque.info[:failed], 'failed jobs'
|
97
|
+
Resque.enqueue(SleepDelay1SecondJob)
|
98
|
+
before = Time.now
|
99
|
+
3.times do
|
100
|
+
perform_next_job(@worker)
|
101
|
+
end
|
102
|
+
actual_delay = Time.now - before
|
103
|
+
|
104
|
+
assert actual_delay >= 1, "did not sleep long enough: #{actual_delay} seconds"
|
105
|
+
assert actual_delay < 2, "slept too long: #{actual_delay} seconds"
|
106
|
+
assert_equal 1, Resque.info[:failed], 'failed jobs'
|
107
|
+
assert_equal 2, Resque.info[:processed], 'processed job'
|
108
|
+
assert_equal 0, Resque.info[:pending], 'pending jobs'
|
109
|
+
end
|
110
|
+
|
85
111
|
def test_can_determine_if_exception_may_be_retried
|
86
112
|
assert_equal true, RetryDefaultsJob.retry_exception?(StandardError), 'StandardError may retry'
|
87
113
|
assert_equal true, RetryDefaultsJob.retry_exception?(CustomException), 'CustomException may retry'
|
data/test/server_test.rb
ADDED
@@ -0,0 +1,25 @@
|
|
1
|
+
require File.dirname(__FILE__) + '/test_helper'
|
2
|
+
|
3
|
+
require 'resque-retry/server'
|
4
|
+
ENV['RACK_ENV'] = 'test'
|
5
|
+
|
6
|
+
# Testing the Resque web interface additions.
|
7
|
+
class ServerTest < MiniTest::Unit::TestCase
|
8
|
+
include Rack::Test::Methods
|
9
|
+
|
10
|
+
def app
|
11
|
+
Resque::Server
|
12
|
+
end
|
13
|
+
|
14
|
+
# make sure the world looks okay first =)
|
15
|
+
def test_were_able_to_access_normal_resque_web_overview
|
16
|
+
get '/overview'
|
17
|
+
assert_equal 200, last_response.status, 'HTTP status code should be 200'
|
18
|
+
end
|
19
|
+
|
20
|
+
def test_should_include_retry_tab
|
21
|
+
get '/overview'
|
22
|
+
assert last_response.body.include?('/retry')
|
23
|
+
end
|
24
|
+
|
25
|
+
end
|
data/test/test_helper.rb
CHANGED
@@ -2,10 +2,10 @@ dir = File.dirname(File.expand_path(__FILE__))
|
|
2
2
|
$LOAD_PATH.unshift dir + '/../lib'
|
3
3
|
$TESTING = true
|
4
4
|
|
5
|
-
require '
|
6
|
-
require '
|
7
|
-
require '
|
8
|
-
require 'simplecov
|
5
|
+
require 'minitest/unit'
|
6
|
+
require 'minitest/pride'
|
7
|
+
require 'rack/test'
|
8
|
+
require 'simplecov'
|
9
9
|
|
10
10
|
SimpleCov.start do
|
11
11
|
add_filter "/test/"
|
@@ -59,7 +59,7 @@ class MockFailureBackend < Resque::Failure::Base
|
|
59
59
|
end
|
60
60
|
|
61
61
|
# Test helpers
|
62
|
-
class
|
62
|
+
class MiniTest::Unit::TestCase
|
63
63
|
def perform_next_job(worker, &block)
|
64
64
|
return unless job = @worker.reserve
|
65
65
|
@worker.perform(job, &block)
|
data/test/test_jobs.rb
CHANGED
@@ -13,6 +13,15 @@ end
|
|
13
13
|
class GoodJob
|
14
14
|
extend Resque::Plugins::Retry
|
15
15
|
@queue = :testing
|
16
|
+
|
17
|
+
def self.perform(*args)
|
18
|
+
end
|
19
|
+
end
|
20
|
+
|
21
|
+
class RetryDefaultSettingsJob
|
22
|
+
extend Resque::Plugins::Retry
|
23
|
+
@queue = :testing
|
24
|
+
|
16
25
|
def self.perform(*args)
|
17
26
|
end
|
18
27
|
end
|
@@ -26,6 +35,15 @@ class RetryDefaultsJob
|
|
26
35
|
end
|
27
36
|
end
|
28
37
|
|
38
|
+
class SleepDelay1SecondJob < RetryDefaultsJob
|
39
|
+
@queue = :testing
|
40
|
+
@sleep_after_requeue = 1
|
41
|
+
|
42
|
+
def self.perform(*args)
|
43
|
+
raise if retry_attempt == 0
|
44
|
+
end
|
45
|
+
end
|
46
|
+
|
29
47
|
class InheritTestJob < RetryDefaultsJob
|
30
48
|
end
|
31
49
|
|
@@ -54,6 +72,11 @@ class RetryWithModifiedArgsJob < RetryDefaultsJob
|
|
54
72
|
end
|
55
73
|
|
56
74
|
class NeverGiveUpJob < RetryDefaultsJob
|
75
|
+
@queue = :testing
|
76
|
+
@retry_limit = -1
|
77
|
+
end
|
78
|
+
|
79
|
+
class NeverRetryJob < RetryDefaultsJob
|
57
80
|
@queue = :testing
|
58
81
|
@retry_limit = 0
|
59
82
|
end
|
@@ -67,6 +90,12 @@ class LimitThreeJob < RetryDefaultsJob
|
|
67
90
|
end
|
68
91
|
end
|
69
92
|
|
93
|
+
class LimitThreeJobDelay1Hour < LimitThreeJob
|
94
|
+
@queue = :testing
|
95
|
+
@retry_limit = 3
|
96
|
+
@retry_delay = 3600
|
97
|
+
end
|
98
|
+
|
70
99
|
class FailFiveTimesJob < RetryDefaultsJob
|
71
100
|
@queue = :testing
|
72
101
|
@retry_limit = 6
|
@@ -204,32 +233,6 @@ class InheritOrderingJobExtendFirst
|
|
204
233
|
end
|
205
234
|
end
|
206
235
|
|
207
|
-
# This job switches to successful after
|
208
|
-
# n +tries+.
|
209
|
-
class SwitchToSuccessJob < GoodJob
|
210
|
-
@queue = :testing
|
211
|
-
@max_retries = 3
|
212
|
-
|
213
|
-
class << self
|
214
|
-
attr_accessor :successful_after
|
215
|
-
attr_accessor :tries
|
216
|
-
|
217
|
-
def reset_defaults
|
218
|
-
self.tries = 0
|
219
|
-
self.successful_after = 2
|
220
|
-
end
|
221
|
-
end
|
222
|
-
|
223
|
-
reset_defaults
|
224
|
-
|
225
|
-
def self.perform(*args)
|
226
|
-
if self.tries < self.successful_after
|
227
|
-
self.tries += 1
|
228
|
-
raise "error"
|
229
|
-
end
|
230
|
-
end
|
231
|
-
end
|
232
|
-
|
233
236
|
class InheritOrderingJobExtendLast
|
234
237
|
class << self
|
235
238
|
attr_accessor :test_value
|
@@ -247,5 +250,22 @@ class InheritOrderingJobExtendLast
|
|
247
250
|
end
|
248
251
|
end
|
249
252
|
|
253
|
+
|
250
254
|
class InheritOrderingJobExtendFirstSubclass < InheritOrderingJobExtendFirst; end
|
251
255
|
class InheritOrderingJobExtendLastSubclass < InheritOrderingJobExtendLast; end
|
256
|
+
|
257
|
+
class CustomIdentifierFailingJob
|
258
|
+
extend Resque::Plugins::Retry
|
259
|
+
|
260
|
+
@queue = :testing
|
261
|
+
@retry_limit = 2
|
262
|
+
@retry_delay = 0
|
263
|
+
|
264
|
+
def self.identifier(*args)
|
265
|
+
args.first.to_s
|
266
|
+
end
|
267
|
+
|
268
|
+
def self.perform(*args)
|
269
|
+
raise 'failed'
|
270
|
+
end
|
271
|
+
end
|
metadata
CHANGED
@@ -1,123 +1,115 @@
|
|
1
|
-
--- !ruby/object:Gem::Specification
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
2
|
name: resque-retry
|
3
|
-
version: !ruby/object:Gem::Version
|
4
|
-
|
5
|
-
prerelease:
|
6
|
-
segments:
|
7
|
-
- 0
|
8
|
-
- 1
|
9
|
-
- 0
|
10
|
-
version: 0.1.0
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 0.2.1
|
5
|
+
prerelease:
|
11
6
|
platform: ruby
|
12
|
-
authors:
|
7
|
+
authors:
|
13
8
|
- Luke Antins
|
14
9
|
- Ryan Carver
|
15
10
|
autorequire:
|
16
11
|
bindir: bin
|
17
12
|
cert_chain: []
|
18
|
-
|
19
|
-
|
20
|
-
|
21
|
-
|
22
|
-
|
23
|
-
|
13
|
+
date: 2011-11-23 00:00:00.000000000Z
|
14
|
+
dependencies:
|
15
|
+
- !ruby/object:Gem::Dependency
|
16
|
+
name: rake
|
17
|
+
requirement: &2165851780 !ruby/object:Gem::Requirement
|
18
|
+
none: false
|
19
|
+
requirements:
|
20
|
+
- - ! '>='
|
21
|
+
- !ruby/object:Gem::Version
|
22
|
+
version: '0'
|
23
|
+
type: :runtime
|
24
24
|
prerelease: false
|
25
|
-
|
25
|
+
version_requirements: *2165851780
|
26
|
+
- !ruby/object:Gem::Dependency
|
27
|
+
name: resque
|
28
|
+
requirement: &2165851020 !ruby/object:Gem::Requirement
|
26
29
|
none: false
|
27
|
-
requirements:
|
28
|
-
- -
|
29
|
-
- !ruby/object:Gem::Version
|
30
|
-
hash: 55
|
31
|
-
segments:
|
32
|
-
- 1
|
33
|
-
- 8
|
34
|
-
- 0
|
30
|
+
requirements:
|
31
|
+
- - ! '>='
|
32
|
+
- !ruby/object:Gem::Version
|
35
33
|
version: 1.8.0
|
36
34
|
type: :runtime
|
37
|
-
version_requirements: *id001
|
38
|
-
- !ruby/object:Gem::Dependency
|
39
|
-
name: resque-scheduler
|
40
35
|
prerelease: false
|
41
|
-
|
36
|
+
version_requirements: *2165851020
|
37
|
+
- !ruby/object:Gem::Dependency
|
38
|
+
name: resque-scheduler
|
39
|
+
requirement: &2165835400 !ruby/object:Gem::Requirement
|
42
40
|
none: false
|
43
|
-
requirements:
|
44
|
-
- -
|
45
|
-
- !ruby/object:Gem::Version
|
46
|
-
hash: 55
|
47
|
-
segments:
|
48
|
-
- 1
|
49
|
-
- 8
|
50
|
-
- 0
|
41
|
+
requirements:
|
42
|
+
- - ! '>='
|
43
|
+
- !ruby/object:Gem::Version
|
51
44
|
version: 1.8.0
|
52
45
|
type: :runtime
|
53
|
-
version_requirements: *id002
|
54
|
-
- !ruby/object:Gem::Dependency
|
55
|
-
name: test-unit
|
56
46
|
prerelease: false
|
57
|
-
|
47
|
+
version_requirements: *2165835400
|
48
|
+
- !ruby/object:Gem::Dependency
|
49
|
+
name: minitest
|
50
|
+
requirement: &2165834800 !ruby/object:Gem::Requirement
|
58
51
|
none: false
|
59
|
-
requirements:
|
60
|
-
- -
|
61
|
-
- !ruby/object:Gem::Version
|
62
|
-
|
63
|
-
segments:
|
64
|
-
- 0
|
65
|
-
version: "0"
|
52
|
+
requirements:
|
53
|
+
- - ! '>='
|
54
|
+
- !ruby/object:Gem::Version
|
55
|
+
version: '0'
|
66
56
|
type: :development
|
67
|
-
version_requirements: *id003
|
68
|
-
- !ruby/object:Gem::Dependency
|
69
|
-
name: turn
|
70
57
|
prerelease: false
|
71
|
-
|
58
|
+
version_requirements: *2165834800
|
59
|
+
- !ruby/object:Gem::Dependency
|
60
|
+
name: rack-test
|
61
|
+
requirement: &2165834000 !ruby/object:Gem::Requirement
|
72
62
|
none: false
|
73
|
-
requirements:
|
74
|
-
- -
|
75
|
-
- !ruby/object:Gem::Version
|
76
|
-
|
77
|
-
segments:
|
78
|
-
- 0
|
79
|
-
version: "0"
|
63
|
+
requirements:
|
64
|
+
- - ! '>='
|
65
|
+
- !ruby/object:Gem::Version
|
66
|
+
version: '0'
|
80
67
|
type: :development
|
81
|
-
|
82
|
-
|
68
|
+
prerelease: false
|
69
|
+
version_requirements: *2165834000
|
70
|
+
- !ruby/object:Gem::Dependency
|
83
71
|
name: yard
|
72
|
+
requirement: &2165833040 !ruby/object:Gem::Requirement
|
73
|
+
none: false
|
74
|
+
requirements:
|
75
|
+
- - ! '>='
|
76
|
+
- !ruby/object:Gem::Version
|
77
|
+
version: '0'
|
78
|
+
type: :development
|
84
79
|
prerelease: false
|
85
|
-
|
80
|
+
version_requirements: *2165833040
|
81
|
+
- !ruby/object:Gem::Dependency
|
82
|
+
name: json
|
83
|
+
requirement: &2165832180 !ruby/object:Gem::Requirement
|
86
84
|
none: false
|
87
|
-
requirements:
|
88
|
-
- -
|
89
|
-
- !ruby/object:Gem::Version
|
90
|
-
|
91
|
-
segments:
|
92
|
-
- 0
|
93
|
-
version: "0"
|
85
|
+
requirements:
|
86
|
+
- - ! '>='
|
87
|
+
- !ruby/object:Gem::Version
|
88
|
+
version: '0'
|
94
89
|
type: :development
|
95
|
-
version_requirements: *id005
|
96
|
-
- !ruby/object:Gem::Dependency
|
97
|
-
name: simplecov-html
|
98
90
|
prerelease: false
|
99
|
-
|
91
|
+
version_requirements: *2165832180
|
92
|
+
- !ruby/object:Gem::Dependency
|
93
|
+
name: simplecov
|
94
|
+
requirement: &2165831320 !ruby/object:Gem::Requirement
|
100
95
|
none: false
|
101
|
-
requirements:
|
102
|
-
- -
|
103
|
-
- !ruby/object:Gem::Version
|
104
|
-
hash: 19
|
105
|
-
segments:
|
106
|
-
- 0
|
107
|
-
- 3
|
108
|
-
- 0
|
96
|
+
requirements:
|
97
|
+
- - ! '>='
|
98
|
+
- !ruby/object:Gem::Version
|
109
99
|
version: 0.3.0
|
110
100
|
type: :development
|
111
|
-
|
112
|
-
|
101
|
+
prerelease: false
|
102
|
+
version_requirements: *2165831320
|
103
|
+
description: ! " resque-retry provides retry, delay and exponential backoff support
|
104
|
+
for\n resque jobs.\n\n Features:\n\n * Redis backed retry count/limit.\n * Retry
|
105
|
+
on all or specific exceptions.\n * Exponential backoff (varying the delay between
|
106
|
+
retrys).\n * Multiple failure backend with retry suppression & resque-web tab.\n
|
107
|
+
\ * Small & Extendable - plenty of places to override retry logic/settings.\n"
|
113
108
|
email: luke@lividpenguin.com
|
114
109
|
executables: []
|
115
|
-
|
116
110
|
extensions: []
|
117
|
-
|
118
111
|
extra_rdoc_files: []
|
119
|
-
|
120
|
-
files:
|
112
|
+
files:
|
121
113
|
- LICENSE
|
122
114
|
- Rakefile
|
123
115
|
- README.md
|
@@ -129,6 +121,7 @@ files:
|
|
129
121
|
- test/retry_criteria_test.rb
|
130
122
|
- test/retry_inheriting_checks_test.rb
|
131
123
|
- test/retry_test.rb
|
124
|
+
- test/server_test.rb
|
132
125
|
- test/test_helper.rb
|
133
126
|
- test/test_jobs.rb
|
134
127
|
- lib/resque/failure/multiple_with_retry_suppression.rb
|
@@ -138,39 +131,30 @@ files:
|
|
138
131
|
- lib/resque-retry/server/views/retry_timestamp.erb
|
139
132
|
- lib/resque-retry/server.rb
|
140
133
|
- lib/resque-retry.rb
|
141
|
-
has_rdoc: false
|
142
134
|
homepage: http://github.com/lantins/resque-retry
|
143
135
|
licenses: []
|
144
|
-
|
145
136
|
post_install_message:
|
146
137
|
rdoc_options: []
|
147
|
-
|
148
|
-
require_paths:
|
138
|
+
require_paths:
|
149
139
|
- lib
|
150
|
-
required_ruby_version: !ruby/object:Gem::Requirement
|
140
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
151
141
|
none: false
|
152
|
-
requirements:
|
153
|
-
- -
|
154
|
-
- !ruby/object:Gem::Version
|
155
|
-
|
156
|
-
|
157
|
-
- 0
|
158
|
-
version: "0"
|
159
|
-
required_rubygems_version: !ruby/object:Gem::Requirement
|
142
|
+
requirements:
|
143
|
+
- - ! '>='
|
144
|
+
- !ruby/object:Gem::Version
|
145
|
+
version: '0'
|
146
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
160
147
|
none: false
|
161
|
-
requirements:
|
162
|
-
- -
|
163
|
-
- !ruby/object:Gem::Version
|
164
|
-
|
165
|
-
segments:
|
166
|
-
- 0
|
167
|
-
version: "0"
|
148
|
+
requirements:
|
149
|
+
- - ! '>='
|
150
|
+
- !ruby/object:Gem::Version
|
151
|
+
version: '0'
|
168
152
|
requirements: []
|
169
|
-
|
170
153
|
rubyforge_project:
|
171
|
-
rubygems_version: 1.
|
154
|
+
rubygems_version: 1.8.10
|
172
155
|
signing_key:
|
173
156
|
specification_version: 3
|
174
|
-
summary: A resque plugin; provides retry, delay and exponential backoff support for
|
157
|
+
summary: A resque plugin; provides retry, delay and exponential backoff support for
|
158
|
+
resque jobs.
|
175
159
|
test_files: []
|
176
|
-
|
160
|
+
has_rdoc: false
|