resque-retry 1.1.1 → 1.1.4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: eaaf3cd5c50ad76cf580f632bd6597db0531ffa5
4
- data.tar.gz: c621c1f3446b60d415ba4d3d696d945b5c824837
3
+ metadata.gz: b1b6a97e99754f141a99af11bd0d77f5f73d8010
4
+ data.tar.gz: 2c6e42f2294034e932b4a12998770214a0eed1ea
5
5
  SHA512:
6
- metadata.gz: 594beb0db07135808c5e761fa1519c3d3c101492cf625f955dcbe8d68749c747458ce02cf782f7383292de86f2e9368a131503aa936692eb4c5e629947fc01de
7
- data.tar.gz: cbdcf78bab11246287808884994446042866c2a39c8c83595258479db7db53b3565e987c1ac919565e179d768f4543dcf4d8417695a7b876023242a86673ed31
6
+ metadata.gz: d748027d00df3b099e92cac7123baedccca96f0240a5f17c27f15277a57c0c527a3207e277950ce42b57a3f440791cf75e2245d4a920db35f897d126f932a895
7
+ data.tar.gz: 56ce881aff8343f9d381861aa38a061a6bf80eaf3e099f90d6a41ca6a3ca3fef52cb30cb12a3014809400d6c52b5f1f1c0103fd8ee6454878b0f512e0f8c2daf
data/HISTORY.md CHANGED
@@ -1,5 +1,11 @@
1
1
  ## HEAD
2
2
 
3
+ ## 1.1.4 (2014-03-17)
4
+
5
+ * Fixed displaying retry information in resque web interface, caused by `Resque::Helpers` being deprecated.
6
+ * Feature: Allow `@fatal_exceptions` as inverse of `@retry_exceptions`, when fatal exception is raised the job will be immediately fail.
7
+ * Feature: Allow a random retry delay (within a range) when using exponential backoff strategy.
8
+
3
9
  ## 1.1.1 (2014-03-12)
4
10
 
5
11
  * Adjust gem dependency `resque-scheduler`.
data/README.md CHANGED
@@ -208,6 +208,8 @@ Use this if you wish to vary the delay between retry attempts:
208
208
 
209
209
  no delay, 1m, 10m, 1h, 3h, 6h
210
210
  @backoff_strategy = [0, 60, 600, 3600, 10800, 21600]
211
+ @retry_delay_multiplicand_min = 1.0
212
+ @retry_delay_multiplicand_max = 1.0
211
213
 
212
214
  The first delay will be 0 seconds, the 2nd will be 60 seconds, etc...
213
215
  Again, tweak to your own needs.
@@ -215,6 +217,13 @@ Again, tweak to your own needs.
215
217
  The number of retries is equal to the size of the `backoff_strategy`
216
218
  array, unless you set `retry_limit` yourself.
217
219
 
220
+ The delay values will be multiplied by a random `Float` value between
221
+ `retry_delay_multiplicand_min` and `retry_delay_multiplicand_max` (both have a
222
+ default of `1.0`). The product (`delay_multiplicand`) is recalculated on every
223
+ attempt. This feature can be useful if you have a lot of jobs fail at the same
224
+ time (e.g. rate-limiting/throttling or connectivity issues) and you don't want
225
+ them all retried on the same schedule.
226
+
218
227
  ### Retry Specific Exceptions
219
228
 
220
229
  The default will allow a retry for any type of exception. You may change
@@ -256,6 +265,26 @@ In the above example, Resque would retry any `DeliverSMS` jobs which throw a
256
265
  will be retried 30 seconds later, if it throws `SystemCallError` it will first
257
266
  retry 120 seconds later then subsequent retry attempts 240 seconds later.
258
267
 
268
+ ### Fail Fast For Specific Exceptions
269
+
270
+ The default will allow a retry for any type of exception. You may change
271
+ it so specific exceptions fail immediately by using `fatal_exceptions`:
272
+
273
+ class DeliverSMS
274
+ extend Resque::Plugins::Retry
275
+ @queue = :mt_divisions
276
+
277
+ @fatal_exceptions = [NetworkError]
278
+
279
+ def self.perform(mt_id, mobile_number, message)
280
+ heavy_lifting
281
+ end
282
+ end
283
+
284
+ In the above example, Resque would retry any `DeliverSMS` jobs that throw any
285
+ type of error other than `NetworkError`. If the job throws a `NetworkError` it
286
+ will be marked as "failed" immediately.
287
+
259
288
  ### Custom Retry Criteria Check Callbacks
260
289
 
261
290
  You may define custom retry criteria callbacks:
@@ -1,3 +1,4 @@
1
1
  web: bundle exec rake start
2
2
  worker: bundle exec rake resque:work QUEUE=*
3
- scheduler: bundle exec rake resque:scheduler
3
+ scheduler: bundle exec rake resque:scheduler
4
+ redis: redis-server
@@ -21,7 +21,7 @@ class FailingWithRetryJob
21
21
 
22
22
  @queue = :testing_failure
23
23
  @retry_limit = 4
24
- @retry_delay = 3
24
+ @retry_delay = 20
25
25
 
26
26
  # Perform that raises an exception, but we will retry the job on failure
27
27
  def self.perform(*args)
@@ -39,14 +39,10 @@ module ResqueRetry
39
39
  module Helpers
40
40
  # builds a retry key for the specified job.
41
41
  def retry_key_for_job(job)
42
- begin
43
- klass = Resque.constantize(job['class'])
44
- if klass.respond_to?(:redis_retry_key)
45
- klass.redis_retry_key(job['args'])
46
- else
47
- nil
48
- end
49
- rescue NameError
42
+ klass = Resque::Job.new(nil, nil).constantize(job['class'])
43
+ if klass.respond_to?(:redis_retry_key)
44
+ klass.redis_retry_key(job['args'])
45
+ else
50
46
  nil
51
47
  end
52
48
  end
@@ -1,3 +1,3 @@
1
1
  module ResqueRetry
2
- VERSION = '1.1.1'
2
+ VERSION = '1.1.4'
3
3
  end
@@ -54,7 +54,7 @@ module Resque
54
54
 
55
55
  # Return the class/module of the failed job.
56
56
  def klass
57
- Resque::Job.new(nil, nil).constantize payload['class']
57
+ Resque::Job.new(nil, nil).constantize(payload['class'])
58
58
  end
59
59
 
60
60
  def retry_delay
@@ -37,6 +37,34 @@ module Resque
37
37
  module ExponentialBackoff
38
38
  include Resque::Plugins::Retry
39
39
 
40
+ # Raised if the min/max retry-delay multiplicand configuration is invalid
41
+ #
42
+ # @api public
43
+ class InvalidRetryDelayMultiplicandConfigurationException < StandardError; end
44
+
45
+ # Constants
46
+ #
47
+ # @api public
48
+ DEFAULT_RETRY_DELAY_MULTIPLICAND_MIN = 1.0
49
+ DEFAULT_RETRY_DELAY_MULTIPLICAND_MAX = 1.0
50
+
51
+ # Fail fast, when extended, if the "receiver" is misconfigured
52
+ #
53
+ # @api private
54
+ def self.extended(receiver)
55
+ retry_delay_multiplicand_min = \
56
+ receiver.instance_variable_get("@retry_delay_multiplicand_min") || \
57
+ DEFAULT_RETRY_DELAY_MULTIPLICAND_MIN
58
+ retry_delay_multiplicand_max = \
59
+ receiver.instance_variable_get("@retry_delay_multiplicand_max") || \
60
+ DEFAULT_RETRY_DELAY_MULTIPLICAND_MAX
61
+ if retry_delay_multiplicand_min > retry_delay_multiplicand_max
62
+ raise InvalidRetryDelayMultiplicandConfigurationException.new(
63
+ %{"@retry_delay_multiplicand_min" must be less than or equal to "@retry_delay_multiplicand_max"}
64
+ )
65
+ end
66
+ end
67
+
40
68
  # Defaults to the number of delays in the backoff strategy
41
69
  #
42
70
  # @return [Number] maximum number of retries
@@ -52,7 +80,32 @@ module Resque
52
80
  #
53
81
  # @api private
54
82
  def retry_delay
55
- backoff_strategy[retry_attempt] || backoff_strategy.last
83
+ delay = backoff_strategy[retry_attempt] || backoff_strategy.last
84
+ delay_multiplicand = \
85
+ rand(retry_delay_multiplicand_min..retry_delay_multiplicand_max)
86
+ (delay * delay_multiplicand).to_i
87
+ end
88
+
89
+ # @abstract
90
+ # The minimum value (lower-bound) for the range that is is used in
91
+ # calculating the retry-delay product
92
+ #
93
+ # @return [Float]
94
+ #
95
+ # @api public
96
+ def retry_delay_multiplicand_min
97
+ @retry_delay_multiplicand_min ||= DEFAULT_RETRY_DELAY_MULTIPLICAND_MIN
98
+ end
99
+
100
+ # @abstract
101
+ # The maximum value (upper-bound) for the range that is is used in
102
+ # calculating the retry-delay product
103
+ #
104
+ # @return [Float]
105
+ #
106
+ # @api public
107
+ def retry_delay_multiplicand_max
108
+ @retry_delay_multiplicand_max ||= DEFAULT_RETRY_DELAY_MULTIPLICAND_MAX
56
109
  end
57
110
 
58
111
  # @abstract
@@ -67,4 +120,4 @@ module Resque
67
120
  end
68
121
 
69
122
  end
70
- end
123
+ end
@@ -36,6 +36,20 @@ module Resque
36
36
  #
37
37
  module Retry
38
38
 
39
+ # Raised if the retry-strategy cannot be determined or has conflicts
40
+ #
41
+ # @api public
42
+ class AmbiguousRetryStrategyException < StandardError; end
43
+
44
+ # Fail fast, when extended, if the "receiver" is misconfigured
45
+ #
46
+ # @api private
47
+ def self.extended(receiver)
48
+ if receiver.instance_variable_get("@fatal_exceptions") && receiver.instance_variable_get("@retry_exceptions")
49
+ raise AmbiguousRetryStrategyException.new(%{You can't define both "@fatal_exceptions" and "@retry_exceptions"})
50
+ end
51
+ end
52
+
39
53
  # Copy retry criteria checks on inheritance.
40
54
  #
41
55
  # @api private
@@ -152,16 +166,45 @@ module Resque
152
166
  #
153
167
  # @api public
154
168
  def retry_exception?(exception)
155
- return true if retry_exceptions.nil?
156
- !! retry_exceptions.any? do |ex|
157
- if exception.is_a?(Class)
158
- ex >= exception
159
- else
160
- ex === exception
169
+ # If both "fatal_exceptions" and "retry_exceptions" are undefined we are
170
+ # done (we should retry the exception)
171
+ #
172
+ # It is intentional that we check "retry_exceptions" first since it is
173
+ # more likely that it will be defined (over "fatal_exceptions") as it
174
+ # has been part of the API for quite a while
175
+ return true if retry_exceptions.nil? && fatal_exceptions.nil?
176
+
177
+ # If "fatal_exceptions" is undefined interrogate "retry_exceptions"
178
+ if fatal_exceptions.nil?
179
+ retry_exceptions.any? do |ex|
180
+ if exception.is_a?(Class)
181
+ ex >= exception
182
+ else
183
+ ex === exception
184
+ end
185
+ end
186
+ # It is safe to assume we need to check "fatal_exceptions" at this point
187
+ else
188
+ fatal_exceptions.none? do |ex|
189
+ if exception.is_a?(Class)
190
+ ex >= exception
191
+ else
192
+ ex === exception
193
+ end
161
194
  end
162
195
  end
163
196
  end
164
197
 
198
+ # @abstract
199
+ # Controls what exceptions may not be retried
200
+ #
201
+ # Default: `nil` - this will retry all exceptions.
202
+ #
203
+ # @return [Array, nil]
204
+ #
205
+ # @api public
206
+ attr_reader :fatal_exceptions
207
+
165
208
  # @abstract
166
209
  # Controls what exceptions may be retried
167
210
  #
@@ -41,6 +41,75 @@ class ExponentialBackoffTest < MiniTest::Unit::TestCase
41
41
  assert_in_delta (start_time + 21_600), delayed[4], 1.00, '6th delay'
42
42
  end
43
43
 
44
+ def test_dont_allow_both_retry_and_ignore_exceptions
45
+ job_types = [
46
+ InvalidRetryDelayMaxConfigurationJob,
47
+ InvalidRetryDelayMinAndMaxConfigurationJob,
48
+ InvalidRetryDelayMinConfigurationJob,
49
+ ]
50
+
51
+ job_types.each do |job_type|
52
+ assert_raises Resque::Plugins::ExponentialBackoff::InvalidRetryDelayMultiplicandConfigurationException do
53
+ job_type.extend(Resque::Plugins::ExponentialBackoff)
54
+ end
55
+ end
56
+ end
57
+
58
+ def test_default_backoff_strategy_with_retry_delay_multiplicands
59
+ job_types = [
60
+ ExponentialBackoffWithRetryDelayMultiplicandMaxJob,
61
+ ExponentialBackoffWithRetryDelayMultiplicandMinJob,
62
+ ExponentialBackoffWithRetryDelayMultiplicandMinAndMaxJob,
63
+ ]
64
+
65
+ job_types.each do |job_type|
66
+ # all of these values are used heavily in assertions below
67
+ start_time = Time.now.to_i
68
+ multiplicand_min = job_type.public_send(:retry_delay_multiplicand_min)
69
+ multiplicand_max = job_type.public_send(:retry_delay_multiplicand_max)
70
+
71
+ Resque.enqueue(job_type)
72
+
73
+ # first attempt, failed but never retried
74
+ perform_next_job(@worker)
75
+ assert_equal 1, Resque.info[:pending]
76
+ assert_equal 1, Resque.info[:processed]
77
+ assert_equal 1, Resque.info[:failed]
78
+
79
+ # second attempt, first retry, should fail again
80
+ perform_next_job(@worker)
81
+ assert_equal 0, Resque.info[:pending]
82
+ assert_equal 2, Resque.info[:processed]
83
+ assert_equal 2, Resque.info[:failed]
84
+
85
+ # second delay
86
+ delayed = Resque.delayed_queue_peek(0, 1)
87
+ assert_in_delta(
88
+ start_time + 60 * multiplicand_min,
89
+ delayed[0],
90
+ 60 * multiplicand_max
91
+ )
92
+
93
+ 5.times do
94
+ Resque.enqueue(job_type)
95
+ perform_next_job(@worker)
96
+ end
97
+
98
+ # third through sixth delays
99
+ delayed = Resque.delayed_queue_peek(1, 5)
100
+ [600, 3600, 10_800, 21_600].each_with_index do |delay, index|
101
+ assert_in_delta(
102
+ start_time + delay * multiplicand_min,
103
+ delayed[index],
104
+ delay * multiplicand_max
105
+ )
106
+ end
107
+
108
+ # always reset the state before the next test case is run
109
+ Resque.redis.flushall
110
+ end
111
+ end
112
+
44
113
  def test_custom_backoff_strategy
45
114
  start_time = Time.now.to_i
46
115
  4.times do
@@ -17,6 +17,7 @@ class RetryTest < MiniTest::Unit::TestCase
17
17
  def test_default_settings
18
18
  assert_equal 1, RetryDefaultSettingsJob.retry_limit, 'default retry limit'
19
19
  assert_equal 0, RetryDefaultSettingsJob.retry_attempt, 'default number of retry attempts'
20
+ assert_equal nil, RetryDefaultSettingsJob.fatal_exceptions, 'default fatal exceptions; nil = none'
20
21
  assert_equal nil, RetryDefaultSettingsJob.retry_exceptions, 'default retry exceptions; nil = any'
21
22
  assert_equal 0, RetryDefaultSettingsJob.retry_delay, 'default seconds until retry'
22
23
  end
@@ -159,6 +160,36 @@ class RetryTest < MiniTest::Unit::TestCase
159
160
  assert_equal 0, Resque.info[:pending], 'pending jobs'
160
161
  end
161
162
 
163
+ def test_dont_allow_both_retry_and_ignore_exceptions
164
+ assert_raises Resque::Plugins::Retry::AmbiguousRetryStrategyException do
165
+ AmbiguousRetryStrategyJob.extend(Resque::Plugins::Retry)
166
+ end
167
+ end
168
+
169
+ def test_will_fail_immediately_if_exception_type_does_not_allow_retry
170
+ Resque.enqueue(FailOnCustomExceptionJob)
171
+
172
+ 3.times do
173
+ perform_next_job(@worker)
174
+ end
175
+
176
+ assert_equal 0, Resque.info[:pending], 'pending jobs'
177
+ assert_equal 1, Resque.info[:failed], 'failed jobs'
178
+ assert_equal 1, Resque.info[:processed], 'processed job'
179
+ end
180
+
181
+ def test_will_retry_if_a_non_fatal_exception_is_raised
182
+ Resque.enqueue(FailOnCustomExceptionButRaiseStandardErrorJob)
183
+
184
+ 3.times do
185
+ perform_next_job(@worker)
186
+ end
187
+
188
+ assert_equal 0, Resque.info[:pending], 'pending jobs'
189
+ assert_equal 2, Resque.info[:failed], 'failed jobs'
190
+ assert_equal 2, Resque.info[:processed], 'processed job'
191
+ end
192
+
162
193
  def test_retry_failed_jobs_in_separate_queue
163
194
  Resque.enqueue(JobWithRetryQueue, 'arg1')
164
195
 
@@ -0,0 +1,25 @@
1
+ require 'test_helper'
2
+
3
+ require 'resque'
4
+ require 'resque-retry/server'
5
+
6
+ class ServerHelpersTest < MiniTest::Unit::TestCase
7
+
8
+ def setup
9
+ Resque.redis.flushall
10
+ @worker = Resque::Worker.new(:testing)
11
+ @worker.register_worker
12
+
13
+ @helpers = Class.new.extend(ResqueRetry::Server::Helpers)
14
+ end
15
+
16
+ def test_retry_key_for_job
17
+ Resque.enqueue(LimitThreeJobDelay1Hour)
18
+ perform_next_job(@worker)
19
+
20
+ timestamp = Resque.delayed_queue_peek(0, 1).first
21
+ job = Resque.delayed_timestamp_peek(timestamp, 0, 1).first
22
+ assert_equal '0', @helpers.retry_attempts_for_job(job), 'should have 0 retry attempt'
23
+ end
24
+
25
+ end
@@ -7,6 +7,12 @@ ENV['RACK_ENV'] = 'test'
7
7
  class ServerTest < MiniTest::Unit::TestCase
8
8
  include Rack::Test::Methods
9
9
 
10
+ def setup
11
+ Resque.redis.flushall
12
+ @worker = Resque::Worker.new(:testing)
13
+ @worker.register_worker
14
+ end
15
+
10
16
  def app
11
17
  Resque::Server
12
18
  end
@@ -22,4 +28,18 @@ class ServerTest < MiniTest::Unit::TestCase
22
28
  assert last_response.body.include?('/retry')
23
29
  end
24
30
 
31
+ def test_display_retry_job
32
+ # to begin with, we should have no retry jobs listed.
33
+ get '/retry'
34
+ assert last_response.body.include?('<b>0</b> timestamps'), 'should have 0 retry jobs'
35
+
36
+ # queue failing job that will retry.
37
+ Resque.enqueue(LimitThreeJobDelay1Hour)
38
+ perform_next_job(@worker)
39
+
40
+ # we should now have the retry job listed.
41
+ get '/retry'
42
+ assert last_response.body.include?('<b>1</b> timestamps'), 'should have 1 retry jobs'
43
+ end
44
+
25
45
  end
@@ -140,6 +140,41 @@ class ExponentialBackoffJob < RetryDefaultsJob
140
140
  @queue = :testing
141
141
  end
142
142
 
143
+ class ExponentialBackoffWithRetryDelayMultiplicandMinJob < RetryDefaultsJob
144
+ extend Resque::Plugins::ExponentialBackoff
145
+ @queue = :testing
146
+ @retry_delay_multiplicand_min = 0.5
147
+ end
148
+
149
+ class ExponentialBackoffWithRetryDelayMultiplicandMaxJob < RetryDefaultsJob
150
+ extend Resque::Plugins::ExponentialBackoff
151
+ @queue = :testing
152
+ @retry_delay_multiplicand_max = 3.0
153
+ end
154
+
155
+ class ExponentialBackoffWithRetryDelayMultiplicandMinAndMaxJob < RetryDefaultsJob
156
+ extend Resque::Plugins::ExponentialBackoff
157
+ @queue = :testing
158
+ @retry_delay_multiplicand_min = 0.5
159
+ @retry_delay_multiplicand_max = 3.0
160
+ end
161
+
162
+ class InvalidRetryDelayMaxConfigurationJob
163
+ @queue = :testing
164
+ @retry_delay_multiplicand_max = 0.9
165
+ end
166
+
167
+ class InvalidRetryDelayMinConfigurationJob
168
+ @queue = :testing
169
+ @retry_delay_multiplicand_min = 1.1
170
+ end
171
+
172
+ class InvalidRetryDelayMinAndMaxConfigurationJob
173
+ @queue = :testing
174
+ @retry_delay_multiplicand_min = 3.0
175
+ @retry_delay_multiplicand_max = 0.5
176
+ end
177
+
143
178
  class CustomExponentialBackoffJob
144
179
  extend Resque::Plugins::ExponentialBackoff
145
180
  @queue = :testing
@@ -169,6 +204,35 @@ class RetryCustomExceptionsJob < RetryDefaultsJob
169
204
  end
170
205
  end
171
206
 
207
+ class AmbiguousRetryStrategyJob
208
+ @queue = :testing
209
+
210
+ @fatal_exceptions = [CustomException]
211
+ @retry_exceptions = [AnotherCustomException]
212
+ end
213
+
214
+ class FailOnCustomExceptionJob
215
+ extend Resque::Plugins::Retry
216
+ @queue = :testing
217
+
218
+ @fatal_exceptions = [CustomException]
219
+
220
+ def self.perform(*args)
221
+ raise CustomException
222
+ end
223
+ end
224
+
225
+ class FailOnCustomExceptionButRaiseStandardErrorJob
226
+ extend Resque::Plugins::Retry
227
+ @queue = :testing
228
+
229
+ @fatal_exceptions = [CustomException]
230
+
231
+ def self.perform(*args)
232
+ raise StandardError
233
+ end
234
+ end
235
+
172
236
  module RetryModuleDefaultsJob
173
237
  extend Resque::Plugins::Retry
174
238
  @queue = :testing
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: resque-retry
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.1.1
4
+ version: 1.1.4
5
5
  platform: ruby
6
6
  authors:
7
7
  - Luke Antins
@@ -9,7 +9,7 @@ authors:
9
9
  autorequire:
10
10
  bindir: bin
11
11
  cert_chain: []
12
- date: 2014-03-12 00:00:00.000000000 Z
12
+ date: 2014-03-17 00:00:00.000000000 Z
13
13
  dependencies:
14
14
  - !ruby/object:Gem::Dependency
15
15
  name: resque
@@ -186,6 +186,7 @@ files:
186
186
  - test/retry_exception_delay_test.rb
187
187
  - test/retry_inheriting_checks_test.rb
188
188
  - test/retry_test.rb
189
+ - test/server_helpers_test.rb
189
190
  - test/server_test.rb
190
191
  - test/test_helper.rb
191
192
  - test/test_jobs.rb