resque-retry 0.2.1 → 0.2.2

data/HISTORY.md

@@ -1,3 +1,9 @@
+## HEAD
+
+## 0.2.1 (2011-12-08)
+
+* Feature: Ability to set `retry_delay` per exception type. (Dave Benvenuti)
+
 ## 0.2.1 (2011-11-23)
 
 * Bugfix: Fixed error when we tried to parse a number/string as JSON on the

data/README.md

@@ -155,8 +155,9 @@ more special.
 
 ### Sleep After Requeuing
 
-Sometimes it is useful to delay the worker that failed a job attempt, but still requeue the job for immediate processing
-by other workers. This can be done with `@sleep_after_requeue`:
+Sometimes it is useful to delay the worker that failed a job attempt, but
+still requeue the job for immediate processing by other workers. This can be
+done with `@sleep_after_requeue`:
 
     class DeliverWebHook
       extend Resque::Plugins::Retry
@@ -169,14 +170,18 @@ by other workers. This can be done with `@sleep_after_requeue`:
       end
     end
 
-This retries the job once and causes the worker that failed to sleep for 5 seconds after requeuing the job.  If there are
-multiple workers in the system this allows the job to be retried immediately while the original worker heals itself.  For
-example failed jobs may cause other (non-worker) OS processes to die.  A system monitor such as [god][god] can fix
-the server while the job is being retried on a different worker.
+This retries the job once and causes the worker that failed to sleep for 5
+seconds after requeuing the job. If there are multiple workers in the system
+this allows the job to be retried immediately while the original worker heals
+itself.For example failed jobs may cause other (non-worker) OS processes to
+die. A system monitor such as [god][god] can fix the server while the job is
+being retried on a different worker.
 
-`@sleep_after_requeue` is independent of `@retry_delay`.  If you set both, they both take effect.
+`@sleep_after_requeue` is independent of `@retry_delay`. If you set both, they
+both take effect.
 
-You can override the method `sleep_after_requeue` to set the sleep value dynamically.
+You can override the method `sleep_after_requeue` to set the sleep value
+dynamically.
 
 ### Exponential Backoff
 
@@ -223,6 +228,28 @@ it so only specific exceptions are retried using `retry_exceptions`:
 The above modification will **only** retry if a `NetworkError` (or subclass)
 exception is thrown.
 
+You may also want to specify different retry delays for different exception
+types. You may optionally set `@retry_exceptions` to a hash where the keys are
+your specific exception classes to retry on, and the values are your retry
+delays in seconds or an array of retry delays to be used similar to
+exponential backoff.
+
+    class DeliverSMS
+      extend Resque::Plugins::Retry
+      @queue = :mt_messages
+
+      @retry_exceptions = { NetworkError => 30, SystemCallError => [120, 240] }
+
+      def self.perform(mt_id, mobile_number, message)
+        heavy_lifting
+      end
+    end
+
+In the above example, Resque would retry any `DeliverSMS` jobs which throw a
+`NetworkError` or `SystemCallError`. If the job throws a `NetworkError` it
+will be retried 30 seconds later, if it throws `SystemCallError` it will first
+retry 120 seconds later then subsequent retry attempts 240 seconds later.
+
 ### Custom Retry Criteria Check Callbacks
 
 You may define custom retry criteria callbacks:

data/lib/resque/plugins/retry.rb

@@ -86,8 +86,14 @@ module Resque
       # Number of seconds to delay until the job is retried.
       # 
       # @return [Number] number of seconds to delay
-      def retry_delay
-        @retry_delay ||= 0
+      def retry_delay(exception_class = nil)
+        if @retry_exceptions.is_a?(Hash)
+          delay = @retry_exceptions[exception_class] || 0
+          # allow an array of delays.
+          delay.is_a?(Array) ? delay[retry_attempt] || delay.last : delay
+        else
+          @retry_delay ||= 0
+        end
       end
 
       # @abstract
@@ -122,7 +128,11 @@ module Resque
       # 
       # @return [Array, nil]
       def retry_exceptions
-        @retry_exceptions ||= nil
+        if @retry_exceptions.is_a?(Hash)
+          @retry_exceptions.keys
+        else
+          @retry_exceptions ||= nil
+        end
       end
 
       # Test if the retry criteria is valid.
@@ -191,12 +201,16 @@ module Resque
       end
 
       # Retries the job.
-      def try_again(*args)
-        if retry_delay <= 0
+      def try_again(exception, *args)
+        # some plugins define retry_delay and have it take no arguments, so rather than break those,
+        # we'll just check here to see whether it takes the additional exception class argument or not
+        temp_retry_delay = ([-1, 1].include?(method(:retry_delay).arity) ? retry_delay(exception.class) : retry_delay)
+
+        if temp_retry_delay <= 0
           # If the delay is 0, no point passing it through the scheduler
           Resque.enqueue(self, *args_for_retry(*args))
         else
-          Resque.enqueue_in(retry_delay, self, *args_for_retry(*args))
+          Resque.enqueue_in(temp_retry_delay, self, *args_for_retry(*args))
         end
         sleep(sleep_after_requeue) if sleep_after_requeue > 0
       end
@@ -223,7 +237,7 @@ module Resque
       # Otherwise the retry attempt count is deleted from Redis.
       def on_failure_retry(exception, *args)
         if retry_criteria_valid?(exception, *args)
-          try_again(*args)
+          try_again(exception, *args)
         else
           Resque.redis.del(redis_retry_key(*args))
         end

data/test/exponential_backoff_test.rb

@@ -1,4 +1,4 @@
-require File.dirname(__FILE__) + '/test_helper'
+require File.expand_path(File.dirname(__FILE__) + '/test_helper')
 
 class ExponentialBackoffTest < MiniTest::Unit::TestCase
   def setup
@@ -13,21 +13,21 @@ class ExponentialBackoffTest < MiniTest::Unit::TestCase
   end
 
   def test_default_backoff_strategy
-    now = Time.now
+    start_time = Time.now.to_i
     Resque.enqueue(ExponentialBackoffJob)
 
     perform_next_job @worker
-    assert_equal 1, Resque.info[:processed], '1 processed job'
-    assert_equal 1, Resque.info[:failed], 'first ever run, and it should have failed, but never retried'
-    assert_equal 1, Resque.info[:pending], '1 pending job, because it never hits the scheduler'
+    assert_equal 1, Resque.info[:processed],  '1 processed job'
+    assert_equal 1, Resque.info[:failed],     'first ever run, and it should have failed, but never retried'
+    assert_equal 1, Resque.info[:pending],    '1 pending job, because it never hits the scheduler'
 
     perform_next_job @worker
-    assert_equal 2, Resque.info[:processed], '2nd run, but first retry'
-    assert_equal 2, Resque.info[:failed], 'should of failed again, this is the first retry attempt'
-    assert_equal 0, Resque.info[:pending], '0 pending jobs, it should be in the delayed queue'
+    assert_equal 2, Resque.info[:processed],  '2nd run, but first retry'
+    assert_equal 2, Resque.info[:failed],     'should of failed again, this is the first retry attempt'
+    assert_equal 0, Resque.info[:pending],    '0 pending jobs, it should be in the delayed queue'
 
     delayed = Resque.delayed_queue_peek(0, 1)
-    assert_equal now.to_i + 60, delayed[0], '2nd delay' # the first had a zero delay.
+    assert_in_delta (start_time + 60), delayed[0], 1.00, '2nd delay' # the first had a zero delay.
 
     5.times do
       Resque.enqueue(ExponentialBackoffJob)
@@ -35,27 +35,28 @@ class ExponentialBackoffTest < MiniTest::Unit::TestCase
     end
 
     delayed = Resque.delayed_queue_peek(0, 5)
-    assert_equal now.to_i + 600, delayed[1], '3rd delay'
-    assert_equal now.to_i + 3600, delayed[2], '4th delay'
-    assert_equal now.to_i + 10_800, delayed[3], '5th delay'
-    assert_equal now.to_i + 21_600, delayed[4], '6th delay'
+    assert_in_delta (start_time + 600),     delayed[1], 1.00, '3rd delay'
+    assert_in_delta (start_time + 3600),    delayed[2], 1.00, '4th delay'
+    assert_in_delta (start_time + 10_800),  delayed[3], 1.00, '5th delay'
+    assert_in_delta (start_time + 21_600),  delayed[4], 1.00, '6th delay'
   end
 
   def test_custom_backoff_strategy
-    now = Time.now
+    start_time = Time.now.to_i
     4.times do
       Resque.enqueue(CustomExponentialBackoffJob, 'http://lividpenguin.com', 1305, 'cd8079192d379dc612f17c660591a6cfb05f1dda')
       perform_next_job @worker
     end
 
     delayed = Resque.delayed_queue_peek(0, 3)
-    assert_equal now.to_i + 10, delayed[0], '1st delay'
-    assert_equal now.to_i + 20, delayed[1], '2nd delay'
-    assert_equal now.to_i + 30, delayed[2], '3rd delay'
+    assert_in_delta (start_time + 10), delayed[0], 1.00, '1st delay'
+    assert_in_delta (start_time + 20), delayed[1], 1.00, '2nd delay'
+    assert_in_delta (start_time + 30), delayed[2], 1.00, '3rd delay'
+
     assert_equal 2, Resque.delayed_timestamp_size(delayed[2]), '4th delay should share delay with 3rd'
 
-    assert_equal 4, Resque.info[:processed], 'processed jobs'
-    assert_equal 4, Resque.info[:failed], 'failed jobs'
-    assert_equal 0, Resque.info[:pending], 'pending jobs'
+    assert_equal 4, Resque.info[:processed],  'processed jobs'
+    assert_equal 4, Resque.info[:failed],     'failed jobs'
+    assert_equal 0, Resque.info[:pending],    'pending jobs'
   end
 end

data/test/multiple_failure_test.rb

@@ -1,4 +1,4 @@
-require File.dirname(__FILE__) + '/test_helper'
+require File.expand_path(File.dirname(__FILE__) + '/test_helper')
 
 class MultipleFailureTest < MiniTest::Unit::TestCase
   def setup

data/test/resque_test.rb

@@ -1,4 +1,4 @@
-require File.dirname(__FILE__) + '/test_helper'
+require File.expand_path(File.dirname(__FILE__) + '/test_helper')
 
 # make sure the worlds not fallen from beneith us.
 class ResqueTest < MiniTest::Unit::TestCase

data/test/retry_criteria_test.rb

@@ -1,4 +1,4 @@
-require File.dirname(__FILE__) + '/test_helper'
+require File.expand_path(File.dirname(__FILE__) + '/test_helper')
 
 class RetryCriteriaTest < MiniTest::Unit::TestCase
   def setup

data/test/retry_exception_delay_test.rb

@@ -0,0 +1,47 @@
+require File.expand_path(File.dirname(__FILE__) + '/test_helper')
+
+class RetryTest < MiniTest::Unit::TestCase
+  def setup
+    Resque.redis.flushall
+    @worker = Resque::Worker.new(:testing)
+    @worker.register_worker
+  end
+
+  def test_retry_delay_per_exception_single_delay
+    # store start time for later comparison with retry delay.
+    start_time = Time.now.to_i
+
+    # work the job a couple of times to build up some delayed jobs.
+    3.times do
+      Resque.enqueue(PerExceptionClassRetryCountJob)
+      perform_next_job(@worker)
+    end
+
+    # double check job counts.
+    assert_equal 3, Resque.info[:failed], 'failed jobs'
+    assert_equal 3, Resque.info[:processed], 'processed job'
+    assert_equal 0, Resque.info[:pending], 'zero pending jobs as their delayed'
+
+    # now lets see if the delays are correct?
+    delayed = Resque.delayed_queue_peek(0, 3)
+    assert_in_delta (start_time + 7), delayed[0], 1.00, 'retry delay timestamp'
+  end
+
+  def test_retry_delay_per_exception_multiple_delay
+    # store start time for later comparison with retry delay.
+    start_time = Time.now.to_i
+
+    # work the job a couple of times to build up some delayed jobs.
+    3.times do
+      Resque.enqueue(PerExceptionClassRetryCountArrayJob)
+      perform_next_job(@worker)
+    end
+
+    # now lets see if the delays are correct?
+    delayed = Resque.delayed_queue_peek(0, 3)
+    assert_in_delta (start_time + 5),  delayed[0], 1.00, '1st retry delay timestamp'
+    assert_in_delta (start_time + 10), delayed[1], 1.00, '2nd retry delay timestamp'
+    assert_in_delta (start_time + 15), delayed[2], 1.00, '3rd retry delay timestamp'
+  end
+
+end

data/test/retry_inheriting_checks_test.rb

@@ -1,4 +1,4 @@
-require File.dirname(__FILE__) + '/test_helper'
+require File.expand_path(File.dirname(__FILE__) + '/test_helper')
 
 class RetryInheritingChecksTest < MiniTest::Unit::TestCase
   def setup

data/test/retry_test.rb

@@ -1,4 +1,4 @@
-require File.dirname(__FILE__) + '/test_helper'
+require File.expand_path(File.dirname(__FILE__) + '/test_helper')
 
 class RetryTest < MiniTest::Unit::TestCase
   def setup
@@ -96,18 +96,18 @@ class RetryTest < MiniTest::Unit::TestCase
     assert_equal 0, Resque.info[:failed], 'failed jobs'
     Resque.enqueue(SleepDelay1SecondJob)
     before = Time.now
-    3.times do
+    2.times do
       perform_next_job(@worker)
     end
     actual_delay = Time.now - before
-    
+
     assert actual_delay >= 1, "did not sleep long enough: #{actual_delay} seconds"
     assert actual_delay < 2, "slept too long: #{actual_delay} seconds"
     assert_equal 1, Resque.info[:failed], 'failed jobs'
     assert_equal 2, Resque.info[:processed], 'processed job'
     assert_equal 0, Resque.info[:pending], 'pending jobs'
   end
-  
+
   def test_can_determine_if_exception_may_be_retried
     assert_equal true, RetryDefaultsJob.retry_exception?(StandardError), 'StandardError may retry'
     assert_equal true, RetryDefaultsJob.retry_exception?(CustomException), 'CustomException may retry'
@@ -180,4 +180,11 @@ class RetryTest < MiniTest::Unit::TestCase
     assert_equal 'resque-retry:GoodJob:arg1-removespace', GoodJob.redis_retry_key('arg1', 'remove space')
   end
 
-end
+  def test_retry_delay
+    assert_equal 3, NormalRetryCountJob.retry_delay
+    assert_equal 7, PerExceptionClassRetryCountJob.retry_delay(RuntimeError)
+    assert_equal 11, PerExceptionClassRetryCountJob.retry_delay(Exception)
+    assert_equal 13, PerExceptionClassRetryCountJob.retry_delay(Timeout::Error)
+  end
+
+end

data/test/server_test.rb

@@ -1,4 +1,4 @@
-require File.dirname(__FILE__) + '/test_helper'
+require File.expand_path(File.dirname(__FILE__) + '/test_helper')
 
 require 'resque-retry/server'
 ENV['RACK_ENV'] = 'test'

data/test/test_helper.rb

@@ -2,6 +2,7 @@ dir = File.dirname(File.expand_path(__FILE__))
 $LOAD_PATH.unshift dir + '/../lib'
 $TESTING = true
 
+gem 'minitest'
 require 'minitest/unit'
 require 'minitest/pride'
 require 'rack/test'

data/test/test_jobs.rb

@@ -269,3 +269,35 @@ class CustomIdentifierFailingJob
     raise 'failed'
   end
 end
+
+class NormalRetryCountJob
+  extend Resque::Plugins::Retry
+
+  @queue = :testing
+  @retry_delay = 3
+  @retry_exceptions = [RuntimeError, Exception, Timeout::Error]
+end
+
+class PerExceptionClassRetryCountJob
+  extend Resque::Plugins::Retry
+
+  @queue = :testing
+  @retry_limit = 3
+  @retry_exceptions = { RuntimeError => 7, Exception => 11, Timeout::Error => 13 }
+
+  def self.perform
+    raise RuntimeError, 'I always fail with a RuntimeError'
+  end
+end
+
+class PerExceptionClassRetryCountArrayJob
+  extend Resque::Plugins::Retry
+
+  @queue = :testing
+  @retry_limit = 3
+  @retry_exceptions = { Exception => 11, RuntimeError => [5, 10, 15],  Timeout::Error => [2, 4, 6, 8, 10] }
+
+  def self.perform
+    raise RuntimeError, 'I always fail with a RuntimeError'
+  end
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: resque-retry
 version: !ruby/object:Gem::Version
-  version: 0.2.1
+  version: 0.2.2
   prerelease: 
 platform: ruby
 authors:
@@ -10,11 +10,11 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2011-11-23 00:00:00.000000000Z
+date: 2011-12-08 00:00:00.000000000Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rake
-  requirement: &2165851780 !ruby/object:Gem::Requirement
+  requirement: &2153685420 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -22,10 +22,10 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *2165851780
+  version_requirements: *2153685420
 - !ruby/object:Gem::Dependency
   name: resque
-  requirement: &2165851020 !ruby/object:Gem::Requirement
+  requirement: &2153679300 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -33,10 +33,10 @@ dependencies:
         version: 1.8.0
   type: :runtime
   prerelease: false
-  version_requirements: *2165851020
+  version_requirements: *2153679300
 - !ruby/object:Gem::Dependency
   name: resque-scheduler
-  requirement: &2165835400 !ruby/object:Gem::Requirement
+  requirement: &2153678300 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -44,10 +44,10 @@ dependencies:
         version: 1.8.0
   type: :runtime
   prerelease: false
-  version_requirements: *2165835400
+  version_requirements: *2153678300
 - !ruby/object:Gem::Dependency
   name: minitest
-  requirement: &2165834800 !ruby/object:Gem::Requirement
+  requirement: &2153677800 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -55,10 +55,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *2165834800
+  version_requirements: *2153677800
 - !ruby/object:Gem::Dependency
   name: rack-test
-  requirement: &2165834000 !ruby/object:Gem::Requirement
+  requirement: &2153660820 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -66,10 +66,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *2165834000
+  version_requirements: *2153660820
 - !ruby/object:Gem::Dependency
   name: yard
-  requirement: &2165833040 !ruby/object:Gem::Requirement
+  requirement: &2153659700 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -77,10 +77,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *2165833040
+  version_requirements: *2153659700
 - !ruby/object:Gem::Dependency
   name: json
-  requirement: &2165832180 !ruby/object:Gem::Requirement
+  requirement: &2153659000 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -88,10 +88,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *2165832180
+  version_requirements: *2153659000
 - !ruby/object:Gem::Dependency
   name: simplecov
-  requirement: &2165831320 !ruby/object:Gem::Requirement
+  requirement: &2153658420 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -99,7 +99,7 @@ dependencies:
         version: 0.3.0
   type: :development
   prerelease: false
-  version_requirements: *2165831320
+  version_requirements: *2153658420
 description: ! "  resque-retry provides retry, delay and exponential backoff support
   for\n  resque jobs.\n\n  Features:\n\n  * Redis backed retry count/limit.\n  * Retry
   on all or specific exceptions.\n  * Exponential backoff (varying the delay between
@@ -119,6 +119,7 @@ files:
 - test/redis-test.conf
 - test/resque_test.rb
 - test/retry_criteria_test.rb
+- test/retry_exception_delay_test.rb
 - test/retry_inheriting_checks_test.rb
 - test/retry_test.rb
 - test/server_test.rb