resque-retry 1.5.0 → 1.5.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: e13254ff1b1a5e06977bbbbfa6a28843629f2133
-  data.tar.gz: e9ebf7d623865a0d067c060a804ccae233a34728
+SHA256:
+  metadata.gz: d298842f059d9cd8ee69821e345e60d8a792e68f8f7b9fb2041508cd57e540c3
+  data.tar.gz: 203fc1782c1bdcf08858075f110e34421d1e5b805869514a36b3122739f2487f
 SHA512:
-  metadata.gz: fa3fba3ab9b3945a9b9de6cd3f16a59ce85ac8365d5ac2fc7c962573e74b0236cc2e0526c0435f925c22a7648aefa0fa9f6dc9015ff9a415cb36137b2f239ff7
-  data.tar.gz: 30220fc7c9f30c3b7c0fa63105f86dbcc4c10f59e7cc1bd0e638581b87abf320d5fdae7deb0c9aadbe327f6c0af501fe5bf76842d6f488e15405ded60222af2f
+  metadata.gz: 69bf34cb35ac34bf9975f4f2798dcb78dec39cfd0d373add776026aeaf5dbb594b70bfe2c0f8352701138a61095274413c3087dc14e251b96b117eea57c56207
+  data.tar.gz: b6602597785e8d13e1bb33883e852aece52d4b1e7ffc14f82843e42042c4219bfaa5cfeeec7a9a94176652b3bc59664535c218860127e7591ccd2f113ee80f83

data/.travis.yml

@@ -1,15 +1,13 @@
 language: ruby
 
+sudo: false
+
 matrix:
   allow_failures:
-    - rvm: jruby-19mode
-    - rvm: rbx-2
+    - rvm: jruby-9.1.9.0
 
 rvm:
-  - 1.9.3
-  - 2.0.0
-  - 2.1.0
-  - 2.1.1
-  - 2.1.2
-  - jruby-19mode
-  - rbx-2
+  - 2.3.7
+  - 2.4.4
+  - 2.5.1
+  - jruby-9.1.9.0

data/HISTORY.md

@@ -1,4 +1,15 @@
-## HEAD
+# 1.5.2 (2018-07-16)
+
+* Remove build support for `ruby < 2.3`, `jruby < 9.1` and `rbx`
+* Improving custom retry criteria documentation (@iloveitaly) 
+
+# 1.5.1 (2018-01-07)
+
+* Added build support for `ruby-2.2.x`, `ruby-2.3.x`, `ruby-2.4.x` and `ruby-2.5.x`
+* Dropped build support for `ruby-1.9.3` and `ruby-2.0.0` (these versions are no unsupported)
+* Fix some deprecation warnings in tests
+* Feature: If no exact exception class is present in `@retry_exceptions` hash, try to find closest superclass (@fanfilmu)
+* Feature: When running Resque inline, explicitly don't try to retry, don't touch Redis (@michaelglass)
 
 # 1.5.0 (2015-10-24)
 
@@ -54,11 +65,11 @@
 
 ** !!! WARNING !!! INCLUDES NON-BACKWARDS COMPATIBLE CHANGES **
 
-* Fixed issues related to infinate job retries and v1.20.0 of resque.
+* Fixed issues related to infinite job retries and v1.20.0 of resque.
 * Minimum gem dependency versions changed: resque >= 1.10.0, resque-scheduler >= 1.9.9
-* Feature: Setting `@retry_job_delegate` allows you to seperate the orignal job from a the retry job. (@tanob/@jniesen)
-* Web interface will work without needing to `require` your job code. (n.b. less details avaialble via web).
-* IMPORTANT: `#identifier` method has been namedspaced to `#retry_identifier`.
+* Feature: Setting `@retry_job_delegate` allows you to separate the original job from a the retry job. (@tanob/@jniesen)
+* Web interface will work without needing to `require` your job code. (n.b. less details available via web).
+* IMPORTANT: `#identifier` method has been namespaced to `#retry_identifier`.
 * Bugfix: `Remove` button on retry web interface was not working.
 * Feature: Allow `tagging` exceptions with a module instead of an exception class. (@tils - Tilmann Singer)
 

data/README.md

@@ -183,9 +183,10 @@ Here are a list of the options provided (click to jump):
  * [Job Retry Identifier/Key](#retry_key)
  * [Expire Retry Counters From Redis](#expire)
  * [Try Again and Give Up Callbacks](#callbacks)
+ * [Ignored Exceptions](#ignored)
  * [Debug Plugin Logging](#debug_log)
 
-### <a name="retry_defaults"></a> Retry Defaults 
+### <a name="retry_defaults"></a> Retry Defaults
 
 Retry the job **once** on failure, with zero delay.
 ```ruby
@@ -357,6 +358,8 @@ In the above example, Resque would retry any `DeliverSMS` jobs that throw any
 type of error other than `NetworkError`. If the job throws a `NetworkError` it
 will be marked as "failed" immediately.
 
+You should use either `@fatal_exceptions` or `@retry_exceptions`. If you specify `@fatal_exceptions` the `@retry_exceptions` are ignored.
+
 ### <a name="custom_check"></a> Custom Retry Criteria Check Callbacks
 
 You may define custom retry criteria callbacks:
@@ -368,11 +371,11 @@ class TurkWorker
   @retry_exceptions = [NetworkError]
 
   retry_criteria_check do |exception, *args|
-    if exception.message =~ /InvalidJobId/
-      false # don't retry if we got passed a invalid job id.
-    else
-      true  # its okay for a retry attempt to continue.
+    if exception.message =~ /SpecialErrorMessageToRetry/
+      return true
     end
+
+    false
   end
 
   def self.perform(job_id)
@@ -385,9 +388,18 @@ Similar to the previous example, this job will retry if either a
 `NetworkError` (or subclass) exception is thrown **or** any of the callbacks
 return true.
 
+You'll want to return false by default in the `retry_criteria_check` callback since
+the result of this callback is OR'd with the result of your `retry_exceptions` or
+`fatal_exceptions` configuration. In other words, if you returned true your
+`retry_exceptions` configuration would never be used.
+
+If you want to AND the result of `fatal_exceptions` or `retry_exceptions` with
+custom retry criteria, you'll need to implement your own logic in a `retry_criteria_check`
+to check for `fatal_exceptions` or `retry_exceptions`.
+
 You can also register a retry criteria check with a Symbol if the method is
 already defined on the job class:
-```
+```ruby
 class AlwaysRetryJob
   extend Resque::Plugins::Retry
 
@@ -397,8 +409,9 @@ class AlwaysRetryJob
     true
   end
 end
+```
 
-Use `@retry_exceptions = []` to **only** use your custom retry criteria checks
+Use `@retry_exceptions = []` and `@fatal_exceptions = []` to **only** use your custom retry criteria checks
 to determine if the job should retry.
 
 NB: Your callback must be able to accept the exception and job arguments as
@@ -419,7 +432,7 @@ class DeliverViaSMSC
     [999, mt_message]
   end
 
-  self.perform(smsc_id, mt_message)
+  def self.perform(smsc_id, mt_message)
     heavy_lifting
   end
 end
@@ -439,7 +452,7 @@ class DeliverViaSMSC
     [999, mt_message + exception.message]
   end
 
-  self.perform(smsc_id, mt_message)
+  def self.perform(smsc_id, mt_message)
     heavy_lifting
   end
 end
@@ -466,7 +479,7 @@ class DeliverSMS
     "#{mobile_number}:#{mt_id}"
   end
 
-  self.perform(mt_id, mobile_number, message)
+  def self.perform(mt_id, mobile_number, message)
     heavy_lifting
   end
 end
@@ -482,7 +495,7 @@ class DeliverSMS
   @queue = :mt_messages
   @expire_retry_key_after = 3600 # expire key after `retry_delay` plus 1 hour
 
-  self.perform(mt_id, mobile_number, message)
+  def self.perform(mt_id, mobile_number, message)
     heavy_lifting
   end
 end
@@ -512,7 +525,7 @@ _not_ retrying, you can add a `give_up_callback`:
 ```ruby
 class LoggedJob
   extend Resque::Plugins::Retry
-  
+
   give_up_callback do |exception, *args|
     logger.error("Received #{exception}, job #{self.name} failed with #{args}")
   end
@@ -524,10 +537,10 @@ the job class:
 ```ruby
 class LoggedJob
   extend Resque::Plugins::Retry
-  
+
   give_up_callback :log_give_up
 
-  def self.log_give_up(ex, *args)
+  def self.log_give_up(exception, *args)
     logger.error("Received #{exception}, job #{self.name} failed with #{args}")
   end
 end
@@ -557,6 +570,25 @@ Warning: Make sure your callbacks do not throw any exceptions. If they do,
 subsequent callbacks will not be triggered, and the job will not be retried
 (if it was trying again). The retry counter also will not be reset.
 
+### <a name="ignored"></a> Ignored Exceptions
+If there is an exception for which you want to retry, but you don't want it to
+increment your retry counter, you can add it to `@ignore_exceptions`.
+
+One use case: Restarting your workers triggers a `Resque::TermException`. You
+may want your workers to retry the job that they were working on, but without
+incrementing the retry counter.
+
+```ruby
+class RestartResilientJob
+  extend Resque::Plugins::Retry
+
+  @retry_exceptions = [Resque::TermException]
+  @ignore_exceptions = [Resque::TermException]
+end
+```
+
+Reminder: `@ignore_exceptions` should be a subset of `@retry_exceptions`.
+
 ### <a name="debug_log"></a> Debug Plugin Logging
 
 The inner-workings of the plugin are output to the Resque [Logger](https://github.com/resque/resque/wiki/Logging)

data/lib/resque-retry/version.rb

@@ -1,3 +1,3 @@
 module ResqueRetry
-  VERSION = '1.5.0'
+  VERSION = '1.5.2'
 end

data/lib/resque/plugins/retry.rb

@@ -43,13 +43,34 @@ module Resque
       # @api public
       class AmbiguousRetryStrategyException < StandardError; end
 
+      # Raised if there is a problem with the configuration of resque-retry.
+      #
+      # @api public
+      class RetryConfigurationException < StandardError; end
+
       # Fail fast, when extended, if the "receiver" is misconfigured
       #
       # @api private
       def self.extended(receiver)
-        if receiver.instance_variable_get('@fatal_exceptions') && receiver.instance_variable_get('@retry_exceptions')
+        retry_exceptions = receiver.instance_variable_get('@retry_exceptions')
+        fatal_exceptions = receiver.instance_variable_get('@fatal_exceptions')
+        ignore_exceptions = receiver.instance_variable_get('@ignore_exceptions')
+
+        if fatal_exceptions && retry_exceptions
           raise AmbiguousRetryStrategyException.new(%{You can't define both "@fatal_exceptions" and "@retry_exceptions"})
         end
+
+        # Check that ignore_exceptions is a subset of retry_exceptions
+        if retry_exceptions.is_a?(Hash)
+          exceptions = retry_exceptions.keys
+        else
+          exceptions = Array(retry_exceptions)
+        end
+
+        excess_exceptions = Array(ignore_exceptions) - exceptions
+        unless excess_exceptions.empty?
+          raise RetryConfigurationException, "The following exceptions are defined in @ignore_exceptions but not in @retry_exceptions: #{excess_exceptions.join(', ')}."
+        end
       end
 
       # Copy retry criteria checks, try again callbacks, and give up callbacks
@@ -115,13 +136,18 @@ module Resque
 
       # @abstract
       # Number of seconds to delay until the job is retried
+      # If @retry_exceptions is a Hash and there is no delay defined for exception_class,
+      # looks for closest superclass and assigns it's delay to @retry_exceptions[exception_class]
       #
       # @return [Number] number of seconds to delay
       #
       # @api public
       def retry_delay(exception_class = nil)
         if @retry_exceptions.is_a?(Hash)
-          delay = @retry_exceptions[exception_class] || 0
+          delay = @retry_exceptions[exception_class] ||= begin
+            relevant_definitions = @retry_exceptions.select { |ex| exception_class <= ex }
+            relevant_definitions.any? ? relevant_definitions.sort.first[1] : 0
+          end
           # allow an array of delays.
           delay.is_a?(Array) ? delay[retry_attempt] || delay.last : delay
         else
@@ -131,7 +157,7 @@ module Resque
 
       # @abstract
       # Number of seconds to sleep after job is requeued
-      # 
+      #
       # @return [Number] number of seconds to sleep
       #
       # @api public
@@ -264,7 +290,7 @@ module Resque
         # if the retry limit was reached, dont bother checking anything else.
         if retry_limit_reached?
           log_message 'retry limit reached', args, exception
-          return false 
+          return false
         end
 
         # We always want to retry if the exception matches.
@@ -369,7 +395,7 @@ module Resque
 
         # remember that this job is now being retried. before_perform_retry will increment
         # this so it represents the retry count, and MultipleWithRetrySuppression uses
-        # the existence of this to determine if the job should be sent to the 
+        # the existence of this to determine if the job should be sent to the
         # parent failure backend (e.g. failed queue) or not.  Removing this means
         # jobs that fail before ::perform will be both retried and sent to the failed queue.
         Resque.redis.setnx(redis_retry_key(*args), -1)
@@ -406,6 +432,7 @@ module Resque
       #
       # @api private
       def before_perform_retry(*args)
+        return if Resque.inline?
         log_message 'before_perform_retry', args
         @on_failure_retry_hook_already_called = false
 
@@ -428,6 +455,7 @@ module Resque
       #
       # @api private
       def after_perform_retry(*args)
+        return if Resque.inline?
         log_message 'after_perform_retry, clearing retry key', args
         clean_retry_key(*args)
       end
@@ -443,6 +471,7 @@ module Resque
       #
       # @api private
       def on_failure_retry(exception, *args)
+        return if Resque.inline?
         log_message 'on_failure_retry', args, exception
         if exception.is_a?(Resque::DirtyExit)
           # This hook is called from a worker processes, not the job process
@@ -453,6 +482,13 @@ module Resque
           return
         end
 
+        # If we are "ignoring" the exception, then we decrement the retry
+        # counter, so that the current attempt didn't count toward the retry
+        # counter.
+        if ignore_exceptions.include?(exception.class)
+          @retry_attempt = Resque.redis.decr(redis_retry_key(*args))
+        end
+
         if retry_criteria_valid?(exception, *args)
           try_again(exception, *args)
         else
@@ -584,6 +620,10 @@ module Resque
         end
       end
 
+      def ignore_exceptions
+        @ignore_exceptions ||= []
+      end
+
       # Helper to call functions that may be passed as Symbols or Procs. If
       # a symbol, it is assumed to refer to a method that is already defined
       # on this class.

data/test/ignore_exceptions_test.rb

@@ -0,0 +1,38 @@
+require 'test_helper'
+
+class IgnoreExceptionsTest < Minitest::Test
+  def setup
+    Resque.redis.flushall
+    @worker = Resque::Worker.new(:testing)
+    @worker.register_worker
+  end
+
+  def test_ignore_exceptions
+    Resque.enqueue(IgnoreExceptionsJob)
+    retry_key = IgnoreExceptionsJob.redis_retry_key
+
+    IgnoreExceptionsJob.stubs(:perform).raises(AnotherCustomException)
+    perform_next_job(@worker)
+    assert_equal '0', Resque.redis.get(retry_key), 'retry counter'
+
+    IgnoreExceptionsJob.stubs(:perform).raises(AnotherCustomException)
+    perform_next_job(@worker)
+    assert_equal '1', Resque.redis.get(retry_key), 'retry counter'
+
+    IgnoreExceptionsJob.stubs(:perform).raises(CustomException)
+    perform_next_job(@worker)
+    assert_equal '1', Resque.redis.get(retry_key), 'retry counter'
+  end
+
+  def test_ignore_exception_configuration_1
+    assert_raises Resque::Plugins::Retry::RetryConfigurationException do
+      IgnoreExceptionsImproperlyConfiguredJob1.extend(Resque::Plugins::Retry)
+    end
+  end
+
+  def test_ignore_exception_configuration_2
+    assert_raises Resque::Plugins::Retry::RetryConfigurationException do
+      IgnoreExceptionsImproperlyConfiguredJob2.extend(Resque::Plugins::Retry)
+    end
+  end
+end

data/test/resque_inline_test.rb

@@ -0,0 +1,23 @@
+require 'test_helper'
+
+class ResqueInlineTest < Minitest::Test
+  def setup
+    Resque.inline = true
+    Resque.expects(:redis).never
+  end
+
+  def teardown
+    Resque.inline = false
+  end
+
+  def test_runs_inline
+    GoodJob.expects :perform
+    Resque.enqueue(GoodJob)
+  end
+
+  def test_fails_inline
+    assert_raises CustomException do
+      Resque.enqueue(RetryCustomExceptionsJob, 'CustomException')
+    end
+  end
+end

data/test/retry_test.rb

@@ -18,8 +18,8 @@ class RetryTest < Minitest::Test
   def test_default_settings
     assert_equal 1, RetryDefaultSettingsJob.retry_limit, 'default retry limit'
     assert_equal 0, RetryDefaultSettingsJob.retry_attempt, 'default number of retry attempts'
-    assert_equal nil, RetryDefaultSettingsJob.fatal_exceptions, 'default fatal exceptions; nil = none'
-    assert_equal nil, RetryDefaultSettingsJob.retry_exceptions, 'default retry exceptions; nil = any'
+    assert_nil RetryDefaultSettingsJob.fatal_exceptions, 'default fatal exceptions; nil = none'
+    assert_nil RetryDefaultSettingsJob.retry_exceptions, 'default retry exceptions; nil = any'
     assert_equal 0, RetryDefaultSettingsJob.retry_delay, 'default seconds until retry'
   end
 
@@ -223,7 +223,7 @@ class RetryTest < Minitest::Test
 
     assert job_from_retry_queue = Resque.pop(:testing_retry)
     assert_equal ['arg1'], job_from_retry_queue['args']
-    assert_equal nil, Resque.redis.get(JobWithRetryQueue.redis_retry_key('arg1'))
+    assert_nil Resque.redis.get(JobWithRetryQueue.redis_retry_key('arg1'))
   end
 
   def test_clean_retry_key_should_splat_args
@@ -244,14 +244,14 @@ class RetryTest < Minitest::Test
   def test_delete_redis_key_when_job_is_successful
     Resque.enqueue(GoodJob, 'arg1')
 
-    assert_equal nil, Resque.redis.get(GoodJob.redis_retry_key('arg1'))
+    assert_nil Resque.redis.get(GoodJob.redis_retry_key('arg1'))
     perform_next_job(@worker)
-    assert_equal nil, Resque.redis.get(GoodJob.redis_retry_key('arg1'))
+    assert_nil Resque.redis.get(GoodJob.redis_retry_key('arg1'))
   end
 
   def test_delete_redis_key_after_final_failed_retry
     Resque.enqueue(FailFiveTimesJob, 'yarrrr')
-    assert_equal nil, Resque.redis.get(FailFiveTimesJob.redis_retry_key('yarrrr'))
+    assert_nil Resque.redis.get(FailFiveTimesJob.redis_retry_key('yarrrr'))
 
     perform_next_job(@worker)
     assert_equal '0', Resque.redis.get(FailFiveTimesJob.redis_retry_key('yarrrr'))
@@ -262,7 +262,7 @@ class RetryTest < Minitest::Test
     5.times do
       perform_next_job(@worker)
     end
-    assert_equal nil, Resque.redis.get(FailFiveTimesJob.redis_retry_key('yarrrr'))
+    assert_nil Resque.redis.get(FailFiveTimesJob.redis_retry_key('yarrrr'))
 
     assert_equal 5, Resque.info[:failed], 'failed jobs'
     assert_equal 6, Resque.info[:processed], 'processed job'
@@ -286,9 +286,12 @@ class RetryTest < Minitest::Test
 
   def test_retry_delay
     assert_equal 3, NormalRetryCountJob.retry_delay
-    assert_equal 7, PerExceptionClassRetryCountJob.retry_delay(RuntimeError)
-    assert_equal 11, PerExceptionClassRetryCountJob.retry_delay(Exception)
-    assert_equal 13, PerExceptionClassRetryCountJob.retry_delay(Timeout::Error)
+    assert_equal 7, PerExceptionClassRetryCountJob.retry_delay(StandardError)
+    assert_equal 7, PerExceptionClassRetryCountJob.retry_delay(CustomException)
+    assert_equal 11, PerExceptionClassRetryCountJob.retry_delay(AnotherCustomException)
+    assert_equal 13, PerExceptionClassRetryCountJob.retry_delay(HierarchyCustomException)
+
+    assert_equal 7, PerExceptionClassRetryCountJob.instance_variable_get("@retry_exceptions")[CustomException]
   end
 
   def test_expire_key_set
@@ -327,7 +330,7 @@ class RetryTest < Minitest::Test
         job.fail(Resque::DirtyExit.new)
       end
 
-      assert_equal nil, @worker.reserve
+      assert_nil @worker.reserve
     end
   end
 end

data/test/test_jobs.rb

@@ -496,7 +496,7 @@ class PerExceptionClassRetryCountJob
 
   @queue = :testing
   @retry_limit = 3
-  @retry_exceptions = { RuntimeError => 7, Exception => 11, Timeout::Error => 13 }
+  @retry_exceptions = { StandardError => 7, AnotherCustomException => 11, HierarchyCustomException => 13 }
 
   def self.perform
     raise RuntimeError, 'I always fail with a RuntimeError'
@@ -579,3 +579,28 @@ class RetryCallbacksJob
 
   give_up_callback :on_give_up_c
 end
+
+class IgnoreExceptionsJob
+  extend Resque::Plugins::Retry
+  @queue = :testing
+  @ignore_exceptions = [CustomException]
+  @retry_exceptions = [CustomException, AnotherCustomException]
+  @retry_limit = 3
+
+  def self.perform
+    "Hello, World!"
+  end
+end
+
+class IgnoreExceptionsImproperlyConfiguredJob1
+  # Manually extend Resque::Plugins::Retry in the test code to catch the
+  # exception.
+  @ignore_exceptions = [CustomException]
+end
+
+class IgnoreExceptionsImproperlyConfiguredJob2
+  # Manually extend Resque::Plugins::Retry in the test code to catch the
+  # exception.
+  @ignore_exceptions = [CustomException]
+  @retry_exceptions = { AnotherCustomException => 0 }
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: resque-retry
 version: !ruby/object:Gem::Version
-  version: 1.5.0
+  version: 1.5.2
 platform: ruby
 authors:
 - Luke Antins
@@ -10,7 +10,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2015-10-24 00:00:00.000000000 Z
+date: 2018-07-16 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: resque
@@ -181,8 +181,10 @@ files:
 - lib/resque/plugins/retry/logging.rb
 - resque-retry.gemspec
 - test/exponential_backoff_test.rb
+- test/ignore_exceptions_test.rb
 - test/multiple_failure_test.rb
 - test/redis-test.conf
+- test/resque_inline_test.rb
 - test/resque_test.rb
 - test/retry_callbacks_test.rb
 - test/retry_criteria_test.rb
@@ -214,10 +216,9 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.4.2
+rubygems_version: 2.7.6
 signing_key: 
 specification_version: 4
 summary: A resque plugin; provides retry, delay and exponential backoff support for
   resque jobs.
 test_files: []
-has_rdoc: false