resque-retry 1.5.0 → 1.7.6

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: e13254ff1b1a5e06977bbbbfa6a28843629f2133
-  data.tar.gz: e9ebf7d623865a0d067c060a804ccae233a34728
+SHA256:
+  metadata.gz: bcd42f2b0761226dd2cfa5f9c9d4ff336cda1be0f6acf972ae15cd553f88fa0b
+  data.tar.gz: d65b4261a3ff1068f9880739ea1f4647c10be957511840203331f714cb8281dc
 SHA512:
-  metadata.gz: fa3fba3ab9b3945a9b9de6cd3f16a59ce85ac8365d5ac2fc7c962573e74b0236cc2e0526c0435f925c22a7648aefa0fa9f6dc9015ff9a415cb36137b2f239ff7
-  data.tar.gz: 30220fc7c9f30c3b7c0fa63105f86dbcc4c10f59e7cc1bd0e638581b87abf320d5fdae7deb0c9aadbe327f6c0af501fe5bf76842d6f488e15405ded60222af2f
+  metadata.gz: 78c8fccf7a670f1fc0246d7e8862941eae3fb2bda8ea4d11fb1bea2a5890ed1867f50ff9389a752f3383bf5383783b32e3f63c5ef623909a28aaeba62f4576a2
+  data.tar.gz: 8e3e0ed80d154282e6dddd85b8b7b95b22949d580ad6c0c7a0fc9c3a16475dca861320f919db0f2ddf6b8dca78e8bdd846a48b86db6a5761848d6d4fcb7012a4

data/.github/workflows/ci.yml

@@ -0,0 +1,24 @@
+on: [push, pull_request]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    services:
+      redis:
+        image: redis
+        ports:
+          - 6379:6379
+    strategy:
+      matrix:
+        ruby-version: [2.7, 2.6]
+
+    steps:
+      - uses: actions/checkout@v2
+      - name: Set up Ruby ${{ matrix.ruby-version }}
+        uses: ruby/setup-ruby@v1
+        with:
+          ruby-version: ${{ matrix.ruby-version }}
+      - name: Install dependencies
+        run: bundle install
+      - name: Run tests
+        run: bundle exec rake

data/.gitignore

@@ -14,3 +14,5 @@ tags.*
 *.gem
 .ruby-version
 .ruby-gemset
+.tags
+dump.rdb

data/HISTORY.md

@@ -1,4 +1,69 @@
-## HEAD
+# 1.7.6 (2021-08-06)
+
+* Fix Redis 4.3 breaking saving job failures with multiple failure backend
+* Disable verbose logging by default during test-runs
+* Remove support for ruby < 2.6
+
+# 1.7.5 (2021-08-06)
+
+* Support `retry_delay` methods with different arity
+* Switch over to _GitHub_ CI for PR test-execution
+
+# 1.7.4 (2020-10-09)
+
+* Rework how the default `retry_limit` is calculated
+* Numerous documentation updates
+
+# 1.7.3 (2019-11-21)
+
+* Do not crash if job class is not found on the resque-web server
+
+# 1.7.2 (2019-11-21)
+
+* Address part 2 of issue #158
+
+# 1.7.1 (2019-07-10)
+
+* Prevent `NoMethodError` when calculating `retry_delay`
+
+# 1.7.0 (2019-06-15)
+
+!!! [Potential] breaking changes -- details below !!!
+
+* Remove support for `args_for_retry` -- it has been deprecated for almost 5
+  years
+* Work w/ shallow-clone(s) of `args` in `MultipleWithRetrySuppression#save` and
+  `Retry.retry_args`
+* Remove deprecated options from `Gem::Specification`
+* Clean-up formatting in HISTORY.md
+
+# 1.6.0 (2019-04-21)
+
+* Clean-up / refactoring in preparation for a release (`ruby` deprecations,
+  etc.)
+* Loosen version requirement to allow resque v2.0 and up (@peret)
+
+# 1.5.3 (2018-11-26)
+
+* Monkey patch in the `requeue_queue` method onto the `Resque::Failure` backend
+  (this _should_ be a temporary fix)
+
+# 1.5.2 (2018-07-16)
+
+* Remove build support for `ruby < 2.3`, `jruby < 9.1` and `rbx`
+* Improving custom retry criteria documentation (@iloveitaly) 
+
+# 1.5.1 (2018-01-07)
+
+* Added build support for `ruby-2.2.x`, `ruby-2.3.x`, `ruby-2.4.x` and
+  `ruby-2.5.x`
+* Dropped build support for `ruby-1.9.3` and `ruby-2.0.0` (these versions are
+  no unsupported)
+* Fix some deprecation warnings in tests
+* Feature: If no exact exception class is present in `@retry_exceptions` hash,
+  try to find closest superclass (@fanfilmu)
+* Feature: When running Resque inline, explicitly don't try to retry, don't
+  touch Redis (@michaelglass)
 
 # 1.5.0 (2015-10-24)
 
@@ -20,25 +85,35 @@
 ## 1.3.0 (2014-07-25)
 
 * Adjust gem dependency on `resque-scheduler` to ~> 3.0
-* Deprecated: `args_for_retry` in favor of `retry_args` (will output deprecation warnings if your using the older method).
-* Feature: Allow changing the args for a given exception using `retry_args_for_exception` (@jonp)
+* Deprecated: `args_for_retry` in favor of `retry_args` (will output deprecation
+  warnings if your using the older method).
+* Feature: Allow changing the args for a given exception using
+  `retry_args_for_exception` (@jonp)
 * Feature: Allow setting `@expire_retry_key_after` on the fly (@orenmazor)
 
 ## 1.2.1 (2014-06-09)
 
-* Fixed Kernel.rand: "invalid argument - 0.0 (ArgumentError)" error with "ExponentialBackoff" (on "Rubinius") when `retry_delay_multiplicand_min` and `retry_delay_multiplicand_max` were the same value (@saizai)
+* Fixed Kernel.rand: "invalid argument - 0.0 (ArgumentError)" error with
+  "ExponentialBackoff" (on "Rubinius") when `retry_delay_multiplicand_min` and
+  `retry_delay_multiplicand_max` were the same value (@saizai)
 
 ## 1.2.0 (2014-05-19)
 
-* Fixed scenario where job does not get retried correctly when `perform` is not called as expected.
-* Feature: Optional `@expire_retry_key_after` settings; expires retry counters from redis to save you cleaning up stale state.
-* Feature: Expose inner-workings of plugin through debug messages using `Resque.logger` (when logging level is Logger:DEBUG).
+* Fixed scenario where job does not get retried correctly when `perform` is not
+  called as expected.
+* Feature: Optional `@expire_retry_key_after` settings; expires retry counters
+  from redis to save you cleaning up stale state.
+* Feature: Expose inner-workings of plugin through debug messages using
+  `Resque.logger` (when logging level is Logger:DEBUG).
 
 ## 1.1.4 (2014-03-17)
 
-* Fixed displaying retry information in resque web interface, caused by `Resque::Helpers` being deprecated.
-* Feature: Allow `@fatal_exceptions` as inverse of `@retry_exceptions`, when fatal exception is raised the job will be immediately fail.
-* Feature: Allow a random retry delay (within a range) when using exponential backoff strategy.
+* Fixed displaying retry information in resque web interface, caused by
+  `Resque::Helpers` being deprecated.
+* Feature: Allow `@fatal_exceptions` as inverse of `@retry_exceptions`, when
+  a fatal exception is raised the job will be immediately fail.
+* Feature: Allow a random retry delay (within a range) when using exponential
+  backoff strategy.
 
 ## 1.1.1 (2014-03-12)
 
@@ -47,20 +122,25 @@
 ## 1.1.0 (2014-03-12)
 
 * Remove dependence on `Resque::Helpers`, will be removed in Resque 2.0
-* Use SHA1 for default `#retry_identifier` to prevents issues with long args gobbling space.
+* Use SHA1 for default `#retry_identifier` to prevents issues with long args
+  gobbling space.
 * Minimum version of Resque is now ~> 1.25
 
 ## 1.0.0 (2012-09-07)
 
 ** !!! WARNING !!! INCLUDES NON-BACKWARDS COMPATIBLE CHANGES **
 
-* Fixed issues related to infinate job retries and v1.20.0 of resque.
-* Minimum gem dependency versions changed: resque >= 1.10.0, resque-scheduler >= 1.9.9
-* Feature: Setting `@retry_job_delegate` allows you to seperate the orignal job from a the retry job. (@tanob/@jniesen)
-* Web interface will work without needing to `require` your job code. (n.b. less details avaialble via web).
-* IMPORTANT: `#identifier` method has been namedspaced to `#retry_identifier`.
+* Fixed issues related to infinite job retries and v1.20.0 of resque.
+* Minimum gem dependency versions changed: resque >= 1.10.0, resque-scheduler
+  >= 1.9.9
+* Feature: Setting `@retry_job_delegate` allows you to separate the original job
+  from a the retry job. (@tanob/@jniesen)
+* Web interface will work without needing to `require` your job code. (n.b.
+  less details available via web).
+* IMPORTANT: `#identifier` method has been namespaced to `#retry_identifier`.
 * Bugfix: `Remove` button on retry web interface was not working.
-* Feature: Allow `tagging` exceptions with a module instead of an exception class. (@tils - Tilmann Singer)
+* Feature: Allow `tagging` exceptions with a module instead of an exception
+  class. (@tils - Tilmann Singer)
 
 ## 0.2.2 (2011-12-08)
 
@@ -68,7 +148,8 @@
 
 ## 0.2.1 (2011-11-23)
 
-* Bugfix: Fixed error when we tried to parse a number/string as JSON on the reque-retry web interface.
+* Bugfix: Fixed error when we tried to parse a number/string as JSON on the
+  reque-retry web interface.
 
 ## 0.2.0 (2011-11-22)
 
@@ -78,9 +159,10 @@
     PREVIOUSLY: 0 == infinite retries.
            NOW: -1 == infinite retries; 0 == means never retry.
 
-* Bugfix: `#redis_retry_key` incorrectly built key when custom identifier was used. (Bogdan Gusiev)
+* Bugfix: `#redis_retry_key` incorrectly built key when custom identifier was
+  used. (Bogdan Gusiev)
 * Feature: Ability to sleep worker after re-queuing a job, may be used to bias
-           against the same worker from picking up the job again. (Michael Keirnan)
+  against the same worker from picking up the job again. (Michael Keirnan)
 * Feature: Ability to remove retry jobs using resque-web. (Thiago Morello)
 * Added example demo application.
 * Added Bundler `Gemfile`.
@@ -110,7 +192,8 @@
 ## 0.0.2 (2010-05-06)
 
 * Bugfix: Were calling non-existent method to delete redis key.
-* Delay no-longer falls back to `sleep`. resque-scheduler is a required dependancy.
+* Delay no-longer falls back to `sleep`. resque-scheduler is a required
+  dependancy.
 * Redis key doesn't include ending colon `:` if no args were passed to the job.
 
 ## 0.0.1 (2010-04-27)

data/README.md

@@ -1,7 +1,7 @@
 resque-retry
 ============
 
-A [Resque][resque] plugin. Requires Resque ~> 1.25 & [resque-scheduler][resque-scheduler] ~> 4.0.
+A [Resque][resque] plugin. Requires Resque ~> 1.25 or Resque ~> 2.0 & [resque-scheduler][resque-scheduler] ~> 4.0.
 
 This gem provides retry, delay and exponential backoff support for resque jobs.
 
@@ -183,9 +183,10 @@ Here are a list of the options provided (click to jump):
  * [Job Retry Identifier/Key](#retry_key)
  * [Expire Retry Counters From Redis](#expire)
  * [Try Again and Give Up Callbacks](#callbacks)
+ * [Ignored Exceptions](#ignored)
  * [Debug Plugin Logging](#debug_log)
 
-### <a name="retry_defaults"></a> Retry Defaults 
+### <a name="retry_defaults"></a> Retry Defaults
 
 Retry the job **once** on failure, with zero delay.
 ```ruby
@@ -223,7 +224,48 @@ end
 The above modification will allow your job to retry up to 10 times, with a delay
 of 120 seconds, or 2 minutes between retry attempts.
 
-You can override the `retry_delay` method to set the delay value dynamically.
+You can override the `retry_delay` method to set the delay value dynamically. For example:
+
+```ruby
+class ExampleJob
+  extend Resque::Plugins::Retry
+  @queue = :testing
+
+  def self.retry_delay(exception_class)
+    if exception_class == SocketError
+      10
+    else
+      1
+    end
+  end
+
+  def self.perform(*args)
+    heavy_lifting
+  end
+end
+```
+
+Or, if you'd like the delay to be dependent on job arguments:
+
+```ruby
+class ExampleJob
+  extend Resque::Plugins::Retry
+  @queue = :testing
+
+  def self.retry_delay(exception, *args)
+    # the delay is dependent on the arguments passed to the job
+    # in this case, "3" is passed as the arg and that is used as the delay
+    # make sure this method returns a integer
+    args.first.to_i
+  end
+
+  def self.perform(*args)
+    heavy_lifting
+  end
+end
+
+Resque.enqueue(ExampleJob, '3')
+```
 
 ### <a name="sleep"></a> Sleep After Requeuing
 
@@ -317,7 +359,12 @@ You may also want to specify different retry delays for different exception
 types. You may optionally set `@retry_exceptions` to a hash where the keys are
 your specific exception classes to retry on, and the values are your retry
 delays in seconds or an array of retry delays to be used similar to exponential
-backoff.
+backoff.  `resque-retry` will attempt to determine your retry strategy's
+`@retry_limit` based on your specified `@retry_exceptions`.  If, however, you
+define `@retry_limit` explicitly, you should define `@retry_limit` such that it
+allows for your retry strategies to complete.  If your `@retry_limit` is less
+than the number of desired retry attempts defined in `@retry_exceptions`, your
+job will only retry `@retry_limit` times.
 ```ruby
 class DeliverSMS
   extend Resque::Plugins::Retry
@@ -332,9 +379,15 @@ end
 ```
 
 In the above example, Resque would retry any `DeliverSMS` jobs which throw a
-`NetworkError` or `SystemCallError`. If the job throws a `NetworkError` it
-will be retried 30 seconds later, if it throws `SystemCallError` it will first
-retry 120 seconds later then subsequent retry attempts 240 seconds later.
+`NetworkError` or `SystemCallError`. The `@retry_limit` would be inferred to be
+2 based on the longest retry strategy defined in `@retry_exceptions`. If the job
+throws a `NetworkError` it will be retried 30 seconds later with a subsequent
+retry 30 seconds after that. If it throws a `SystemCallError` it will first
+retry 120 seconds later then a subsequent retry attempt 240 seconds later.  If
+the job fails due to a `NetworkError`, Resque would retry the job in 30 seconds.
+If the job fails a second time, this time due to a `SystemCallError`, the next
+retry would occur 240 seconds later as specified in the `SystemCallError`
+array defined in `@retry_exceptions`.
 
 ### <a name="fail_fast"></a> Fail Fast For Specific Exceptions
 
@@ -357,6 +410,8 @@ In the above example, Resque would retry any `DeliverSMS` jobs that throw any
 type of error other than `NetworkError`. If the job throws a `NetworkError` it
 will be marked as "failed" immediately.
 
+You should use either `@fatal_exceptions` or `@retry_exceptions`. If you specify `@fatal_exceptions` the `@retry_exceptions` are ignored.
+
 ### <a name="custom_check"></a> Custom Retry Criteria Check Callbacks
 
 You may define custom retry criteria callbacks:
@@ -368,11 +423,11 @@ class TurkWorker
   @retry_exceptions = [NetworkError]
 
   retry_criteria_check do |exception, *args|
-    if exception.message =~ /InvalidJobId/
-      false # don't retry if we got passed a invalid job id.
-    else
-      true  # its okay for a retry attempt to continue.
+    if exception.message =~ /SpecialErrorMessageToRetry/
+      return true
     end
+
+    false
   end
 
   def self.perform(job_id)
@@ -385,9 +440,18 @@ Similar to the previous example, this job will retry if either a
 `NetworkError` (or subclass) exception is thrown **or** any of the callbacks
 return true.
 
+You'll want to return false by default in the `retry_criteria_check` callback since
+the result of this callback is OR'd with the result of your `retry_exceptions` or
+`fatal_exceptions` configuration. In other words, if you returned true your
+`retry_exceptions` configuration would never be used.
+
+If you want to AND the result of `fatal_exceptions` or `retry_exceptions` with
+custom retry criteria, you'll need to implement your own logic in a `retry_criteria_check`
+to check for `fatal_exceptions` or `retry_exceptions`.
+
 You can also register a retry criteria check with a Symbol if the method is
 already defined on the job class:
-```
+```ruby
 class AlwaysRetryJob
   extend Resque::Plugins::Retry
 
@@ -397,8 +461,9 @@ class AlwaysRetryJob
     true
   end
 end
+```
 
-Use `@retry_exceptions = []` to **only** use your custom retry criteria checks
+Use `@retry_exceptions = []` and `@fatal_exceptions = []` to **only** use your custom retry criteria checks
 to determine if the job should retry.
 
 NB: Your callback must be able to accept the exception and job arguments as
@@ -419,7 +484,7 @@ class DeliverViaSMSC
     [999, mt_message]
   end
 
-  self.perform(smsc_id, mt_message)
+  def self.perform(smsc_id, mt_message)
     heavy_lifting
   end
 end
@@ -439,11 +504,52 @@ class DeliverViaSMSC
     [999, mt_message + exception.message]
   end
 
-  self.perform(smsc_id, mt_message)
+  def self.perform(smsc_id, mt_message)
+    heavy_lifting
+  end
+end
+```
+
+### Custom Retry Queues
+
+By default, when a job is retried, it is added to the `@queue` specified in the worker. However, you may want to push the job into another (lower or higher priority) queue when the job fails. You can do this by dynamically specifying the retry queue. For example:
+
+```ruby
+class ExampleJob
+  extend Resque::Plugins::Retry
+  @queue = :testing
+  @retry_delay = 1
+
+  def self.work(*args)
+    user_id, user_mode, record_id = *args
+
+    Resque.enqueue_to(
+      target_queue_for_args(user_id, user_mode, record_id),
+      self,
+      *args
+     )
+  end
+
+  def self.retry_queue(exception, *args)
+    target_queue_for_args(*args)
+  end
+
+  def self.perform(*args)
     heavy_lifting
   end
+
+  def self.target_queue_for_args(*args)
+    user_id, user_mode, record_id = *args
+
+    if user_mode
+      'high'
+    else
+      'low'
+    end
+  end
 end
 ```
+
 ### <a name="retry_key"></a> Job Retry Identifier/Key
 
 The retry attempt is incremented and stored in a Redis key. The key is built
@@ -466,7 +572,7 @@ class DeliverSMS
     "#{mobile_number}:#{mt_id}"
   end
 
-  self.perform(mt_id, mobile_number, message)
+  def self.perform(mt_id, mobile_number, message)
     heavy_lifting
   end
 end
@@ -482,7 +588,7 @@ class DeliverSMS
   @queue = :mt_messages
   @expire_retry_key_after = 3600 # expire key after `retry_delay` plus 1 hour
 
-  self.perform(mt_id, mobile_number, message)
+  def self.perform(mt_id, mobile_number, message)
     heavy_lifting
   end
 end
@@ -490,7 +596,7 @@ end
 
 This saves you from having to run a "house cleaning" or "errand" job.
 
-The expiary timeout is "pushed forward" or "touched" after each failure to
+The expiry timeout is "pushed forward" or "touched" after each failure to
 ensure it's not expired too soon.
 
 ### <a name="callbacks"></a> Try Again and Give Up Callbacks
@@ -512,7 +618,7 @@ _not_ retrying, you can add a `give_up_callback`:
 ```ruby
 class LoggedJob
   extend Resque::Plugins::Retry
-  
+
   give_up_callback do |exception, *args|
     logger.error("Received #{exception}, job #{self.name} failed with #{args}")
   end
@@ -524,10 +630,10 @@ the job class:
 ```ruby
 class LoggedJob
   extend Resque::Plugins::Retry
-  
+
   give_up_callback :log_give_up
 
-  def self.log_give_up(ex, *args)
+  def self.log_give_up(exception, *args)
     logger.error("Received #{exception}, job #{self.name} failed with #{args}")
   end
 end
@@ -557,11 +663,42 @@ Warning: Make sure your callbacks do not throw any exceptions. If they do,
 subsequent callbacks will not be triggered, and the job will not be retried
 (if it was trying again). The retry counter also will not be reset.
 
+### <a name="ignored"></a> Ignored Exceptions
+If there is an exception for which you want to retry, but you don't want it to
+increment your retry counter, you can add it to `@ignore_exceptions`.
+
+One use case: Restarting your workers triggers a `Resque::TermException`. You
+may want your workers to retry the job that they were working on, but without
+incrementing the retry counter.
+
+```ruby
+class RestartResilientJob
+  extend Resque::Plugins::Retry
+
+  @retry_exceptions = [Resque::TermException]
+  @ignore_exceptions = [Resque::TermException]
+end
+```
+
+Reminder: `@ignore_exceptions` should be a subset of `@retry_exceptions`.
+
 ### <a name="debug_log"></a> Debug Plugin Logging
 
 The inner-workings of the plugin are output to the Resque [Logger](https://github.com/resque/resque/wiki/Logging)
 when `Resque.logger.level` is set to `Logger::DEBUG`.
 
+Add `VVERBOSE=true` as an environment variable to easily set the log level to debug.
+
+### Testing
+
+To run a specific test and inspect logging output
+
+```
+bundle exec rake TEST=the_test_file.rb VVERBOSE=true
+```
+
+There are many example jobs implementing various use-cases for this gem in `test_jobs.rb`
+
 Contributing/Pull Requests
 --------------------------