resque-retry 0.0.6 → 0.1.0

data/HISTORY.md

@@ -1,4 +1,10 @@
-## 0.0.6 (2010-07-11)
+## 0.1.0 (2010-08-29)
+
+* Feature: Multiple failure backend with retry suppression.
+* Feature: resque-web tab showing retry information.
+* Improved README documentation, added a 'Quick Start' section.
+
+## 0.0.6 (2010-07-12)
 
 * Feature: Added support for custom retry criteria check callbacks.
 

data/README.md

@@ -1,27 +1,120 @@
 resque-retry
 ============
 
-A [Resque][rq] plugin. Requires Resque 1.8.0 & [resque-scheduler][rqs]
+A [Resque][rq] plugin. Requires Resque 1.8.0 & [resque-scheduler][rqs].
 
 resque-retry provides retry, delay and exponential backoff support for
 resque jobs.
 
-### Features
+  * Redis backed retry count/limit.
+  * Retry on all or specific exceptions.
+  * Exponential backoff (varying the delay between retrys).
+  * Multiple failure backend with retry suppression & resque-web tab.
+  * Small & Extendable - plenty of places to override retry logic/settings.
 
-  - Redis backed retry count/limit.
-  - Retry on all or specific exceptions.
-  - Exponential backoff (varying the delay between retrys).
-  - Small & Extendable - plenty of places to override retry logic/settings.
+Install & Quick Start
+---------------------
 
-Usage / Examples
-----------------
+To install:
 
-Just extend your module/class with this module, and your ready to retry!
+    $ gem install resque-retry
+
+You'll want add this to your `Rakefile`:
+
+    require 'resque/tasks'
+    require 'resque_scheduler/tasks'
+
+The delay between retry attempts is provided by [resque-scheduler][rqs].
+You'll want to run the scheduler process, otherwise delayed retry attempts
+will never perform:
+
+    $ rake resque:scheduler
+
+Use the plugin:
+
+    require 'resque-retry'
+
+    class ExampleRetryJob
+      extend Resque::Plugins::Retry
+      @queue = :example_queue
+
+      @retry_limit = 3
+      @retry_delay = 60
+
+      def self.perform(*args)
+        # your magic/heavy lifting goes here.
+      end
+    end
+
+Then start up a resque worker as normal:
+
+    $ QUEUE=* rake resque:work
+
+Now if you ExampleRetryJob fails, it will be retried 3 times, with a 60 second
+delay between attempts.
+
+For more explanation and examples, please see the remaining documentation.
+
+Failure Backend & Resque Web Additions
+--------------------------------------
+
+Lets say your using the Redis failure backend of resque (the default).
+Every time a job fails, the failure queue is populated with the job and
+exception details.
+
+Normally this is useful, but if your jobs retry... it can cause a bit of a mess.
+
+For example: given a job that retried 4 times before completing successful.
+You'll have a lot of failures for the same job and you wont be sure if it
+actually completed successfully just by just using the resque-web interface.
+
+### Failure Backend
+
+`MultipleWithRetrySuppression` is a multiple failure backend, with retry suppression.
+
+Here's an example, using the Redis failure backend:
+
+    require 'resque-retry'
+    require 'resque/failure/redis'
+
+    # require your jobs & application code.
+
+    Resque::Failure::MultipleWithRetrySuppression.classes = [Resque::Failure::Redis]
+    Resque::Failure.backend = Resque::Failure::MultipleWithRetrySuppression
+
+If a job fails, but **can and will** retry, the failure details wont be
+logged in the Redis failed queue *(visible via resque-web)*.
+
+If the job fails, but **can't or won't** retry, the failure will be logged in
+the Redis failed queue, like a normal failure *(without retry)* would.
+
+### Resque Web Additions
+
+If your using the `MultipleWithRetrySuppression` failure backend, you should
+also checkout the resque-web additions!
+
+The new Retry tab displays delayed jobs with retry information; the number of
+attempts and the exception details from the last failure.
+
+Make sure you include this in your `config.ru` or similar file:
+
+    require 'resque-retry'
+    require 'resque-retry/server'
+
+    # require your jobs & application code.
+
+    run Resque::Server.new
+
+Retry Options & Logic
+---------------------
+
+Please take a look at the yardoc/code for more details on methods you may
+wish to override.
 
 Customisation is pretty easy, the below examples should give you
 some ideas =), adapt for your own usage and feel free to pick and mix!
 
-### Retry
+### Retry Defaults
 
 Retry the job **once** on failure, with zero delay.
 
@@ -135,13 +228,24 @@ return true.
 Use `@retry_exceptions = []` to **only** use callbacks, to determine if the
 job should retry.
 
-Customise & Extend
-------------------
+### Retry Arguments
 
-Please take a look at the yardoc/code for more details on methods you may
-wish to override.
+You may override `args_for_retry`, which is passed the current
+job arguments, to modify the arguments for the next retry attempt.
+
+    class DeliverViaSMSC
+      extend Resque::Plugins::Retry
+      @queue = :mt_smsc_messages
 
-Some things worth noting:
+      # retry using the emergency SMSC.
+      def self.args_for_retry(smsc_id, mt_message)
+        [999, mt_message]
+      end
+
+      self.perform(smsc_id, mt_message)
+        heavy_lifting
+      end
+    end
 
 ### Job Identifier/Key
 
@@ -170,29 +274,16 @@ Or you can define the entire key by overriding `redis_retry_key`.
       end
     end
 
-### Retry Arguments
+Contributing/Pull Requests
+--------------------------
 
-You may override `args_for_retry`, which is passed the current
-job arguments, to modify the arguments for the next retry attempt.
-
-    class DeliverViaSMSC
-      extend Resque::Plugins::Retry
-      @queue = :mt_smsc_messages
-      
-      # retry using the emergency SMSC.
-      def self.args_for_retry(smsc_id, mt_message)
-        [999, mt_message]
-      end
-
-      self.perform(smsc_id, mt_message)
-        heavy_lifting
-      end
-    end
-
-Install
--------
-
-    $ gem install resque-retry
+  * Yes please!
+  * Fork the project.
+  * Make your feature addition or bug fix.
+  * Add tests for it.
+  * Commit.
+  * Send me a pull request. Bonus points for topic branches.
+  * If you edit the gemspec/version etc, do it in another commit please.
 
 [rq]: http://github.com/defunkt/resque
-[rqs]: http://github.com/bvandenbos/resque-scheduler
+[rqs]: http://github.com/bvandenbos/resque-scheduler

data/lib/resque-retry.rb

@@ -2,4 +2,5 @@ require 'resque'
 require 'resque_scheduler'
 
 require 'resque/plugins/retry'
-require 'resque/plugins/exponential_backoff'
+require 'resque/plugins/exponential_backoff'
+require 'resque/failure/multiple_with_retry_suppression'

data/lib/resque-retry/server.rb

@@ -0,0 +1,51 @@
+# Extend Resque::Server to add tabs.
+module ResqueRetry
+  module Server
+
+    def self.included(base)
+      base.class_eval {
+        helpers do
+          # builds a retry key for the specified job.
+          def retry_key_for_job(job)
+            klass = Resque.constantize(job['class'])
+            if klass.respond_to?(:redis_retry_key)
+              klass.redis_retry_key(job['args'])
+            else
+              nil
+            end
+          end
+
+          # gets the number of retry attempts for a job.
+          def retry_attempts_for_job(job)
+            Resque.redis.get(retry_key_for_job(job))
+          end
+          
+          # gets the failure details hash for a job.
+          def retry_failure_details(retry_key)
+            Resque.decode(Resque.redis["failure_#{retry_key}"])
+          end
+
+          # reads a 'local' template file.
+          def local_template(path)
+            # Is there a better way to specify alternate template locations with sinatra?
+            File.read(File.join(File.dirname(__FILE__), "server/views/#{path}"))
+          end
+        end
+
+        get '/retry' do
+          erb local_template('retry.erb')
+        end
+
+        get '/retry/:timestamp' do
+          erb local_template('retry_timestamp.erb')
+        end
+      }
+    end
+
+  end
+end
+
+Resque::Server.tabs << 'Retry'
+Resque::Server.class_eval do
+  include ResqueRetry::Server
+end

data/lib/resque-retry/server/views/retry.erb

@@ -0,0 +1,48 @@
+<h1>Delayed Jobs with Retry Information</h1>
+
+<p class="intro">
+  This list below contains the timestamps for scheduled delayed jobs with
+  retry information.
+</p>
+
+<p class="sub">
+  Showing <%= start = params[:start].to_i %> to <%= start + 20 %> of
+  <b><%= size = resque.delayed_queue_schedule_size %></b> timestamps
+</p>
+
+<table>
+  <tr>
+    <th></th>
+    <th>Timestamp</th>
+    <th>Job count</th>
+    <th>Class</th>
+    <th>Args</th>
+    <th>Retry Attempts</th>
+  </tr>
+  <% timestamps = resque.delayed_queue_peek(start, start+20) %>
+  <% timestamps.each do |timestamp| %>
+    <% job = resque.delayed_timestamp_peek(timestamp, 0, 1).first %>
+    <% retry_key = retry_key_for_job(job) %>
+    <tr>
+      <td>
+        <form action="<%= url "/delayed/queue_now" %>" method="post">
+          <input type="hidden" name="timestamp" value="<%= timestamp.to_i %>">
+          <input type="submit" value="Queue now">
+        </form>
+      </td>
+      <td><a href="<%= url "retry/#{timestamp}" %>"><%= format_time(Time.at(timestamp)) %></a></td>
+      <td><%= delayed_timestamp_size = resque.delayed_timestamp_size(timestamp) %></td>
+      <% if job && delayed_timestamp_size == 1 %>
+        <td><%= h job['class'] %></td>
+        <td><%= h job['args'].inspect %></td>
+        <td><%= retry_attempts_for_job(job) || '<i>n/a</i>' %></td>
+      <% else %>
+        <td><a href="<%= url "retry/#{timestamp}" %>">see details</a></td>
+        <td></td>
+        <td></td>
+      <% end %>
+    </tr>
+  <% end %>
+</table>
+
+<%= partial :next_more, :start => start, :size => size %>

data/lib/resque-retry/server/views/retry_timestamp.erb

@@ -0,0 +1,59 @@
+<% timestamp = params[:timestamp].to_i %>
+
+<h1>
+  Delayed Jobs scheduled for <%= format_time(Time.at(timestamp)) %>
+  (with Retry Information)
+</h1>
+
+<p class="intro">
+  This list below contains the delayed jobs scheduled for the current
+  timestamp, with retry information.
+</p>
+
+<p class="sub">
+  Showing <%= start = params[:start].to_i %> to <%= start + 20 %> of
+  <b><%= size = resque.delayed_timestamp_size(timestamp) %></b> jobs
+</p>
+
+<table class="jobs">
+  <tr>
+    <th>Class</th>
+    <th>Args</th>
+    <th>Retry Attempts</th>
+    <th>Exception</th>
+    <th>Backtrace</th>
+  </tr>
+  <% jobs = resque.delayed_timestamp_peek(timestamp, start, 20) %>
+  <% jobs.each do |job| %>
+    <% retry_key = retry_key_for_job(job) %>
+    <% retry_attempts = retry_attempts_for_job(job) %>
+    <tr>
+      <td class="class"><%= h job['class'] %></td>
+      <td class="args"><%= h job['args'].inspect %></td>
+      <% if retry_attempts.nil? %>
+        <td colspan="3"><i>n/a - normal delayed job</i></td>
+      <% else %>
+        <td><%= retry_attempts %></td>
+        <% failure = retry_failure_details(retry_key) %>
+        <td><code><%= failure['exception'] %></code></td>
+        <td class="error">
+          <% if failure['backtrace'] %>
+            <a href="#" class="backtrace"><%= h(failure['error']) %></a>
+            <pre style="display:none"><%= h failure['backtrace'].join("\n") %></pre>
+          <% else %>
+            <%= h failure['error'] %>
+          <% end %>
+        </td>
+      <% end %>
+    </tr>
+  <% end %>
+  <% if jobs.empty? %>
+    <tr>
+      <td class="no-data" colspan="5">
+        There are no pending jobs scheduled for this time.
+      </td>
+    </tr>
+  <% end %>
+</table>
+
+<%= partial :next_more, :start => start, :size => size %>

data/lib/resque/failure/multiple_with_retry_suppression.rb

@@ -0,0 +1,93 @@
+require 'resque/failure/multiple'
+
+module Resque
+  module Failure
+
+    # A multiple failure backend, with retry suppression.
+    #
+    # For example: if you had a job that could retry 5 times, your failure 
+    # backends are not notified unless the _final_ retry attempt also fails.
+    #
+    # Example:
+    #
+    #   require 'resque-retry'
+    #   require 'resque/failure/redis'
+    #
+    #   Resque::Failure::MultipleWithRetrySuppression.classes = [Resque::Failure::Redis]
+    #   Resque::Failure.backend = Resque::Failure::MultipleWithRetrySuppression
+    #
+    class MultipleWithRetrySuppression < Multiple
+      include Resque::Helpers
+
+      module CleanupHooks
+        # Resque after_perform hook.
+        #
+        # Deletes retry failure information from Redis.
+        def after_perform_retry_failure_cleanup(*args)
+          retry_key = redis_retry_key(*args)
+          failure_key = Resque::Failure::MultipleWithRetrySuppression.failure_key(retry_key)
+          Resque.redis.del(failure_key)
+        end
+      end
+
+      # Called when the job fails.
+      #
+      # If the job will retry, suppress the failure from the other backends.
+      # Store the lastest failure information in redis, used by the web
+      # interface.
+      def save
+        unless retryable? && retrying?
+          cleanup_retry_failure_log!
+          super
+        else
+          data = {
+            :failed_at => Time.now.strftime("%Y/%m/%d %H:%M:%S"),
+            :payload   => payload,
+            :exception => exception.class.to_s,
+            :error     => exception.to_s,
+            :backtrace => Array(exception.backtrace),
+            :worker    => worker.to_s,
+            :queue     => queue
+          }
+
+          # Register cleanup hooks.
+          unless klass.respond_to?(:after_perform_retry_failure_cleanup)
+            klass.send(:extend, CleanupHooks)
+          end
+
+          redis[failure_key] = Resque.encode(data)
+        end
+      end
+
+      # Expose this for the hook's use.
+      def self.failure_key(retry_key)
+        'failure_' + retry_key
+      end
+
+      protected
+      def klass
+        constantize(payload['class'])
+      end
+
+      def retry_key
+        klass.redis_retry_key(payload['args'])
+      end
+
+      def failure_key
+        self.class.failure_key(retry_key)
+      end
+
+      def retryable?
+        klass.respond_to?(:redis_retry_key)
+      end
+
+      def retrying?
+        redis.exists(retry_key)
+      end
+
+      def cleanup_retry_failure_log!
+        redis.del(failure_key) if retryable?
+      end
+    end
+  end
+end

data/lib/resque/plugins/retry.rb

@@ -19,7 +19,7 @@ module Resque
     #     extend Resque::Plugins::Retry
     #     @queue = :web_hooks
     #
-    #     @retry_limit = 8          # default: 1
+    #     @retry_limit = 8  # default: 1
     #     @retry_delay = 60 # default: 0
     #
     #     # used to build redis key, for counting job attempts.

data/test/exponential_backoff_test.rb

@@ -16,13 +16,16 @@ class ExponentialBackoffTest < Test::Unit::TestCase
   def test_default_backoff_strategy
     now = Time.now
     Resque.enqueue(ExponentialBackoffJob)
-    2.times do
-      perform_next_job @worker
-    end
 
-    assert_equal 2, Resque.info[:processed], 'processed jobs'
-    assert_equal 2, Resque.info[:failed], 'failed jobs'
-    assert_equal 0, Resque.info[:pending], 'pending jobs'
+    perform_next_job @worker
+    assert_equal 1, Resque.info[:processed], '1 processed job'
+    assert_equal 1, Resque.info[:failed], 'first ever run, and it should of failed, but never retried'
+    assert_equal 1, Resque.info[:pending], '1 pending job, because it never hits the scheduler'
+
+    perform_next_job @worker
+    assert_equal 2, Resque.info[:processed], '2nd run, but first retry'
+    assert_equal 2, Resque.info[:failed], 'should of failed again, this is the first retry attempt'
+    assert_equal 0, Resque.info[:pending], '0 pending jobs, it should be in the delayed queue'
 
     delayed = Resque.delayed_queue_peek(0, 1)
     assert_equal now.to_i + 60, delayed[0], '2nd delay' # the first had a zero delay.

data/test/multiple_failure_test.rb

@@ -0,0 +1,86 @@
+require File.dirname(__FILE__) + '/test_helper'
+
+class MultipleFailureTest < Test::Unit::TestCase
+  def setup
+    Resque.redis.flushall
+    @worker = Resque::Worker.new(:testing)
+    @worker.register_worker
+
+    @old_failure_backend = Resque::Failure.backend
+    MockFailureBackend.errors = []
+    Resque::Failure::MultipleWithRetrySuppression.classes = [MockFailureBackend]
+    Resque::Failure.backend = Resque::Failure::MultipleWithRetrySuppression
+  end
+
+  def failure_key_for(klass)
+    args = []
+    key = "failure_" + klass.redis_retry_key(args)
+  end
+
+  def test_last_failure_is_saved_in_redis
+    Resque.enqueue(LimitThreeJob)
+    perform_next_job(@worker)
+
+    # I don't like this, but...
+    key = failure_key_for(LimitThreeJob)
+    assert Resque.redis.exists(key)
+  end
+
+  def test_last_failure_removed_from_redis_after_error_limit
+    Resque.enqueue(LimitThreeJob)
+    3.times do
+      perform_next_job(@worker)
+    end
+
+    key = failure_key_for(LimitThreeJob)
+    assert Resque.redis.exists(key)
+
+    perform_next_job(@worker)
+    assert !Resque.redis.exists(key)
+  end
+
+  def test_on_success_failure_log_removed_from_redis
+    SwitchToSuccessJob.successful_after = 1
+    Resque.enqueue(SwitchToSuccessJob)
+    perform_next_job(@worker)
+
+    key = failure_key_for(SwitchToSuccessJob)
+    assert Resque.redis.exists(key)
+
+    perform_next_job(@worker)
+    assert !Resque.redis.exists(key), 'key removed on success'
+  ensure
+    SwitchToSuccessJob.reset_defaults
+  end
+
+  def test_errors_are_suppressed_up_to_retry_limit
+    Resque.enqueue(LimitThreeJob)
+    3.times do
+      perform_next_job(@worker)
+    end
+
+    assert_equal 0, MockFailureBackend.errors.size
+  end
+
+  def test_errors_are_logged_after_retry_limit
+    Resque.enqueue(LimitThreeJob)
+    4.times do
+      perform_next_job(@worker)
+    end
+
+    assert_equal 1, MockFailureBackend.errors.size
+  end
+
+  def test_jobs_without_retry_log_errors
+    5.times do
+      Resque.enqueue(NoRetryJob)
+      perform_next_job(@worker)
+    end
+
+    assert_equal 5, MockFailureBackend.errors.size
+  end
+
+  def teardown
+    Resque::Failure.backend = @old_failure_backend
+  end
+end

data/test/test_helper.rb

@@ -5,6 +5,11 @@ $TESTING = true
 require 'test/unit'
 require 'rubygems'
 require 'turn'
+require 'simplecov-html'
+
+SimpleCov.start do
+  add_filter "/test/"
+end
 
 require 'resque-retry'
 require dir + '/test_jobs'
@@ -40,6 +45,19 @@ puts "Starting redis for testing at localhost:9736..."
 `redis-server #{dir}/redis-test.conf`
 Resque.redis = '127.0.0.1:9736'
 
+# Mock failure backend for testing MultipleWithRetrySuppression
+class MockFailureBackend < Resque::Failure::Base
+  class << self
+    attr_accessor :errors
+  end
+
+  def save
+    self.class.errors << exception.to_s
+  end
+
+  self.errors = []
+end
+
 # Test helpers
 class Test::Unit::TestCase
   def perform_next_job(worker, &block)
@@ -57,4 +75,4 @@ class Test::Unit::TestCase
     worker.perform(job)
     worker.done_working
   end
-end
+end

data/test/test_jobs.rb

@@ -2,6 +2,14 @@ CustomException = Class.new(StandardError)
 HierarchyCustomException = Class.new(CustomException)
 AnotherCustomException = Class.new(StandardError)
 
+class NoRetryJob
+  @queue = :testing
+
+  def self.perform(*args)
+    raise "error"
+  end
+end
+
 class GoodJob
   extend Resque::Plugins::Retry
   @queue = :testing
@@ -50,6 +58,15 @@ class NeverGiveUpJob < RetryDefaultsJob
   @retry_limit = 0
 end
 
+class LimitThreeJob < RetryDefaultsJob
+  @queue = :testing
+  @retry_limit = 3
+
+  def self.perform(*args)
+    raise ArgumentError, "custom message"
+  end
+end
+
 class FailFiveTimesJob < RetryDefaultsJob
   @queue = :testing
   @retry_limit = 6
@@ -187,6 +204,32 @@ class InheritOrderingJobExtendFirst
   end
 end
 
+# This job switches to successful after
+# n +tries+.
+class SwitchToSuccessJob < GoodJob
+  @queue = :testing
+  @max_retries = 3
+
+  class << self
+    attr_accessor :successful_after
+    attr_accessor :tries
+
+    def reset_defaults
+      self.tries = 0
+      self.successful_after = 2
+    end
+  end
+
+  reset_defaults
+
+  def self.perform(*args)
+    if self.tries < self.successful_after
+      self.tries += 1
+      raise "error"
+    end
+  end
+end
+
 class InheritOrderingJobExtendLast
   class << self
     attr_accessor :test_value

metadata

@@ -1,12 +1,13 @@
 --- !ruby/object:Gem::Specification 
 name: resque-retry
 version: !ruby/object:Gem::Version 
+  hash: 27
   prerelease: false
   segments: 
   - 0
+  - 1
   - 0
-  - 6
-  version: 0.0.6
+  version: 0.1.0
 platform: ruby
 authors: 
 - Luke Antins
@@ -15,16 +16,18 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2010-07-12 00:00:00 +01:00
+date: 2010-08-29 00:00:00 +01:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency 
   name: resque
   prerelease: false
   requirement: &id001 !ruby/object:Gem::Requirement 
+    none: false
     requirements: 
     - - ">="
       - !ruby/object:Gem::Version 
+        hash: 55
         segments: 
         - 1
         - 8
@@ -36,9 +39,11 @@ dependencies:
   name: resque-scheduler
   prerelease: false
   requirement: &id002 !ruby/object:Gem::Requirement 
+    none: false
     requirements: 
     - - ">="
       - !ruby/object:Gem::Version 
+        hash: 55
         segments: 
         - 1
         - 8
@@ -47,30 +52,64 @@ dependencies:
   type: :runtime
   version_requirements: *id002
 - !ruby/object:Gem::Dependency 
-  name: turn
+  name: test-unit
   prerelease: false
   requirement: &id003 !ruby/object:Gem::Requirement 
+    none: false
     requirements: 
     - - ">="
       - !ruby/object:Gem::Version 
+        hash: 3
         segments: 
         - 0
         version: "0"
   type: :development
   version_requirements: *id003
 - !ruby/object:Gem::Dependency 
-  name: yard
+  name: turn
   prerelease: false
   requirement: &id004 !ruby/object:Gem::Requirement 
+    none: false
     requirements: 
     - - ">="
       - !ruby/object:Gem::Version 
+        hash: 3
         segments: 
         - 0
         version: "0"
   type: :development
   version_requirements: *id004
-description: "  resque-retry provides retry, delay and exponential backoff support for\n  resque jobs.\n\n  Features:\n\n  * Redis backed retry count/limit.\n  * Retry on all or specific exceptions.\n  * Exponential backoff (varying the delay between retrys).\n  * Small & Extendable - plenty of places to override retry logic/settings.\n"
+- !ruby/object:Gem::Dependency 
+  name: yard
+  prerelease: false
+  requirement: &id005 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        hash: 3
+        segments: 
+        - 0
+        version: "0"
+  type: :development
+  version_requirements: *id005
+- !ruby/object:Gem::Dependency 
+  name: simplecov-html
+  prerelease: false
+  requirement: &id006 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        hash: 19
+        segments: 
+        - 0
+        - 3
+        - 0
+        version: 0.3.0
+  type: :development
+  version_requirements: *id006
+description: "  resque-retry provides retry, delay and exponential backoff support for\n  resque jobs.\n\n  Features:\n\n  * Redis backed retry count/limit.\n  * Retry on all or specific exceptions.\n  * Exponential backoff (varying the delay between retrys).\n  * Multiple failure backend with retry suppression & resque-web tab.\n  * Small & Extendable - plenty of places to override retry logic/settings.\n"
 email: luke@lividpenguin.com
 executables: []
 
@@ -84,6 +123,7 @@ files:
 - README.md
 - HISTORY.md
 - test/exponential_backoff_test.rb
+- test/multiple_failure_test.rb
 - test/redis-test.conf
 - test/resque_test.rb
 - test/retry_criteria_test.rb
@@ -91,8 +131,12 @@ files:
 - test/retry_test.rb
 - test/test_helper.rb
 - test/test_jobs.rb
+- lib/resque/failure/multiple_with_retry_suppression.rb
 - lib/resque/plugins/exponential_backoff.rb
 - lib/resque/plugins/retry.rb
+- lib/resque-retry/server/views/retry.erb
+- lib/resque-retry/server/views/retry_timestamp.erb
+- lib/resque-retry/server.rb
 - lib/resque-retry.rb
 has_rdoc: false
 homepage: http://github.com/lantins/resque-retry
@@ -104,23 +148,27 @@ rdoc_options: []
 require_paths: 
 - lib
 required_ruby_version: !ruby/object:Gem::Requirement 
+  none: false
   requirements: 
   - - ">="
     - !ruby/object:Gem::Version 
+      hash: 3
       segments: 
       - 0
       version: "0"
 required_rubygems_version: !ruby/object:Gem::Requirement 
+  none: false
   requirements: 
   - - ">="
     - !ruby/object:Gem::Version 
+      hash: 3
       segments: 
       - 0
       version: "0"
 requirements: []
 
 rubyforge_project: 
-rubygems_version: 1.3.6
+rubygems_version: 1.3.7
 signing_key: 
 specification_version: 3
 summary: A resque plugin; provides retry, delay and exponential backoff support for resque jobs.