kbaum-resque-retry 0.0.5

data/HISTORY.md

@@ -0,0 +1,23 @@
+## 0.0.5 (2010-06-27)
+
+* Handle our own dependancies.
+
+## 0.0.4 (2010-06-16)
+
+* Relax gemspec dependancies.
+
+## 0.0.3 (2010-06-02)
+
+* Bugfix: Make sure that `redis_retry_key` has no whitespace.
+
+## 0.0.2 (2010-05-06)
+
+* Bugfix: Were calling non-existent method to delete redis key.
+* Delay no-longer falls back to `sleep`. resque-scheduler is a required
+  dependancy.
+* Redis key doesn't include ending colon `:` if no args were passed
+  to the job.
+
+## 0.0.1 (2010-04-27)
+
+* First release.

data/LICENSE

@@ -0,0 +1,21 @@
+Copyright (c) 2010 Luke Antins
+Copyright (c) 2010 Ryan Carver
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+Software), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED AS IS, WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,168 @@
+resque-retry
+============
+
+A [Resque][rq] plugin. Requires Resque 1.8.0 & [resque-scheduler][rqs]
+
+resque-retry provides retry, delay and exponential backoff support for
+resque jobs.
+
+### Features
+
+  - Redis backed retry count/limit.
+  - Retry on all or specific exceptions.
+  - Exponential backoff (varying the delay between retrys).
+  - Small & Extendable - plenty of places to override retry logic/settings.
+
+Usage / Examples
+----------------
+
+Just extend your module/class with this module, and your ready to retry!
+
+Customisation is pretty easy, the below examples should give you
+some ideas =), adapt for your own usage and feel free to pick and mix!
+
+### Retry
+
+Retry the job **once** on failure, with zero delay.
+
+    require 'require-retry'
+
+    class DeliverWebHook
+      extend Resque::Plugins::Retry
+      @queue = :web_hooks
+
+      def self.perform(url, hook_id, hmac_key)
+        heavy_lifting
+      end
+    end
+
+When a job runs, the number of retry attempts is checked and incremented
+in Redis. If your job fails, the number of retry attempts is used to
+determine if we can requeue the job for another go.
+
+### Custom Retry
+
+    class DeliverWebHook
+      extend Resque::Plugins::Retry
+      @queue = :web_hooks
+
+      @retry_limit = 10
+      @retry_delay = 120
+
+      def self.perform(url, hook_id, hmac_key)
+        heavy_lifting
+      end
+    end
+
+The above modification will allow your job to retry upto 10 times, with
+a delay of 120 seconds, or 2 minutes between retry attempts.
+
+Alternatively you could override the `retry_delay` method to do something
+more special.
+
+### Exponential Backoff
+
+Use this if you wish to vary the delay between retry attempts:
+
+    class DeliverSMS
+      extend Resque::Plugins::ExponentialBackoff
+      @queue = :mt_messages
+
+      def self.perform(mt_id, mobile_number, message)
+        heavy_lifting
+      end
+    end
+
+**Default Settings**
+
+    key: m = minutes, h = hours
+
+                  no delay, 1m, 10m,   1h,    3h,    6h
+    @backoff_strategy = [0, 60, 600, 3600, 10800, 21600]
+
+The first delay will be 0 seconds, the 2nd will be 60 seconds, etc...
+Again, tweak to your own needs.
+
+The number if retrys is equal to the size of the `backoff_strategy`
+array, unless you set `retry_limit` yourself.
+
+### Retry Specific Exceptions
+
+The default will allow a retry for any type of exception. You may change
+it so only specific exceptions are retried using `retry_exceptions`:
+
+    class DeliverSMS
+      extend Resque::Plugins::Retry
+      @queue = :mt_messages
+
+      @retry_exceptions = [NetworkError]
+
+      def self.perform(mt_id, mobile_number, message)
+        heavy_lifting
+      end
+    end
+
+The above modification will **only** retry if a `NetworkError` (or subclass)
+exception is thrown.
+
+Customise & Extend
+------------------
+
+Please take a look at the yardoc/code for more details on methods you may
+wish to override.
+
+Some things worth noting:
+
+### Job Identifier/Key
+
+The retry attempt is incremented and stored in a Redis key. The key is
+built using the `identifier`. If you have a lot of arguments or really long
+ones, you should consider overriding `identifier` to define a more precise
+or loose custom identifier.
+
+The default identifier is just your job arguments joined with a dash `-`.
+
+By default the key uses this format: 
+`resque-retry:<job class name>:<identifier>`.
+
+Or you can define the entire key by overriding `redis_retry_key`.
+
+    class DeliverSMS
+      extend Resque::Plugins::Retry
+      @queue = :mt_messages
+
+      def self.identifier(mt_id, mobile_number, message)
+        "#{mobile_number}:#{mt_id}"
+      end
+
+      self.perform(mt_id, mobile_number, message)
+        heavy_lifting
+      end
+    end
+
+### Retry Arguments
+
+You may override `args_for_retry`, which is passed the current
+job arguments, to modify the arguments for the next retry attempt.
+
+    class DeliverViaSMSC
+      extend Resque::Plugins::Retry
+      @queue = :mt_smsc_messages
+      
+      # retry using the emergency SMSC.
+      def self.args_for_retry(smsc_id, mt_message)
+        [999, mt_message]
+      end
+
+      self.perform(smsc_id, mt_message)
+        heavy_lifting
+      end
+    end
+
+Install
+-------
+
+    $ gem install resque-retry
+
+[rq]: http://github.com/defunkt/resque
+[rqs]: http://github.com/bvandenbos/resque-scheduler

data/Rakefile

@@ -0,0 +1,25 @@
+$LOAD_PATH.unshift 'lib'
+
+require 'rake/testtask'
+require 'fileutils'
+require 'yard'
+require 'yard/rake/yardoc_task'
+
+task :default => :test
+
+##
+# Test task.
+Rake::TestTask.new(:test) do |task|
+  task.test_files = FileList['test/*_test.rb']
+  task.verbose = true
+end
+
+##
+# docs task.
+YARD::Rake::YardocTask.new :yardoc do |t|
+    t.files   = ['lib/**/*.rb']
+    t.options = ['--output-dir', "doc/",
+                 '--files', 'LICENSE',
+                 '--readme', 'README.md',
+                 '--title', 'resque-retry documentation']
+end

data/lib/resque-retry.rb

@@ -0,0 +1,7 @@
+require 'resque'
+require 'resque_scheduler'
+
+require 'resque/plugins/retry'
+require 'resque/plugins/exponential_backoff'
+require 'resque/plugins/retry_failure_backend'
+require 'resque-retry/server'

data/lib/resque-retry/server.rb

@@ -0,0 +1,34 @@
+# Extend Resque::Server to add tabs
+module ResqueRetry
+
+  module Server
+
+    def self.included(base)
+      base.class_eval do
+
+
+        get "/retry" do
+          # Is there a better way to specify alternate template locations with sinatra?
+          erb File.read(File.join(File.dirname(__FILE__), 'server/views/retry.erb'))
+        end
+
+        get "/retry/:timestamp" do
+          # Is there a better way to specify alternate template locations with sinatra?
+          erb File.read(File.join(File.dirname(__FILE__), 'server/views/retry_timestamp.erb'))
+        end
+
+
+      end
+
+    end
+
+
+  end
+
+end
+
+Resque::Server.tabs << 'Retry'
+
+Resque::Server.class_eval do
+  include ResqueRetry::Server
+end

data/lib/resque-retry/server/views/retry.erb

@@ -0,0 +1,44 @@
+<h1>Jobs Scheduled to Retry</h1>
+
+<p class='intro'>
+  This list below contains the timestamps for scheduled delayed jobs.
+</p>
+
+<p class='sub'>
+  Showing <%= start = params[:start].to_i %> to <%= start + 20 %> of <b><%=size = resque.delayed_queue_schedule_size %></b> timestamps
+</p>
+
+<table>
+  <tr>
+    <th></th>
+    <th>Timestamp</th>
+    <th>Job count</th>
+    <th>Class</th>
+    <th>Args</th>
+    <th>Retry Count</th>
+  </tr>
+  <% resque.delayed_queue_peek(start, start+20).each do |timestamp| %>
+    <tr>
+      <td>
+        <form action="<%= url "/delayed/queue_now" %>" method="post">
+          <input type="hidden" name="timestamp" value="<%= timestamp.to_i %>">
+          <input type="submit" value="Queue now">
+        </form>
+      </td>
+      <td><a href="<%= url "retry/#{timestamp}" %>"><%= format_time(Time.at(timestamp)) %></a></td>
+      <td><%= delayed_timestamp_size = resque.delayed_timestamp_size(timestamp) %></td>
+      <% job = resque.delayed_timestamp_peek(timestamp, 0, 1).first %>
+      <td>
+        <% if job && delayed_timestamp_size == 1 %>
+          <%= h(job['class']) %>
+        <% else %>
+          <a href="<%= url "delayed/#{timestamp}" %>">see details</a>
+        <% end %>
+      </td>
+      <td><%= h(job['args'].inspect) if job && delayed_timestamp_size == 1 %></td>
+      <td><%=Resque.redis.get(Resque.constantize(job['class']).redis_retry_key(job["args"]))%> </td>
+    </tr>
+  <% end %>
+</table>
+
+<%= partial :next_more, :start => start, :size => size %>

data/lib/resque-retry/server/views/retry_timestamp.erb

@@ -0,0 +1,42 @@
+<% timestamp = params[:timestamp].to_i %>
+
+<h1>Delayed jobs scheduled for <%= format_time(Time.at(timestamp)) %></h1>
+
+<p class='sub'>Showing <%= start = params[:start].to_i %> to <%= start + 20 %> of
+  <b><%= size = resque.delayed_timestamp_size(timestamp) %></b> jobs</p>
+
+<table class='jobs'>
+  <tr>
+    <th>Class</th>
+    <th>Args</th>
+    <th>Retry Count</th>
+    <th>Exception</th>
+    <th>Backtrace</th>
+  </tr>
+  <% jobs = resque.delayed_timestamp_peek(timestamp, start, 20) %>
+  <% jobs.each do |job| %>
+      <% retry_key = Resque.constantize(job['class']).redis_retry_key(job["args"]) %>
+      <tr>
+        <td class='class'><%= job['class'] %></td>
+        <td class='args'><%= h job['args'].inspect %></td>
+        <td><%= Resque.redis.get(retry_key) %> </td>
+        <% failure =  Resque.decode(Resque.redis["failure_#{retry_key}"] )%>
+        <td><code><%= failure['exception'] %></code></td>
+        <td class='error'>
+          <% if failure['backtrace'] %>
+              <a href="#" class="backtrace"><%= h(failure['error']) %></a>
+              <pre style='display:none'><%= h failure['backtrace'].join("\n") %></pre>
+          <% else %>
+              <%= h failure['error'] %>
+          <% end %>
+        </td>
+      </tr>
+  <% end %>
+  <% if jobs.empty? %>
+      <tr>
+        <td class='no-data' colspan='2'>There are no pending jobs scheduled for this time.</td>
+      </tr>
+  <% end %>
+</table>
+
+<%= partial :next_more, :start => start, :size => size %>

data/lib/resque/plugins/exponential_backoff.rb

@@ -0,0 +1,68 @@
+module Resque
+  module Plugins
+
+    ##
+    # If you want your job to retry on failure using a varying delay, simply
+    # extend your module/class with this module:
+    #
+    #   class DeliverSMS
+    #     extend Resque::Plugins::ExponentialBackoff
+    #     @queue = :mt_messages
+    #
+    #     def self.perform(mt_id, mobile_number, message)
+    #       heavy_lifting
+    #     end
+    #   end
+    #
+    # Easily do something custom:
+    #
+    #   class DeliverSMS
+    #     extend Resque::Plugins::ExponentialBackoff
+    #     @queue = :mt_messages
+    #
+    #     @retry_limit = 4
+    #
+    #     # retry delay in seconds; [0] => 1st retry, [1] => 2nd..4th retry.
+    #     @backoff_strategy = [0, 60]
+    #
+    #     # used to build redis key, for counting job attempts.
+    #     def self.identifier(mt_id, mobile_number, message)
+    #       "#{mobile_number}:#{mt_id}"
+    #     end
+    #
+    #     self.perform(mt_id, mobile_number, message)
+    #       heavy_lifting
+    #     end
+    #   end
+    #
+    module ExponentialBackoff
+      include Resque::Plugins::Retry
+
+      ##
+      # Defaults to the number of delays in the backoff strategy.
+      #
+      # @return [Number] maximum number of retries
+      def retry_limit
+        @retry_limit ||= backoff_strategy.length
+      end
+
+      ##
+      # Selects the delay from the backoff strategy.
+      #
+      # @return [Number] seconds to delay until the next retry.
+      def retry_delay
+        backoff_strategy[retry_attempt] || backoff_strategy.last
+      end
+
+      ##
+      # @abstract
+      # The backoff strategy is used to vary the delay between retry attempts.
+      # 
+      # @return [Array] array of delays. index = retry attempt
+      def backoff_strategy
+        @backoff_strategy ||= [0, 60, 600, 3600, 10_800, 21_600]
+      end
+    end
+
+  end
+end

data/lib/resque/plugins/retry.rb

@@ -0,0 +1,178 @@
+module Resque
+  module Plugins
+
+    ##
+    # If you want your job to retry on failure, simply extend your module/class
+    # with this module:
+    #
+    #   class DeliverWebHook
+    #     extend Resque::Plugins::Retry # allows 1 retry by default.
+    #     @queue = :web_hooks
+    #
+    #     def self.perform(url, hook_id, hmac_key)
+    #       heavy_lifting
+    #     end
+    #   end
+    #
+    # Easily do something custom:
+    #
+    #   class DeliverWebHook
+    #     extend Resque::Plugins::Retry
+    #     @queue = :web_hooks
+    #
+    #     @retry_limit = 8          # default: 1
+    #     @retry_delay = 60 # default: 0
+    #
+    #     # used to build redis key, for counting job attempts.
+    #     def self.identifier(url, hook_id, hmac_key)
+    #       "#{url}-#{hook_id}"
+    #     end
+    #
+    #     def self.perform(url, hook_id, hmac_key)
+    #       heavy_lifting
+    #     end
+    #   end
+    #
+    module Retry
+      ##
+      # @abstract You may override to implement a custom identifier,
+      #           you should consider doing this if your job arguments
+      #           are many/long or may not cleanly cleanly to strings.
+      #
+      # Builds an identifier using the job arguments. This identifier
+      # is used as part of the redis key.
+      #
+      # @param [Array] args job arguments
+      # @return [String] job identifier
+      def identifier(*args)
+        args_string = args.join('-')
+        args_string.empty? ? nil : args_string
+      end
+
+      ##
+      # Builds the redis key to be used for keeping state of the job
+      # attempts.
+      #
+      # @return [String] redis key
+      def redis_retry_key(*args)
+        ['resque-retry', name, identifier(*args)].compact.join(":").gsub(/\s/, '')
+      end
+
+      ##
+      # Maximum number of retrys we can attempt to successfully perform the job.
+      # A retry limit of 0 or below will retry forever.
+      #
+      # @return [Fixnum]
+      def retry_limit
+        @retry_limit ||= 1
+      end
+
+      ##
+      # Number of retry attempts used to try and perform the job.
+      #
+      # The real value is kept in Redis, it is accessed and incremented using
+      # a before_perform hook.
+      #
+      # @return [Fixnum] number of attempts
+      def retry_attempt
+        @retry_attempt ||= 0
+      end
+
+      ##
+      # @abstract
+      # Number of seconds to delay until the job is retried.
+      # 
+      # @return [Number] number of seconds to delay
+      def retry_delay
+        @retry_delay ||= 0
+      end
+
+      ##
+      # @abstract
+      # Modify the arguments used to retry the job. Use this to do something
+      # other than try the exact same job again.
+      #
+      # @return [Array] new job arguments
+      def args_for_retry(*args)
+        args
+      end
+
+      ##
+      # Convenience method to test whether you may retry on a given exception.
+      #
+      # @return [Boolean]
+      def retry_exception?(exception)
+        return true if retry_exceptions.nil?
+        !! retry_exceptions.any? { |ex| ex >= exception }
+      end
+
+      ##
+      # @abstract
+      # Controls what exceptions may be retried.
+      #
+      # Default: `nil` - this will retry all exceptions.
+      # 
+      # @return [Array, nil]
+      def retry_exceptions
+        @retry_exceptions ||= nil
+      end
+
+      ##
+      # Test if the retry criteria is valid.
+      #
+      # @param [Exception] exception
+      # @param [Array] args job arguments
+      # @return [Boolean]
+      def retry_criteria_valid?(exception, *args)
+        # FIXME: let people extend retry criteria, give them a chance to say no.
+        if retry_limit > 0
+          return false if retry_attempt >= retry_limit
+        end
+        retry_exception?(exception.class)
+      end
+
+      ##
+      # Will retry the job.
+      def try_again(*args)
+        if retry_delay <= 0
+          # If the delay is 0, no point passing it through the scheduler
+          Resque.enqueue(self, *args_for_retry(*args))
+        else
+          Resque.enqueue_in(retry_delay, self, *args_for_retry(*args))
+        end
+      end
+
+      ##
+      # Resque before_perform hook.
+      #
+      # Increments and sets the `@retry_attempt` count.
+      def before_perform_retry(*args)
+        retry_key = redis_retry_key(*args)
+        Resque.redis.setnx(retry_key, -1)             # default to -1 if not set.
+        @retry_attempt = Resque.redis.incr(retry_key) # increment by 1.
+      end
+
+      ##
+      # Resque after_perform hook.
+      #
+      # Deletes retry attempt count from Redis.
+      def after_perform_retry(*args)
+        Resque.redis.del(redis_retry_key(*args))
+      end
+
+      ##
+      # Resque on_failure hook.
+      #
+      # Checks if our retry criteria is valid, if it is we try again.
+      # Otherwise the retry attempt count is deleted from Redis.
+      def on_failure_retry(exception, *args)
+        if retry_criteria_valid?(exception, *args)
+          try_again(*args)
+        else
+          Resque.redis.del(redis_retry_key(*args))
+        end
+      end
+    end
+
+  end
+end

data/lib/resque/plugins/retry_failure_backend.rb

@@ -0,0 +1,42 @@
+require 'resque/failure/multiple'
+
+class RetryFailureBackend < Resque::Failure::Multiple
+
+  include Resque::Helpers
+
+  def save
+    unless retrying?
+      super
+    else
+      data = {
+              :failed_at => Time.now.strftime("%Y/%m/%d %H:%M:%S"),
+              :payload   => payload,
+              :exception => exception.class.to_s,
+              :error     => exception.to_s,
+              :backtrace => Array(exception.backtrace),
+              :worker    => worker.to_s,
+              :queue     => queue
+      }
+      data = Resque.encode(data)
+      Resque.redis[failure_key]=data
+    end
+  end
+
+  protected
+
+  def retrying?
+    Resque.redis.get(retry_key)
+  end
+
+  def failure_key
+    "failure_#{retry_key}"
+  end
+
+  def retry_key
+    klass.redis_retry_key(payload["args"])
+  end
+
+  def klass
+    constantize(payload["class"])
+  end
+end

data/test/exponential_backoff_test.rb

@@ -0,0 +1,59 @@
+require File.dirname(__FILE__) + '/test_helper'
+
+class ExponentialBackoffTest < Test::Unit::TestCase
+  def setup
+    Resque.redis.flushall
+    @worker = Resque::Worker.new(:testing)
+    @worker.register_worker
+  end
+
+  def test_resque_plugin_lint
+    assert_nothing_raised do
+      Resque::Plugin.lint(Resque::Plugins::ExponentialBackoff)
+    end
+  end
+
+  def test_default_backoff_strategy
+    now = Time.now
+    Resque.enqueue(ExponentialBackoffJob)
+    2.times do
+      perform_next_job @worker
+    end
+
+    assert_equal 2, Resque.info[:processed], 'processed jobs'
+    assert_equal 2, Resque.info[:failed], 'failed jobs'
+    assert_equal 0, Resque.info[:pending], 'pending jobs'
+
+    delayed = Resque.delayed_queue_peek(0, 1)
+    assert_equal now.to_i + 60, delayed[0], '2nd delay' # the first had a zero delay.
+
+    5.times do
+      Resque.enqueue(ExponentialBackoffJob)
+      perform_next_job @worker
+    end
+
+    delayed = Resque.delayed_queue_peek(0, 5)
+    assert_equal now.to_i + 600, delayed[1], '3rd delay'
+    assert_equal now.to_i + 3600, delayed[2], '4th delay'
+    assert_equal now.to_i + 10_800, delayed[3], '5th delay'
+    assert_equal now.to_i + 21_600, delayed[4], '6th delay'
+  end
+  
+  def test_custom_backoff_strategy
+    now = Time.now
+    4.times do
+      Resque.enqueue(CustomExponentialBackoffJob, 'http://lividpenguin.com', 1305, 'cd8079192d379dc612f17c660591a6cfb05f1dda')
+      perform_next_job @worker
+    end
+    
+    delayed = Resque.delayed_queue_peek(0, 3)
+    assert_equal now.to_i + 10, delayed[0], '1st delay'
+    assert_equal now.to_i + 20, delayed[1], '2nd delay'
+    assert_equal now.to_i + 30, delayed[2], '3rd delay'
+    assert_equal 2, Resque.delayed_timestamp_size(delayed[2]), '4th delay should share delay with 3rd'
+    
+    assert_equal 4, Resque.info[:processed], 'processed jobs'
+    assert_equal 4, Resque.info[:failed], 'failed jobs'
+    assert_equal 0, Resque.info[:pending], 'pending jobs'
+  end
+end

data/test/redis-test.conf

@@ -0,0 +1,132 @@
+# Redis configuration file example
+
+# By default Redis does not run as a daemon. Use 'yes' if you need it.
+# Note that Redis will write a pid file in /var/run/redis.pid when daemonized.
+daemonize yes
+
+# When run as a daemon, Redis write a pid file in /var/run/redis.pid by default.
+# You can specify a custom pid file location here.
+pidfile ./test/redis-test.pid
+
+# Accept connections on the specified port, default is 6379
+port 9736
+
+# If you want you can bind a single interface, if the bind option is not
+# specified all the interfaces will listen for connections.
+#
+# bind 127.0.0.1
+
+# Close the connection after a client is idle for N seconds (0 to disable)
+timeout 300
+
+# Save the DB on disk:
+#
+#   save <seconds> <changes>
+#
+#   Will save the DB if both the given number of seconds and the given
+#   number of write operations against the DB occurred.
+#
+#   In the example below the behaviour will be to save:
+#   after 900 sec (15 min) if at least 1 key changed
+#   after 300 sec (5 min) if at least 10 keys changed
+#   after 60 sec if at least 10000 keys changed
+save 900 1
+save 300 10
+save 60 10000
+
+# The filename where to dump the DB
+dbfilename dump.rdb
+
+# For default save/load DB in/from the working directory
+# Note that you must specify a directory not a file name.
+dir ./test/
+
+# Set server verbosity to 'debug'
+# it can be one of:
+# debug (a lot of information, useful for development/testing)
+# notice (moderately verbose, what you want in production probably)
+# warning (only very important / critical messages are logged)
+loglevel debug
+
+# Specify the log file name. Also 'stdout' can be used to force
+# the demon to log on the standard output. Note that if you use standard
+# output for logging but daemonize, logs will be sent to /dev/null
+logfile stdout
+
+# Set the number of databases. The default database is DB 0, you can select
+# a different one on a per-connection basis using SELECT <dbid> where
+# dbid is a number between 0 and 'databases'-1
+databases 16
+
+################################# REPLICATION #################################
+
+# Master-Slave replication. Use slaveof to make a Redis instance a copy of
+# another Redis server. Note that the configuration is local to the slave
+# so for example it is possible to configure the slave to save the DB with a
+# different interval, or to listen to another port, and so on.
+
+# slaveof <masterip> <masterport>
+
+################################## SECURITY ###################################
+
+# Require clients to issue AUTH <PASSWORD> before processing any other
+# commands.  This might be useful in environments in which you do not trust
+# others with access to the host running redis-server.
+#
+# This should stay commented out for backward compatibility and because most
+# people do not need auth (e.g. they run their own servers).
+
+# requirepass foobared
+
+################################### LIMITS ####################################
+
+# Set the max number of connected clients at the same time. By default there
+# is no limit, and it's up to the number of file descriptors the Redis process
+# is able to open. The special value '0' means no limts.
+# Once the limit is reached Redis will close all the new connections sending
+# an error 'max number of clients reached'.
+
+# maxclients 128
+
+# Don't use more memory than the specified amount of bytes.
+# When the memory limit is reached Redis will try to remove keys with an
+# EXPIRE set. It will try to start freeing keys that are going to expire
+# in little time and preserve keys with a longer time to live.
+# Redis will also try to remove objects from free lists if possible.
+#
+# If all this fails, Redis will start to reply with errors to commands
+# that will use more memory, like SET, LPUSH, and so on, and will continue
+# to reply to most read-only commands like GET.
+#
+# WARNING: maxmemory can be a good idea mainly if you want to use Redis as a
+# 'state' server or cache, not as a real DB. When Redis is used as a real
+# database the memory usage will grow over the weeks, it will be obvious if
+# it is going to use too much memory in the long run, and you'll have the time
+# to upgrade. With maxmemory after the limit is reached you'll start to get
+# errors for write operations, and this may even lead to DB inconsistency.
+
+# maxmemory <bytes>
+
+############################### ADVANCED CONFIG ###############################
+
+# Glue small output buffers together in order to send small replies in a
+# single TCP packet. Uses a bit more CPU but most of the times it is a win
+# in terms of number of queries per second. Use 'yes' if unsure.
+glueoutputbuf yes
+
+# Use object sharing. Can save a lot of memory if you have many common
+# string in your dataset, but performs lookups against the shared objects
+# pool so it uses more CPU and can be a bit slower. Usually it's a good
+# idea.
+#
+# When object sharing is enabled (shareobjects yes) you can use
+# shareobjectspoolsize to control the size of the pool used in order to try
+# object sharing. A bigger pool size will lead to better sharing capabilities.
+# In general you want this value to be at least the double of the number of
+# very common strings you have in your dataset.
+#
+# WARNING: object sharing is experimental, don't enable this feature
+# in production before of Redis 1.0-stable. Still please try this feature in
+# your development environment so that we can test it better.
+# shareobjects no
+# shareobjectspoolsize 1024

data/test/resque_test.rb

@@ -0,0 +1,18 @@
+require File.dirname(__FILE__) + '/test_helper'
+
+# make sure the worlds not fallen from beneith us.
+class ResqueTest < Test::Unit::TestCase
+  def test_resque_version
+    major, minor, patch = Resque::Version.split('.')
+    assert_equal 1, major.to_i, 'major version does not match'
+    assert_operator minor.to_i, :>=, 8, 'minor version is too low'
+  end
+
+  def test_good_job
+    clean_perform_job(GoodJob, 1234, { :cats => :maiow }, [true, false, false])
+
+    assert_equal 0, Resque.info[:failed], 'failed jobs'
+    assert_equal 1, Resque.info[:processed], 'processed job'
+    assert_equal 0, Resque.delayed_queue_schedule_size
+  end
+end

data/test/retry_test.rb

@@ -0,0 +1,145 @@
+require File.dirname(__FILE__) + '/test_helper'
+
+class RetryTest < Test::Unit::TestCase
+  def setup
+    Resque.redis.flushall
+    @worker = Resque::Worker.new(:testing)
+    @worker.register_worker
+  end
+
+  def test_resque_plugin_lint
+    assert_nothing_raised do
+      Resque::Plugin.lint(Resque::Plugins::Retry)
+    end
+  end
+
+  def test_default_settings
+    assert_equal 1, RetryDefaultsJob.retry_limit, 'default retry limit'
+    assert_equal 0, RetryDefaultsJob.retry_attempt, 'default number of retry attempts'
+    assert_equal nil, RetryDefaultsJob.retry_exceptions, 'default retry exceptions; nil = any'
+    assert_equal 0, RetryDefaultsJob.retry_delay, 'default seconds until retry'
+  end
+
+  def test_retry_once_by_default
+    Resque.enqueue(RetryDefaultsJob)
+    3.times do
+      perform_next_job(@worker)
+    end
+
+    assert_equal 0, Resque.info[:pending], 'pending jobs'
+    assert_equal 2, Resque.info[:failed], 'failed jobs'
+    assert_equal 2, Resque.info[:processed], 'processed job'
+  end
+
+  def test_job_args_are_maintained
+    test_args = ['maiow', 'cat', [42, 84]]
+
+    Resque.enqueue(RetryDefaultsJob, *test_args)
+    perform_next_job(@worker)
+
+    assert job = Resque.pop(:testing)
+    assert_equal test_args, job['args']
+  end
+
+  def test_job_args_may_be_modified
+    Resque.enqueue(RetryWithModifiedArgsJob, 'foo', 'bar')
+    perform_next_job(@worker)
+
+    assert job = Resque.pop(:testing)
+    assert_equal ['foobar', 'barbar'], job['args']
+  end
+
+  def test_retry_never_give_up
+    Resque.enqueue(NeverGiveUpJob)
+    10.times do
+      perform_next_job(@worker)
+    end
+
+    assert_equal 1, Resque.info[:pending], 'pending jobs'
+    assert_equal 10, Resque.info[:failed], 'failed jobs'
+    assert_equal 10, Resque.info[:processed], 'processed job'
+  end
+
+  def test_fail_five_times_then_succeed
+    Resque.enqueue(FailFiveTimesJob)
+    7.times do
+      perform_next_job(@worker)
+    end
+
+    assert_equal 5, Resque.info[:failed], 'failed jobs'
+    assert_equal 6, Resque.info[:processed], 'processed job'
+    assert_equal 0, Resque.info[:pending], 'pending jobs'
+  end
+
+  def test_can_determine_if_exception_may_be_retried
+    assert_equal true, RetryDefaultsJob.retry_exception?(StandardError), 'StandardError may retry'
+    assert_equal true, RetryDefaultsJob.retry_exception?(CustomException), 'CustomException may retry'
+    assert_equal true, RetryDefaultsJob.retry_exception?(HierarchyCustomException), 'HierarchyCustomException may retry'
+
+    assert_equal true, RetryCustomExceptionsJob.retry_exception?(CustomException), 'CustomException may retry'
+    assert_equal true, RetryCustomExceptionsJob.retry_exception?(HierarchyCustomException), 'HierarchyCustomException may retry'
+    assert_equal false, RetryCustomExceptionsJob.retry_exception?(AnotherCustomException), 'AnotherCustomException may not retry'
+  end
+
+  def test_retry_if_failed_and_exception_may_retry
+    Resque.enqueue(RetryCustomExceptionsJob, CustomException)
+    Resque.enqueue(RetryCustomExceptionsJob, HierarchyCustomException)
+    4.times do
+      perform_next_job(@worker)
+    end
+
+    assert_equal 4, Resque.info[:failed], 'failed jobs'
+    assert_equal 4, Resque.info[:processed], 'processed job'
+    assert_equal 2, Resque.info[:pending], 'pending jobs'
+  end
+
+  def test_do_not_retry_if_failed_and_exception_does_not_allow_retry
+    Resque.enqueue(RetryCustomExceptionsJob, AnotherCustomException)
+    Resque.enqueue(RetryCustomExceptionsJob, RuntimeError)
+    4.times do
+      perform_next_job(@worker)
+    end
+
+    assert_equal 2, Resque.info[:failed], 'failed jobs'
+    assert_equal 2, Resque.info[:processed], 'processed job'
+    assert_equal 0, Resque.info[:pending], 'pending jobs'
+  end
+
+  def test_delete_redis_key_when_job_is_successful
+    Resque.enqueue(GoodJob, 'arg1')
+
+    assert_equal nil, Resque.redis.get(GoodJob.redis_retry_key('arg1'))
+    perform_next_job(@worker)
+    assert_equal nil, Resque.redis.get(GoodJob.redis_retry_key('arg1'))
+  end
+
+  def test_delete_redis_key_after_final_failed_retry
+    Resque.enqueue(FailFiveTimesJob, 'yarrrr')
+    assert_equal nil, Resque.redis.get(FailFiveTimesJob.redis_retry_key('yarrrr'))
+
+    perform_next_job(@worker)
+    assert_equal '0', Resque.redis.get(FailFiveTimesJob.redis_retry_key('yarrrr'))
+
+    perform_next_job(@worker)
+    assert_equal '1', Resque.redis.get(FailFiveTimesJob.redis_retry_key('yarrrr'))
+
+    5.times do
+      perform_next_job(@worker)
+    end
+    assert_equal nil, Resque.redis.get(FailFiveTimesJob.redis_retry_key('yarrrr'))
+
+    assert_equal 5, Resque.info[:failed], 'failed jobs'
+    assert_equal 6, Resque.info[:processed], 'processed job'
+    assert_equal 0, Resque.info[:pending], 'pending jobs'
+  end
+
+  def test_job_without_args_has_no_ending_colon_in_redis_key
+    assert_equal 'resque-retry:GoodJob:yarrrr', GoodJob.redis_retry_key('yarrrr')
+    assert_equal 'resque-retry:GoodJob:foo', GoodJob.redis_retry_key('foo')
+    assert_equal 'resque-retry:GoodJob', GoodJob.redis_retry_key
+  end
+
+  def test_redis_retry_key_removes_whitespace
+    assert_equal 'resque-retry:GoodJob:arg1-removespace', GoodJob.redis_retry_key('arg1', 'remove space')
+  end
+end

data/test/test_helper.rb

@@ -0,0 +1,63 @@
+dir = File.dirname(File.expand_path(__FILE__))
+$LOAD_PATH.unshift dir + '/../lib'
+$TESTING = true
+
+require 'test/unit'
+require 'rubygems'
+require 'turn'
+
+require 'resque-retry'
+require dir + '/test_jobs'
+
+
+##
+# make sure we can run redis
+if !system("which redis-server")
+  puts '', "** can't find `redis-server` in your path"
+  puts "** try running `sudo rake install`"
+  abort ''
+end
+
+
+##
+# start our own redis when the tests start,
+# kill it when they end
+at_exit do
+  next if $!
+
+  if defined?(MiniTest)
+    exit_code = MiniTest::Unit.new.run(ARGV)
+  else
+    exit_code = Test::Unit::AutoRunner.run
+  end
+
+  pid = `ps -e -o pid,command | grep [r]edis-test`.split(" ")[0]
+  puts "Killing test redis server..."
+  `rm -f #{dir}/dump.rdb`
+  Process.kill("KILL", pid.to_i)
+  exit exit_code
+end
+
+puts "Starting redis for testing at localhost:9736..."
+`redis-server #{dir}/redis-test.conf`
+Resque.redis = '127.0.0.1:9736'
+
+##
+# Test helpers
+class Test::Unit::TestCase
+  def perform_next_job(worker, &block)
+    return unless job = @worker.reserve
+    @worker.perform(job, &block)
+    @worker.done_working
+  end
+
+  def clean_perform_job(klass, *args)
+    Resque.redis.flushall
+    Resque.enqueue(klass, *args)
+
+    worker = Resque::Worker.new(:testing)
+    return false unless job = worker.reserve
+    worker.perform(job)
+    worker.done_working
+  end
+end

data/test/test_jobs.rb

@@ -0,0 +1,74 @@
+CustomException = Class.new(StandardError)
+HierarchyCustomException = Class.new(CustomException)
+AnotherCustomException = Class.new(StandardError)
+
+class GoodJob
+  extend Resque::Plugins::Retry
+  @queue = :testing
+  def self.perform(*args)
+  end
+end
+
+class RetryDefaultsJob
+  extend Resque::Plugins::Retry
+  @queue = :testing
+
+  def self.perform(*args)
+    raise
+  end
+end
+
+class RetryWithModifiedArgsJob < RetryDefaultsJob
+  @queue = :testing
+
+  def self.args_for_retry(*args)
+    args.each { |arg| arg << 'bar' }
+  end
+end
+
+class NeverGiveUpJob < RetryDefaultsJob
+  @queue = :testing
+  @retry_limit = 0
+end
+
+class FailFiveTimesJob < RetryDefaultsJob
+  @queue = :testing
+  @retry_limit = 6
+
+  def self.perform(*args)
+    raise if retry_attempt <= 4
+  end
+end
+
+class ExponentialBackoffJob < RetryDefaultsJob
+  extend Resque::Plugins::ExponentialBackoff
+  @queue = :testing
+end
+
+class CustomExponentialBackoffJob
+  extend Resque::Plugins::ExponentialBackoff
+  @queue = :testing
+  
+  @retry_limit = 4
+  @backoff_strategy = [10, 20, 30]
+  
+  def self.perform(url, hook_id, hmac_key)
+    raise
+  end
+end
+
+class RetryCustomExceptionsJob < RetryDefaultsJob
+  @queue = :testing
+
+  @retry_limit = 5
+  @retry_exceptions = [CustomException, HierarchyCustomException]
+
+  def self.perform(exception)
+    case exception
+    when 'CustomException' then raise CustomException
+    when 'HierarchyCustomException' then raise HierarchyCustomException
+    when 'AnotherCustomException' then raise AnotherCustomException
+    else raise StandardError
+    end
+  end
+end

metadata

@@ -0,0 +1,143 @@
+--- !ruby/object:Gem::Specification 
+name: kbaum-resque-retry
+version: !ruby/object:Gem::Version 
+  hash: 21
+  prerelease: false
+  segments: 
+  - 0
+  - 0
+  - 5
+  version: 0.0.5
+platform: ruby
+authors: 
+- Luke Antins
+- Ryan Carver
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2010-07-11 00:00:00 -04:00
+default_executable: 
+dependencies: 
+- !ruby/object:Gem::Dependency 
+  name: resque
+  prerelease: false
+  requirement: &id001 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        hash: 55
+        segments: 
+        - 1
+        - 8
+        - 0
+        version: 1.8.0
+  type: :runtime
+  version_requirements: *id001
+- !ruby/object:Gem::Dependency 
+  name: resque-scheduler
+  prerelease: false
+  requirement: &id002 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        hash: 55
+        segments: 
+        - 1
+        - 8
+        - 0
+        version: 1.8.0
+  type: :runtime
+  version_requirements: *id002
+- !ruby/object:Gem::Dependency 
+  name: turn
+  prerelease: false
+  requirement: &id003 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        hash: 3
+        segments: 
+        - 0
+        version: "0"
+  type: :development
+  version_requirements: *id003
+- !ruby/object:Gem::Dependency 
+  name: yard
+  prerelease: false
+  requirement: &id004 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        hash: 3
+        segments: 
+        - 0
+        version: "0"
+  type: :development
+  version_requirements: *id004
+description: "  resque-retry provides retry, delay and exponential backoff support for\n  resque jobs.\n\n  Features:\n\n  * Redis backed retry count/limit.\n  * Retry on all or specific exceptions.\n  * Exponential backoff (varying the delay between retrys).\n  * Small & Extendable - plenty of places to override retry logic/settings.\n"
+email: luke@lividpenguin.com
+executables: []
+
+extensions: []
+
+extra_rdoc_files: []
+
+files: 
+- LICENSE
+- Rakefile
+- README.md
+- HISTORY.md
+- test/exponential_backoff_test.rb
+- test/redis-test.conf
+- test/resque_test.rb
+- test/retry_test.rb
+- test/test_helper.rb
+- test/test_jobs.rb
+- lib/resque/plugins/exponential_backoff.rb
+- lib/resque/plugins/retry.rb
+- lib/resque/plugins/retry_failure_backend.rb
+- lib/resque-retry/server/views/retry.erb
+- lib/resque-retry/server/views/retry_timestamp.erb
+- lib/resque-retry/server.rb
+- lib/resque-retry.rb
+has_rdoc: true
+homepage: http://github.com/lantins/resque-retry
+licenses: []
+
+post_install_message: 
+rdoc_options: []
+
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      hash: 3
+      segments: 
+      - 0
+      version: "0"
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      hash: 3
+      segments: 
+      - 0
+      version: "0"
+requirements: []
+
+rubyforge_project: 
+rubygems_version: 1.3.7
+signing_key: 
+specification_version: 3
+summary: A resque plugin; provides retry, delay and exponential backoff support for resque jobs.
+test_files: []
+