onyx-resque-retry 0.1.0

data/HISTORY.md

@@ -0,0 +1,33 @@
+## 0.1.0 (2010-08-29)
+
+* Feature: Multiple failure backend with retry suppression.
+* Feature: resque-web tab showing retry information.
+* Improved README documentation, added a 'Quick Start' section.
+
+## 0.0.6 (2010-07-12)
+
+* Feature: Added support for custom retry criteria check callbacks.
+
+## 0.0.5 (2010-06-27)
+
+* Handle our own dependancies.
+
+## 0.0.4 (2010-06-16)
+
+* Relax gemspec dependancies.
+
+## 0.0.3 (2010-06-02)
+
+* Bugfix: Make sure that `redis_retry_key` has no whitespace.
+
+## 0.0.2 (2010-05-06)
+
+* Bugfix: Were calling non-existent method to delete redis key.
+* Delay no-longer falls back to `sleep`. resque-scheduler is a required
+  dependancy.
+* Redis key doesn't include ending colon `:` if no args were passed
+  to the job.
+
+## 0.0.1 (2010-04-27)
+
+* First release.

data/LICENSE

@@ -0,0 +1,21 @@
+Copyright (c) 2010 Luke Antins
+Copyright (c) 2010 Ryan Carver
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+Software), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED AS IS, WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,289 @@
+resque-retry
+============
+
+A [Resque][rq] plugin. Requires Resque 1.8.0 & [resque-scheduler][rqs].
+
+resque-retry provides retry, delay and exponential backoff support for
+resque jobs.
+
+  * Redis backed retry count/limit.
+  * Retry on all or specific exceptions.
+  * Exponential backoff (varying the delay between retrys).
+  * Multiple failure backend with retry suppression & resque-web tab.
+  * Small & Extendable - plenty of places to override retry logic/settings.
+
+Install & Quick Start
+---------------------
+
+To install:
+
+    $ gem install resque-retry
+
+Add this to your `Rakefile`:
+
+    require 'resque/tasks'
+    require 'resque_scheduler/tasks'
+
+The delay between retry attempts is provided by [resque-scheduler][rqs].
+You'll want to run the scheduler process, otherwise delayed retry attempts
+will never perform:
+
+    $ rake resque:scheduler
+
+Use the plugin:
+
+    require 'resque-retry'
+
+    class ExampleRetryJob
+      extend Resque::Plugins::Retry
+      @queue = :example_queue
+
+      @retry_limit = 3
+      @retry_delay = 60
+
+      def self.perform(*args)
+        # your magic/heavy lifting goes here.
+      end
+    end
+
+Then start up a resque worker as normal:
+
+    $ QUEUE=* rake resque:work
+
+Now if you ExampleRetryJob fails, it will be retried 3 times, with a 60 second
+delay between attempts.
+
+For more explanation and examples, please see the remaining documentation.
+
+Failure Backend & Resque Web Additions
+--------------------------------------
+
+Lets say your using the Redis failure backend of resque (the default).
+Every time a job fails, the failure queue is populated with the job and
+exception details.
+
+Normally this is useful, but if your jobs retry... it can cause a bit of a mess.
+
+For example: given a job that retried 4 times before completing successful.
+You'll have a lot of failures for the same job and you wont be sure if it
+actually completed successfully just by just using the resque-web interface.
+
+### Failure Backend
+
+`MultipleWithRetrySuppression` is a multiple failure backend, with retry suppression.
+
+Here's an example, using the Redis failure backend:
+
+    require 'resque-retry'
+    require 'resque/failure/redis'
+
+    # require your jobs & application code.
+
+    Resque::Failure::MultipleWithRetrySuppression.classes = [Resque::Failure::Redis]
+    Resque::Failure.backend = Resque::Failure::MultipleWithRetrySuppression
+
+If a job fails, but **can and will** retry, the failure details wont be
+logged in the Redis failed queue *(visible via resque-web)*.
+
+If the job fails, but **can't or won't** retry, the failure will be logged in
+the Redis failed queue, like a normal failure *(without retry)* would.
+
+### Resque Web Additions
+
+If your using the `MultipleWithRetrySuppression` failure backend, you should
+also checkout the resque-web additions!
+
+The new Retry tab displays delayed jobs with retry information; the number of
+attempts and the exception details from the last failure.
+
+Make sure you include this in your `config.ru` or similar file:
+
+    require 'resque-retry'
+    require 'resque-retry/server'
+
+    # require your jobs & application code.
+
+    run Resque::Server.new
+
+Retry Options & Logic
+---------------------
+
+Please take a look at the yardoc/code for more details on methods you may
+wish to override.
+
+Customisation is pretty easy, the below examples should give you
+some ideas =), adapt for your own usage and feel free to pick and mix!
+
+### Retry Defaults
+
+Retry the job **once** on failure, with zero delay.
+
+    require 'resque-retry'
+
+    class DeliverWebHook
+      extend Resque::Plugins::Retry
+      @queue = :web_hooks
+
+      def self.perform(url, hook_id, hmac_key)
+        heavy_lifting
+      end
+    end
+
+When a job runs, the number of retry attempts is checked and incremented
+in Redis. If your job fails, the number of retry attempts is used to
+determine if we can requeue the job for another go.
+
+### Custom Retry
+
+    class DeliverWebHook
+      extend Resque::Plugins::Retry
+      @queue = :web_hooks
+
+      @retry_limit = 10
+      @retry_delay = 120
+
+      def self.perform(url, hook_id, hmac_key)
+        heavy_lifting
+      end
+    end
+
+The above modification will allow your job to retry upto 10 times, with
+a delay of 120 seconds, or 2 minutes between retry attempts.
+
+Alternatively you could override the `retry_delay` method to do something
+more special.
+
+### Exponential Backoff
+
+Use this if you wish to vary the delay between retry attempts:
+
+    class DeliverSMS
+      extend Resque::Plugins::ExponentialBackoff
+      @queue = :mt_messages
+
+      def self.perform(mt_id, mobile_number, message)
+        heavy_lifting
+      end
+    end
+
+**Default Settings**
+
+    key: m = minutes, h = hours
+
+                  no delay, 1m, 10m,   1h,    3h,    6h
+    @backoff_strategy = [0, 60, 600, 3600, 10800, 21600]
+
+The first delay will be 0 seconds, the 2nd will be 60 seconds, etc...
+Again, tweak to your own needs.
+
+The number if retrys is equal to the size of the `backoff_strategy`
+array, unless you set `retry_limit` yourself.
+
+### Retry Specific Exceptions
+
+The default will allow a retry for any type of exception. You may change
+it so only specific exceptions are retried using `retry_exceptions`:
+
+    class DeliverSMS
+      extend Resque::Plugins::Retry
+      @queue = :mt_messages
+
+      @retry_exceptions = [NetworkError]
+
+      def self.perform(mt_id, mobile_number, message)
+        heavy_lifting
+      end
+    end
+
+The above modification will **only** retry if a `NetworkError` (or subclass)
+exception is thrown.
+
+### Custom Retry Criteria Check Callbacks
+
+You may define custom retry criteria callbacks:
+
+    class TurkWorker
+      extend Resque::Plugins::Retry
+      @queue = :turk_job_processor
+
+      @retry_exceptions = [NetworkError]
+
+      retry_criteria_check do |exception, *args|
+        if exception.message =~ /InvalidJobId/
+          false # don't retry if we got passed a invalid job id.
+        else
+          true  # its okay for a retry attempt to continue.
+        end
+      end
+
+      def self.perform(job_id)
+        heavy_lifting
+      end
+    end
+
+Similar to the previous example, this job will retry if either a
+`NetworkError` (or subclass) exception is thrown **or** any of the callbacks
+return true.
+
+Use `@retry_exceptions = []` to **only** use callbacks, to determine if the
+job should retry.
+
+### Retry Arguments
+
+You may override `args_for_retry`, which is passed the current
+job arguments, to modify the arguments for the next retry attempt.
+
+    class DeliverViaSMSC
+      extend Resque::Plugins::Retry
+      @queue = :mt_smsc_messages
+
+      # retry using the emergency SMSC.
+      def self.args_for_retry(smsc_id, mt_message)
+        [999, mt_message]
+      end
+
+      self.perform(smsc_id, mt_message)
+        heavy_lifting
+      end
+    end
+
+### Job Identifier/Key
+
+The retry attempt is incremented and stored in a Redis key. The key is
+built using the `identifier`. If you have a lot of arguments or really long
+ones, you should consider overriding `identifier` to define a more precise
+or loose custom identifier.
+
+The default identifier is just your job arguments joined with a dash `-`.
+
+By default the key uses this format: 
+`resque-retry:<job class name>:<identifier>`.
+
+Or you can define the entire key by overriding `redis_retry_key`.
+
+    class DeliverSMS
+      extend Resque::Plugins::Retry
+      @queue = :mt_messages
+
+      def self.identifier(mt_id, mobile_number, message)
+        "#{mobile_number}:#{mt_id}"
+      end
+
+      self.perform(mt_id, mobile_number, message)
+        heavy_lifting
+      end
+    end
+
+Contributing/Pull Requests
+--------------------------
+
+  * Yes please!
+  * Fork the project.
+  * Make your feature addition or bug fix.
+  * Add tests for it.
+  * Commit.
+  * Send me a pull request. Bonus points for topic branches.
+  * If you edit the gemspec/version etc, do it in another commit please.
+
+[rq]: http://github.com/defunkt/resque
+[rqs]: http://github.com/bvandenbos/resque-scheduler

data/Rakefile

@@ -0,0 +1,25 @@
+$LOAD_PATH.unshift 'lib'
+
+require 'rake/testtask'
+require 'fileutils'
+require 'yard'
+require 'yard/rake/yardoc_task'
+
+task :default => :test
+
+##
+# Test task.
+Rake::TestTask.new(:test) do |task|
+  task.test_files = FileList['test/*_test.rb']
+  task.verbose = true
+end
+
+##
+# docs task.
+YARD::Rake::YardocTask.new :yardoc do |t|
+    t.files   = ['lib/**/*.rb']
+    t.options = ['--output-dir', "doc/",
+                 '--files', 'LICENSE',
+                 '--readme', 'README.md',
+                 '--title', 'resque-retry documentation']
+end

data/lib/resque-retry.rb

@@ -0,0 +1,6 @@
+require 'resque'
+require 'resque_scheduler'
+
+require 'resque/plugins/retry'
+require 'resque/plugins/exponential_backoff'
+require 'resque/failure/multiple_with_retry_suppression'

data/lib/resque-retry/server.rb

@@ -0,0 +1,51 @@
+# Extend Resque::Server to add tabs.
+module ResqueRetry
+  module Server
+
+    def self.included(base)
+      base.class_eval {
+        helpers do
+          # builds a retry key for the specified job.
+          def retry_key_for_job(job)
+            klass = Resque.constantize(job['class'])
+            if klass.respond_to?(:redis_retry_key)
+              klass.redis_retry_key(job['args'])
+            else
+              nil
+            end
+          end
+
+          # gets the number of retry attempts for a job.
+          def retry_attempts_for_job(job)
+            Resque.redis.get(retry_key_for_job(job))
+          end
+          
+          # gets the failure details hash for a job.
+          def retry_failure_details(retry_key)
+            Resque.decode(Resque.redis["failure_#{retry_key}"])
+          end
+
+          # reads a 'local' template file.
+          def local_template(path)
+            # Is there a better way to specify alternate template locations with sinatra?
+            File.read(File.join(File.dirname(__FILE__), "server/views/#{path}"))
+          end
+        end
+
+        get '/retry' do
+          erb local_template('retry.erb')
+        end
+
+        get '/retry/:timestamp' do
+          erb local_template('retry_timestamp.erb')
+        end
+      }
+    end
+
+  end
+end
+
+Resque::Server.tabs << 'Retry'
+Resque::Server.class_eval do
+  include ResqueRetry::Server
+end

data/lib/resque-retry/server/views/retry.erb

@@ -0,0 +1,48 @@
+<h1>Delayed Jobs with Retry Information</h1>
+
+<p class="intro">
+  This list below contains the timestamps for scheduled delayed jobs with
+  retry information.
+</p>
+
+<p class="sub">
+  Showing <%= start = params[:start].to_i %> to <%= start + 20 %> of
+  <b><%= size = resque.delayed_queue_schedule_size %></b> timestamps
+</p>
+
+<table>
+  <tr>
+    <th></th>
+    <th>Timestamp</th>
+    <th>Job count</th>
+    <th>Class</th>
+    <th>Args</th>
+    <th>Retry Attempts</th>
+  </tr>
+  <% timestamps = resque.delayed_queue_peek(start, start+20) %>
+  <% timestamps.each do |timestamp| %>
+    <% job = resque.delayed_timestamp_peek(timestamp, 0, 1).first %>
+    <% retry_key = retry_key_for_job(job) %>
+    <tr>
+      <td>
+        <form action="<%= url "/delayed/queue_now" %>" method="post">
+          <input type="hidden" name="timestamp" value="<%= timestamp.to_i %>">
+          <input type="submit" value="Queue now">
+        </form>
+      </td>
+      <td><a href="<%= url "retry/#{timestamp}" %>"><%= format_time(Time.at(timestamp)) %></a></td>
+      <td><%= delayed_timestamp_size = resque.delayed_timestamp_size(timestamp) %></td>
+      <% if job && delayed_timestamp_size == 1 %>
+        <td><%= h job['class'] %></td>
+        <td><%= h job['args'].inspect %></td>
+        <td><%= retry_attempts_for_job(job) || '<i>n/a</i>' %></td>
+      <% else %>
+        <td><a href="<%= url "retry/#{timestamp}" %>">see details</a></td>
+        <td></td>
+        <td></td>
+      <% end %>
+    </tr>
+  <% end %>
+</table>
+
+<%= partial :next_more, :start => start, :size => size %>