resque-pertry 1.0.0

data/.gitignore

@@ -0,0 +1,2 @@
+*.gem
+Gemfile.lock

data/Gemfile

@@ -0,0 +1,3 @@
+source 'https://rubygems.org'
+
+gemspec

data/LICENSE

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) Anthony 'YoGiN' Powles
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,190 @@
+resque-pertry
+=============
+
+Setup
+-----
+
+```ruby
+gem 'resque-pertry'
+```
+
+Wherever you include resque's tasks, you should add:
+
+```ruby
+require 'resque/tasks'
+require 'resque/pertry/tasks'
+```
+
+You'll need to run a migration to create the persistence table.
+
+```ruby
+require 'resque/pertry/migrations/create_resque_pertry_table'
+
+class AddResquePertry < Resque::Pertry::Migrations::CreateResquePertryTable
+  # if you are using db-charmer, you can easily specify the connection to use
+  db_magic :connection => :resque
+end
+```
+
+The associated model is `Resque::Pertry::ResquePertryPersistence`, feel free to add your own code in there if you need (such as db_magic)
+
+You will also need to start [resque-scheduler](https://github.com/bvandenbos/resque-scheduler) and the resque pertry purger
+
+```
+$ VERBOSE=1 rake resque:scheduler
+$ VERBOSE=1 rake resque:pertry:purger
+```
+
+The purger's default values will have it run every 5 minutes, and purge 100 failed jobs from redis and another 100 failed jobs from the database.
+You can add hooks in your application if you want to process the purged jobs further (you may want to log the jobs you are purging):
+
+```ruby
+Resque::Pertry::Purger.after_redis_purge do |job,redis|
+  # do something with this job from resque's failed queue
+end
+
+Resque::Pertry::Purger.after_database_purge do |job|
+  # do something with this job from the persistence table
+end
+```
+
+Usage
+-----
+
+Your resque jobs need to inherit from `Resque::Plugins::Pertry` :
+
+```ruby
+class SomeJob < Resque::Plugins::Pertry
+
+  # specify Resque queue
+  in_queue :critical
+
+  def perform
+    # do something
+  end
+end
+```
+
+You can enqueue the job:
+
+```ruby
+SomeJob.enqueue
+```
+
+You can pass arguments to your job:
+
+```ruby
+SomeJob.enqueue(:user_id => 42, :image_id => 24)
+```
+
+In your job class, you specify the arguments with `needs` :
+
+```ruby
+class SomeJob < Resque::Plugins::Pertry
+
+  # specify Resque queue
+  in_queue :critical
+
+  needs :user_id, :image_id
+
+  def perform
+    # do something with @user_id and @image_id
+  end
+end
+```
+
+The arguments `user_id` and `image_id` will then be accessible as instance variables `@user_id` and `@image_id`
+You can also specify a default value for your needs:
+
+```ruby
+needs :user_id, :image_id, :action => "delete"
+```
+
+Finally you can specify if your job should be persistent or not, and if it is, the properties to retry failed persistent jobs.
+All jobs including `Resque::Plugins::Pertry` are persistent by default. Non persistent jobs act just like regular resque jobs.
+
+To set a job as non persistent, simply:
+
+```ruby
+class SomeJob < Resque::Plugins::Pertry
+
+  # set job as non persistent, acts like a regular resque job
+  non_persistent
+
+  ...
+end
+```
+
+To retry a persistent job, you have several properties you can set:
+
+  * `set_retry_attempts <max_retries>`
+
+    Sets the maximum number of times we will retry a failed job. 
+    The total number of times a job will run is max_retries + 1 (the original run).
+
+  * `set_retry_delay <seconds>`
+
+    Sets the number of seconds to wait before enqueueing a failed job again
+
+  * `set_retry_delays <seconds>, <seconds>, ...`
+
+    Set an array of delays. 
+    The number of items will be used to defined `max_retries`, while each value will be used in turn for each consecutive retry delay.
+    If this is set, `set_retry_attempts` and `set_retry_delay` will be ignored
+
+  * `set_retry_exceptions <SomeException>, <SomeOtherException>, ...`
+
+    Sets a whitelist of exceptions you want to retry. Any other exceptions will not be retried.
+    If not set, all exceptions will be retried (default).
+    This property can be combined with any other property.
+
+  * `set_retry_ttl <seconds>`
+
+    Sets the maximum time to live for a job, after which the job will not be retried, even if it has attempts lefts.
+    This property can be combined with any other property.
+
+The default settings for all persistent jobs are:
+
+```ruby
+# retry a failed job once, enqueue the job immediately, job expires in 1 week
+set_retry_attempts 1
+set_retry_ttl 1.week
+```
+
+Here are some exemples of retry settings:
+
+```ruby
+# retry 5 times, enqueue job immediately
+set_retry_attempts 5
+```
+
+```ruby
+# retry 10 times, wait 1 minute before enqueueing the job again
+set_retry_attempts 10
+set_retry_delay 1.minute
+```
+
+```ruby
+# retry 3 times, wait time before enqueueing job increases after each attempt
+set_retry_delays 1.minute, 5.minutes, 15.minutes
+```
+
+```ruby
+# retry every 30 seconds for 1 hour, as long as we are getting the exception ActiveRecord::ConnectionTimeoutError
+set_retry_ttl 1.hour
+set_retry_delay 30.seconds
+set_retry_exceptions ActiveRecord::ConnectionTimeoutError
+```
+
+TODO
+----
+
+Write some specs!
+
+How to contribute
+-----------------
+
+Tell me what you think about this gem, things you like or don't about it, how it could be better.
+Pull Requests most welcomed.
+
+

data/lib/resque/pertry.rb

@@ -0,0 +1,7 @@
+require 'resque'
+require 'resque_scheduler'
+require 'resque/pertry/resque_pertry_persistence'
+require 'resque/pertry/persistence'
+require 'resque/pertry/retry'
+require 'resque/pertry/purger'
+require 'resque/plugins/pertry'

data/lib/resque/pertry/migrations/create_resque_pertry_table.rb

@@ -0,0 +1,30 @@
+module Resque
+  module Pertry
+    module Migrations
+      class CreateResquePertryTable < ActiveRecord::Migration
+
+        def up
+          create_table :resque_pertry_persistence do |t|
+            t.string :audit_id, :null => false, :limit => 64
+            t.string :job, :null => false
+            t.text :arguments, :null => false, :limit => 64.kilobytes + 1
+            t.integer :attempt, :default => 1
+            t.datetime :enqueued_at
+            t.datetime :completed_at
+            t.datetime :failed_at
+            t.datetime :last_tried_at
+            t.datetime :expires_at
+            t.timestamps
+          end
+
+          add_index :resque_pertry_persistence, :audit_id, :unique => true
+        end
+
+        def down
+          drop_table :resque_pertry_persistence
+        end
+
+      end
+    end
+  end
+end

data/lib/resque/pertry/persistence.rb

@@ -0,0 +1,67 @@
+module Resque
+  module Pertry
+    module Persistence
+      extend ActiveSupport::Concern
+
+      included do
+        class_attribute :_persistent
+
+        # define all jobs as persistent by default
+        persistent
+      end
+
+      module ClassMethods
+        
+        # Set job as persistent
+        def persistent
+          self._persistent = true
+        end
+        alias_method :durable, :persistent
+
+        def non_persistent
+          self._persistent = false
+        end
+        alias_method :non_durable, :non_persistent
+
+        # Check if job is persistent
+        def persistent?
+          !!self._persistent
+        end
+
+        # Resque before_enqueue hook
+        def before_enqueue_pertry_99_persistence(args = {})
+          pertry_key = Resque::Plugins::Pertry::JOB_HASH.to_s
+
+          args[pertry_key] ||= {}
+          args[pertry_key]['audit_id'] ||= UUIDTools::UUID.random_create.to_s
+          args[pertry_key]['queue_time'] ||= Time.now
+          args[pertry_key]['persist'] = persistent?
+
+          if persistent?
+            ResquePertryPersistence.create_job_if_needed(self, args)
+          end
+
+          # continue with enqueue
+          true
+        end
+
+        # Resque after_perform hook (job completed successfully)
+        def after_perform_pertry_00_persistence(args = {})
+          return unless persistent?
+
+          ResquePertryPersistence.finnish_job(self, args)
+        end
+
+      end
+
+      def audit_id
+        @_job_properties['audit_id']
+      end
+
+      def queue_time
+        Time.parse(@_job_properties['queue_time'])
+      end
+
+    end
+  end
+end

data/lib/resque/pertry/purger.rb

@@ -0,0 +1,214 @@
+module Resque
+  module Pertry
+    class Purger
+
+      class << self
+
+        attr_accessor :verbose
+
+        # sleep time between purges
+        attr_writer :sleep_time
+
+        # sets the number of failed jobs to purge per run
+        attr_writer :failed_jobs_limit
+
+        def sleep_time
+          @sleep_time ||= 5.minutes
+        end
+
+        def failed_jobs_limit
+          @failed_jobs_limit ||= 100
+        end
+
+        def stats
+          @stats ||= {}
+        end
+
+        # main loop
+        def run
+          setup
+
+          loop do
+            purge
+            wait
+          end
+        end
+
+        # run a purge cycle
+        def purge
+          procline("working")
+          add_stat(:loops, 1)
+
+          purge_resque
+          purge_database
+        end
+
+        # display status of failed queues and persistence table
+        def status
+          show_config
+          show_info
+        end
+
+        # allows an app to set a hook to deal with the failed redis job
+        def after_redis_purge(&block)
+          @after_redis_purge = block
+        end
+
+        # allows an app to set a hook to deal with the failed persistence table job
+        def after_database_purge(&block)
+          @after_database_purge = block
+        end
+
+        private
+
+        # purger setup and init
+        def setup
+          stats[:pid] = Process.pid
+          stats[:started] = Time.now
+          stats[:loops] = 0
+
+          procline("starting")
+          status
+          register_signal_handlers
+        end
+
+        # intercept signals
+        def register_signal_handlers
+          trap("TERM") { shutdown }
+          trap("INT")  { shutdown }
+          trap("QUIT") { shutdown }
+          trap("USR1") { status }
+        end
+
+        def show_config
+          log("Configuration:")
+          [ :sleep_time, :failed_jobs_limit ].each do |v|
+            log("\tconfig #{v} = #{send(v)}")
+          end
+        end
+
+        def show_info
+          with_redis do |redis|
+            set_stat("failed queue length on #{redis.id}", redis.llen(:failed))
+          end
+          
+          log!("Status:")
+          stats.each do |key, value|
+            log!("\t#{key} : #{value}")
+          end
+        end
+
+        # just sleep for a while
+        def wait
+          procline("sleeping for #{sleep_time} seconds")
+          sleep(sleep_time)
+        end
+
+        # shutdown the process
+        def shutdown
+          procline("shutting down")
+          status
+          exit
+        end
+
+        # update process line
+        def procline(string)
+          log(string)
+          $0 = "resque-pertry purger: #{string}"
+        end
+
+        # purge jobs in the failed queue
+        def purge_resque
+          with_redis do |redis|
+            purge_redis(redis)
+          end
+       end
+
+        def with_redis(&block)
+          return {} unless block_given?
+
+          # testing the redis class name so we don't have to require resque_redis_composite
+          case Resque.redis.redis.class.name
+          when "Redis"
+            { redis.id => block.call(Resque.redis) }
+          when "Resque::RedisComposite"
+            Resque.redis.mapping.reduce({}) do |results, (queue, redis)|
+              results[redis.id] = block.call(redis)
+              results
+            end
+          else
+            raise NotImplementedError, "Unsupported redis client #{Resque.redis.inspect}"
+          end
+        end
+
+        # purge resque's failed queue on a redis client
+        def purge_redis(redis)
+          failed_jobs = redis.lrange(:failed, 0, failed_jobs_limit - 1)
+          return 0 if failed_jobs.empty?
+
+          log("purging #{failed_jobs.size} failed jobs from #{redis.id}")
+          redis.ltrim(:failed, failed_jobs.size, -1)
+
+          failed_jobs.each do |failed_job|
+            run_after_redis_purge(failed_job, redis)
+          end if @after_redis_purge
+
+          add_stat("purged from #{redis.id}", failed_jobs.size)
+        end
+
+        # purge resque-pertry persistence table
+        def purge_database
+          jobs = ResquePertryPersistence.finnished_or_expired.limit(failed_jobs_limit)
+          return 0 if jobs.empty?
+
+          log("purging #{jobs.size} completed, failed, or expired jobs from database")
+
+          jobs.each do |job|
+            ResquePertryPersistence.destroy(job.id)
+            run_after_database_purge(job)
+          end if @after_database_purge
+
+          add_stat("purged from database", jobs.size)
+        end
+
+        # run hook after_redis_purge
+        def run_after_redis_purge(job, redis)
+          return unless @after_redis_purge
+          @after_redis_purge.call(job, redis)
+        rescue => e
+          log!("exception #{e.inspect} while running hook after_redis_purge on job #{job.inspect}")
+        end
+
+        # run hook after_database_purge
+        def run_after_database_purge(job)
+          return unless @after_database_purge
+          @after_database_purge.call(job)
+        rescue => e
+          log!("exception #{e.inspect} while running hook after_database_purge on job #{job.inspect}")
+        end
+
+        # only print if verbose is turned on
+        def log(string)
+          log!(string) if verbose
+        end
+
+        # always print this string
+        def log!(string)
+          $stdout.puts "#{Time.now.strftime("%Y-%m-%d %H:%M:%S.%L")} - #{string}"
+        end
+
+        def add_stat(stat, value)
+          stats[stat] ||= 0
+          stats[stat] += value
+          value
+        end
+
+        def set_stat(stat, value)
+          stats[stat] = value
+        end
+
+      end
+
+    end
+  end
+end

data/lib/resque/pertry/resque_pertry_persistence.rb

@@ -0,0 +1,87 @@
+module Resque
+  module Pertry
+    class ResquePertryPersistence < ActiveRecord::Base
+
+      self.table_name = "resque_pertry_persistence"
+
+      default_scope -> { order(:updated_at) }
+      scope :completed, -> { where("completed_at IS NOT NULL") }
+      scope :failed,    -> { where("failed_at IS NOT NULL") }
+      scope :finnished, -> { where("completed_at IS NOT NULL OR failed_at IS NOT NULL") }
+      scope :ongoing,   -> { where(:completed_at => nil, :failed_at => nil) }
+      scope :expired,   -> { where("expires_at < ?", Time.now) }
+      scope :finnished_or_expired, -> { where("completed_at IS NOT NULL OR failed_at IS NOT NULL OR expires_at < ?", Time.now) }
+
+      class << self
+
+        def create_job_if_needed(klass, args)
+          with_job(klass, args) do |job|
+            # we already have a job for this, we don't want another
+            return if job
+
+            # creating a new job
+            params = {
+              :audit_id => field_from_args('audit_id', args),
+              :job => klass.to_s,
+              :arguments => Resque.encode(args), 
+              :attempt => 0,
+              :enqueued_at => field_from_args('queue_time', args)
+            }
+
+            params[:expires_at] = params[:enqueued_at] + klass.retry_ttl if klass.retry_ttl
+
+            new(params).save!
+          end
+        end
+
+
+        def finnish_job(klass, args)
+          with_job(klass, args) do |job|
+            return unless job
+            job.update_attribute(:completed_at, Time.now)
+          end
+        end
+
+        def trying_job(klass, args)
+          with_job(klass, args) do |job|
+            return unless job
+            job.update_attributes(  :last_tried_at => Time.now,
+                                    :attempt => job.attempt + 1)
+          end
+        end
+
+        def fail_job(klass, args)
+          with_job(klass, args) do |job|
+            return unless job
+            job.update_attribute(:failed_at, Time.now)
+          end
+        end
+
+        def with_job(klass, args, &block)
+          audit_id = field_from_args('audit_id', args)
+          return unless audit_id
+          
+          job = find_by_audit_id(audit_id)
+          block.call(job) if block_given?
+        end
+
+        private
+
+        def field_from_args(field, args)
+          pertry_hash = args[Resque::Plugins::Pertry::JOB_HASH] || args[Resque::Plugins::Pertry::JOB_HASH.to_s] || {}
+          pertry_hash[field]
+        end
+
+      end
+
+      def has_completed?
+        completed_at
+      end
+
+      def payload
+        @payload ||= Resque.decode(arguments)
+      end
+
+    end
+  end
+end

data/lib/resque/pertry/retry.rb

@@ -0,0 +1,164 @@
+module Resque
+  module Pertry
+    module Retry
+      extend ActiveSupport::Concern
+
+      included do
+        class_attribute :_retry_delays
+        class_attribute :_retry_delay
+        class_attribute :_retry_attempts
+        class_attribute :_retry_ttl
+        class_attribute :_retry_exceptions
+
+        # set a default max number of retry attempts
+        set_retry_attempts 1
+
+        # set job to expire in 1 week (we need a default so we can purge the database)
+        set_retry_ttl 1.week
+      end
+
+      module ClassMethods
+        
+        # Sets a number of seconds to wait before retrying
+        def set_retry_delay(delay)
+          self._retry_delays = nil
+          self._retry_delay = Integer(delay)
+        end
+        
+        def retry_delay
+          self._retry_delay
+        end
+
+        # Sets a list of delays (list length will be the # of attempts)
+        def set_retry_delays(*delays)
+          self._retry_attempts = nil
+          self._retry_delay = nil
+          self._retry_delays = Array(delays).map { |delay| Integer(delay) }
+        end
+
+        def retry_delays
+          self._retry_delays
+        end
+
+        # Sets the maximum number of times we will retry
+        def set_retry_attempts(count)
+          self._retry_delays = nil
+          self._retry_attempts = Integer(count)
+        end
+
+        def retry_attempts
+          self._retry_attempts
+        end
+
+        # Sets the maximum time-to-live of the job, after which no attempts will ever be made
+        def set_retry_ttl(ttl)
+          self._retry_ttl = Integer(ttl)
+        end
+
+        def retry_ttl
+          self._retry_ttl
+        end
+
+        # Sets a list of exceptions that we want to retry
+        # If none are set, we will retry every exceptions
+        def set_retry_exceptions(*exceptions)
+          self._retry_exceptions = Array(exceptions)
+        end
+
+        def retry_exceptions
+          self._retry_exceptions
+        end
+
+        # Check if we will retry this job on failure
+        # There has to be a constraint on the number of times we will retry a failing job
+        # or have a ttl, otherwise we could be retrying job endlessly
+        def retryable?
+          retry_attempts || retry_delays || retry_ttl
+        end
+
+        # Resque around_perform hook
+        def around_perform_pertry_00_retry(args = {})
+          ResquePertryPersistence.trying_job(self, args)
+
+          yield
+        end
+
+        # Resque on_failure hook (job failed)
+        def on_failure_pertry_00_retry(exception, args = {})
+          return unless retryable?
+
+          ResquePertryPersistence.with_job(self, args) do |job_model|
+            job = instance(args)
+
+            unless job.retry!(job_model, exception)
+              ResquePertryPersistence.fail_job(self, args)
+            end
+          end
+        end
+
+      end
+
+      # Checks if we can retry
+      def retry?(model, exception)
+        # check the obvious
+        return false unless model
+        return false if model.has_completed?
+
+        # job has used up all it's allowed attempts
+        return false if max_attempt_reached?(model)
+
+        # job exception is not whitelisted for retries
+        return false unless exception_whitelisted?(model, exception)
+
+        # job has expired
+        return false if ttl_expired?(model)
+
+        # seems like we should be able to retry this job
+        return true
+      end
+
+      # Retry the job
+      def retry!(model, exception)
+        return false unless retry?(model, exception)
+
+        delay = delay_before_retry(model)
+        return false unless delay
+
+        Resque.enqueue_in(delay, self.class, payload)
+      end
+
+      def exception_whitelisted?(model, exception)
+        # all exceptions are whitelisted implicitly if we didn't set the exception list
+        return true unless self.class.retry_exceptions
+
+        self.class.retry_exceptions.include?(exception.class)
+      end
+
+      def ttl_expired?(model)
+        # if we didn't set a ttl, it hasn't expired
+        return false unless self.class.retry_ttl
+
+        model.expires_at < Time.now
+      end
+
+      def max_attempt_reached?(model)
+        if self.class.retry_attempts && self.class.retry_attempts < model.attempt
+          true
+        elsif self.class.retry_delays && self.class.retry_delays.size < model.attempt
+          true
+        else
+          false
+        end
+      end
+
+      def delay_before_retry(model)
+        if self.class.retry_delays
+          self.class.retry_delays[ model.attempt - 1 ]
+        else
+          self.class.retry_delay || 0
+        end
+      end
+
+    end
+  end
+end

data/lib/resque/pertry/tasks.rb

@@ -0,0 +1,22 @@
+require 'resque/tasks'
+require 'resque_scheduler/tasks'
+
+namespace :resque do
+  task :setup
+
+  namespace :pertry do
+    desc "Start Resque Pertry database and failed queue purger"
+    task :purger => :pertry_setup do
+      File.open(ENV['PIDFILE'], 'w') { |f| f << Process.pid.to_s } if ENV['PIDFILE']
+
+      Resque::Pertry::Purger.sleep_time = Integer(ENV['PURGER_SLEEP']) if ENV['PURGER_SLEEP']
+      Resque::Pertry::Purger.failed_jobs_limit = Integer(ENV['PURGER_LIMIT']) if ENV['PURGER_LIMIT']
+      Resque::Pertry::Purger.verbose = true if ENV['VERBOSE']
+      Resque::Pertry::Purger.run
+    end
+
+    task :pertry_setup do
+      Rake::Task['resque:setup'].invoke
+    end
+  end
+end

data/lib/resque/plugins/pertry.rb

@@ -0,0 +1,115 @@
+module Resque
+  module Plugins
+    class Pertry
+      JOB_HASH = :_pertry
+
+      include Resque::Pertry::Persistence
+      include Resque::Pertry::Retry
+
+      class << self
+
+        # Enqueue a job
+        def enqueue(args = {})
+          raise ArgumentError, "Invalid arguments, expecting a Hash but got: #{args.inspect}" unless Hash === args
+
+          args.symbolize_keys!
+          args = check_arguments(args)
+          raise ArgumentError, "Invalid arguments, #{JOB_HASH} is a reserved argument!" if args.key?(JOB_HASH)
+
+          Resque.enqueue(self, args)
+        end
+
+        # Perform a job
+        def perform(args = {})
+          raise ArgumentError, "Invalid arguments, expecting a Hash but got: #{args.inspect}" unless Hash === args
+
+          instance(args).perform
+        end
+
+        def instance(args = {})
+          args.symbolize_keys!
+          raise ArgumentError, "Job is not supported, missing key #{JOB_HASH} from payload #{args.inspect}" unless args.key?(JOB_HASH)
+
+          new(check_arguments(args), args[JOB_HASH])
+        end
+
+        # Specificy job queue
+        def in_queue(queue)
+          @queue = queue.to_sym
+        end
+        
+        # Get job queue
+        def queue
+          @queue or raise ArgumentError, "No queue defined for job #{self.name}!"
+        end
+
+        # Define required job attributes
+        def needs(*arguments)
+          arguments.each do |argument|
+            if Hash === argument
+              argument.each do |key, default|
+                self.required_arguments << { :name => key, :default => default }
+              end
+            else
+              self.required_arguments << { :name => argument }
+            end
+          end
+        end
+
+        # List of required attributes
+        def required_arguments
+          @required_arguments ||= []
+        end
+
+        private
+
+        # Check that job arguments match required arguments
+        def check_arguments(provided_arguments)
+          required_arguments.inject({}) do |checked_arguments, argument|
+            raise ArgumentError, "#{self} is missing required argument #{argument[:name]} from #{provided_arguments.inspect}" unless provided_arguments.member?(argument[:name]) || argument.member?(:default)
+
+            provided_argument = provided_arguments[argument[:name]] || argument[:default]
+            # TODO check that provided_argument is serializable as json
+
+            checked_arguments[argument[:name]] = provided_argument
+            checked_arguments
+          end
+        end
+
+      end
+
+      def initialize(arguments, job_properties)
+        set_job_arguments(arguments)
+        set_job_properties(job_properties)
+      end
+
+      # Perform method needs to be overridden in job classes
+      def perform
+        raise NoMethodError, "No method #{self.class.name}#perform defined!"
+      end
+
+      def arguments
+        @_arguments
+      end
+
+      def payload
+        @_arguments.merge(JOB_HASH => @_job_properties)
+      end
+
+      private
+
+      def set_job_properties(hash)
+        @_job_properties ||= {}
+        @_job_properties.merge!(hash)
+      end
+
+      def set_job_arguments(hash)
+        @_arguments = hash
+        arguments.each do |key, val|
+          instance_variable_set("@#{key}", val)
+        end
+      end
+
+    end
+  end
+end

data/resque-pertry.gemspec

@@ -0,0 +1,21 @@
+Gem::Specification.new do |s|
+  s.name        = "resque-pertry"
+  s.description = "Adds persistence to Resque jobs, and retry properties"
+  s.version     = "1.0.0"
+  s.authors     = [ "Anthony Powles" ]
+  s.email       = "rubygems+resque-pertry@idreamz.net"
+  s.summary     = "Allows job to be persistent, and be retried in case of failure"
+  s.homepage    = "https://github.com/yogin/resque-pertry"
+  s.license     = "MIT"
+  s.files       = `git ls-files`.split($/)
+
+  s.add_development_dependency "bundler", "~> 1.3"
+  s.add_development_dependency "rake"
+  s.add_development_dependency "pry"
+
+  s.add_dependency "activesupport", ">= 3.0.0"
+  s.add_dependency "activerecord", ">= 3.0.0"
+  s.add_dependency "resque"
+  s.add_dependency "resque-scheduler", ">= 2.0.0"
+  s.add_dependency "uuidtools", "~> 2.1.4"
+end

data/tasks/resque_pertry.rake

@@ -0,0 +1,2 @@
+$LOAD_PATH.unshift File.dirname(__FILE__) + '/../lib'
+require 'resque/pertry/tasks'

metadata

@@ -0,0 +1,187 @@
+--- !ruby/object:Gem::Specification
+name: resque-pertry
+version: !ruby/object:Gem::Version
+  version: 1.0.0
+  prerelease: 
+platform: ruby
+authors:
+- Anthony Powles
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2013-08-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.3'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.3'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: pry
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: activesupport
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: 3.0.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: 3.0.0
+- !ruby/object:Gem::Dependency
+  name: activerecord
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: 3.0.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: 3.0.0
+- !ruby/object:Gem::Dependency
+  name: resque
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: resque-scheduler
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: 2.0.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: 2.0.0
+- !ruby/object:Gem::Dependency
+  name: uuidtools
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 2.1.4
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 2.1.4
+description: Adds persistence to Resque jobs, and retry properties
+email: rubygems+resque-pertry@idreamz.net
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- .gitignore
+- Gemfile
+- LICENSE
+- README.md
+- lib/resque/pertry.rb
+- lib/resque/pertry/migrations/create_resque_pertry_table.rb
+- lib/resque/pertry/persistence.rb
+- lib/resque/pertry/purger.rb
+- lib/resque/pertry/resque_pertry_persistence.rb
+- lib/resque/pertry/retry.rb
+- lib/resque/pertry/tasks.rb
+- lib/resque/plugins/pertry.rb
+- resque-pertry.gemspec
+- tasks/resque_pertry.rake
+homepage: https://github.com/yogin/resque-pertry
+licenses:
+- MIT
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.23
+signing_key: 
+specification_version: 3
+summary: Allows job to be persistent, and be retried in case of failure
+test_files: []