blaxter-delayed_job 1.7.99 → 2.0.0

data/README.markdown

@@ -0,0 +1,199 @@
+Delayed::Job
+============
+
+Delayed_job (or DJ) encapsulates the common pattern of asynchronously executing longer tasks in the background.
+
+It is a direct extraction from Shopify where the job table is responsible for a multitude of core tasks. Amongst those tasks are:
+
+* sending massive newsletters
+* image resizing
+* http downloads
+* updating smart collections
+* updating solr, our search server, after product changes
+* batch imports
+* spam checks
+
+What is this fork for?
+----------------------
+
+My purpose with this fork is make delayed_job for flexible. That's means you can customize how your workers behave.
+
+The common use will be to have several workers running concurrently (in one process) and each one with differents constraints so they'll run different kind of jobs.
+
+
+Setup
+-----
+
+The library evolves around a delayed_jobs table which looks as follows:
+
+    create_table :delayed_jobs, :force => true do |table|
+        table.integer  :priority, :default => 0      # Allows some jobs to jump to the front of the queue
+        table.integer  :attempts, :default => 0      # Provides for retries, but still fail eventually.
+        table.text     :handler                      # YAML-encoded string of the object that will do work
+        table.string   :job_type                     # Class name of the job object, for type-specific workers
+        table.string   :name                         # The display name, an informative field or can be use to filter jobs
+        table.string   :last_error                   # reason for last failure (See Note below)
+        table.datetime :run_at                       # When to run. Could be Time.now for immediately, or sometime in the future.
+        table.datetime :locked_at                    # Set when a client is working on this object
+        table.datetime :failed_at                    # Set when all retries have failed (actually, by default, the record is deleted instead)
+        table.string   :locked_by                    # Who is working on this object (if locked)
+        table.datetime :finished_at			 # Used for statiscics / monitoring
+        table.timestamps
+    end
+
+You can generate the migration executing:
+
+    $ script/generate delayed_job
+      exists  db/migrate
+      create  db/migrate/20090807090217_create_delayed_jobs.rb
+
+
+On failure, the job is scheduled again in 5 seconds + N ** 4, where N is the number of retries.
+
+The default `MAX_ATTEMPTS` is 25 (jobs can override this value by responding to `:max_attempts`). After this, the job either deleted (default), or left in the database with "failed_at" set. With the default of 25 attempts, the last retry will be 20 days later, with the last interval being almost 100 hours.
+
+The default `MAX_RUN_TIME` is 4.hours. If your job takes longer than that, another computer could pick it up. It's up to you to make sure your job doesn't exceed this time. You should set this to the longest time you think the job could take.
+
+By default, it will delete failed jobs. If you want to keep failed jobs, set `Delayed::Job.destroy_failed_jobs = false`. The failed jobs will be marked with non-null failed_at.
+
+Same thing for successful jobs. They're deleted by default and, to keep them, set `Delayed::Job.destroy_successful_jobs = false`. They will be marked with finished_at. This is useful for gathering statistics like how long jobs took between entering the queue (created_at) and being finished (finished_at).
+
+You have a couble of named scopes for searching unfinished/finsihed jobs, very useful when destroy_successful_jobs is false `Delayed::Job.unfinished` and `Delayed::Job.finsihed`.
+
+Here is an example of changing job parameters in Rails:
+
+    # config/initializers/delayed_job_config.rb
+    Delayed::Job.destroy_failed_jobs     = false
+    Delayed::Job.destroy_successful_jobs = false
+    silence_warnings do
+      Delayed::Job.const_set("MAX_ATTEMPTS", 3)
+      Delayed::Job.const_set("MAX_RUN_TIME", 5.hours)
+
+      Delayed::Worker.const_set("SLEEP", 5.minutes.to_i)
+    end
+
+Note: If your error messages are long, consider changing last_error field to a :text instead of a :string (255 character limit).
+
+
+Usage
+-----
+
+Jobs are simple ruby objects with a method called perform. Any object which responds to perform can be stuffed into the jobs table.
+Job objects are serialized to yaml so that they can later be resurrected by the job runner.
+
+    class NewsletterJob < Struct.new(:text, :emails)
+      def perform
+        emails.each { |e| NewsletterMailer.deliver_text_to_email(text, e) }
+      end
+    end
+
+    Delayed::Job.enqueue NewsletterJob.new('lorem ipsum...', Customers.find(:all).collect(&:email))
+
+There is also a second way to get jobs in the queue: send_later.
+
+    BatchImporter.new(Shop.find(1)).send_later(:import_massive_csv, massive_csv)
+
+And also you can specified priority as second parameter and the time the job should execute as thrird one
+
+
+    class FooJob
+      def perform
+        ...
+      end
+    end
+
+    important_job = FooJob.new
+    normal_job    = FooJob.new
+
+    # Delayed::Job.enqueue( job, priority, start_at )
+    Delayed::Job.enqueue important_job, 100
+    Delayed::Job.enqueue normal_job, 1, 2.hours.from_now
+
+This will simply create a `Delayed::PerformableMethod` job in the jobs table which serializes all the parameters you pass to it. There are some special smarts for active record objects which are stored as their text representation and loaded from the database fresh when the job is actually run later.
+
+
+Running the jobs
+----------------
+
+You can invoke `rake jobs:work` which will start working off jobs. You can cancel the rake task with `CTRL-C`.
+
+You can also run by writing a simple `script/job_runner`, and invoking it externally:
+
+
+    require File.dirname(__FILE__) + '/../config/environment'
+
+    Delayed::Worker.new.start
+
+Workers can be running on any computer, as long as they have access to the database and their clock is in sync. You can even run multiple workers on per computer, but you must give each one a unique name.
+
+
+    require File.dirname(__FILE__) + '/../config/environment'
+    N = 10
+    workers = []
+    N.times do |n|
+    workers << Thread.new do
+      Delayed::Worker.new( :name => "Worker #{n}" ).start
+    end
+    end
+
+    workers.first.join # wait until finish (signal catched)
+
+Keep in mind that each worker will check the database at least every 5 seconds.
+
+Note: The rake task will exit if the database has any network connectivity problems.
+
+If you only want to run specific types of jobs in a given worker, include them when initializing the worker:
+
+    Delayed::Worker.new(:job_types => "SimpleJob").start
+    Delayed::Worker.new(:job_types => ["SimpleJob", "NewsletterJob"]).start
+
+Also for a more specific restriction you can define in your job's classes a `display_name` method, and create workers to specific kind of jobs
+
+    # 1 - The job class that does the real work
+    class MyJob
+      def initialize( data )
+        @some_data = data
+      end
+
+      def perform
+        # do the real work
+      end
+
+      def display_name
+        "foobar #{@some_data}"
+      end
+    end
+
+    # 2 - Enqueue jobs
+    Delayed::Job.enqueue MyJob.new("foobar")
+    Delayed::Job.enqueue MyJob.new("arrrr")
+
+    # 3 - Create workers, one for each type of "data"
+    Thread.new {
+      # This worker will only perform jobs which display name is like "%foobar%"
+      Delayed::Worker.new :name => "Worker for foobar", :only_for => "foobar"
+    }
+    Thread.new {
+      Delayed::Worker.new :name => "Worker for arrr", :only_for => "arrr"
+    }
+
+
+Cleaning up
+-----------
+
+You can invoke `rake jobs:clear` to delete all jobs in the queue.
+
+Changes
+-------
+
+* 2.0.0: Contains the changes made in this fork, the ability to create workers with individual constraints without interfere to other workers
+
+* 1.7.0: Added failed_at column which can optionally be set after a certain amount of failed job attempts. By default failed job attempts are destroyed after about a month.
+
+* 1.6.0: Renamed locked_until to locked_at. We now store when we start a given job instead of how long it will be locked by the worker. This allows us to get a reading on how long a job took to execute.
+
+* 1.5.0: Job runners can now be run in parallel. Two new database columns are needed: locked_until and locked_by. This allows us to use   pessimistic locking instead of relying on row level locks. This enables us to run as many worker processes as we need to speed up queue processing.
+
+* 1.2.0: Added #send_later to Object for simpler job creation
+
+* 1.0.0: Initial release

data/VERSION

@@ -0,0 +1 @@
+2.0.0

data/delayed_job.gemspec

@@ -2,21 +2,19 @@
 
 Gem::Specification.new do |s|
   s.name = %q{delayed_job}
-  s.version = "1.7.99"
+  s.version = "2.0.0"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
   s.authors = ["Tobias L\303\274tke"]
-  s.date = %q{2009-08-06}
+  s.date = %q{2009-08-10}
   s.description = %q{Delayed_job (or DJ) encapsulates the common pattern of asynchronously executing longer tasks in the background. It is a direct extraction from Shopify where the job table is responsible for a multitude of core tasks.}
   s.email = %q{tobi@leetsoft.com}
-  s.extra_rdoc_files = [
-    "README.textile"
-  ]
   s.files = [
     ".gitignore",
      "MIT-LICENSE",
-     "README.textile",
+     "README.markdown",
      "Rakefile",
+     "VERSION",
      "delayed_job.gemspec",
      "generators/delayed_job/delayed_job_generator.rb",
      "generators/delayed_job/templates/migration.rb",

data/generators/delayed_job/templates/migration.rb

@@ -5,6 +5,7 @@ class CreateDelayedJobs < ActiveRecord::Migration
       t.integer  :attempts, :default => 0
       t.text     :handler
       t.string   :job_type
+      t.string   :name
       t.string   :last_error
       t.datetime :run_at
       t.datetime :locked_at

data/lib/delayed/job.rb

@@ -22,28 +22,110 @@ module Delayed
     # If you want to keep them around (for statistics/monitoring),
     # set this to false.
     cattr_accessor :destroy_successful_jobs
-    self.destroy_successful_jobs = true
-
-    # Every worker has a unique name which by default is the pid of the process.
-    # There are some advantages to overriding this with something which survives worker retarts:
-    # Workers can safely resume working on tasks which are locked by themselves. The worker will assume that it crashed before.
-    cattr_accessor :worker_name
-    self.worker_name = "host:#{Socket.gethostname} pid:#{Process.pid}" rescue "pid:#{Process.pid}"
+    self.destroy_successful_jobs = false
 
     NextTaskSQL         = '(run_at <= ? AND (locked_at IS NULL OR locked_at < ?) OR (locked_by = ?)) AND failed_at IS NULL AND finished_at IS NULL'
     NextTaskOrder       = 'priority DESC, run_at ASC'
 
     ParseObjectFromYaml = /\!ruby\/\w+\:([^\s]+)/
 
-    cattr_accessor :min_priority, :max_priority, :job_types
-    self.min_priority = nil
-    self.max_priority = nil
-    self.job_types    = nil
+    named_scope :unfinished, :conditions => { :finished_at => nil }
+    named_scope :finished,   :conditions => [ "finished_at IS NOT NULL" ]
 
-    # When a worker is exiting, make sure we don't have any locked jobs.
-    def self.clear_locks!
-      update_all("locked_by = null, locked_at = null", ["locked_by = ?", worker_name])
-    end
+    class << self
+      # When a worker is exiting, make sure we don't have any locked jobs.
+      def clear_locks!( worker_name )
+        update_all("locked_by = null, locked_at = null", ["locked_by = ?", worker_name])
+      end
+
+      # Add a job to the queue
+      def enqueue(*args, &block)
+        object = block_given? ? EvaledJob.new(&block) : args.shift
+
+        unless object.respond_to?(:perform) || block_given?
+          raise ArgumentError, 'Cannot enqueue items which do not respond to perform'
+        end
+
+        priority = args.first || 0
+        run_at   = args[1]
+
+        Job.create(:payload_object => object, :priority => priority.to_i, :run_at => run_at)
+      end
+
+      # Find a few candidate jobs to run (in case some immediately get locked by others).
+      # Return in random order prevent everyone trying to do same head job at once.
+      def find_available( options = {} )
+        limit        = options[:limit]        || 5
+        max_run_time = options[:max_run_time] || MAX_RUN_TIME
+        worker_name  = options[:worker_name]  || Worker::DEFAULT_WORKER_NAME
+
+        sql        = NextTaskSQL.dup
+        time_now   = db_time_now
+        conditions = [time_now, time_now - max_run_time, worker_name]
+        if options[:min_priority]
+          sql << ' AND (priority >= ?)'
+          conditions << options[:min_priority]
+        end
+
+        if options[:max_priority]
+          sql << ' AND (priority <= ?)'
+          conditions << options[:max_priority]
+        end
+
+        if options[:job_types]
+          sql << ' AND (job_type IN (?))'
+          conditions << options[:job_types]
+        end
+
+        if options[:only_for]
+          sql << ' AND name LIKE ?'
+          conditions << "%#{options[:only_for]}%"
+        end
+        conditions.unshift(sql)
+
+        ActiveRecord::Base.silence do
+          find(:all, :conditions => conditions, :order => NextTaskOrder, :limit => limit)
+        end
+      end
+
+      # Run the next job we can get an exclusive lock on.
+      # If no jobs are left we return nil
+      def reserve_and_run_one_job( options = {} )
+        max_run_time = options[:max_run_time] || MAX_RUN_TIME
+        worker_name  = options[:worker_name]  || Worker::DEFAULT_WORKER_NAME
+        # For the next jobs availables, try to get lock. In case we cannot get exclusive
+        # access to a job we try the next.
+        # This leads to a more even distribution of jobs across the worker processes.
+        find_available( options ).each do |job|
+          t = job.run_with_lock( max_run_time, worker_name )
+          return t unless t.nil?  # return if we did work (good or bad)
+        end
+        # we didn't do any work, all 5 were not lockable
+        nil
+      end
+
+      # Do num jobs and return stats on success/failure.
+      # Exit early if interrupted.
+      def work_off( options = {} )
+        n = options[:n] || 100
+        success, failure = 0, 0
+
+        n.times do
+          case reserve_and_run_one_job( options )
+            when true
+              success += 1
+            when false
+              failure += 1
+            else
+              break  # leave if no work could be done
+          end
+          break if Worker.exit # leave if we're exiting
+        end
+
+        return [success, failure]
+      end
+
+    end # class << self
 
     def failed?
       failed_at
@@ -55,18 +137,25 @@ module Delayed
     end
 
     def name
-      @name ||= begin
+      if new_record?
         payload = payload_object
         if payload.respond_to?(:display_name)
           payload.display_name
         else
           payload.class.name
         end
+      else
+        self['name']
       end
     end
 
     def payload_object=(object)
-      self['job_type'] = object.class.to_s
+      job_type = if object.respond_to? :job_type
+        object.job_type
+      else
+        object.class.to_s
+      end
+      self['job_type'] = job_type
       self['handler']  = object.to_yaml
     end
 
@@ -87,9 +176,9 @@ module Delayed
       end
     end
 
-
-    # Try to run one job. Returns true/false (work done/work failed) or nil if job can't be locked.
-    def run_with_lock(max_run_time, worker_name)
+    # Try to run one job.
+    # Returns true/false (work done/work failed) or nil if job can't be locked.
+    def run_with_lock(max_run_time = MAX_RUN_TIME, worker_name = Worker::DEFAULT_WORKER_NAME)
       Delayed::Worker.logger.info "* [JOB] aquiring lock on #{name}"
       unless lock_exclusively!(max_run_time, worker_name)
         # We did not get the lock, some other worker process must have
@@ -103,7 +192,7 @@ module Delayed
         end
         destroy_successful_jobs ? destroy :
           update_attribute(:finished_at, Time.now)
-        Delayed::Worker.logger.info "* [JOB] #{name} completed after %.4f" % runtime        
+        Delayed::Worker.logger.info "* [JOB] #{name} completed after %.4f" % runtime
         return true  # did work
       rescue Exception => e
         reschedule e.message, e.backtrace
@@ -112,71 +201,9 @@ module Delayed
       end
     end
 
-    # Add a job to the queue
-    def self.enqueue(*args, &block)
-      object = block_given? ? EvaledJob.new(&block) : args.shift
-
-      unless object.respond_to?(:perform) || block_given?
-        raise ArgumentError, 'Cannot enqueue items which do not respond to perform'
-      end
-    
-      priority = args.first || 0
-      run_at   = args[1]
-
-      Job.create(:payload_object => object, :priority => priority.to_i, :run_at => run_at)
-    end
-
-    # Find a few candidate jobs to run (in case some immediately get locked by others).
-    # Return in random order prevent everyone trying to do same head job at once.
-    def self.find_available(limit = 5, max_run_time = MAX_RUN_TIME)
-
-      time_now = db_time_now
-
-      sql = NextTaskSQL.dup
-
-      conditions = [time_now, time_now - max_run_time, worker_name]
-
-      if self.min_priority
-        sql << ' AND (priority >= ?)'
-        conditions << min_priority
-      end
-
-      if self.max_priority
-        sql << ' AND (priority <= ?)'
-        conditions << max_priority
-      end
-
-      if self.job_types
-        sql << ' AND (job_type IN (?))'
-        conditions << job_types
-      end
-
-      conditions.unshift(sql)
-
-      records = ActiveRecord::Base.silence do
-        find(:all, :conditions => conditions, :order => NextTaskOrder, :limit => limit)
-      end
-
-      records.sort_by { rand() }
-    end
-
-    # Run the next job we can get an exclusive lock on.
-    # If no jobs are left we return nil
-    def self.reserve_and_run_one_job(max_run_time = MAX_RUN_TIME)
-
-      # We get up to 5 jobs from the db. In case we cannot get exclusive access to a job we try the next.
-      # this leads to a more even distribution of jobs across the worker processes
-      find_available(5, max_run_time).each do |job|
-        t = job.run_with_lock(max_run_time, worker_name)
-        return t unless t == nil  # return if we did work (good or bad)
-      end
-
-      nil # we didn't do any work, all 5 were not lockable
-    end
-
-    # Lock this job for this worker.
+    # Lock this job for the worker given as parameter (the name).
     # Returns true if we have the lock, false otherwise.
-    def lock_exclusively!(max_run_time, worker = worker_name)
+    def lock_exclusively!(max_run_time, worker)
       now = self.class.db_time_now
       affected_rows = if locked_by != worker
         # We don't own this job so we will update the locked_by name and the locked_at
@@ -207,26 +234,6 @@ module Delayed
       Delayed::Worker.logger.error(error)
     end
 
-    # Do num jobs and return stats on success/failure.
-    # Exit early if interrupted.
-    def self.work_off(num = 100)
-      success, failure = 0, 0
-
-      num.times do
-        case self.reserve_and_run_one_job
-        when true
-            success += 1
-        when false
-            failure += 1
-        else
-          break  # leave if no work could be done
-        end
-        break if $exit # leave if we're exiting
-      end
-
-      return [success, failure]
-    end
-
     # Moved into its own method so that new_relic can trace it.
     def invoke_job
       payload_object.perform
@@ -271,6 +278,7 @@ module Delayed
 
     def before_save
       self.run_at ||= self.class.db_time_now
+      self['name'] = name
     end
 
   end

data/lib/delayed/message_sending.rb

@@ -3,7 +3,7 @@ module Delayed
     def send_later(method, *args)
       Delayed::Job.enqueue Delayed::PerformableMethod.new(self, method.to_sym, args)
     end
-    
+
     module ClassMethods
       def handle_asynchronously(method)
         without_name = "#{method}_without_send_later"
@@ -13,5 +13,5 @@ module Delayed
         alias_method_chain method, :send_later
       end
     end
-  end                               
-end
+  end
+end

data/lib/delayed/performable_method.rb

@@ -10,14 +10,14 @@ module Delayed
       self.args   = args.map { |a| dump(a) }
       self.method = method.to_sym
     end
-    
-    def display_name  
+
+    def display_name
       case self.object
       when CLASS_STRING_FORMAT then "#{$1}.#{method}"
       when AR_STRING_FORMAT    then "#{$1}##{method}"
       else "Unknown##{method}"
-      end      
-    end    
+      end
+    end
 
     def perform
       load(object).send(method, *args.map{|a| load(a)})
@@ -26,6 +26,15 @@ module Delayed
       true
     end
 
+    def job_type
+      class_name = case self.object
+        when CLASS_STRING_FORMAT then $1.to_s
+        when AR_STRING_FORMAT    then $1.to_s
+        else self.object.class.to_s
+      end
+      "#{class_name}##{self.method.to_s}"
+    end
+
     private
 
     def load(arg)
@@ -52,4 +61,4 @@ module Delayed
       "CLASS:#{obj.name}"
     end
   end
-end
+end

data/lib/delayed/worker.rb

@@ -1,6 +1,12 @@
 module Delayed
   class Worker
     SLEEP = 5
+    JOBS_EACH = 100 # Jobs executed in each iteration
+
+    DEFAULT_WORKER_NAME = "host:#{Socket.gethostname} pid:#{Process.pid}" rescue "pid:#{Process.pid}"
+    # Indicates that we have catched a signal and we have to exit asap
+    cattr_accessor :exit
+    self.exit = false
 
     cattr_accessor :logger
     self.logger = if defined?(Merb::Logger)
@@ -9,47 +15,82 @@ module Delayed
       RAILS_DEFAULT_LOGGER
     end
 
+    # Every worker has a unique name which by default is the pid of the process (so you only are
+    # be able to have one unless override this in the constructor).
+    #
+    #     Thread.new { Delayed::Worker.new( :name => "Worker 1" ).start }
+    #     Thread.new { Delayed::Worker.new( :name => "Worker 2" ).start }
+    #
+    # There are some advantages to overriding this with something which survives worker retarts:
+    # Workers can safely resume working on tasks which are locked by themselves.
+    # The worker will assume that it crashed before.
+    attr_accessor :name
+
+    # Constraints for this worker, what kind of jobs is gonna execute?
+    attr_accessor :min_priority, :max_priority, :job_types
+
+    attr_accessor :quiet
+
+    # A worker will be in a loop trying to execute pending jobs, you can also set
+    # a few constraints to customize the worker's behaviour.
+    #
+    # Named parameters:
+    #   - name: the name of the worker, mandatory if you are going to create several workers
+    #   - quiet: log to stdout (besides the normal logger)
+    #   - min_priority: constraint for selecting what jobs to execute (integer)
+    #   - max_priority: constraint for selecting what jobs to execute (integer)
+    #   - job_types: constraint for selecting what jobs to execute (String or Array)
     def initialize(options={})
-      @quiet = options[:quiet]
-      Delayed::Job.min_priority = options[:min_priority] if options.has_key?(:min_priority)
-      Delayed::Job.max_priority = options[:max_priority] if options.has_key?(:max_priority)
-      Delayed::Job.job_types    = options[:job_types]    if options.has_key?(:job_types)
+      [ :quiet, :name, :min_priority, :max_priority, :job_types ].each do |attr_name|
+        send "#{attr_name}=", options[attr_name]
+      end
+      # Default values
+      self.name  = DEFAULT_WORKER_NAME if self.name.nil?
+      self.quiet = true                if self.quiet.nil?
     end
 
     def start
-      say "*** Starting job worker #{Delayed::Job.worker_name}"
+      say "*** Starting job worker #{name}"
 
-      trap('TERM') { say 'Exiting...'; $exit = true }
-      trap('INT')  { say 'Exiting...'; $exit = true }
+      trap('TERM') { say 'Exiting...'; self.exit = true }
+      trap('INT')  { say 'Exiting...'; self.exit = true }
 
       loop do
         result = nil
 
         realtime = Benchmark.realtime do
-          result = Delayed::Job.work_off
+          result = Job.work_off( constraints )
         end
 
         count = result.sum
 
-        break if $exit
-
         if count.zero?
-          sleep(SLEEP)
+          sleep(SLEEP) unless self.exit
         else
           say "#{count} jobs processed at %.4f j/s, %d failed ..." % [count / realtime, result.last]
         end
-
-        break if $exit
+        break if self.exit
       end
 
     ensure
-      Delayed::Job.clear_locks!
+      Job.clear_locks! name
     end
 
     def say(text)
-      puts text unless @quiet
+      text = "#{name}: #{text}"
+      puts text unless self.quiet
       logger.info text if logger
     end
 
+    protected
+      def constraints
+        { :max_run_time => Job::MAX_RUN_TIME,
+          :worker_name  => name,
+          :n            => JOBS_EACH,
+          :limit        => 5,
+          :min_priority => min_priority,
+          :max_priority => max_priority,
+          :job_types    => job_types }
+      end
   end
 end

data/spec/database.rb

@@ -23,6 +23,7 @@ ActiveRecord::Schema.define do
     table.integer  :attempts, :default => 0
     table.text     :handler
     table.string   :job_type
+    table.string   :name
     table.string   :last_error
     table.datetime :run_at
     table.datetime :locked_at

data/spec/delayed_method_spec.rb

@@ -37,12 +37,10 @@ class StoryReader
 end
 
 describe 'random ruby objects' do
-  before       { Delayed::Job.delete_all }
+  before { Delayed::Job.delete_all }
 
   it "should respond_to :send_later method" do
-
     RandomRubyObject.new.respond_to?(:send_later)
-
   end
 
   it "should raise a ArgumentError if send_later is called but the target method doesn't exist" do
@@ -73,15 +71,21 @@ describe 'random ruby objects' do
   end
 
   it "should ignore ActiveRecord::RecordNotFound errors because they are permanent" do
-
     ErrorObject.new.send_later(:throw)
 
-    Delayed::Job.count.should == 1
+    Delayed::Job.unfinished.count.should == 1
 
-    Delayed::Job.reserve_and_run_one_job
+    Delayed::Job.work_off
 
-    Delayed::Job.count.should == 0
+    Delayed::Job.unfinished.count.should == 0
+  end
+
+  it "should store the job type properly when its an normal class" do
+    story = Story.create :text => 'Once upon...'
+    story.send_later :tell
 
+    job = Delayed::Job.first
+    job.job_type.should == "Story#tell"
   end
 
   it "should store the object as string if its an active record" do

data/spec/job_spec.rb

@@ -7,12 +7,16 @@ end
 
 class CustomJob < SimpleJob
   def max_attempts; 3; end
+
+  def display_name
+    "custom name for the job"
+  end
 end
 
 class ErrorJob
   cattr_accessor :runs; self.runs = 0
   def perform; raise 'did not work'; end
-end             
+end
 
 class LongRunningJob
   def perform; sleep 250; end
@@ -21,19 +25,15 @@ end
 module M
   class ModuleJob
     cattr_accessor :runs; self.runs = 0
-    def perform; @@runs += 1; end    
+    def perform; @@runs += 1; end
   end
-  
 end
 
 describe Delayed::Job do
-  before  do               
-    Delayed::Job.max_priority = nil
-    Delayed::Job.min_priority = nil      
-    
+  before do
     Delayed::Job.delete_all
   end
-  
+
   before(:each) do
     SimpleJob.runs = 0
   end
@@ -78,40 +78,55 @@ describe Delayed::Job do
     SimpleJob.runs.should == 1
   end
 
+  it "should set name properly according to the class" do
+    job = Delayed::Job.enqueue SimpleJob.new
+    job.name.should == "SimpleJob"
+  end
+
+  it "should set name properly when display_name method is defined in the object" do
+    job = Delayed::Job.enqueue CustomJob.new
+    job.name.should_not == "CustomJob"
+    job.name.should == "custom name for the job"
+  end
+
+  it "should work on specified job types for common objects" do
+    Delayed::Job.unfinished.should have(0).jobs
+    Delayed::Job.finished.should have(0).jobs
+
+    "a string".send_later :size
+    Delayed::Job.unfinished.should have(1).jobs
+    Delayed::Job.finished.should have(0).jobs
+
+    Delayed::Job.work_off :job_types => "String#size"
+    Delayed::Job.unfinished.should have(0).jobs
+    Delayed::Job.finished.should have(1).job
+  end
+
   it "should work on specified job types" do
     SimpleJob.runs.should == 0
 
-    Delayed::Job.job_types = "SimpleJob"
     Delayed::Job.enqueue SimpleJob.new
-    Delayed::Job.work_off
+    Delayed::Job.work_off :job_types => "SimpleJob"
 
     SimpleJob.runs.should == 1
-
-    Delayed::Job.job_types = nil
   end
 
   it "should not work on unspecified job types" do
     SimpleJob.runs.should == 0
 
-    Delayed::Job.job_types = "AnotherJob"
     Delayed::Job.enqueue SimpleJob.new
-    Delayed::Job.work_off
+    Delayed::Job.work_off :job_types => "AnotherJob"
 
     SimpleJob.runs.should == 0
-
-    Delayed::Job.job_types = nil
   end
 
   it "should work on specified job types even when it's a list" do
     SimpleJob.runs.should == 0
 
-    Delayed::Job.job_types = %w( Whatever SimpleJob )
     Delayed::Job.enqueue SimpleJob.new
-    Delayed::Job.work_off
+    Delayed::Job.work_off :job_types => %w( Whatever SimpleJob )
 
     SimpleJob.runs.should == 1
-
-    Delayed::Job.job_types = nil
   end
 
   it "should work with eval jobs" do
@@ -126,7 +141,7 @@ describe Delayed::Job do
 
     $eval_job_ran.should == true
   end
-                   
+
   it "should work with jobs in modules" do
     M::ModuleJob.runs.should == 0
 
@@ -175,17 +190,17 @@ describe Delayed::Job do
   it "should never find finished jobs" do
     @job = Delayed::Job.create :payload_object => SimpleJob.new,
       :finished_at => Time.now
-    Delayed::Job.find_available(1).length.should == 0
+    Delayed::Job.find_available( :limit => 1 ).length.should == 0
   end
 
   it "should re-schedule by about 1 second at first and increment this more and more minutes when it fails to execute properly" do
-    Delayed::Job.enqueue ErrorJob.new
-    Delayed::Job.work_off(1)
+    job = Delayed::Job.enqueue ErrorJob.new
+    Delayed::Job.work_off( :n => 1 )
 
-    job = Delayed::Job.find(:first)
+    job.reload
 
     job.last_error.should =~ /did not work/
-    job.last_error.should =~ /job_spec.rb:14:in `perform'/
+    job.last_error.should =~ /job_spec.rb:\d+:in `perform'/
     job.attempts.should == 1
 
     job.run_at.should > Delayed::Job.db_time_now - 10.minutes
@@ -232,7 +247,7 @@ describe Delayed::Job do
     job.should_receive(:attempt_to_load).with('Delayed::JobThatDoesNotExist').and_return(true)
     lambda { job.payload_object.perform }.should raise_error(Delayed::DeserializationError)
   end
-  
+
   it "should be failed if it failed more than MAX_ATTEMPTS times and we don't want to destroy jobs" do
     default = Delayed::Job.destroy_failed_jobs
     Delayed::Job.destroy_failed_jobs = false
@@ -269,21 +284,41 @@ describe Delayed::Job do
 
   it "should fail after MAX_RUN_TIME" do
     @job = Delayed::Job.create :payload_object => LongRunningJob.new
-    Delayed::Job.reserve_and_run_one_job(1.second)
+    Delayed::Job.reserve_and_run_one_job( :max_run_time => 1.second )
     @job.reload.last_error.should =~ /expired/
     @job.attempts.should == 1
   end
 
+  it "should find high priority jobs first" do
+    @job_10 = Delayed::Job.create :payload_object => SimpleJob.new, :priority => 10
+    @job_20 = Delayed::Job.create :payload_object => SimpleJob.new, :priority => 20
+
+    Delayed::Job.find_available( :limit => 1 ).first.should == @job_20
+  end
+
+  it "should find only jobs like the parameter given" do
+    Delayed::Job.create :payload_object => SimpleJob.new
+    Delayed::Job.create :payload_object => CustomJob.new
+    Delayed::Job.unfinished.should have(2).jobs
+
+    Delayed::Job.work_off :only_for => 'Simple'
+    Delayed::Job.unfinished.should have(1).jobs
+
+    Delayed::Job.work_off :only_for => 'custom'
+    Delayed::Job.unfinished.should have(0).jobs
+  end
+
   it "should never find failed jobs" do
     @job = Delayed::Job.create :payload_object => SimpleJob.new, :attempts => 50, :failed_at => Time.now
-    Delayed::Job.find_available(1).length.should == 0
+    Delayed::Job.find_available( :limit => 1 ).length.should == 0
   end
 
   context "when another worker is already performing an task, it" do
 
     before :each do
-      Delayed::Job.worker_name = 'worker1'
-      @job = Delayed::Job.create :payload_object => SimpleJob.new, :locked_by => 'worker1', :locked_at => Delayed::Job.db_time_now - 5.minutes
+      @job = Delayed::Job.create :payload_object => SimpleJob.new,
+                                 :locked_by => 'worker1',
+                                 :locked_at => Delayed::Job.db_time_now - 5.minutes
     end
 
     it "should not allow a second worker to get exclusive access" do
@@ -292,8 +327,8 @@ describe Delayed::Job do
 
     it "should allow a second worker to get exclusive access if the timeout has passed" do
       @job.lock_exclusively!(1.minute, 'worker2').should == true
-    end      
-    
+    end
+
     it "should be able to get access to the task if it was started more then max_age ago" do
       @job.locked_at = 5.hours.ago
       @job.save
@@ -305,24 +340,20 @@ describe Delayed::Job do
     end
 
     it "should not be found by another worker" do
-      Delayed::Job.worker_name = 'worker2'
-
-      Delayed::Job.find_available(1, 6.minutes).length.should == 0
+      Delayed::Job.find_available( :worker_name => 'worker2', :max_run_time => 6.minutes ).should have(0).jobs
     end
 
     it "should be found by another worker if the time has expired" do
-      Delayed::Job.worker_name = 'worker2'
-
-      Delayed::Job.find_available(1, 4.minutes).length.should == 1
+      Delayed::Job.find_available( :worker_name => 'worker2', :max_run_time => 4.minutes ).should have(1).job
     end
 
     it "should be able to get exclusive access again when the worker name is the same" do
       @job.lock_exclusively! 5.minutes, 'worker1'
       @job.lock_exclusively! 5.minutes, 'worker1'
       @job.lock_exclusively! 5.minutes, 'worker1'
-    end                                        
-  end            
-  
+    end
+  end
+
   context "#name" do
     it "should be the class name of the job that was enqueued" do
       Delayed::Job.create(:payload_object => ErrorJob.new ).name.should == 'ErrorJob'
@@ -334,93 +365,78 @@ describe Delayed::Job do
 
     end
     it "should be the instance method that will be called if its a performable method object" do
-      story = Story.create :text => "..."                 
-      
+      story = Story.create :text => "..."
+
       story.send_later(:save)
-      
+
       Delayed::Job.last.name.should == 'Story#save'
     end
   end
-  
+
   context "worker prioritization" do
-    
-    before(:each) do
-      Delayed::Job.max_priority = nil
-      Delayed::Job.min_priority = nil      
-    end
-  
     it "should only work_off jobs that are >= min_priority" do
-      Delayed::Job.min_priority = -5
-      Delayed::Job.max_priority = 5
       SimpleJob.runs.should == 0
-    
+
       Delayed::Job.enqueue SimpleJob.new, -10
       Delayed::Job.enqueue SimpleJob.new, 0
-      Delayed::Job.work_off
-    
+      Delayed::Job.work_off :min_priority => -5, :max_priority => 5
+
       SimpleJob.runs.should == 1
     end
-  
+
     it "should only work_off jobs that are <= max_priority" do
-      Delayed::Job.min_priority = -5
-      Delayed::Job.max_priority = 5
       SimpleJob.runs.should == 0
-    
+
       Delayed::Job.enqueue SimpleJob.new, 10
       Delayed::Job.enqueue SimpleJob.new, 0
 
-      Delayed::Job.work_off
+      Delayed::Job.work_off :min_priority => -5, :max_priority => 5
 
       SimpleJob.runs.should == 1
-    end                         
-   
+    end
   end
-  
+
   context "when pulling jobs off the queue for processing, it" do
     before(:each) do
       @job = Delayed::Job.create(
-        :payload_object => SimpleJob.new, 
-        :locked_by => 'worker1', 
-        :locked_at => Delayed::Job.db_time_now - 5.minutes)
+        :payload_object => SimpleJob.new,
+        :locked_by      => 'worker1',
+        :locked_at      => Delayed::Job.db_time_now - 5.minutes )
     end
 
     it "should leave the queue in a consistent state and not run the job if locking fails" do
-      SimpleJob.runs.should == 0     
+      SimpleJob.runs.should == 0
       @job.stub!(:lock_exclusively!).with(any_args).once.and_return(false)
       Delayed::Job.should_receive(:find_available).once.and_return([@job])
-      Delayed::Job.work_off(1)
+      Delayed::Job.work_off( :n => 1 )
       SimpleJob.runs.should == 0
     end
-  
+
   end
-  
+
   context "while running alongside other workers that locked jobs, it" do
     before(:each) do
-      Delayed::Job.worker_name = 'worker1'
-      Delayed::Job.create(:payload_object => SimpleJob.new, :locked_by => 'worker1', :locked_at => (Delayed::Job.db_time_now - 1.minutes))
-      Delayed::Job.create(:payload_object => SimpleJob.new, :locked_by => 'worker2', :locked_at => (Delayed::Job.db_time_now - 1.minutes))
-      Delayed::Job.create(:payload_object => SimpleJob.new)
-      Delayed::Job.create(:payload_object => SimpleJob.new, :locked_by => 'worker1', :locked_at => (Delayed::Job.db_time_now - 1.minutes))
+      Delayed::Job.create :payload_object => SimpleJob.new, :locked_by => 'worker1', :locked_at => (Delayed::Job.db_time_now - 1.minutes)
+      Delayed::Job.create :payload_object => SimpleJob.new, :locked_by => 'worker2', :locked_at => (Delayed::Job.db_time_now - 1.minutes)
+      Delayed::Job.create :payload_object => SimpleJob.new
+      Delayed::Job.create :payload_object => SimpleJob.new, :locked_by => 'worker1', :locked_at => (Delayed::Job.db_time_now - 1.minutes)
     end
 
     it "should ingore locked jobs from other workers" do
-      Delayed::Job.worker_name = 'worker3'
       SimpleJob.runs.should == 0
-      Delayed::Job.work_off
+      Delayed::Job.work_off :worker_name => 'worker3'
       SimpleJob.runs.should == 1 # runs the one open job
     end
 
     it "should find our own jobs regardless of locks" do
-      Delayed::Job.worker_name = 'worker1'
       SimpleJob.runs.should == 0
-      Delayed::Job.work_off
+      Delayed::Job.work_off :worker_name => 'worker1'
       SimpleJob.runs.should == 3 # runs open job plus worker1 jobs that were already locked
     end
   end
 
   context "while running with locked and expired jobs, it" do
     before(:each) do
-      Delayed::Job.worker_name = 'worker1'
       exp_time = Delayed::Job.db_time_now - (1.minutes + Delayed::Job::MAX_RUN_TIME)
       Delayed::Job.create(:payload_object => SimpleJob.new, :locked_by => 'worker1', :locked_at => exp_time)
       Delayed::Job.create(:payload_object => SimpleJob.new, :locked_by => 'worker2', :locked_at => (Delayed::Job.db_time_now - 1.minutes))
@@ -429,20 +445,19 @@ describe Delayed::Job do
     end
 
     it "should only find unlocked and expired jobs" do
-      Delayed::Job.worker_name = 'worker3'
       SimpleJob.runs.should == 0
-      Delayed::Job.work_off
+      Delayed::Job.work_off :worker_name => 'worker3'
       SimpleJob.runs.should == 2 # runs the one open job and one expired job
     end
 
     it "should ignore locks when finding our own jobs" do
-      Delayed::Job.worker_name = 'worker1'
       SimpleJob.runs.should == 0
-      Delayed::Job.work_off
+      Delayed::Job.work_off :worker_name => 'worker1'
       SimpleJob.runs.should == 3 # runs open job plus worker1 jobs
-      # This is useful in the case of a crash/restart on worker1, but make sure multiple workers on the same host have unique names!
+      # This is useful in the case of a crash/restart on worker1,
+      # but make sure multiple workers on the same host have unique names!
     end
 
   end
-  
+
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification 
 name: blaxter-delayed_job
 version: !ruby/object:Gem::Version 
-  version: 1.7.99
+  version: 2.0.0
 platform: ruby
 authors: 
 - "Tobias L\xC3\xBCtke"
@@ -9,7 +9,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2009-08-06 00:00:00 -07:00
+date: 2009-08-10 00:00:00 -07:00
 default_executable: 
 dependencies: []
 
@@ -19,13 +19,14 @@ executables: []
 
 extensions: []
 
-extra_rdoc_files: 
-- README.textile
+extra_rdoc_files: []
+
 files: 
 - .gitignore
 - MIT-LICENSE
-- README.textile
+- README.markdown
 - Rakefile
+- VERSION
 - delayed_job.gemspec
 - generators/delayed_job/delayed_job_generator.rb
 - generators/delayed_job/templates/migration.rb

data/README.textile

@@ -1,118 +0,0 @@
-h1. Delayed::Job
-
-Delayed_job (or DJ) encapsulates the common pattern of asynchronously executing longer tasks in the background. 
-
-It is a direct extraction from Shopify where the job table is responsible for a multitude of core tasks. Amongst those tasks are:
-
-* sending massive newsletters
-* image resizing
-* http downloads
-* updating smart collections
-* updating solr, our search server, after product changes
-* batch imports 
-* spam checks 
-            
-h2. Setup
-            
-The library evolves around a delayed_jobs table which looks as follows: 
-<pre><code>create_table :delayed_jobs, :force => true do |table|
-    table.integer  :priority, :default => 0      # Allows some jobs to jump to the front of the queue
-    table.integer  :attempts, :default => 0      # Provides for retries, but still fail eventually.
-    table.text     :handler                      # YAML-encoded string of the object that will do work
-    table.string   :job_type                     # Class name of the job object, for type-specific workers
-    table.string   :last_error                   # reason for last failure (See Note below)
-    table.datetime :run_at                       # When to run. Could be Time.now for immediately, or sometime in the future.
-    table.datetime :locked_at                    # Set when a client is working on this object
-    table.datetime :failed_at                    # Set when all retries have failed (actually, by default, the record is deleted instead)
-    table.string   :locked_by                    # Who is working on this object (if locked)
-    table.datetime :finished_at			 # Used for statiscics / monitoring
-    table.timestamps
-  end
-</code></pre>
-On failure, the job is scheduled again in 5 seconds + N ** 4, where N is the number of retries.
-
-The default MAX_ATTEMPTS is 25- jobs can override this value by responding to :max_attempts. After this, the job
-either deleted (default), or left in the database with "failed_at" set. With the default of 25 attempts, 
-the last retry will be 20 days later, with the last interval being almost 100 hours.
-
-The default MAX_RUN_TIME is 4.hours. If your job takes longer than that, another computer could pick it up. It's up to you to
-make sure your job doesn't exceed this time. You should set this to the longest time you think the job could take.
-
-By default, it will delete failed jobs. If you want to keep failed jobs, set
-@Delayed::Job.destroy_failed_jobs = false@. The failed jobs will be marked with non-null failed_at.
-
-Same thing for successful jobs. They're deleted by default and, to keep them, set @Delayed::Job.destroy_successful_jobs = false@. They will be marked with finished_at. This is useful for gathering statistics like how long jobs took between entering the queue (created_at) and being finished (finished_at).
-
-Here is an example of changing job parameters in Rails:
-<pre><code># config/initializers/delayed_job_config.rb
-  Delayed::Job.destroy_failed_jobs = false
-  Delayed::Job.destroy_successful_jobs = false
-  silence_warnings do
-    Delayed::Job.const_set("MAX_ATTEMPTS", 3)
-    Delayed::Job.const_set("MAX_RUN_TIME", 5.minutes)
-  end
-</code></pre>
-Note: If your error messages are long, consider changing last_error field to a :text instead of a :string (255 character limit).
-
-
-h2. Usage
-
-Jobs are simple ruby objects with a method called perform. Any object which responds to perform can be stuffed into the jobs table.
-Job objects are serialized to yaml so that they can later be resurrected by the job runner. 
-<pre><code>class NewsletterJob < Struct.new(:text, :emails)
-    def perform
-      emails.each { |e| NewsletterMailer.deliver_text_to_email(text, e) }
-    end    
-  end  
-  
-  Delayed::Job.enqueue NewsletterJob.new('lorem ipsum...', Customers.find(:all).collect(&:email))
-</code></pre>
-There is also a second way to get jobs in the queue: send_later. 
-
-<pre><code>BatchImporter.new(Shop.find(1)).send_later(:import_massive_csv, massive_csv)
-</code></pre>
-
-This will simply create a Delayed::PerformableMethod job in the jobs table which serializes all the parameters you pass to it. There are some special smarts for active record objects
-which are stored as their text representation and loaded from the database fresh when the job is actually run later.
-                                                                                                                              
-                                                                                                                    
-h2. Running the jobs
-
-You can invoke @rake jobs:work@ which will start working off jobs. You can cancel the rake task with @CTRL-C@. 
-
-You can also run by writing a simple @script/job_runner@, and invoking it externally:
-  
-<pre><code>#!/usr/bin/env ruby
-  require File.dirname(__FILE__) + '/../config/environment'
-  
-  Delayed::Worker.new.start  
-</code></pre>
-
-Workers can be running on any computer, as long as they have access to the database and their clock is in sync. You can even
-run multiple workers on per computer, but you must give each one a unique name. (TODO: put in an example)
-Keep in mind that each worker will check the database at least every 5 seconds.
-
-Note: The rake task will exit if the database has any network connectivity problems.
-
-If you only want to run specific types of jobs in a given worker, include them when initializing the worker:
-
-<pre><code>
-  Delayed::Worker.new(:job_types => "SimpleJob").start
-  Delayed::Worker.new(:job_types => ["SimpleJob", "NewsletterJob"]).start
-</pre></code>
-
-h3. Cleaning up
-
-You can invoke @rake jobs:clear@ to delete all jobs in the queue.
-
-h3. Changes
-
-* 1.7.0: Added failed_at column which can optionally be set after a certain amount of failed job attempts. By default failed job attempts are destroyed after about a month. 
-
-* 1.6.0: Renamed locked_until to locked_at. We now store when we start a given job instead of how long it will be locked by the worker. This allows us to get a reading on how long a job took to execute.                    
-
-* 1.5.0: Job runners can now be run in parallel. Two new database columns are needed: locked_until and locked_by. This allows us to use   pessimistic locking instead of relying on row level locks. This enables us to run as many worker processes as we need to speed up queue processing.
-
-* 1.2.0: Added #send_later to Object for simpler job creation
-
-* 1.0.0: Initial release