delayed_job 1.8.5 → 1.9.0pre

data/README.textile

@@ -36,13 +36,24 @@ To install as a plugin:
 script/plugin install git://github.com/collectiveidea/delayed_job.git
 </pre>
 
-After delayed_job is installed, run:
+After delayed_job is installed, you will need to setup the backend.
+
+h2. Backends
+
+delayed_job supports multiple backends for storing the job queue. The default is Active Record, which requires a jobs table.
 
 <pre>
 script/generate delayed_job
 rake db:migrate
 </pre>
 
+You can change the backend in an initializer:
+
+<pre>
+# config/initializers/delayed_job.rb
+Delayed::Worker.backend = :mongo_mapper
+</pre>
+
 h2. Upgrading to 1.8
 
 If you are upgrading from a previous release, you will need to generate the new @script/delayed_job@:
@@ -51,6 +62,8 @@ If you are upgrading from a previous release, you will need to generate the new
 script/generate delayed_job --skip-migration
 </pre>
 
+Known Issues: script/delayed_job does not work properly with anything besides the Active Record backend.  That will be resolved before the next gem release.
+
 h2. Queuing Jobs
 
 Call @#send_later(method, params)@ on any object and it will be processed in the background.
@@ -112,39 +125,39 @@ h2. Gory Details
 
 The library evolves around a delayed_jobs table which looks as follows: 
 
-  create_table :delayed_jobs, :force => true do |table|
-    table.integer  :priority, :default => 0      # Allows some jobs to jump to the front of the queue
-    table.integer  :attempts, :default => 0      # Provides for retries, but still fail eventually.
-    table.text     :handler                      # YAML-encoded string of the object that will do work
-    table.text   :last_error                   # reason for last failure (See Note below)
-    table.datetime :run_at                       # When to run. Could be Time.zone.now for immediately, or sometime in the future.
-    table.datetime :locked_at                    # Set when a client is working on this object
-    table.datetime :failed_at                    # Set when all retries have failed (actually, by default, the record is deleted instead)
-    table.string   :locked_by                    # Who is working on this object (if locked)
-    table.timestamps
-  end
+<pre>
+create_table :delayed_jobs, :force => true do |table|
+  table.integer  :priority, :default => 0      # Allows some jobs to jump to the front of the queue
+  table.integer  :attempts, :default => 0      # Provides for retries, but still fail eventually.
+  table.text     :handler                      # YAML-encoded string of the object that will do work
+  table.text   :last_error                   # reason for last failure (See Note below)
+  table.datetime :run_at                       # When to run. Could be Time.zone.now for immediately, or sometime in the future.
+  table.datetime :locked_at                    # Set when a client is working on this object
+  table.datetime :failed_at                    # Set when all retries have failed (actually, by default, the record is deleted instead)
+  table.string   :locked_by                    # Who is working on this object (if locked)
+  table.timestamps
+end
+</pre>
 
 On failure, the job is scheduled again in 5 seconds + N ** 4, where N is the number of retries.
 
-The default Job::max_attempts is 25. After this, the job either deleted (default), or left in the database with "failed_at" set.
+The default Worker.max_attempts is 25. After this, the job either deleted (default), or left in the database with "failed_at" set.
 With the default of 25 attempts, the last retry will be 20 days later, with the last interval being almost 100 hours.
 
-The default Job::max_run_time is 4.hours. If your job takes longer than that, another computer could pick it up. It's up to you to
+The default Worker.max_run_time is 4.hours. If your job takes longer than that, another computer could pick it up. It's up to you to
 make sure your job doesn't exceed this time. You should set this to the longest time you think the job could take.
 
 By default, it will delete failed jobs (and it always deletes successful jobs). If you want to keep failed jobs, set
-Delayed::Job.destroy_failed_jobs = false. The failed jobs will be marked with non-null failed_at.
+Delayed::Worker.destroy_failed_jobs = false. The failed jobs will be marked with non-null failed_at.
 
 Here is an example of changing job parameters in Rails:
 
 <pre>
 # config/initializers/delayed_job_config.rb
-Delayed::Job.destroy_failed_jobs = false
-silence_warnings do
-  Delayed::Worker::sleep_delay = 60
-  Delayed::Job::max_attempts = 3
-  Delayed::Job::max_run_time = 5.minutes
-end
+Delayed::Worker.destroy_failed_jobs = false
+Delayed::Worker.sleep_delay = 60
+Delayed::Worker.max_attempts = 3
+Delayed::Worker.max_run_time = 5.minutes
 </pre>
 
 h3. Cleaning up

data/Rakefile

@@ -11,7 +11,7 @@ Jeweler::Tasks.new do |s|
   s.summary  = "Database-backed asynchronous priority queue system -- Extracted from Shopify"
   s.email    = "tobi@leetsoft.com"
   s.homepage = "http://github.com/collectiveidea/delayed_job"
-  s.description = "Delayed_job (or DJ) encapsulates the common pattern of asynchronously executing longer tasks in the background. It is a direct extraction from Shopify where the job table is responsible for a multitude of core tasks."
+  s.description = "Delayed_job (or DJ) encapsulates the common pattern of asynchronously executing longer tasks in the background. It is a direct extraction from Shopify where the job table is responsible for a multitude of core tasks.\n\nThis gem is collectiveidea's fork (http://github.com/collectiveidea/delayed_job)."
   s.authors  = ["Brandon Keepers", "Tobias Lütke"]
   
   s.has_rdoc = true
@@ -19,6 +19,10 @@ Jeweler::Tasks.new do |s|
   s.extra_rdoc_files = ["README.textile"]
   
   s.test_files = Dir['spec/**/*']
+  
+  s.add_dependency "daemons"
+  s.add_development_dependency "rspec"
+  s.add_development_dependency "sqlite3-ruby"
 end
 
 require 'spec/rake/spectask'
@@ -28,7 +32,8 @@ task :default => :spec
 desc 'Run the specs'
 Spec::Rake::SpecTask.new(:spec) do |t|
   t.libs << 'lib'
-  t.pattern = 'spec/**/*_spec.rb'
+  t.pattern = 'spec/*_spec.rb'
   t.verbose = true
 end
+task :spec => :check_dependencies
 

data/VERSION

@@ -1 +1 @@
-1.8.5
+1.9.0pre

data/delayed_job.gemspec

@@ -5,12 +5,14 @@
 
 Gem::Specification.new do |s|
   s.name = %q{delayed_job}
-  s.version = "1.8.5"
+  s.version = "1.9.0pre"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
   s.authors = ["Brandon Keepers", "Tobias L\303\274tke"]
-  s.date = %q{2010-03-15}
-  s.description = %q{Delayed_job (or DJ) encapsulates the common pattern of asynchronously executing longer tasks in the background. It is a direct extraction from Shopify where the job table is responsible for a multitude of core tasks.}
+  s.date = %q{2010-03-26}
+  s.description = %q{Delayed_job (or DJ) encapsulates the common pattern of asynchronously executing longer tasks in the background. It is a direct extraction from Shopify where the job table is responsible for a multitude of core tasks.
+
+This gem is collectiveidea's fork (http://github.com/collectiveidea/delayed_job).}
   s.email = %q{tobi@leetsoft.com}
   s.extra_rdoc_files = [
     "README.textile"
@@ -26,20 +28,29 @@ Gem::Specification.new do |s|
      "generators/delayed_job/delayed_job_generator.rb",
      "generators/delayed_job/templates/migration.rb",
      "generators/delayed_job/templates/script",
-     "init.rb",
+     "lib/delayed/backend/active_record.rb",
+     "lib/delayed/backend/base.rb",
+     "lib/delayed/backend/mongo_mapper.rb",
      "lib/delayed/command.rb",
-     "lib/delayed/job.rb",
      "lib/delayed/message_sending.rb",
      "lib/delayed/performable_method.rb",
      "lib/delayed/recipes.rb",
      "lib/delayed/tasks.rb",
      "lib/delayed/worker.rb",
      "lib/delayed_job.rb",
+     "rails/init.rb",
      "recipes/delayed_job.rb",
-     "spec/database.rb",
+     "spec/backend/active_record_job_spec.rb",
+     "spec/backend/mongo_mapper_job_spec.rb",
+     "spec/backend/shared_backend_spec.rb",
      "spec/delayed_method_spec.rb",
-     "spec/job_spec.rb",
+     "spec/performable_method_spec.rb",
+     "spec/sample_jobs.rb",
+     "spec/setup/active_record.rb",
+     "spec/setup/mongo_mapper.rb",
+     "spec/spec_helper.rb",
      "spec/story_spec.rb",
+     "spec/worker_spec.rb",
      "tasks/jobs.rake"
   ]
   s.homepage = %q{http://github.com/collectiveidea/delayed_job}
@@ -48,10 +59,19 @@ Gem::Specification.new do |s|
   s.rubygems_version = %q{1.3.5}
   s.summary = %q{Database-backed asynchronous priority queue system -- Extracted from Shopify}
   s.test_files = [
-    "spec/database.rb",
+    "spec/backend",
+     "spec/backend/active_record_job_spec.rb",
+     "spec/backend/mongo_mapper_job_spec.rb",
+     "spec/backend/shared_backend_spec.rb",
      "spec/delayed_method_spec.rb",
-     "spec/job_spec.rb",
-     "spec/story_spec.rb"
+     "spec/performable_method_spec.rb",
+     "spec/sample_jobs.rb",
+     "spec/setup",
+     "spec/setup/active_record.rb",
+     "spec/setup/mongo_mapper.rb",
+     "spec/spec_helper.rb",
+     "spec/story_spec.rb",
+     "spec/worker_spec.rb"
   ]
 
   if s.respond_to? :specification_version then
@@ -59,9 +79,18 @@ Gem::Specification.new do |s|
     s.specification_version = 3
 
     if Gem::Version.new(Gem::RubyGemsVersion) >= Gem::Version.new('1.2.0') then
+      s.add_runtime_dependency(%q<daemons>, [">= 0"])
+      s.add_development_dependency(%q<rspec>, [">= 0"])
+      s.add_development_dependency(%q<sqlite3-ruby>, [">= 0"])
     else
+      s.add_dependency(%q<daemons>, [">= 0"])
+      s.add_dependency(%q<rspec>, [">= 0"])
+      s.add_dependency(%q<sqlite3-ruby>, [">= 0"])
     end
   else
+    s.add_dependency(%q<daemons>, [">= 0"])
+    s.add_dependency(%q<rspec>, [">= 0"])
+    s.add_dependency(%q<sqlite3-ruby>, [">= 0"])
   end
 end
 

data/generators/delayed_job/delayed_job_generator.rb

@@ -4,7 +4,7 @@ class DelayedJobGenerator < Rails::Generator::Base
   def manifest
     record do |m|
       m.template 'script', 'script/delayed_job', :chmod => 0755
-      unless options[:skip_migration]
+      if !options[:skip_migration] && defined?(ActiveRecord)
         m.migration_template "migration.rb", 'db/migrate',
                              :migration_file_name => "create_delayed_jobs"
       end

data/generators/delayed_job/templates/migration.rb

@@ -11,7 +11,8 @@ class CreateDelayedJobs < ActiveRecord::Migration
       table.string   :locked_by                    # Who is working on this object (if locked)
       table.timestamps
     end
-
+	
+    add_index :delayed_jobs, [:priority, :run_at], :name => 'delayed_jobs_priority'
   end
   
   def self.down

data/lib/delayed/backend/active_record.rb

@@ -0,0 +1,90 @@
+require 'active_record'
+
+class ActiveRecord::Base
+  def self.load_for_delayed_job(id)
+    if id
+      find(id)
+    else
+      super
+    end
+  end
+  
+  def dump_for_delayed_job
+    "#{self.class};#{id}"
+  end
+end
+
+module Delayed
+  module Backend
+    module ActiveRecord
+      # A job object that is persisted to the database.
+      # Contains the work object as a YAML field.
+      class Job < ::ActiveRecord::Base
+        include Delayed::Backend::Base
+        set_table_name :delayed_jobs
+        
+        before_save :set_default_run_at
+
+        named_scope :ready_to_run, lambda {|worker_name, max_run_time|
+          {:conditions => ['(run_at <= ? AND (locked_at IS NULL OR locked_at < ?) OR locked_by = ?) AND failed_at IS NULL', db_time_now, db_time_now - max_run_time, worker_name]}
+        }
+        named_scope :by_priority, :order => 'priority ASC, run_at ASC'
+        
+        def self.after_fork
+          ActiveRecord::Base.connection.reconnect!
+        end
+
+        # When a worker is exiting, make sure we don't have any locked jobs.
+        def self.clear_locks!(worker_name)
+          update_all("locked_by = null, locked_at = null", ["locked_by = ?", worker_name])
+        end
+
+        # Find a few candidate jobs to run (in case some immediately get locked by others).
+        def self.find_available(worker_name, limit = 5, max_run_time = Worker.max_run_time)
+          scope = self.ready_to_run(worker_name, max_run_time)
+          scope = scope.scoped(:conditions => ['priority >= ?', Worker.min_priority]) if Worker.min_priority
+          scope = scope.scoped(:conditions => ['priority <= ?', Worker.max_priority]) if Worker.max_priority
+      
+          ::ActiveRecord::Base.silence do
+            scope.by_priority.all(:limit => limit)
+          end
+        end
+
+        # Lock this job for this worker.
+        # Returns true if we have the lock, false otherwise.
+        def lock_exclusively!(max_run_time, worker)
+          now = self.class.db_time_now
+          affected_rows = if locked_by != worker
+            # We don't own this job so we will update the locked_by name and the locked_at
+            self.class.update_all(["locked_at = ?, locked_by = ?", now, worker], ["id = ? and (locked_at is null or locked_at < ?) and (run_at <= ?)", id, (now - max_run_time.to_i), now])
+          else
+            # We already own this job, this may happen if the job queue crashes.
+            # Simply resume and update the locked_at
+            self.class.update_all(["locked_at = ?", now], ["id = ? and locked_by = ?", id, worker])
+          end
+          if affected_rows == 1
+            self.locked_at    = now
+            self.locked_by    = worker
+            return true
+          else
+            return false
+          end
+        end
+
+        # Get the current time (GMT or local depending on DB)
+        # Note: This does not ping the DB to get the time, so all your clients
+        # must have syncronized clocks.
+        def self.db_time_now
+          if Time.zone
+            Time.zone.now
+          elsif ::ActiveRecord::Base.default_timezone == :utc
+            Time.now.utc
+          else
+            Time.now
+          end
+        end
+
+      end
+    end
+  end
+end

data/lib/delayed/backend/base.rb

@@ -0,0 +1,106 @@
+module Delayed
+  module Backend
+    class DeserializationError < StandardError
+    end
+
+    module Base
+      def self.included(base)
+        base.extend ClassMethods
+      end
+      
+      module ClassMethods
+        # Add a job to the queue
+        def enqueue(*args)
+          object = args.shift
+          unless object.respond_to?(:perform)
+            raise ArgumentError, 'Cannot enqueue items which do not respond to perform'
+          end
+    
+          priority = args.first || 0
+          run_at   = args[1]
+          self.create(:payload_object => object, :priority => priority.to_i, :run_at => run_at)
+        end
+        
+        # Hook method that is called before a new worker is forked
+        def before_fork
+        end
+        
+        # Hook method that is called after a new worker is forked
+        def after_fork
+        end
+      end
+      
+      ParseObjectFromYaml = /\!ruby\/\w+\:([^\s]+)/
+
+      def failed?
+        failed_at
+      end
+      alias_method :failed, :failed?
+
+      def payload_object
+        @payload_object ||= deserialize(self['handler'])
+      end
+
+      def name
+        @name ||= begin
+          payload = payload_object
+          if payload.respond_to?(:display_name)
+            payload.display_name
+          else
+            payload.class.name
+          end
+        end
+      end
+
+      def payload_object=(object)
+        self['handler'] = object.to_yaml
+      end
+      
+      # Moved into its own method so that new_relic can trace it.
+      def invoke_job
+        payload_object.perform
+      end
+      
+      # Unlock this job (note: not saved to DB)
+      def unlock
+        self.locked_at    = nil
+        self.locked_by    = nil
+      end
+      
+    private
+
+      def deserialize(source)
+        handler = YAML.load(source) rescue nil
+
+        unless handler.respond_to?(:perform)
+          if handler.nil? && source =~ ParseObjectFromYaml
+            handler_class = $1
+          end
+          attempt_to_load(handler_class || handler.class)
+          handler = YAML.load(source)
+        end
+
+        return handler if handler.respond_to?(:perform)
+
+        raise DeserializationError,
+          'Job failed to load: Unknown handler. Try to manually require the appropriate file.'
+      rescue TypeError, LoadError, NameError => e
+        raise DeserializationError,
+          "Job failed to load: #{e.message}. Try to manually require the required file."
+      end
+
+      # Constantize the object so that ActiveSupport can attempt
+      # its auto loading magic. Will raise LoadError if not successful.
+      def attempt_to_load(klass)
+         klass.constantize
+      end
+
+    protected
+
+      def set_default_run_at
+        self.run_at ||= self.class.db_time_now
+      end
+    
+    end
+  end
+end

data/lib/delayed/backend/mongo_mapper.rb

@@ -0,0 +1,110 @@
+require 'mongo_mapper'
+
+module ::MongoMapper
+  module Document
+    module ClassMethods
+      def load_for_delayed_job(id)
+        find!(id)
+      end
+    end
+
+    module InstanceMethods
+      def dump_for_delayed_job
+        "#{self.class};#{id}"
+      end
+    end
+  end
+end
+
+module Delayed
+  module Backend
+    module MongoMapper
+      class Job
+        include ::MongoMapper::Document
+        include Delayed::Backend::Base
+        set_collection_name 'delayed_jobs'
+        
+        key :priority,    Integer, :default => 0
+        key :attempts,    Integer, :default => 0
+        key :handler,     String
+        key :run_at,      Time
+        key :locked_at,   Time
+        key :locked_by,   String, :index => true
+        key :failed_at,   Time
+        key :last_error,  String
+        timestamps!
+        
+        before_save :set_default_run_at
+
+        ensure_index [[:priority, 1], [:run_at, 1]]
+        
+        def self.before_fork
+          ::MongoMapper.connection.close
+        end
+        
+        def self.after_fork
+          ::MongoMapper.connection.connect_to_master
+        end
+        
+        def self.db_time_now
+          ::MongoMapper.time_class.now.utc
+        end
+        
+        def self.find_available(worker_name, limit = 5, max_run_time = Worker.max_run_time)
+          right_now = db_time_now
+
+          conditions = {
+            :run_at => {"$lte" => right_now},
+            :limit => -limit, # In mongo, positive limits are 'soft' and negative are 'hard'
+            :failed_at => nil,
+            :sort => [['priority', 1], ['run_at', 1]]
+          }
+
+          where = "this.locked_at == null || this.locked_at < #{make_date(right_now - max_run_time)}"
+          
+          (conditions[:priority] ||= {})['$gte'] = Worker.min_priority if Worker.min_priority
+          (conditions[:priority] ||= {})['$lte'] = Worker.max_priority if Worker.max_priority
+
+          results = all(conditions.merge(:locked_by => worker_name))
+          results += all(conditions.merge('$where' => where)) if results.size < limit
+          results
+        end
+        
+        # When a worker is exiting, make sure we don't have any locked jobs.
+        def self.clear_locks!(worker_name)
+          collection.update({:locked_by => worker_name}, {"$set" => {:locked_at => nil, :locked_by => nil}}, :multi => true)
+        end
+        
+        # Lock this job for this worker.
+        # Returns true if we have the lock, false otherwise.
+        def lock_exclusively!(max_run_time, worker = worker_name)
+          right_now = self.class.db_time_now
+          overtime = right_now - max_run_time.to_i
+          
+          query = "this.locked_at == null || this.locked_at < #{make_date(overtime)} || this.locked_by == #{worker.to_json}"
+          conditions = {:_id => id, :run_at => {"$lte" => right_now}, "$where" => query}
+
+          collection.update(conditions, {"$set" => {:locked_at => right_now, :locked_by => worker}})
+          affected_rows = collection.find({:_id => id, :locked_by => worker}).count
+          if affected_rows == 1
+            self.locked_at = right_now
+            self.locked_by = worker
+            return true
+          else
+            return false
+          end
+        end
+        
+      private
+      
+        def self.make_date(date_or_seconds)
+          "new Date(#{date_or_seconds.to_f * 1000})"
+        end
+
+        def make_date(date)
+          self.class.make_date(date)
+        end
+      end
+    end
+  end
+end