delayed_job 2.0.8 → 2.1.0.pre

data/.gitignore

@@ -0,0 +1,2 @@
+*.gem
+*.swp

data/README.textile

@@ -1,6 +1,6 @@
 h1. Delayed::Job
 
-Delated_job (or DJ) encapsulates the common pattern of asynchronously executing longer tasks in the background.
+Delated_job (or DJ) encapsulates the common pattern of asynchronously executing longer tasks in the background. 
 
 It is a direct extraction from Shopify where the job table is responsible for a multitude of core tasks. Amongst those tasks are:
 
@@ -9,24 +9,21 @@ It is a direct extraction from Shopify where the job table is responsible for a
 * http downloads
 * updating smart collections
 * updating solr, our search server, after product changes
-* batch imports
-* spam checks
+* batch imports 
+* spam checks 
 
 h2. Installation
 
-This version is for Rails 2.x only.  For rails 3 support, install delayed_job 2.1.
-
 To install as a gem, add the following to @config/environment.rb@:
 
 <pre>
-config.gem 'delayed_job', :version => '~>2.0.4'
+config.gem 'delayed_job'
 </pre>
 
 Rake tasks are not automatically loaded from gems, so you'll need to add the following to your Rakefile:
 
 <pre>
 begin
-  gem 'delayed_job', '~>2.0.4'
   require 'delayed/tasks'
 rescue LoadError
   STDERR.puts "Run `rake gems:install` to install delayed_job"
@@ -36,15 +33,11 @@ end
 To install as a plugin:
 
 <pre>
-script/plugin install git://github.com/collectiveidea/delayed_job.git -r v2.0
+script/plugin install git://github.com/collectiveidea/delayed_job.git
 </pre>
 
 After delayed_job is installed, you will need to setup the backend.
 
-h3. Dependencies
-
-delayed_job depends upon version 1.0.10 of the daemons gem. delayed_job is incompatible with the newest (1.1.0) daemons gem.
-
 h2. Backends
 
 delayed_job supports multiple backends for storing the job queue. There are currently implementations for Active Record, MongoMapper, and DataMapper.
@@ -60,12 +53,8 @@ $ rake db:migrate
 
 h3. MongoMapper
 
-You must use @MongoMapper.setup@ in the initializer:
-
 <pre>
-config = YAML::load(File.read(Rails.root.join('config/mongo.yml')))
-MongoMapper.setup(config, Rails.env)
-
+# config/initializers/delayed_job.rb
 Delayed::Worker.backend = :mongo_mapper
 </pre>
 
@@ -103,39 +92,6 @@ device = Device.new
 device.deliver
 </pre>
 
-handle_asynchronously can take as options anything you can pass to delay. In addition the values can be Proc objects allowing call time evaluation of the value. For some examples:
-
-<pre>
-class LongTasks
-  def send_mailer
-    # Some other code
-  end
-  handle_asynchronously :send_mailer, :priority => 20
-
-  def in_the_future
-    # Some other code
-  end
-  # 5.minutes.from_now will be evaluated when in_the_future is called
-  handle_asynchronously :in_the_future, :run_at => Proc.new { 5.minutes.from_now }
-
-  def self.when_to_run
-    2.hours.from_now
-  end
-
-  def call_a_class_method
-    # Some other code
-  end
-  handle_asynchronously :call_a_class_method, :run_at => Proc.new { when_to_run }
-
-  attr_reader :how_important
-
-  def call_an_instance_method
-    # Some other code
-  end
-  handle_asynchronously :call_an_instance_method, :priority => Proc.new {|i| i.how_important }
-end
-</pre>
-
 h2. Running Jobs
 
 @script/delayed_job@ can be used to manage a background process which will start working off jobs.  Make sure you've run `script/generate delayed_job`.
@@ -151,19 +107,19 @@ $ RAILS_ENV=production script/delayed_job stop
 
 Workers can be running on any computer, as long as they have access to the database and their clock is in sync. Keep in mind that each worker will check the database at least every 5 seconds.
 
-You can also invoke @rake jobs:work@ which will start working off jobs. You can cancel the rake task with @CTRL-C@.
+You can also invoke @rake jobs:work@ which will start working off jobs. You can cancel the rake task with @CTRL-C@. 
 
 h2. Custom Jobs
 
-Jobs are simple ruby objects with a method called perform. Any object which responds to perform can be stuffed into the jobs table. Job objects are serialized to yaml so that they can later be resurrected by the job runner.
+Jobs are simple ruby objects with a method called perform. Any object which responds to perform can be stuffed into the jobs table. Job objects are serialized to yaml so that they can later be resurrected by the job runner. 
 
 <pre>
 class NewsletterJob < Struct.new(:text, :emails)
   def perform
     emails.each { |e| NewsletterMailer.deliver_text_to_email(text, e) }
-  end
-end
-
+  end    
+end  
+  
 Delayed::Job.enqueue NewsletterJob.new('lorem ipsum...', Customers.find(:all).collect(&:email))
 </pre>
 
@@ -173,17 +129,17 @@ You can also add an optional on_permanent_failure method which will run if the j
 class ParanoidNewsletterJob < NewsletterJob
   def perform
     emails.each { |e| NewsletterMailer.deliver_text_to_email(text, e) }
-  end
+  end    
 
   def on_permanent_failure
     page_sysadmin_in_the_middle_of_the_night
   end
-end
+end  
 </pre>
 
 h2. Gory Details
 
-The library evolves around a delayed_jobs table which looks as follows:
+The library evolves around a delayed_jobs table which looks as follows: 
 
 <pre>
 create_table :delayed_jobs, :force => true do |table|

data/Rakefile

@@ -0,0 +1,53 @@
+# -*- encoding: utf-8 -*-
+begin
+  require 'jeweler'
+rescue LoadError
+  puts "Jeweler not available. Install it with: sudo gem install jeweler"
+  exit 1
+end
+
+Jeweler::Tasks.new do |s|
+  s.name     = "delayed_job"
+  s.summary  = "Database-backed asynchronous priority queue system -- Extracted from Shopify"
+  s.email    = "tobi@leetsoft.com"
+  s.homepage = "http://github.com/collectiveidea/delayed_job"
+  s.description = "Delayed_job (or DJ) encapsulates the common pattern of asynchronously executing longer tasks in the background. It is a direct extraction from Shopify where the job table is responsible for a multitude of core tasks.\n\nThis gem is collectiveidea's fork (http://github.com/collectiveidea/delayed_job)."
+  s.authors  = ["Brandon Keepers", "Tobias Lütke"]
+  
+  s.has_rdoc = true
+  s.rdoc_options = ["--main", "README.textile", "--inline-source", "--line-numbers"]
+  s.extra_rdoc_files = ["README.textile"]
+  
+  s.test_files = Dir['spec/*_spec.rb']
+  
+  s.add_dependency "daemons"
+  s.add_development_dependency "rspec"
+  s.add_development_dependency "sqlite3-ruby"
+  s.add_development_dependency "activerecord"
+  s.add_development_dependency "mongo_mapper"
+  s.add_development_dependency "dm-core"
+  s.add_development_dependency "dm-observer"
+  s.add_development_dependency "dm-aggregates"
+  s.add_development_dependency "dm-validations"
+  s.add_development_dependency "do_sqlite3"
+  s.add_development_dependency "couchrest"
+end
+
+require 'spec/rake/spectask'
+
+
+task :default do
+  %w(2.3.5 3.0.0.beta3).each do |version|
+    puts "Running specs with Rails #{version}"
+    system("RAILS_VERSION=#{version} rake -s spec;")
+  end
+end
+
+desc 'Run the specs'
+Spec::Rake::SpecTask.new(:spec) do |t|
+  t.libs << 'lib'
+  t.pattern = 'spec/*_spec.rb'
+  t.verbose = true
+end
+task :spec => :check_dependencies
+

data/VERSION

@@ -0,0 +1 @@
+2.1.0.pre

data/benchmarks.rb

@@ -0,0 +1,33 @@
+$:.unshift(File.dirname(__FILE__) + '/lib')
+require 'rubygems'
+require 'logger'
+require 'delayed_job'
+require 'benchmark'
+
+RAILS_ENV = 'test'
+
+Delayed::Worker.logger = Logger.new('/dev/null')
+
+BACKENDS = []
+Dir.glob("#{File.dirname(__FILE__)}/spec/setup/*.rb") do |backend|
+  begin
+    backend = File.basename(backend, '.rb')
+    require "spec/setup/#{backend}"
+    BACKENDS << backend.to_sym
+  rescue LoadError
+    puts "Unable to load #{backend} backend! #{$!}"
+  end
+end
+
+
+Benchmark.bm(10) do |x|
+  BACKENDS.each do |backend|
+    require "spec/setup/#{backend}"
+    Delayed::Worker.backend = backend
+  
+    n = 10000
+    n.times { "foo".delay.length }
+
+    x.report(backend.to_s) { Delayed::Worker.new(:quiet => true).work_off(n) }
+  end  
+end

data/delayed_job.gemspec

@@ -0,0 +1,125 @@
+# Generated by jeweler
+# DO NOT EDIT THIS FILE DIRECTLY
+# Instead, edit Jeweler::Tasks in Rakefile, and run the gemspec command
+# -*- encoding: utf-8 -*-
+
+Gem::Specification.new do |s|
+  s.name = %q{delayed_job}
+  s.version = "2.1.0.pre"
+
+  s.required_rubygems_version = Gem::Requirement.new("> 1.3.1") if s.respond_to? :required_rubygems_version=
+  s.authors = ["Brandon Keepers", "Tobias L\303\274tke"]
+  s.date = %q{2010-05-21}
+  s.description = %q{Delayed_job (or DJ) encapsulates the common pattern of asynchronously executing longer tasks in the background. It is a direct extraction from Shopify where the job table is responsible for a multitude of core tasks.
+
+This gem is collectiveidea's fork (http://github.com/collectiveidea/delayed_job).}
+  s.email = %q{tobi@leetsoft.com}
+  s.extra_rdoc_files = [
+    "README.textile"
+  ]
+  s.files = [
+    ".gitignore",
+     "MIT-LICENSE",
+     "README.textile",
+     "Rakefile",
+     "VERSION",
+     "benchmarks.rb",
+     "contrib/delayed_job.monitrc",
+     "contrib/delayed_job_multiple.monitrc",
+     "delayed_job.gemspec",
+     "generators/delayed_job/delayed_job_generator.rb",
+     "generators/delayed_job/templates/migration.rb",
+     "generators/delayed_job/templates/script",
+     "init.rb",
+     "lib/delayed/backend/active_record.rb",
+     "lib/delayed/backend/base.rb",
+     "lib/delayed/backend/couch_rest.rb",
+     "lib/delayed/backend/data_mapper.rb",
+     "lib/delayed/backend/mongo_mapper.rb",
+     "lib/delayed/command.rb",
+     "lib/delayed/message_sending.rb",
+     "lib/delayed/performable_method.rb",
+     "lib/delayed/railtie.rb",
+     "lib/delayed/recipes.rb",
+     "lib/delayed/tasks.rb",
+     "lib/delayed/worker.rb",
+     "lib/delayed/yaml_ext.rb",
+     "lib/delayed_job.rb",
+     "lib/generators/delayed_job/delayed_job_generator.rb",
+     "lib/generators/delayed_job/templates/migration.rb",
+     "lib/generators/delayed_job/templates/script",
+     "rails/init.rb",
+     "recipes/delayed_job.rb",
+     "spec/autoloaded/clazz.rb",
+     "spec/autoloaded/struct.rb",
+     "spec/backend/active_record_job_spec.rb",
+     "spec/backend/couch_rest_job_spec.rb",
+     "spec/backend/data_mapper_job_spec.rb",
+     "spec/backend/mongo_mapper_job_spec.rb",
+     "spec/backend/shared_backend_spec.rb",
+     "spec/message_sending_spec.rb",
+     "spec/performable_method_spec.rb",
+     "spec/sample_jobs.rb",
+     "spec/setup/active_record.rb",
+     "spec/setup/couch_rest.rb",
+     "spec/setup/data_mapper.rb",
+     "spec/setup/mongo_mapper.rb",
+     "spec/spec_helper.rb",
+     "spec/worker_spec.rb",
+     "tasks/jobs.rake"
+  ]
+  s.homepage = %q{http://github.com/collectiveidea/delayed_job}
+  s.rdoc_options = ["--main", "README.textile", "--inline-source", "--line-numbers"]
+  s.require_paths = ["lib"]
+  s.rubygems_version = %q{1.3.6}
+  s.summary = %q{Database-backed asynchronous priority queue system -- Extracted from Shopify}
+  s.test_files = [
+    "spec/message_sending_spec.rb",
+     "spec/performable_method_spec.rb",
+     "spec/worker_spec.rb"
+  ]
+
+  if s.respond_to? :specification_version then
+    current_version = Gem::Specification::CURRENT_SPECIFICATION_VERSION
+    s.specification_version = 3
+
+    if Gem::Version.new(Gem::RubyGemsVersion) >= Gem::Version.new('1.2.0') then
+      s.add_runtime_dependency(%q<daemons>, [">= 0"])
+      s.add_development_dependency(%q<rspec>, [">= 0"])
+      s.add_development_dependency(%q<sqlite3-ruby>, [">= 0"])
+      s.add_development_dependency(%q<activerecord>, [">= 0"])
+      s.add_development_dependency(%q<mongo_mapper>, [">= 0"])
+      s.add_development_dependency(%q<dm-core>, [">= 0"])
+      s.add_development_dependency(%q<dm-observer>, [">= 0"])
+      s.add_development_dependency(%q<dm-aggregates>, [">= 0"])
+      s.add_development_dependency(%q<dm-validations>, [">= 0"])
+      s.add_development_dependency(%q<do_sqlite3>, [">= 0"])
+      s.add_development_dependency(%q<couchrest>, [">= 0"])
+    else
+      s.add_dependency(%q<daemons>, [">= 0"])
+      s.add_dependency(%q<rspec>, [">= 0"])
+      s.add_dependency(%q<sqlite3-ruby>, [">= 0"])
+      s.add_dependency(%q<activerecord>, [">= 0"])
+      s.add_dependency(%q<mongo_mapper>, [">= 0"])
+      s.add_dependency(%q<dm-core>, [">= 0"])
+      s.add_dependency(%q<dm-observer>, [">= 0"])
+      s.add_dependency(%q<dm-aggregates>, [">= 0"])
+      s.add_dependency(%q<dm-validations>, [">= 0"])
+      s.add_dependency(%q<do_sqlite3>, [">= 0"])
+      s.add_dependency(%q<couchrest>, [">= 0"])
+    end
+  else
+    s.add_dependency(%q<daemons>, [">= 0"])
+    s.add_dependency(%q<rspec>, [">= 0"])
+    s.add_dependency(%q<sqlite3-ruby>, [">= 0"])
+    s.add_dependency(%q<activerecord>, [">= 0"])
+    s.add_dependency(%q<mongo_mapper>, [">= 0"])
+    s.add_dependency(%q<dm-core>, [">= 0"])
+    s.add_dependency(%q<dm-observer>, [">= 0"])
+    s.add_dependency(%q<dm-aggregates>, [">= 0"])
+    s.add_dependency(%q<dm-validations>, [">= 0"])
+    s.add_dependency(%q<do_sqlite3>, [">= 0"])
+    s.add_dependency(%q<couchrest>, [">= 0"])
+  end
+end
+

data/init.rb

@@ -0,0 +1 @@
+require File.join(File.dirname(__FILE__), 'rails', 'init')

data/lib/delayed/backend/active_record.rb

@@ -1,16 +1,16 @@
 require 'active_record'
 
 class ActiveRecord::Base
-  def self.load_for_delayed_job(id)
-    if id
-      find(id)
-    else
-      super
-    end
-  end
+  yaml_as "tag:ruby.yaml.org,2002:ActiveRecord"
   
-  def dump_for_delayed_job
-    "#{self.class};#{id}"
+  def self.yaml_new(klass, tag, val)
+    klass.find(val['attributes']['id'])
+  rescue ActiveRecord::RecordNotFound
+    nil
+  end
+
+  def to_yaml_properties
+    ['@attributes']
   end
 end
 
@@ -63,10 +63,8 @@ module Delayed
             self.class.update_all(["locked_at = ?", now], ["id = ? and locked_by = ?", id, worker])
           end
           if affected_rows == 1
-            self.locked_at = now
-            self.locked_by = worker
-            self.locked_at_will_change!
-            self.locked_by_will_change!
+            self.locked_at    = now
+            self.locked_by    = worker
             return true
           else
             return false

data/lib/delayed/backend/base.rb

@@ -1,5 +1,8 @@
 module Delayed
   module Backend
+    class DeserializationError < StandardError
+    end
+
     module Base
       def self.included(base)
         base.extend ClassMethods
@@ -13,19 +16,11 @@ module Delayed
             raise ArgumentError, 'Cannot enqueue items which do not respond to perform'
           end
     
-          priority = args.first || Delayed::Worker.default_priority
+          priority = args.first || 0
           run_at   = args[1]
           self.create(:payload_object => object, :priority => priority.to_i, :run_at => run_at)
         end
-
-        def reserve(worker, max_run_time = Worker.max_run_time)
-          # We get up to 5 jobs from the db. In case we cannot get exclusive access to a job we try the next.
-          # this leads to a more even distribution of jobs across the worker processes
-          find_available(worker.name, 5, max_run_time).detect do |job|
-            job.lock_exclusively!(max_run_time, worker.name)
-          end
-        end
-
+        
         # Hook method that is called before a new worker is forked
         def before_fork
         end
@@ -50,20 +45,19 @@ module Delayed
       def name
         @name ||= begin
           payload = payload_object
-          if payload.respond_to?(:display_name)
-            payload.display_name
-          else
-            payload.class.name
-          end
+          payload.respond_to?(:display_name) ? payload.display_name : payload.class.name
         end
       end
 
       def payload_object=(object)
-        self['handler'] = object.to_yaml
+        self.handler = object.to_yaml
       end
       
       def payload_object
-        @payload_object ||= deserialize(self['handler'])
+        @payload_object ||= YAML.load(self.handler)
+      rescue TypeError, LoadError, NameError => e
+          raise DeserializationError,
+            "Job failed to load: #{e.message}. Try to manually require the required file. Handler: #{handler.inspect}"
       end
 
       # Moved into its own method so that new_relic can trace it.
@@ -76,51 +70,13 @@ module Delayed
         self.locked_at    = nil
         self.locked_by    = nil
       end
-    
-      def reschedule_at
-        payload_object.respond_to?(:reschedule_at) ? 
-          payload_object.reschedule_at(self.class.db_time_now, attempts) :
-          self.class.db_time_now + (attempts ** 4) + 5
-      end
-
-      def max_attempts
-        payload_object.max_attempts if payload_object.respond_to?(:max_attempts)
-      end
-      
-    private
-
-      def deserialize(source)
-        handler = YAML.load(source) rescue nil
-
-        unless handler.respond_to?(:perform)
-          if handler.nil? && source =~ ParseObjectFromYaml
-            handler_class = $1
-          end
-          attempt_to_load(handler_class || handler.class)
-          handler = YAML.load(source)
-        end
-
-        return handler if handler.respond_to?(:perform)
-
-        raise DeserializationError,
-          'Job failed to load: Unknown handler. Try to manually require the appropriate file.'
-      rescue TypeError, LoadError, NameError, ArgumentError => e
-        raise DeserializationError,
-          "Job failed to load: #{e.message}. Try to manually require the required file."
-      end
-
-      # Constantize the object so that ActiveSupport can attempt
-      # its auto loading magic. Will raise LoadError if not successful.
-      def attempt_to_load(klass)
-         klass.constantize
-      end
       
     protected
 
       def set_default_run_at
         self.run_at ||= self.class.db_time_now
-      end    
-
+      end
+    
     end
   end
-end
+end