que 0.10.0 → 0.11.0

data/docs/writing_reliable_jobs.md

@@ -4,60 +4,68 @@ Que does everything it can to ensure that jobs are worked exactly once, but if s
 
 The safest type of job is one that reads in data, either from the database or from external APIs, then does some number crunching and writes the results to the database. These jobs are easy to make safe - simply write the results to the database inside a transaction, and also have the job destroy itself inside that transaction, like so:
 
-    class UpdateWidgetPrice < Que::Job
-      def run(widget_id)
-        widget = Widget[widget_id]
-        price  = ExternalService.get_widget_price(widget_id)
-
-        ActiveRecord::Base.transaction do
-          # Make changes to the database.
-          widget.update :price => price
-
-          # Destroy the job.
-          destroy
-        end
-      end
+```ruby
+class UpdateWidgetPrice < Que::Job
+  def run(widget_id)
+    widget = Widget[widget_id]
+    price  = ExternalService.get_widget_price(widget_id)
+
+    ActiveRecord::Base.transaction do
+      # Make changes to the database.
+      widget.update :price => price
+
+      # Destroy the job.
+      destroy
     end
+  end
+end
+```
 
 Here, you're taking advantage of the guarantees of an [ACID](https://en.wikipedia.org/wiki/ACID) database. The job is destroyed along with the other changes, so either the write will succeed and the job will be run only once, or it will fail and the database will be left untouched. But even if it fails, the job can simply be retried, and there are no lingering effects from the first attempt, so no big deal.
 
 The more difficult type of job is one that makes changes that can't be controlled transactionally. For example, writing to an external service:
 
-    class ChargeCreditCard < Que::Job
-      def run(user_id, credit_card_id)
-        CreditCardService.charge(credit_card_id, :amount => "$10.00")
+```ruby
+class ChargeCreditCard < Que::Job
+  def run(user_id, credit_card_id)
+    CreditCardService.charge(credit_card_id, :amount => "$10.00")
 
-        ActiveRecord::Base.transaction do
-          User.where(:id => user_id).update_all :charged_at => Time.now
-          destroy
-        end
-      end
+    ActiveRecord::Base.transaction do
+      User.where(:id => user_id).update_all :charged_at => Time.now
+      destroy
     end
+  end
+end
+```
 
 What if the process abruptly dies after we tell the provider to charge the credit card, but before we finish the transaction? Que will retry the job, but there's no way to tell where (or even if) it failed the first time. The credit card will be charged a second time, and then you've got an angry customer. The ideal solution in this case is to make the job [idempotent](https://en.wikipedia.org/wiki/Idempotence), meaning that it will have the same effect no matter how many times it is run:
 
-    class ChargeCreditCard < Que::Job
-      def run(user_id, credit_card_id)
-        unless CreditCardService.check_for_previous_charge(credit_card_id)
-          CreditCardService.charge(credit_card_id, :amount => "$10.00")
-        end
-
-        ActiveRecord::Base.transaction do
-          User.where(:id => user_id).update_all :charged_at => Time.now
-          destroy
-        end
-      end
+```ruby
+class ChargeCreditCard < Que::Job
+  def run(user_id, credit_card_id)
+    unless CreditCardService.check_for_previous_charge(credit_card_id)
+      CreditCardService.charge(credit_card_id, :amount => "$10.00")
     end
 
+    ActiveRecord::Base.transaction do
+      User.where(:id => user_id).update_all :charged_at => Time.now
+      destroy
+    end
+  end
+end
+```
+
 This makes the job slightly more complex, but reliable (or, at least, as reliable as your credit card service).
 
 Finally, there are some jobs where you won't want to write to the database at all:
 
-    class SendVerificationEmail < Que::Job
-      def run(email_address)
-        Mailer.verification_email(email_address).deliver
-      end
-    end
+```ruby
+class SendVerificationEmail < Que::Job
+  def run(email_address)
+    Mailer.verification_email(email_address).deliver
+  end
+end
+```
 
 In this case, we don't have any no way to prevent the occasional double-sending of an email. But, for ease of use, you can leave out the transaction and the `destroy` call entirely - Que will recognize that the job wasn't destroyed and will clean it up for you.
 
@@ -69,36 +77,40 @@ Que doesn't offer a general way to kill jobs that have been running too long, be
 
 However, if there's part of your job that is prone to hang (due to an API call or other HTTP request that never returns, for example), you can timeout those individual parts of your job relatively safely. For example, consider a job that needs to make an HTTP request and then write to the database:
 
-    require 'net/http'
+```ruby
+require 'net/http'
 
-    class ScrapeStuff < Que::Job
-      def run(domain_to_scrape, path_to_scrape)
-        result = Net::HTTP.get(domain_to_scrape, path_to_scrape)
+class ScrapeStuff < Que::Job
+  def run(domain_to_scrape, path_to_scrape)
+    result = Net::HTTP.get(domain_to_scrape, path_to_scrape)
 
-        ActiveRecord::Base.transaction do
-          # Insert result...
+    ActiveRecord::Base.transaction do
+      # Insert result...
 
-          destroy
-        end
-      end
+      destroy
     end
+  end
+end
+```
 
 That request could take a very long time, or never return at all. Let's wrap it in a five-second timeout:
 
-    require 'net/http'
-    require 'timeout'
+```ruby
+require 'net/http'
+require 'timeout'
 
-    class ScrapeStuff < Que::Job
-      def run(domain_to_scrape, path_to_scrape)
-        result = Timeout.timeout(5){Net::HTTP.get(domain_to_scrape, path_to_scrape)}
+class ScrapeStuff < Que::Job
+  def run(domain_to_scrape, path_to_scrape)
+    result = Timeout.timeout(5){Net::HTTP.get(domain_to_scrape, path_to_scrape)}
 
-        ActiveRecord::Base.transaction do
-          # Insert result...
+    ActiveRecord::Base.transaction do
+      # Insert result...
 
-          destroy
-        end
-      end
+      destroy
     end
+  end
+end
+```
 
 Now, if the request takes more than five seconds, a `Timeout::Error` will be raised and Que will just retry the job later. This solution isn't perfect, since Timeout uses Thread#kill under the hood, which can lead to unpredictable behavior. But it's separate from our transaction, so there's no risk of losing data - even a catastrophic error that left Net::HTTP in a bad state would be fixable by restarting the process.
 

data/lib/que/adapters/active_record.rb

@@ -14,6 +14,20 @@ module Que
         end
       end
 
+      def cleanup!
+        # ActiveRecord will check out connections to the current thread when
+        # queries are executed and not return them to the pool until
+        # explicitly requested to. The wisdom of this API is questionable, and
+        # it doesn't pose a problem for the typical case of workers using a
+        # single PG connection (since we ensure that connection is checked in
+        # and checked out responsibly), but since ActiveRecord supports
+        # connections to multiple databases, it's easy for people using that
+        # feature to unknowingly leak connections to other databases. So, take
+        # the additional step of telling ActiveRecord to check in all of the
+        # current thread's connections between jobs.
+        ::ActiveRecord::Base.clear_active_connections!
+      end
+
       class TransactionCallback
         def has_transactional_callbacks?
           true

data/lib/que/adapters/base.rb

@@ -21,6 +21,11 @@ module Que
         raise NotImplementedError
       end
 
+      # Called after Que has returned its connection to whatever pool it's
+      # using.
+      def cleanup!
+      end
+
       # Called after a job is queued in async mode, to prompt a worker to
       # wake up after the current transaction commits. Not all adapters will
       # implement this.
@@ -60,7 +65,7 @@ module Que
         checkout do |conn|
           # Prepared statement errors have the potential to foul up the entire
           # transaction, so if we're in one, err on the side of safety.
-          return execute_sql(SQL[name], params) if in_transaction?
+          return execute_sql(SQL[name], params) if Que.disable_prepared_statements || in_transaction?
 
           statements = @prepared_statements[conn] ||= {}
 
@@ -86,21 +91,6 @@ module Que
         end
       end
 
-      HASH_DEFAULT_PROC = proc { |hash, key| hash[key.to_s] if Symbol === key }
-
-      INDIFFERENTIATOR = proc do |object|
-        case object
-        when Array
-          object.each(&INDIFFERENTIATOR)
-        when Hash
-          object.default_proc = HASH_DEFAULT_PROC
-          object.each { |key, value| object[key] = INDIFFERENTIATOR.call(value) }
-          object
-        else
-          object
-        end
-      end
-
       CAST_PROCS = {}
 
       # Integer, bigint, smallint:
@@ -128,11 +118,7 @@ module Que
           end
         end
 
-        if result.first.respond_to?(:with_indifferent_access)
-          output.map(&:with_indifferent_access)
-        else
-          output.each(&INDIFFERENTIATOR)
-        end
+        output.map!(&Que.json_converter)
       end
     end
   end

data/lib/que/job.rb

@@ -79,65 +79,70 @@ module Que
         # Since we're taking session-level advisory locks, we have to hold the
         # same connection throughout the process of getting a job, working it,
         # deleting it, and removing the lock.
-        Que.adapter.checkout do
-          begin
-            if job = Que.execute(:lock_job, [queue]).first
-              # Edge case: It's possible for the lock_job query to have
-              # grabbed a job that's already been worked, if it took its MVCC
-              # snapshot while the job was processing, but didn't attempt the
-              # advisory lock until it was finished. Since we have the lock, a
-              # previous worker would have deleted it by now, so we just
-              # double check that it still exists before working it.
-
-              # Note that there is currently no spec for this behavior, since
-              # I'm not sure how to reliably commit a transaction that deletes
-              # the job in a separate thread between lock_job and check_job.
-              if Que.execute(:check_job, job.values_at(:queue, :priority, :run_at, :job_id)).none?
-                {:event => :job_race_condition}
+        return_value =
+          Que.adapter.checkout do
+            begin
+              if job = Que.execute(:lock_job, [queue]).first
+                # Edge case: It's possible for the lock_job query to have
+                # grabbed a job that's already been worked, if it took its MVCC
+                # snapshot while the job was processing, but didn't attempt the
+                # advisory lock until it was finished. Since we have the lock, a
+                # previous worker would have deleted it by now, so we just
+                # double check that it still exists before working it.
+
+                # Note that there is currently no spec for this behavior, since
+                # I'm not sure how to reliably commit a transaction that deletes
+                # the job in a separate thread between lock_job and check_job.
+                if Que.execute(:check_job, job.values_at(:queue, :priority, :run_at, :job_id)).none?
+                  {:event => :job_race_condition}
+                else
+                  klass = class_for(job[:job_class])
+                  klass.new(job)._run
+                  {:event => :job_worked, :job => job}
+                end
               else
-                klass = class_for(job[:job_class])
-                klass.new(job)._run
-                {:event => :job_worked, :job => job}
+                {:event => :job_unavailable}
               end
-            else
-              {:event => :job_unavailable}
-            end
-          rescue => error
-            begin
-              if job
-                count    = job[:error_count].to_i + 1
-                interval = klass && klass.respond_to?(:retry_interval) && klass.retry_interval || retry_interval
-                delay    = interval.respond_to?(:call) ? interval.call(count) : interval
-                message  = "#{error.message}\n#{error.backtrace.join("\n")}"
-                Que.execute :set_error, [count, delay, message] + job.values_at(:queue, :priority, :run_at, :job_id)
+            rescue => error
+              begin
+                if job
+                  count    = job[:error_count].to_i + 1
+                  interval = klass && klass.respond_to?(:retry_interval) && klass.retry_interval || retry_interval
+                  delay    = interval.respond_to?(:call) ? interval.call(count) : interval
+                  message  = "#{error.message}\n#{error.backtrace.join("\n")}"
+                  Que.execute :set_error, [count, delay, message] + job.values_at(:queue, :priority, :run_at, :job_id)
+                end
+              rescue
+                # If we can't reach the database for some reason, too bad, but
+                # don't let it crash the work loop.
               end
-            rescue
-              # If we can't reach the database for some reason, too bad, but
-              # don't let it crash the work loop.
-            end
 
-            if Que.error_handler
-              # Similarly, protect the work loop from a failure of the error handler.
-              Que.error_handler.call(error, job) rescue nil
-            end
+              if Que.error_handler
+                # Similarly, protect the work loop from a failure of the error handler.
+                Que.error_handler.call(error, job) rescue nil
+              end
 
-            return {:event => :job_errored, :error => error, :job => job}
-          ensure
-            # Clear the advisory lock we took when locking the job. Important
-            # to do this so that they don't pile up in the database. Again, if
-            # we can't reach the database, don't crash the work loop.
-            begin
-              Que.execute "SELECT pg_advisory_unlock($1)", [job[:job_id]] if job
-            rescue
+              return {:event => :job_errored, :error => error, :job => job}
+            ensure
+              # Clear the advisory lock we took when locking the job. Important
+              # to do this so that they don't pile up in the database. Again, if
+              # we can't reach the database, don't crash the work loop.
+              begin
+                Que.execute "SELECT pg_advisory_unlock($1)", [job[:job_id]] if job
+              rescue
+              end
             end
           end
-        end
+
+        Que.adapter.cleanup!
+
+        return_value
       end
 
       private
 
       def class_for(string)
-        string.split('::').inject(Object, &:const_get)
+        Que.constantize(string)
       end
     end
   end

data/lib/que/railtie.rb

@@ -2,28 +2,13 @@ module Que
   class Railtie < Rails::Railtie
     config.que = Que
 
-    Que.logger     = proc { Rails.logger }
-    Que.mode       = :sync if Rails.env.test?
-    Que.connection = ::ActiveRecord if defined? ::ActiveRecord
+    Que.logger         = proc { Rails.logger }
+    Que.mode           = :sync if Rails.env.test?
+    Que.connection     = ::ActiveRecord if defined? ::ActiveRecord
+    Que.json_converter = :with_indifferent_access.to_proc
 
     rake_tasks do
       load 'que/rake_tasks.rb'
     end
-
-    initializer 'que.setup' do
-      ActiveSupport.on_load :after_initialize do
-        # Only start up the worker pool if running as a server.
-        Que.mode ||= :async if defined? Rails::Server
-
-        at_exit do
-          if Que.mode == :async
-            $stdout.puts "Finishing Que's current jobs before exiting..."
-            Que.worker_count = 0
-            Que.mode = :off
-            $stdout.puts "Que's jobs finished, exiting..."
-          end
-        end
-      end
-    end
   end
 end

data/lib/que/rake_tasks.rb

@@ -12,6 +12,7 @@ namespace :que do
     Que.logger.level  = Logger.const_get((ENV['QUE_LOG_LEVEL'] || 'INFO').upcase)
     Que.worker_count  = (ENV['QUE_WORKER_COUNT'] || 4).to_i
     Que.wake_interval = (ENV['QUE_WAKE_INTERVAL'] || 0.1).to_f
+    Que.queue_name    = ENV['QUE_QUEUE'] if ENV['QUE_QUEUE']
     Que.mode          = :async
 
     # When changing how signals are caught, be sure to test the behavior with

data/lib/que/sql.rb

@@ -1,8 +1,47 @@
 module Que
   SQL = {
-    # Thanks to RhodiumToad in #postgresql for help with the job lock CTE.
+    # Locks a job using a Postgres recursive CTE [1].
+    #
+    # As noted by the Postgres documentation, it may be slightly easier to
+    # think about this expression as iteration rather than recursion, despite
+    # the `RECURSION` nomenclature defined by the SQL standards committee.
+    # Recursion is used here so that jobs in the table can be iterated one-by-
+    # one until a lock can be acquired, where a non-recursive `SELECT` would
+    # have the undesirable side-effect of locking multiple jobs at once. i.e.
+    # Consider that the following would have the worker lock *all* unlocked
+    # jobs:
+    #
+    #   SELECT (j).*, pg_try_advisory_lock((j).job_id) AS locked
+    #   FROM que_jobs AS j;
+    #
+    # The CTE will initially produce an "anchor" from the non-recursive term
+    # (i.e. before the `UNION`), and then use it as the contents of the
+    # working table as it continues to iterate through `que_jobs` looking for
+    # a lock. The jobs table has a sort on (priority, run_at, job_id) which
+    # allows it to walk the jobs table in a stable manner. As noted above, the
+    # recursion examines one job at a time so that it only ever acquires a
+    # single lock.
+    #
+    # The recursion has two possible end conditions:
+    #
+    # 1. If a lock *can* be acquired, it bubbles up to the top-level `SELECT`
+    #    outside of the `job` CTE which stops recursion because it is
+    #    constrained with a `LIMIT` of 1.
+    #
+    # 2. If a lock *cannot* be acquired, the recursive term of the expression
+    #    (i.e. what's after the `UNION`) will return an empty result set
+    #    because there are no more candidates left that could possibly be
+    #    locked. This empty result automatically ends recursion.
+    #
+    # Note that this query can be easily modified to lock any number of jobs
+    # by tweaking the LIMIT clause in the main SELECT statement.
+    #
+    # [1] http://www.postgresql.org/docs/devel/static/queries-with.html
+    #
+    # Thanks to RhodiumToad in #postgresql for help with the original version
+    # of the job lock CTE.
     :lock_job => %{
-      WITH RECURSIVE job AS (
+      WITH RECURSIVE jobs AS (
         SELECT (j).*, pg_try_advisory_lock((j).job_id) AS locked
         FROM (
           SELECT j
@@ -20,18 +59,18 @@ module Que
               FROM que_jobs AS j
               WHERE queue = $1::text
               AND run_at <= now()
-              AND (priority, run_at, job_id) > (job.priority, job.run_at, job.job_id)
+              AND (priority, run_at, job_id) > (jobs.priority, jobs.run_at, jobs.job_id)
               ORDER BY priority, run_at, job_id
               LIMIT 1
             ) AS j
-            FROM job
-            WHERE NOT job.locked
+            FROM jobs
+            WHERE jobs.job_id IS NOT NULL
             LIMIT 1
           ) AS t1
         )
       )
       SELECT queue, priority, run_at, job_id, job_class, args, error_count
-      FROM job
+      FROM jobs
       WHERE locked
       LIMIT 1
     }.freeze,

data/lib/que/version.rb

@@ -1,3 +1,3 @@
 module Que
-  Version = '0.10.0'
+  Version = '0.11.0'
 end

data/lib/que/worker.rb

@@ -119,6 +119,7 @@ module Que
 
     class << self
       attr_reader :mode, :wake_interval, :worker_count
+      attr_accessor :queue_name
 
       # In order to work in a forking webserver, we need to be able to accept
       # worker_count and wake_interval settings without actually instantiating
@@ -162,7 +163,7 @@ module Que
 
       def set_up_workers
         if worker_count > workers.count
-          workers.push(*(worker_count - workers.count).times.map{new(ENV['QUE_QUEUE'] || '')})
+          workers.push(*(worker_count - workers.count).times.map{new(queue_name || '')})
         elsif worker_count < workers.count
           workers.pop(workers.count - worker_count).each(&:stop).each(&:wait_until_stopped)
         end

data/lib/que.rb

@@ -16,9 +16,38 @@ module Que
     JSON_MODULE = JSON
   end
 
+  HASH_DEFAULT_PROC = proc { |hash, key| hash[key.to_s] if Symbol === key }
+
+  INDIFFERENTIATOR = proc do |object|
+    case object
+    when Array
+      object.each(&INDIFFERENTIATOR)
+    when Hash
+      object.default_proc = HASH_DEFAULT_PROC
+      object.each { |key, value| object[key] = INDIFFERENTIATOR.call(value) }
+      object
+    else
+      object
+    end
+  end
+
+  SYMBOLIZER = proc do |object|
+    case object
+    when Hash
+      object.keys.each do |key|
+        object[key.to_sym] = SYMBOLIZER.call(object.delete(key))
+      end
+      object
+    when Array
+      object.map! { |e| SYMBOLIZER.call(e) }
+    else
+      object
+    end
+  end
+
   class << self
     attr_accessor :error_handler
-    attr_writer :logger, :adapter, :log_formatter
+    attr_writer :logger, :adapter, :log_formatter, :disable_prepared_statements, :json_converter
 
     def connection=(connection)
       self.adapter =
@@ -96,6 +125,19 @@ module Que
       @log_formatter ||= JSON_MODULE.method(:dump)
     end
 
+    def disable_prepared_statements
+      @disable_prepared_statements || false
+    end
+
+    def constantize(camel_cased_word)
+      if camel_cased_word.respond_to?(:constantize)
+        # Use ActiveSupport's version if it exists.
+        camel_cased_word.constantize
+      else
+        string.split('::').inject(Object, &:const_get)
+      end
+    end
+
     # A helper method to manage transactions, used mainly by the migration
     # system. It's available for general use, but if you're using an ORM that
     # provides its own transaction helper, be sure to use that instead, or the
@@ -122,8 +164,12 @@ module Que
       end
     end
 
+    def json_converter
+      @json_converter ||= INDIFFERENTIATOR
+    end
+
     # Copy some of the Worker class' config methods here for convenience.
-    [:mode, :mode=, :worker_count, :worker_count=, :wake_interval, :wake_interval=, :wake!, :wake_all!].each do |meth|
+    [:mode, :mode=, :worker_count, :worker_count=, :wake_interval, :wake_interval=, :queue_name, :queue_name=, :wake!, :wake_all!].each do |meth|
       define_method(meth) { |*args| Worker.send(meth, *args) }
     end
   end

data/que.gemspec

@@ -14,7 +14,7 @@ Gem::Specification.new do |spec|
   spec.license       = 'MIT'
 
   spec.files         = `git ls-files`.split($/)
-  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.executables   = ['que']
   spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
   spec.require_paths = ['lib']
 

data/spec/adapters/active_record_spec.rb

@@ -67,10 +67,17 @@ unless defined?(RUBY_ENGINE) && RUBY_ENGINE == 'jruby'
     end
 
     it "should instantiate args as ActiveSupport::HashWithIndifferentAccess" do
-      ArgsJob.enqueue :param => 2
-      Que::Job.work
-      $passed_args.first[:param].should == 2
-      $passed_args.first.should be_an_instance_of ActiveSupport::HashWithIndifferentAccess
+      begin
+        # Mimic the setting in the Railtie.
+        Que.json_converter = :with_indifferent_access.to_proc
+
+        ArgsJob.enqueue :param => 2
+        Que::Job.work
+        $passed_args.first[:param].should == 2
+        $passed_args.first.should be_an_instance_of ActiveSupport::HashWithIndifferentAccess
+      ensure
+        Que.json_converter = Que::INDIFFERENTIATOR
+      end
     end
 
     it "should support Rails' special extensions for times" do
@@ -119,5 +126,25 @@ unless defined?(RUBY_ENGINE) && RUBY_ENGINE == 'jruby'
         Que.adapter.should be_in_transaction
       end
     end
+
+    it "should not leak connections to other databases when using ActiveRecord's multiple database support" do
+      class SecondDatabaseModel < ActiveRecord::Base
+        establish_connection(QUE_URL)
+      end
+
+      SecondDatabaseModel.clear_active_connections!
+      SecondDatabaseModel.connection_handler.active_connections?.should == false
+
+      class SecondDatabaseModelJob < Que::Job
+        def run(*args)
+          SecondDatabaseModel.connection.execute("SELECT 1")
+        end
+      end
+
+      SecondDatabaseModelJob.enqueue
+      Que::Job.work
+
+      SecondDatabaseModel.connection_handler.active_connections?.should == false
+    end
   end
 end