que 0.2.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: f76b4a74e17281ea1007a2c313e146bcda43e1fb
-  data.tar.gz: 6ff1046d4cf009062327e79fb9be628032b252ce
+  metadata.gz: eca6d367104b9c0dcb8c5fb591641bebdc7c4a8f
+  data.tar.gz: a71620a15051793942059b7c423a2f78cc3b6f16
 SHA512:
-  metadata.gz: 319ea6bea73cdd606b4c3103beee120564aa29976255b2eb14992029b2f13768e33164b5134ad5e0b39a038086380294f02d7226a5c2e1f5d092ef32ef03a2e0
-  data.tar.gz: 44144ce58a99d38ced64a4e6259bafa66c7d6373ad125844efc27f45c990c764ad1e04e8e1e1b33f834e9b9be6ae3e812f2cfd3f9ce65303d19a070070b63832
+  metadata.gz: d59625b0b10cca06e54f9a6714bc5cb701d4895936ad9a5ff4a6500ab1f165f0ad5f3c293fded447979e3aa34379480591518e0819160c857a2a22a754c70d8b
+  data.tar.gz: 92741560f486b993a950538cb8187fa7183ef02822d386479b978401d5fc08d5750a93418b44c68fa66eb5ace6d90691fb9c9c86e472caad59e5650cab3b27cb

data/CHANGELOG.md

@@ -1,3 +1,15 @@
+### 0.3.0 (2013-12-21)
+
+*   Add Que.stop!, which immediately kills all jobs being worked in the process.
+
+    This can leave database connections and such in an unpredictable state, and so should only be used when the process is exiting.
+
+*   Use Que.stop! to safely handle processes that exit while Que is running.
+
+    Previously, a job that was in the middle of a transaction when the process was killed with SIGINT or SIGTERM would have had its work committed prematurely.
+
+*   Clean up internals and hammer out several race conditions.
+
 ### 0.2.0 (2013-11-30)
 
 *   Officially support JRuby 1.7.5+. Earlier versions may work.

data/README.md

@@ -1,44 +1,28 @@
 # Que
 
-Que is a queue for Ruby applications that manages jobs using PostgreSQL's advisory locks. There are several advantages to this design:
+**TL;DR: Que is a high-performance alternative to DelayedJob or QueueClassic that improves the reliability of your application by helping you keep your jobs [consistent](https://en.wikipedia.org/wiki/ACID#Consistency) with the rest of your data.**
 
-* **Safety** - If a Ruby process dies, its jobs won't be lost, or left in a locked or ambiguous state - they immediately become available for any other worker to pick up.
-* **Efficiency** - Locking a job doesn't incur a disk write or hold open a transaction.
-* **Concurrency** - There's no locked_at column or SELECT FOR UPDATE-style locking, so workers don't block each other when locking jobs.
+Que is a queue for Ruby and PostgreSQL that manages jobs using [advisory locks](http://www.postgresql.org/docs/current/static/explicit-locking.html#ADVISORY-LOCKS), which gives it several advantages over other RDBMS-backed queues:
 
-Additionally, there are the general benefits of storing jobs in Postgres rather than a dedicated queue:
+* **Concurrency** - Workers don't block each other when trying to lock jobs, as often occurs with "SELECT FOR UPDATE"-style locking. This allows for very high throughput with a large number of workers.
+* **Efficiency** - Locks are held in memory, so locking a job doesn't incur a disk write. These first two points are what limit performance with other queues - all workers trying to lock jobs have to wait behind one that's persisting its UPDATE on a locked_at column to disk (and the disks of however many other servers your database is replicating to). Under heavy load, Que's bottleneck is CPU, not I/O.
+* **Safety** - If a Ruby process dies, the jobs it is working won't be lost, or left in a locked or ambiguous state - they immediately become available for any other worker to pick up.
 
-* **Transactional control** - Queue a job along with other changes to your database, and it'll commit or rollback with everything else.
-* **Atomic backups** - Your jobs and data can be backed up together and restored as a snapshot. If your jobs relate to your data (and they usually do), this is important if you don't want to lose anything during a restoration.
-* **Fewer dependencies** - If you're already using Postgres (and you probably should be), a separate queue is another moving part that can break.
+Additionally, there are the general benefits of storing jobs in Postgres, alongside the rest of your data, rather than in Redis or a dedicated queue:
 
-Que's primary goal is reliability. When it's stable, you should be able to leave your application running indefinitely without worrying about jobs being lost due to a lack of transactional support, or left in limbo due to a crashing process. Que works hard to make sure that jobs you queue are performed exactly once.
+* **Transactional Control** - Queue a job along with other changes to your database, and it'll commit or rollback with everything else. If you're using ActiveRecord or Sequel, Que can piggyback on their connections, so setup is simple and jobs are protected by the transactions you're already using.
+* **Atomic Backups** - Your jobs and data can be backed up together and restored as a snapshot. If your jobs relate to your data (and they usually do), there's no risk of jobs falling through the cracks during a recovery.
+* **Fewer Dependencies** - If you're already using Postgres (and you probably should be), a separate queue is another moving part that can break.
 
-Que's secondary goal is performance. It won't be able to match the speed or throughput of a dedicated queue, or maybe even a Redis-backed queue, but it should be plenty fast for most use cases. It also includes a worker pool, so that multiple threads can process jobs in the same process. It can even do this in the background of your web process - if you're running on Heroku, for example, you won't need to run a separate worker dyno.
+Que's primary goal is reliability. When it's stable, you should be able to leave your application running indefinitely without worrying about jobs being lost due to a lack of transactional support, or left in limbo due to a crashing process. Que does everything it can to ensure that jobs you queue are performed exactly once (though the occasional repetition of a job can be impossible to avoid - see the wiki page on [how to write a reliable job](https://github.com/chanks/que/wiki/Writing-Reliable-Jobs)).
 
-The rakefile includes a benchmark that tries to compare the performance and concurrency of Que's locking mechanism to that of DelayedJob and QueueClassic. On my i5 quad-core laptop, the results are along the lines of:
+Que's secondary goal is performance. It won't be able to match the speed or throughput of a dedicated queue, or maybe even a Redis-backed queue, but it should be fast enough for most use cases. In [benchmarks](https://github.com/chanks/queue-shootout) on an AWS c3.8xlarge instance, Que approaches 10,000 jobs per second, or about twenty times the throughput of DelayedJob or QueueClassic. You are encouraged to try things out on your own production hardware, though.
 
-    ~/que $ rake benchmark_queues
-    Benchmarking 1000 jobs, 10 workers and synchronous_commit = on...
-    Benchmarking delayed_job... 1000 jobs in 30.086127964 seconds = 33 jobs per second
-    Benchmarking queue_classic... 1000 jobs in 19.642309724 seconds = 51 jobs per second
-    Benchmarking que... 1000 jobs in 2.31483287 seconds = 432 jobs per second
-    Benchmarking que_lateral... 1000 jobs in 2.383887915 seconds = 419 jobs per second
+Que also includes a worker pool, so that multiple threads can process jobs in the same process. It can even do this in the background of your web process - if you're running on Heroku, for example, you won't need to run a separate worker dyno.
 
-Or, minus the I/O limitations of my 5400 rpm hard drive:
+*Please be careful when running Que in production. It's still very new compared to other RDBMS-backed queues, and there may be issues that haven't been ironed out yet. Bug reports are welcome.*
 
-    ~/que $ SYNCHRONOUS_COMMIT=off rake benchmark_queues
-    Benchmarking 1000 jobs, 10 workers and synchronous_commit = off...
-    Benchmarking delayed_job... 1000 jobs in 4.906474583 seconds = 204 jobs per second
-    Benchmarking queue_classic... 1000 jobs in 1.587542394 seconds = 630 jobs per second
-    Benchmarking que... 1000 jobs in 0.39063824 seconds = 2560 jobs per second
-    Benchmarking que_lateral... 1000 jobs in 0.392068154 seconds = 2551 jobs per second
-
-As always, this is a single naive benchmark that doesn't represent anything real, take it with a grain of salt, try it for yourself, etc.
-
-**Que was extracted from an app of mine that ran in production for a few months. That queue worked well, but Que has been adapted somewhat from that design in order to support multiple ORMs and other features. Please don't trust Que with your production data until we've all tried to break it a few times.**
-
-Right now, Que is only tested on Ruby 2.0 - it may work on other versions. It requires Postgres 9.2+ for the JSON type. The benchmark requires Postgres 9.3, since it also tests a variant of the typical locking query that uses the new LATERAL syntax.
+Que is tested on Ruby 2.0, Rubinius and JRuby (with the `jruby-pg` gem, which is [not yet functional with ActiveRecord](https://github.com/chanks/que/issues/4#issuecomment-29561356)). It requires Postgres 9.2+ for the JSON datatype.
 
 ## Installation
 
@@ -58,7 +42,7 @@ Or install it yourself as:
 
 The following is assuming you're using Rails 4.0. Que hasn't been tested with previous versions of Rails.
 
-First, generate a migration for the jobs table.
+First, generate a migration for the job table.
 
     rails generate que:install
     rake db:migrate
@@ -71,7 +55,7 @@ Create a class for each type of job you want to run:
       @default_priority = 3
       @default_run_at = proc { 1.minute.from_now }
 
-      def run(user_id, card_id)
+      def run(user_id, card_id, your_options = {})
         # Do stuff.
 
         ActiveRecord::Base.transaction do
@@ -87,24 +71,24 @@ Create a class for each type of job you want to run:
       end
     end
 
-Queue your job. Again, it's best to do this in a transaction with other changes you're making.
+Queue your job. Again, it's best to do this in a transaction with other changes you're making. Also note that any arguments you pass will be serialized to JSON and back again, so stick to simple types (strings, integers, floats, hashes, and arrays).
 
     ActiveRecord::Base.transaction do
       # Persist credit card information
       card = CreditCard.create(params[:credit_card])
-      ChargeCreditCard.queue(current_user.id, card.id)
+      ChargeCreditCard.queue(current_user.id, card.id, :your_custom_option => 'whatever')
     end
 
 You can also schedule it to run at a specific time, or with a specific priority:
 
     # 1 is high priority, 5 is low priority.
-    ChargeCreditCard.queue current_user.id, card.id, :run_at => 1.day.from_now, :priority => 5
+    ChargeCreditCard.queue current_user.id, card.id, :your_custom_option => 'whatever', :run_at => 1.day.from_now, :priority => 5
 
-There are a few ways to work jobs. In development and production, the default is for Que to run a pool of workers to process jobs in their own background threads. If you like, you can disable this behavior when configuring your Rails app:
+To determine what happens when a job is queued, you can set Que's mode with `Que.mode = :off` or `config.que.mode = :off` in your application configuration. There are a few options for the mode:
 
-    config.que.mode = :off
-
-You can also change the mode at any time with Que.mode = :off. The other options are :async and :sync. :async runs the background workers, while :sync will run any jobs you queue synchronously (that is, MyJob.queue runs the job immediately and won't return until it's completed). This makes your application's behavior easier to test, so it's the default in the test environment.
+* `:off` - In this mode, queueing a job will simply insert it into the database - the current process will make no effort to run it. You should use this if you want to use a dedicated process to work tasks (there's a rake task to do this, see below). This is the default when running `rails console` in the development or production environments.
+* `:async` - In this mode, a pool of background workers is spun up, each running in their own thread. They will intermittently check for new jobs. This is the default when running `rails server` in the development or production environments. By default, there are 4 workers and they'll check for a new job every 5 seconds. You can modify these options with `Que.worker_count = 8` or `config.que.worker_count = 8` and `Que.sleep_period = 1` or `config.que.sleep_period = 1`.
+* `:sync` - In this mode, any jobs you queue will be run in the same thread, synchronously (that is, `MyJob.queue` runs the job and won't return until it's completed). This makes your application's behavior easier to test, so it's the default in the test environment.
 
 If you don't want to run workers in your web process, you can also work jobs in a rake task, similar to how other queueing systems work:
 
@@ -114,25 +98,16 @@ If you don't want to run workers in your web process, you can also work jobs in
     # Or configure the number of workers.
     WORKER_COUNT=8 rake que:work
 
-    # If your app code isn't thread-safe, you can stick to one worker.
+    # If your app code isn't thread-safe, be sure to stick to one worker.
     WORKER_COUNT=1 rake que:work
 
-If an error causes a job to fail, Que will repeat that job at intervals that increase exponentially with each error (using the same algorithm as DelayedJob). You can also hook Que into whatever error notification system you're using:
+If an error causes a job to fail, Que will repeat that job at exponentially-increasing intervals, similar to DelayedJob (the job will be retried at 4 seconds, 19 seconds, 84 seconds, 259 seconds...). You can also hook Que into whatever error notification system you're using:
 
     config.que.error_handler = proc do |error|
       # Do whatever you want with the error object.
     end
 
-You can find documentation on more issues at the project's [Github wiki](https://github.com/chanks/que/wiki).
-
-## TODO
-
-These aren't promises, just ideas for possible features:
-
-  * Use LISTEN/NOTIFY to check for new jobs (or simpler, just wake a worker in the same process after a transaction commits a new job).
-  * Multiple queues (in multiple tables?)
-  * Integration with ActionMailer for easier mailings.
-  * Add options for max_run_time and max_attempts. Make them specific to job classes.
+You can find more documentation on the [Github wiki](https://github.com/chanks/que/wiki).
 
 ## Contributing
 
@@ -141,3 +116,13 @@ These aren't promises, just ideas for possible features:
 3. Commit your changes (`git commit -am 'Add some feature'`)
 4. Push to the branch (`git push origin my-new-feature`)
 5. Create new Pull Request
+
+A note on running specs - Que's worker system is multithreaded and therefore prone to race conditions (especially on Rubinius). As such, if you've touched that code, a single spec run passing isn't a guarantee that any changes you've made haven't introduced bugs. One thing I like to do before pushing changes is rerun the specs many times and watching for hangs. You can do this from the command line with something like:
+
+    for i in {1..1000}; do rspec -b --seed $i; done
+
+This will iterate the specs one thousand times, each with a different ordering. If the specs hang, note what the seed number was on that iteration. For example, if the previous specs finished with a "Randomized with seed 328", you know that there's a hang with seed 329, and you can narrow it down to a specific spec with:
+
+    for i in {1..1000}; do LOG_SPEC=true rspec -b --seed 329; done
+
+Note that we iterate because there's no guarantee that the hang would reappear with a single additional run, so we need to rerun the specs until it reappears. The LOG_SPEC parameter will output the name and file location of each spec before it is run, so you can easily tell which spec is hanging, and you can continue narrowing things down from there.

data/lib/generators/que/install_generator.rb

@@ -6,7 +6,7 @@ module Que
   class InstallGenerator < Rails::Generators::Base
     include Rails::Generators::Migration
 
-    namespace "que:install"
+    namespace 'que:install'
     self.source_paths << File.join(File.dirname(__FILE__), 'templates')
     desc "Generates a migration to add Que's job table."
 

data/lib/que.rb

@@ -50,9 +50,9 @@ module Que
       logger.send level, "[Que] #{text}" if logger
     end
 
-    # Duplicate some Worker config methods to the Que module for convenience.
-    [:mode, :mode=, :worker_count=, :sleep_period, :sleep_period=].each do |meth|
-      define_method(meth){|*args| Worker.send(meth, *args)}
+    # Copy some of the Worker class' config methods here for convenience.
+    [:mode, :mode=, :worker_count=, :sleep_period, :sleep_period=, :stop!].each do |meth|
+      define_method(meth) { |*args| Worker.send(meth, *args) }
     end
   end
 end

data/lib/que/job.rb

@@ -7,7 +7,7 @@ module Que
     end
 
     # Subclasses should define their own run methods, but keep an empty one
-    # here so we can just do Que::Job.queue in testing.
+    # here so that Que::Job.queue can queue an empty job in testing.
     def run(*args)
     end
 
@@ -54,11 +54,11 @@ module Que
       end
 
       def work
-        # Job.work will typically be called in a loop, where we'd sleep when
-        # there's no more work to be done, so its return value should reflect
-        # whether we should hit the database again or not. So, return truthy
-        # if we worked a job or encountered a typical error while working a
-        # job, and falsy if we found nothing to do or hit a connection error.
+        # Job.work is typically called in a loop, where we sleep when there's
+        # no more work to be done, so its return value should reflect whether
+        # we should look for another job or not. So, return truthy if we
+        # worked a job or encountered a typical error while working a job, and
+        # falsy if we found nothing to do or hit a connection error.
 
         # Since we're taking session-level advisory locks, we have to hold the
         # same connection throughout the process of getting a job, working it,
@@ -66,16 +66,16 @@ module Que
         Que.adapter.checkout do
           begin
             if row = Que.execute(:lock_job).first
-              # Edge case: It's possible to have grabbed a job that's already
-              # been worked, if the SELECT took its MVCC snapshot while the
-              # job was processing, but didn't attempt the advisory lock until
-              # it was finished. Now that we have the job lock, we know that a
+              # Edge case: It's possible for the lock_job query to have
+              # grabbed a job that's already been worked, if it took its MVCC
+              # snapshot while the job was processing, but didn't attempt the
+              # advisory lock until it was finished. Since we have the lock, a
               # previous worker would have deleted it by now, so we just
               # double check that it still exists before working it.
 
               # Note that there is currently no spec for this behavior, since
               # I'm not sure how to reliably commit a transaction that deletes
-              # the job in a separate thread between this lock and check.
+              # the job in a separate thread between lock_job and check_job.
               return true if Que.execute(:check_job, [row['priority'], row['run_at'], row['job_id']]).none?
 
               run_job(row)
@@ -86,25 +86,27 @@ module Que
           rescue => error
             begin
               if row
-                # Borrowed the exponential backoff formula and error data format from delayed_job.
+                # Borrowed the backoff formula and error data format from delayed_job.
                 count   = row['error_count'].to_i + 1
-                run_at  = Time.now + (count ** 4 + 3)
+                run_at  = count ** 4 + 3
                 message = "#{error.message}\n#{error.backtrace.join("\n")}"
                 Que.execute :set_error, [count, run_at, message, row['priority'], row['run_at'], row['job_id']]
               end
             rescue
-              # If we can't reach the DB for some reason, too bad, but don't
-              # let it crash the work loop.
+              # If we can't reach the database for some reason, too bad, but
+              # don't let it crash the work loop.
             end
 
             if Que.error_handler
+              # Similarly, protect the work loop from a failure of the error handler.
               Que.error_handler.call(error) rescue nil
             end
 
             # If it's a garden variety error, we can just return true, pick up
             # another job, no big deal. If it's a PG::Error, though, assume
             # it's a disconnection or something and that we shouldn't just hit
-            # the database again right away.
+            # the database again right away. We could be a lot more
+            # sophisticated about what errors we delay for, though.
             return !error.is_a?(PG::Error)
           ensure
             # Clear the advisory lock we took when locking the job. Important

data/lib/que/railtie.rb

@@ -1,19 +1,23 @@
 module Que
   class Railtie < Rails::Railtie
     config.que = Que
-    config.que.connection = ::ActiveRecord if defined?(::ActiveRecord)
-    config.que.mode = :sync if Rails.env.test?
+
+    Que.mode       = :sync if Rails.env.test?
+    Que.connection = ::ActiveRecord if defined?(::ActiveRecord)
 
     rake_tasks do
       load 'que/rake_tasks.rb'
     end
 
-    initializer "que.setup" do
-      ActiveSupport.on_load(:after_initialize) do
+    initializer 'que.setup' do
+      ActiveSupport.on_load :after_initialize do
         Que.logger ||= Rails.logger
 
         # Only start up the worker pool if running as a server.
         Que.mode ||= :async if defined? Rails::Server
+
+        # When the process exits, safely interrupt any jobs that are still running.
+        at_exit { Que.stop! }
       end
     end
   end

data/lib/que/rake_tasks.rb

@@ -1,21 +1,26 @@
-require 'logger'
-
 namespace :que do
   desc "Process Que's jobs using a worker pool"
   task :work => :environment do
+    require 'logger'
+
     Que.logger       = Logger.new(STDOUT)
     Que.mode         = :async
     Que.worker_count = (ENV['WORKER_COUNT'] || 4).to_i
 
-    %w(INT TERM).each do |signal|
-      trap signal do
-        puts "SIG#{signal} caught, finishing current jobs and shutting down..."
-        Que.mode = :off
-        $stop = true
-      end
+    # When changing how signals are caught, be sure to test the behavior with
+    # the rake task in tasks/safe_shutdown.rb.
+    at_exit do
+      puts "Stopping Que..."
+      Que.stop!
     end
 
-    loop { sleep 0.01; break if $stop }
+    stop = false
+    trap('INT'){stop = true}
+
+    loop do
+      sleep 0.01
+      break if stop
+    end
   end
 
   desc "Create Que's job table"

data/lib/que/sql.rb

@@ -1,7 +1,7 @@
 module Que
   SQL = {
-    # Thanks to RhodiumToad in #postgresql for the job lock CTE and its lateral
-    # variant. They were modified only slightly from his design.
+    # Thanks to RhodiumToad in #postgresql for the job lock CTE. It was
+    # modified only slightly from his design.
     :lock_job => (
       <<-SQL
         WITH RECURSIVE cte AS (
@@ -29,64 +29,13 @@ module Que
             ) AS t1
           )
         )
-        SELECT job_id, priority, run_at, args, job_class, error_count
+        SELECT priority, run_at, job_id, job_class, args, error_count
         FROM cte
         WHERE locked
         LIMIT 1
       SQL
     ).freeze,
 
-    # Here's an alternate scheme using LATERAL, which will work in Postgres 9.3+.
-    # Basically the same, but benchmark to see if it's faster/just as reliable.
-
-    # WITH RECURSIVE cte AS (
-    #   SELECT *, pg_try_advisory_lock(s.job_id) AS locked
-    #   FROM (
-    #     SELECT *
-    #     FROM que_jobs
-    #     WHERE run_at <= now()
-    #     ORDER BY priority, run_at, job_id
-    #     LIMIT 1
-    #   ) s
-    #   UNION ALL (
-    #     SELECT j.*, pg_try_advisory_lock(j.job_id) AS locked
-    #     FROM (
-    #       SELECT *
-    #       FROM cte
-    #       WHERE NOT locked
-    #     ) t,
-    #     LATERAL (
-    #       SELECT *
-    #       FROM que_jobs
-    #       WHERE run_at <= now()
-    #       AND (priority, run_at, job_id) > (t.priority, t.run_at, t.job_id)
-    #       ORDER BY priority, run_at, job_id
-    #       LIMIT 1
-    #     ) j
-    #   )
-    # )
-    # SELECT *
-    # FROM cte
-    # WHERE locked
-    # LIMIT 1
-
-    :create_table => (
-      <<-SQL
-        CREATE TABLE que_jobs
-        (
-          priority    integer     NOT NULL DEFAULT 1,
-          run_at      timestamptz NOT NULL DEFAULT now(),
-          job_id      bigserial   NOT NULL,
-          job_class   text        NOT NULL,
-          args        json        NOT NULL DEFAULT '[]'::json,
-          error_count integer     NOT NULL DEFAULT 0,
-          last_error  text,
-
-          CONSTRAINT que_jobs_pkey PRIMARY KEY (priority, run_at, job_id)
-        )
-      SQL
-    ).freeze,
-
     :check_job => (
       <<-SQL
         SELECT 1 AS one
@@ -101,7 +50,7 @@ module Que
       <<-SQL
         UPDATE que_jobs
         SET error_count = $1::integer,
-            run_at      = $2::timestamptz,
+            run_at      = now() + $2::integer * '1 second'::interval,
             last_error  = $3::text
         WHERE priority  = $4::integer
         AND   run_at    = $5::timestamptz
@@ -116,6 +65,23 @@ module Que
         AND   run_at   = $2::timestamptz
         AND   job_id   = $3::bigint
       SQL
+    ).freeze,
+
+    :create_table => (
+      <<-SQL
+        CREATE TABLE que_jobs
+        (
+          priority    integer     NOT NULL DEFAULT 1,
+          run_at      timestamptz NOT NULL DEFAULT now(),
+          job_id      bigserial   NOT NULL,
+          job_class   text        NOT NULL,
+          args        json        NOT NULL DEFAULT '[]'::json,
+          error_count integer     NOT NULL DEFAULT 0,
+          last_error  text,
+
+          CONSTRAINT que_jobs_pkey PRIMARY KEY (priority, run_at, job_id)
+        )
+      SQL
     ).freeze
   }
 end