que 0.0.1 → 0.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: df82eb95360d4f2dd2d6ae5185a209836946a5a3
-  data.tar.gz: edbd98c7b468295a1d67ff135c076def834a1f0f
+  metadata.gz: 12eec9dd8d0889bfe3253afef4fa850332fcd541
+  data.tar.gz: 7419a5b65a4f96e48b7e4817b9281fc5b0e0e1aa
 SHA512:
-  metadata.gz: d93d47004b33e2f1e7e82e05351b629238d36eb9e48b8a142f421d267217f99ba51c6916267e9fb2c636a0d0ddfc5ab680c8b1168b6f089ace6cb396bb2989a1
-  data.tar.gz: c697670fcd1525986e22fa743cbae99b7e0107143c7e2b7e5b26ab52a4bdc4ec8f1d0fcb295cc3dab1861499af42186480058fde8e5ebc1827ad6a7ddf0a515c
+  metadata.gz: 8cc387df0177ea5374563b75079e2ae573b44ad041bda0175b09aa68a23cc31cacdd0741e5f725f83d95f553eca61ae73230c3207aa5fae935e34a653d3ed42d
+  data.tar.gz: ae5e7d28eca45c3cbeeb5fee8323d5e8c4bc62dcdd192e7d7b570144dea5a43c239d9f086165163189ad15fb4df30bb862b5fc06e6d48ae8d3e4a4d6fcf7b699

data/.rspec

@@ -1,3 +1,2 @@
 --color
---profile
 --order random

data/README.md

@@ -1,14 +1,44 @@
 # Que
 
-A job queue that uses Postgres' advisory lock system. The aim is for job queuing to be efficient and highly concurrent (there's no need for a locked_at column or SELECT FOR UPDATE NOWAIT queries) but still very durable (with all the stability that Postgres offers).
+Que is a queue for Ruby applications that manages jobs using PostgreSQL's advisory locks. There are several advantages to this design:
 
-It was extracted from an app of mine where it worked well for me, but I wouldn't recommend anyone use it until it's generalized some more. It currently requires the use of Sequel as an ORM, but I expect that it could be expanded to support ActiveRecord.
+* **Safety** - If a Ruby process dies, its jobs won't be lost, or left in a locked or ambiguous state - they immediately become available for any other worker to pick up.
+* **Efficiency** - Locking a job doesn't incur a disk write or hold open a transaction.
+* **Concurrency** - There's no locked_at column or SELECT FOR UPDATE-style locking, so workers don't block each other when locking jobs.
 
-Features:
-  * Jobs can be queued transactionally, alongside every other change to your database.
-  * If a worker process crashes or segfaults, the jobs it was working are immediately released to be picked up by other workers.
-  * Workers are multi-threaded, similar to Sidekiq, so the same process can work many jobs at the same time.
-  * Workers can run in your web processes. This means if you're on Heroku, there's no need to have a worker process constantly running.
+Additionally, there are the general benefits of storing jobs in Postgres rather than a dedicated queue:
+
+* **Transactional control** - Queue a job along with other changes to your database, and it'll commit or rollback with everything else.
+* **Atomic backups** - Your jobs and data can be backed up together and restored as a snapshot. If your jobs relate to your data (and they usually do), this is important if you don't want to lose anything during a restoration.
+* **Fewer dependencies** - If you're already using Postgres (and you probably should be), a separate queue is another moving part that can break.
+
+Que's primary goal is reliability. When it's stable, you should be able to leave your application running indefinitely without worrying about jobs being lost due to a lack of transactional support, or left in limbo due to a crashing process. Que works hard to make sure that jobs you queue are performed exactly once.
+
+Que's secondary goal is performance. It won't be able to match the speed or throughput of a dedicated queue, or maybe even a Redis-backed queue, but it should be plenty fast for most use cases. It also includes a worker pool, so that multiple threads can process jobs in the same process. It can even do this in the background of your web process - if you're running on Heroku, for example, you won't need to run a separate worker dyno.
+
+The rakefile includes a benchmark that tries to compare the performance and concurrency of Que's locking mechanism to that of DelayedJob and QueueClassic. On my i5 quad-core laptop, the results are along the lines of:
+
+    ~/que $ rake benchmark
+    Benchmarking 1000 jobs, 10 workers and synchronous_commit = on...
+    Benchmarking delayed_job... 1000 jobs in 30.086127964 seconds = 33 jobs per second
+    Benchmarking queue_classic... 1000 jobs in 19.642309724 seconds = 51 jobs per second
+    Benchmarking que... 1000 jobs in 2.31483287 seconds = 432 jobs per second
+    Benchmarking que_lateral... 1000 jobs in 2.383887915 seconds = 419 jobs per second
+
+Or, minus the I/O limitations of my 5400 rpm hard drive:
+
+    ~/que $ SYNCHRONOUS_COMMIT=off rake benchmark
+    Benchmarking 1000 jobs, 10 workers and synchronous_commit = off...
+    Benchmarking delayed_job... 1000 jobs in 4.906474583 seconds = 204 jobs per second
+    Benchmarking queue_classic... 1000 jobs in 1.587542394 seconds = 630 jobs per second
+    Benchmarking que... 1000 jobs in 0.39063824 seconds = 2560 jobs per second
+    Benchmarking que_lateral... 1000 jobs in 0.392068154 seconds = 2551 jobs per second
+
+As always, this is a single naive benchmark that doesn't represent anything real, take it with a grain of salt, try it for yourself, etc.
+
+**Que was extracted from an app of mine that ran in production for a few months. That queue worked well, but Que has been adapted somewhat from that design in order to support multiple ORMs and other features. Please don't trust Que with your production data until we've all tried to break it a few times.**
+
+Right now, Que is only tested on Ruby 2.0 - it may work on other versions. It requires Postgres 9.2+ for the JSON type. The benchmark requires Postgres 9.3, since it also tests a variant of the typical locking query that uses the new LATERAL syntax.
 
 ## Installation
 
@@ -26,72 +56,83 @@ Or install it yourself as:
 
 ## Usage
 
-Create a jobs table that looks something like:
+The following is assuming you're using Rails 4.0. Que hasn't been tested with previous versions of Rails.
 
-    CREATE TABLE jobs
-    (
-      priority integer NOT NULL,
-      run_at timestamp with time zone NOT NULL DEFAULT now(),
-      job_id bigserial NOT NULL,
-      created_at timestamp with time zone NOT NULL DEFAULT now(),
-      type text NOT NULL,
-      args json NOT NULL DEFAULT '[]'::json,
-      data json NOT NULL DEFAULT '{}'::json,
-      CONSTRAINT jobs_pkey PRIMARY KEY (priority, run_at, job_id),
-      CONSTRAINT valid_priority CHECK (priority >= 1 AND priority <= 5)
-    );
+First, generate a migration for the jobs table.
 
-Start up the worker to process jobs:
-
-    # In an initializer:
-    Que::Worker.state = :async
-
-    # Alternatively, you can set state to :sync to process jobs in the same
-    # thread as they're queued, which is useful for testing, or set it to :off
-    # to simply leave them in the jobs table.
+    rails generate que:install
+    rake db:migrate
 
 Create a class for each type of job you want to run:
 
+    # app/jobs/charge_credit_card.rb
     class ChargeCreditCard < Que::Job
-      @default_priority = 1 # Highest priority.
+      # Custom job options.
+      @default_priority = 3
+      @default_run_at = proc { 1.minute.from_now }
 
-      def perform(user_id, card_id)
+      def run(user_id, card_id)
         # Do stuff.
 
-        db.transaction do
+        ActiveRecord::Base.transaction do
           # Write any changes you'd like to the database.
 
           # It's best to destroy the job in the same transaction as any other
-          # changes you make. Que will destroy the job for you after the
-          # perform method if you don't do it here, but if your job writes to
-          # the DB but doesn't destroy the job in the same transaction, it's
+          # changes you make. Que will destroy the job for you after the run
+          # method if you don't do it yourself, but if your job writes to the
+          # DB but doesn't destroy the job in the same transaction, it's
           # possible that the job could be repeated in the event of a crash.
           destroy
         end
       end
     end
 
-Queue that type of job. Again, it's best to take advantage of Postgres by doing this in a transaction with other changes you're making:
+Queue your job. Again, it's best to do this in a transaction with other changes you're making.
 
-    DB.transaction do
+    ActiveRecord::Base.transaction do
       # Persist credit card information
       card = CreditCard.create(params[:credit_card])
       ChargeCreditCard.queue(current_user.id, card.id)
     end
 
+You can also schedule it to run at a specific time, or with a specific priority:
+
+    # 1 is high priority, 5 is low priority.
+    ChargeCreditCard.queue current_user.id, card.id, :run_at => 1.day.from_now, :priority => 5
+
+There are a few ways to work jobs. In development and production, the default is for Que to run a pool of workers to process jobs in their own background threads. If you like, you can disable this behavior when configuring your Rails app:
+
+    config.que.mode = :off
+
+You can also change the mode at any time with Que.mode = :off. The other options are :async and :sync. :async runs the background workers, while :sync will run any jobs you queue synchronously (that is, MyJob.queue runs the job immediately and won't return until it's completed). This makes your application's behavior easier to test, so it's the default in the test environment.
+
+If you don't want to run workers in your web process, you can also work jobs in a rake task, similar to how other queueing systems work:
+
+    # Run a pool of 4 workers.
+    rake que:work
+
+    # Or configure the number of workers.
+    WORKER_COUNT=8 rake que:work
+
+    # If your app code isn't thread-safe, you can stick to one worker.
+    WORKER_COUNT=1 rake que:work
+
+If an error causes a job to fail, Que will repeat that job at intervals that increase exponentially with each error (using the same algorithm as DelayedJob). You can also hook Que into whatever error notification system you're using:
+
+    config.que.error_handler = proc do |error|
+      # Do whatever you want with the error object.
+    end
+
+You can find documentation on more issues at the project's [Github wiki](https://github.com/chanks/que/wiki).
+
 ## TODO
 
-These aren't promises, just ideas for stuff that could be done in the future.
+These aren't promises, just ideas for possible features:
 
-  * Railtie, to make default setup easy.
-  * ActiveRecord support.
-  * Keep deleted jobs.
-  * Use LISTEN/NOTIFY for checking for new jobs, rather than signaling the worker within the same process.
+  * Use LISTEN/NOTIFY to check for new jobs (or simpler, just wake a worker in the same process after a transaction commits a new job).
   * Multiple queues (in multiple tables?)
-  * More configurable logging that isn't tied to Rails.
   * Integration with ActionMailer for easier mailings.
-  * Options for max_run_time and max_attempts that are specific to job classes.
-  * Rake tasks for creating/working/dropping/clearing queues.
+  * Add options for max_run_time and max_attempts. Make them specific to job classes.
 
 ## Contributing
 

data/Rakefile

@@ -1 +1,413 @@
 require 'bundler/gem_tasks'
+
+require 'rspec/core/rake_task'
+
+RSpec::Core::RakeTask.new :default do |spec|
+  spec.pattern = './spec/**/*_spec.rb'
+end
+
+# Shortcut to skip the adapter specs, and run only with the basic PG
+# connection. I use this occasionally to make sure ActiveRecord isn't loaded,
+# so any accidental Rails-isms are caught.
+RSpec::Core::RakeTask.new :pg do |spec|
+  spec.pattern = './spec/*_spec.rb'
+end
+
+task :benchmark do
+  # The following is a somewhat simplistic benchmark (aren't they all) meant
+  # to compare the speed and concurrency of the locking mechanisms used by Que
+  # (standard and lateral queries), DelayedJob and QueueClassic - it does this
+  # by simply sending the raw queries that each system sends during operation.
+
+  # It is NOT meant to benchmark the overall performance of each system (which
+  # would include the time each spends working in Ruby), but to see which one
+  # supports the highest concurrency under load, assuming that there will be
+  # many workers and that Postgres will be the bottleneck. I'm unsure how
+  # useful it is for this, but it's a start.
+
+  JOB_COUNT          = (ENV['JOB_COUNT'] || 1000).to_i
+  WORKER_COUNT       = (ENV['WORKER_COUNT'] || 10).to_i
+  SYNCHRONOUS_COMMIT = ENV['SYNCHRONOUS_COMMIT'] || 'on'
+
+  require 'uri'
+  require 'pg'
+  require 'connection_pool'
+
+  uri = URI.parse ENV["DATABASE_URL"] || "postgres://postgres:@localhost/que-test"
+
+  new_connection = proc do
+    PG::Connection.open :host     => uri.host,
+                        :user     => uri.user,
+                        :password => uri.password,
+                        :port     => uri.port || 5432,
+                        :dbname   => uri.path[1..-1]
+  end
+
+  pg = new_connection.call
+
+
+
+  # Necessary setup, mostly for QueueClassic. I apologize for this - I hope your editor supports code folding.
+  pg.async_exec <<-SQL
+    SET SESSION client_min_messages = 'WARNING';
+
+    -- Que table.
+    DROP TABLE IF EXISTS que_jobs;
+    CREATE TABLE que_jobs
+    (
+      priority    integer     NOT NULL DEFAULT 1,
+      run_at      timestamptz NOT NULL DEFAULT now(),
+      job_id      bigserial   NOT NULL,
+      job_class   text        NOT NULL,
+      args        json        NOT NULL DEFAULT '[]'::json,
+      error_count integer     NOT NULL DEFAULT 0,
+      last_error  text,
+
+      CONSTRAINT que_jobs_pkey PRIMARY KEY (priority, run_at, job_id)
+    );
+
+    DROP TABLE IF EXISTS que_lateral_jobs;
+    CREATE TABLE que_lateral_jobs
+    (
+      priority    integer     NOT NULL DEFAULT 1,
+      run_at      timestamptz NOT NULL DEFAULT now(),
+      job_id      bigserial   NOT NULL,
+      job_class   text        NOT NULL,
+      args        json        NOT NULL DEFAULT '[]'::json,
+      error_count integer     NOT NULL DEFAULT 0,
+      last_error  text,
+
+      CONSTRAINT que_lateral_jobs_pkey PRIMARY KEY (priority, run_at, job_id)
+    );
+
+    DROP TABLE IF EXISTS delayed_jobs;
+    -- DelayedJob table.
+    CREATE TABLE delayed_jobs
+    (
+      id serial NOT NULL,
+      priority integer NOT NULL DEFAULT 0,
+      attempts integer NOT NULL DEFAULT 0,
+      handler text NOT NULL,
+      last_error text,
+      run_at timestamp without time zone,
+      locked_at timestamp without time zone,
+      failed_at timestamp without time zone,
+      locked_by character varying(255),
+      queue character varying(255),
+      created_at timestamp without time zone,
+      updated_at timestamp without time zone,
+      CONSTRAINT delayed_jobs_pkey PRIMARY KEY (id)
+    );
+    ALTER TABLE delayed_jobs
+      OWNER TO postgres;
+
+    CREATE INDEX delayed_jobs_priority
+      ON delayed_jobs
+      USING btree
+      (priority, run_at);
+
+
+
+    -- QueueClassic table and functions.
+    DROP FUNCTION IF EXISTS lock_head(tname varchar);
+    DROP FUNCTION IF EXISTS lock_head(q_name varchar, top_boundary integer);
+    DROP FUNCTION IF EXISTS queue_classic_notify() cascade;
+    DROP TABLE IF EXISTS queue_classic_jobs;
+
+    CREATE TABLE queue_classic_jobs (
+      id bigserial PRIMARY KEY,
+      q_name text not null check (length(q_name) > 0),
+      method text not null check (length(method) > 0),
+      args   text not null,
+      locked_at timestamptz
+    );
+
+    alter table queue_classic_jobs alter column args type json using (args::json);
+
+    create function queue_classic_notify() returns trigger as $$ begin
+      perform pg_notify(new.q_name, '');
+      return null;
+    end $$ language plpgsql;
+
+    create trigger queue_classic_notify
+    after insert on queue_classic_jobs
+    for each row
+    execute procedure queue_classic_notify();
+
+    CREATE INDEX idx_qc_on_name_only_unlocked ON queue_classic_jobs (q_name, id) WHERE locked_at IS NULL;
+
+    CREATE OR REPLACE FUNCTION lock_head(q_name varchar, top_boundary integer)
+    RETURNS SETOF queue_classic_jobs AS $$
+    DECLARE
+      unlocked bigint;
+      relative_top integer;
+      job_count integer;
+    BEGIN
+      -- The purpose is to release contention for the first spot in the table.
+      -- The select count(*) is going to slow down dequeue performance but allow
+      -- for more workers. Would love to see some optimization here...
+
+      EXECUTE 'SELECT count(*) FROM '
+        || '(SELECT * FROM queue_classic_jobs WHERE q_name = '
+        || quote_literal(q_name)
+        || ' LIMIT '
+        || quote_literal(top_boundary)
+        || ') limited'
+      INTO job_count;
+
+      SELECT TRUNC(random() * (top_boundary - 1))
+      INTO relative_top;
+
+      IF job_count < top_boundary THEN
+        relative_top = 0;
+      END IF;
+
+      LOOP
+        BEGIN
+          EXECUTE 'SELECT id FROM queue_classic_jobs '
+            || ' WHERE locked_at IS NULL'
+            || ' AND q_name = '
+            || quote_literal(q_name)
+            || ' ORDER BY id ASC'
+            || ' LIMIT 1'
+            || ' OFFSET ' || quote_literal(relative_top)
+            || ' FOR UPDATE NOWAIT'
+          INTO unlocked;
+          EXIT;
+        EXCEPTION
+          WHEN lock_not_available THEN
+            -- do nothing. loop again and hope we get a lock
+        END;
+      END LOOP;
+
+      RETURN QUERY EXECUTE 'UPDATE queue_classic_jobs '
+        || ' SET locked_at = (CURRENT_TIMESTAMP)'
+        || ' WHERE id = $1'
+        || ' AND locked_at is NULL'
+        || ' RETURNING *'
+      USING unlocked;
+
+      RETURN;
+    END;
+    $$ LANGUAGE plpgsql;
+
+    CREATE OR REPLACE FUNCTION lock_head(tname varchar)
+    RETURNS SETOF queue_classic_jobs AS $$
+    BEGIN
+      RETURN QUERY EXECUTE 'SELECT * FROM lock_head($1,10)' USING tname;
+    END;
+    $$ LANGUAGE plpgsql;
+
+
+
+
+
+    INSERT INTO que_jobs (job_class, args, priority)
+    SELECT 'Que::Job', ('[' || i || ',{}]')::json, 1
+    FROM generate_Series(1,#{JOB_COUNT}) AS i;
+
+    INSERT INTO que_lateral_jobs (job_class, args, priority)
+    SELECT 'Que::Job', ('[' || i || ',{}]')::json, 1
+    FROM generate_Series(1,#{JOB_COUNT}) AS i;
+
+    INSERT INTO delayed_jobs (handler, run_at, created_at, updated_at)
+    SELECT '--- !ruby/struct:NewsletterJob\ntext: lorem ipsum...\nemails: blah@blah.com\n', now(), now(), now()
+    FROM generate_Series(1,#{JOB_COUNT}) AS i;
+
+    INSERT INTO queue_classic_jobs (q_name, method, args)
+    SELECT 'default', 'Kernel.puts', '["hello world"]'
+    FROM generate_Series(1,#{JOB_COUNT}) AS i;
+
+
+
+
+    -- Necessary tables and functions made, now stock them with jobs and analyze.
+    ANALYZE;
+  SQL
+
+
+  queries = {
+    :que => (
+      <<-SQL
+        WITH RECURSIVE cte AS (
+          SELECT (job).*, pg_try_advisory_lock((job).job_id) AS locked
+          FROM (
+            SELECT job
+            FROM que_jobs AS job
+            WHERE run_at <= now()
+            ORDER BY priority, run_at, job_id
+            LIMIT 1
+          ) AS t1
+          UNION ALL (
+            SELECT (job).*, pg_try_advisory_lock((job).job_id) AS locked
+            FROM (
+              SELECT (
+               SELECT job
+                FROM que_jobs AS job
+                WHERE run_at <= now() AND (priority, run_at, job_id) > (cte.priority, cte.run_at, cte.job_id)
+                ORDER BY priority, run_at, job_id
+                LIMIT 1
+              ) AS job
+              FROM cte
+              WHERE NOT cte.locked
+              LIMIT 1
+            ) AS t1
+          )
+        )
+        SELECT job_id, priority, run_at, args, job_class, error_count
+        FROM cte
+        WHERE locked
+        LIMIT 1
+      SQL
+    ),
+    :que_lateral => (
+      <<-SQL
+        WITH RECURSIVE cte AS (
+          SELECT *, pg_try_advisory_lock(s.job_id) AS locked
+          FROM (
+            SELECT *
+            FROM que_lateral_jobs
+            WHERE run_at <= now()
+            ORDER BY priority, run_at, job_id
+            LIMIT 1
+          ) s
+          UNION ALL (
+            SELECT j.*, pg_try_advisory_lock(j.job_id) AS locked
+            FROM (
+              SELECT *
+              FROM cte
+              WHERE NOT locked
+            ) t,
+            LATERAL (
+              SELECT *
+              FROM que_lateral_jobs
+              WHERE run_at <= now()
+              AND (priority, run_at, job_id) > (t.priority, t.run_at, t.job_id)
+              ORDER BY priority, run_at, job_id
+              LIMIT 1
+            ) j
+          )
+        )
+        SELECT *
+        FROM cte
+        WHERE locked
+        LIMIT 1
+      SQL
+    ),
+    :delayed_job => (
+      # From delayed_job_active_record
+      <<-SQL
+        UPDATE delayed_jobs
+        SET locked_at = now(),
+            locked_by = $1::text
+        WHERE id IN (
+          SELECT id
+          FROM delayed_jobs
+          WHERE (
+            (run_at <= now() AND (locked_at IS NULL OR locked_at < now()) OR locked_by = $1) AND failed_at IS NULL
+          )
+          ORDER BY priority ASC, run_at ASC
+          LIMIT 1
+          FOR UPDATE
+        )
+        RETURNING *
+      SQL
+    )
+  }
+
+  connections = WORKER_COUNT.times.map do
+    conn = new_connection.call
+    conn.async_exec "SET SESSION synchronous_commit = #{SYNCHRONOUS_COMMIT}"
+    queries.each do |name, sql|
+      conn.prepare(name.to_s, sql)
+    end
+    conn
+  end
+
+
+
+  # Track the ids that are worked, to make sure they're all hit.
+  $results = {
+    :delayed_job   => [],
+    :queue_classic => [],
+    :que           => [],
+    :que_lateral   => []
+  }
+
+  def work_job(type, conn)
+    case type
+    when :delayed_job
+      return unless r = conn.exec_prepared("delayed_job", [conn.object_id]).first
+      $results[type] << r['id']
+      conn.async_exec "DELETE FROM delayed_jobs WHERE id = $1", [r['id']]
+
+    when :queue_classic
+      return unless r = conn.async_exec("SELECT * FROM lock_head($1, $2)", ['default', 9]).first
+      $results[type] << r['id']
+      conn.async_exec "DELETE FROM queue_classic_jobs WHERE id = $1", [r['id']]
+
+    when :que
+      begin
+        return unless r = conn.exec_prepared("que").first
+        # Have to double-check that the job is valid, as explained at length in Que::Job.work.
+        return true unless conn.async_exec("SELECT * FROM que_jobs WHERE priority = $1 AND run_at = $2 AND job_id = $3", [r['priority'], r['run_at'], r['job_id']]).first
+        conn.async_exec "DELETE FROM que_jobs WHERE priority = $1 AND run_at = $2 AND job_id = $3", [r['priority'], r['run_at'], r['job_id']]
+        $results[type] << r['job_id']
+      ensure
+        conn.async_exec "SELECT pg_advisory_unlock_all()" if r
+      end
+
+    when :que_lateral
+      begin
+        return unless r = conn.exec_prepared("que_lateral").first
+        return true unless conn.async_exec("SELECT * FROM que_lateral_jobs WHERE priority = $1 AND run_at = $2 AND job_id = $3", [r['priority'], r['run_at'], r['job_id']]).first
+        conn.async_exec "DELETE FROM que_lateral_jobs WHERE priority = $1 AND run_at = $2 AND job_id = $3", [r['priority'], r['run_at'], r['job_id']]
+        $results[type] << r['job_id']
+      ensure
+        conn.async_exec "SELECT pg_advisory_unlock_all()" if r
+      end
+
+    end
+  end
+
+  puts "Benchmarking #{JOB_COUNT} jobs, #{WORKER_COUNT} workers and synchronous_commit = #{SYNCHRONOUS_COMMIT}..."
+
+  {
+    :delayed_job   => :delayed_jobs,
+    :queue_classic => :queue_classic_jobs,
+    :que           => :que_jobs,
+    :que_lateral   => :que_lateral_jobs
+  }.each do |type, table|
+    print "Benchmarking #{type}... "
+    start = Time.now
+
+    threads = connections.map do |conn|
+      Thread.new do
+        loop do
+          begin
+            break unless work_job(type, conn)
+          rescue
+            # DelayedJob deadlocks sometimes.
+          end
+        end
+      end
+    end
+
+    threads.each &:join
+    time = Time.now - start
+    puts "#{JOB_COUNT} jobs in #{time} seconds = #{(JOB_COUNT / time).round} jobs per second"
+
+
+    # These checks are commented out because I can't seem to get DelayedJob to
+    # pass them (Que and QueueClassic don't have the same problem). It seems
+    # to repeat some jobs multiple times on every run, and its run times are
+    # also highly variable.
+
+    # worked = $results[type].map(&:to_i).sort
+    # puts "Jobs worked more than once! #{worked.inspect}" unless worked == worked.uniq
+    # puts "Jobs worked less than once! #{worked.inspect}" unless worked.length == JOB_COUNT
+
+    puts "Jobs left in DB" unless pg.async_exec("SELECT count(*) FROM #{table}").first['count'].to_i == 0
+    puts "Advisory locks left over!" if pg.async_exec("SELECT * FROM pg_locks WHERE locktype = 'advisory'").first
+  end
+end