que 0.14.3 → 1.0.0.beta

data/docs/job_helper_methods.md

@@ -0,0 +1,27 @@
+## Job Helper Methods
+
+There are a number of instance methods on Que::Job that you can use in your jobs, preferably in transactions. See [Writing Reliable Jobs](/writing_reliable_jobs.md) for more information on where to use these methods.
+
+### destroy
+
+This method deletes the job from the queue table, ensuring that it won't be worked a second time.
+
+### finish
+
+This method marks the current job as finished, ensuring that it won't be worked a second time. This is like destroy, in that it finalizes a job, but this method leaves the job in the table, in case you want to query it later.
+
+### expire
+
+This method marks the current job as expired. It will be left in the table and won't be retried, but it will be easy to query for expired jobs. This method is called if the job exceeds its maximum_retry_count.
+
+### retry_in
+
+This method marks the current job to be retried later. You can pass a numeric to this method, in which case that is the number of seconds after which it can be retried (`retry_in(10)`, `retry_in(0.5)`), or, if you're using ActiveSupport, you can pass in a duration object (`retry_in(10.minutes)`). This automatically happens, with an exponentially-increasing interval, when the job encounters an error.
+
+### error_count
+
+This method returns the total number of times the job has errored, in case you want to modify the job's behavior after it has failed a given number of times.
+
+### default_resolve_action
+
+If you don't perform a resolve action (destroy, finish, expire, retry_in) while the job is worked, Que will call this method for you. By default it simply calls `destroy`, but you can override it in your Job subclasses if you wish - for example, to call `finish`, or to invoke some more complicated logic.

data/docs/logging.md

@@ -3,7 +3,7 @@
 By default, Que logs important information in JSON to either Rails' logger (when running in a Rails web process) or STDOUT (when running via the `que` executable). So, your logs will look something like:
 
 ```
-I, [2014-01-12T05:07:31.094201 #4687]  INFO -- : {"lib":"que","thread":104928,"event":"job_worked","elapsed":0.01045,"job":{"priority":"1","run_at":"2014-01-12 05:07:31.081877+00","job_id":"4","job_class":"MyJob","args":[],"error_count":"0"}}
+I, [2017-08-12T05:07:31.094201 #4687]  INFO -- : {"lib":"que","hostname":"lovelace","pid":21626,"thread":21471100,"event":"job_worked","job_id":6157665,"elapsed":0.531411}
 ```
 
 Of course you can have it log wherever you like:
@@ -12,26 +12,7 @@ Of course you can have it log wherever you like:
 Que.logger = Logger.new(...)
 ```
 
-You can use Que's logger in your jobs anywhere you like:
-
-```ruby
-class MyJob
-  def run
-    Que.log :my_output => "my string"
-  end
-end
-
-#=> I, [2014-01-12T05:13:11.006776 #4914]  INFO -- : {"lib":"que","thread":24960,"my_output":"my string"}
-```
-
-Que will always add a 'lib' key, so you can easily filter its output from that of other sources, and the object_id of the thread that emitted the log, so you can follow the actions of a particular worker if you wish. You can also pass a :level key to set the level of the output:
-
-```ruby
-Que.log :level => :debug, :my_output => 'my string'
-#=> D, [2014-01-12T05:16:15.221941 #5088] DEBUG -- : {"lib":"que","thread":24960,"my_output":"my string"}
-```
-
-If you don't like JSON, you can also customize the format of the logging output by passing a callable object (such as a proc) to Que.log_formatter=. The proc should take a hash (the keys are symbols) and return a string. The keys and values are just as you would expect from the JSON output:
+If you don't like logging in JSON, you can also customize the format of the logging output by passing a callable object (such as a proc) to Que.log_formatter=. The proc should take a hash (the keys are symbols) and return a string. The keys and values are just as you would expect from the JSON output:
 
 ```ruby
 Que.log_formatter = proc do |data|
@@ -39,7 +20,7 @@ Que.log_formatter = proc do |data|
 end
 ```
 
-If the log formatter returns nil or false, a nothing will be logged at all. You could use this to narrow down what you want to emit, for example:
+If the log formatter returns nil or false, nothing will be logged at all. You could use this to narrow down what you want to emit, for example:
 
 ```ruby
 Que.log_formatter = proc do |data|

data/docs/managing_workers.md

@@ -1,80 +1,25 @@
 ## Managing Workers
 
-Que provides a pool of workers to process jobs in a multithreaded fashion - this allows you to save memory by working many jobs simultaneously in the same process.
+Que uses a multithreaded pool of workers to run jobs in parallel - this allows you to save memory by working many jobs simultaneously in the same process. The `que` executable starts up a pool of 6 workers by default. This is fine for most use cases, but the ideal number for your app will depend on your interpreter and what types of jobs you're running.
 
-When the worker pool is active (when you set `Que.mode = :async`), the default number of workers is 4. This is fine for most use cases, but the ideal number for your app will depend on your interpreter and what types of jobs you're running.
-
-Ruby MRI has a global interpreter lock (GIL), which prevents it from using more than one CPU core at a time. Having multiple workers running makes sense if your jobs tend to spend a lot of time in I/O (waiting on complex database queries, sending emails, making HTTP requests, etc.), as most jobs do. However, if your jobs are doing a lot of work in Ruby, they'll be spending a lot of time blocking each other, and having too many workers running will just slow everything down.
-
-JRuby and Rubinius, on the other hand, have no global interpreter lock, and so can make use of multiple CPU cores - you could potentially set the number of workers very high for them. You should experiment to find the best setting for your use case.
-
-You can change the number of workers in the pool whenever you like by setting the `worker_count` option:
-
-```ruby
-Que.worker_count = 8
-```
+Ruby MRI has a global interpreter lock (GIL), which prevents it from using more than one CPU core at a time. Having multiple workers running makes sense if your jobs tend to spend a lot of time in I/O (waiting on complex database queries, sending emails, making HTTP requests, etc.), as most jobs do. However, if your jobs are doing a lot of work in Ruby, they'll be spending a lot of time blocking each other, and having too many workers running will cause you to lose efficiency to context-switching. So, you'll want to choose the appropriate number of workers for your use case.
 
 ### Working Jobs Via Executable
 
-If you don't want to burden your web processes with too much work and want to run workers in a background process instead, similar to how most other queues work, you can:
-
 ```shell
-# Run a pool of 4 workers:
+# Run a pool of 6 workers:
 que
 
 # Or configure the number of workers:
-que --worker-count 8
+que --worker-count 10
 ```
 
-See `que -h` for a list of command-line options.
+See `que -h` for a complete list of command-line options.
 
 ### Thread-Unsafe Application Code
 
-If your application code is not thread-safe, you won't want any workers to be processing jobs while anything else is happening in the Ruby process. So, you'll want to turn the worker pool off by default:
-
-```ruby
-Que.mode = :off
-```
-
-This will prevent Que from trying to process jobs in the background of your web processes. In order to actually work jobs, you'll want to run a single worker at a time, and to do so via a separate process, like so:
+If your application code is not thread-safe, you won't want any workers to be processing jobs while anything else is happening in the Ruby process. So, you'll want to run a single worker at a time, like so:
 
 ```shell
 que --worker-count 1
 ```
-
-### The Wake Interval
-
-If a worker checks the job queue and finds no jobs ready for it to work, it will fall asleep. In order to make sure that newly-available jobs don't go unworked, a worker is awoken every so often to check for available work. By default, this happens every five seconds, but you can make it happen more or less often by setting a custom wake_interval:
-
-```ruby
-Que.wake_interval = 2 # In Rails, 2.seconds also works fine.
-```
-
-You can also choose to never let workers wake up on their own:
-
-```ruby
-# Never wake up any workers:
-Que.wake_interval = nil
-```
-
-If you do this, though, you'll need to wake workers manually.
-
-### Manually Waking Workers
-
-Regardless of the `wake_interval` setting, you can always wake workers manually:
-
-```ruby
-# Wake up a single worker to check the queue for work:
-Que.wake!
-
-# Wake up all workers in this process to check for work:
-Que.wake_all!
-```
-
-`Que.wake_all!` is helpful if there are no jobs available and all your workers go to sleep, and then you queue a large number of jobs. Typically, it will take a little while for the entire pool of workers get going again - a new one will wake up every `wake_interval` seconds, but it will take up to `wake_interval * worker_count` seconds for all of them to get going. `Que.wake_all!` can get them all moving immediately.
-
-### Connection Pool Size
-
-For the job locking system to work properly, each worker thread needs to reserve a database connection from the connection pool for the period of time between when it locks a job and when it releases that lock (which won't happen until the job has been finished and deleted from the queue).
-
-So, for example, if you're running 6 workers via the executable, you'll want to make sure that whatever connection pool Que is using (usually ActiveRecord's) has a maximum size of at least 6. If you're running those workers in a web process, you'll want the size to be at least 6 plus however many connections you expect your application to need for serving web requests (which may only be one if you're using Rails in single-threaded mode, or many more if you're running a threaded web server like Puma).

data/docs/middleware.md

@@ -0,0 +1,15 @@
+## Defining Middleware For Jobs
+
+You can define middleware to wrap jobs. For example:
+
+``` ruby
+Que.middleware.push(
+  -> (job, &block) {
+    # Do stuff with the job object - report on it, count time elapsed, etc.
+    block.call
+    nil # Doesn't matter what's returned.
+  }
+)
+```
+
+This API is experimental for the 1.0 beta and may change.

data/docs/migrating.md

@@ -3,13 +3,13 @@
 Some new releases of Que may require updates to the database schema. It's recommended that you integrate these updates alongside your other database migrations. For example, when Que released version 0.6.0, the schema version was updated from 2 to 3. If you're running ActiveRecord, you could make a migration to perform this upgrade like so:
 
 ```ruby
-class UpdateQue < ActiveRecord::Migration
+class UpdateQue < ActiveRecord::Migration[5.0]
   def self.up
-    Que.migrate! :version => 3
+    Que.migrate! version: 3
   end
 
   def self.down
-    Que.migrate! :version => 2
+    Que.migrate! version: 2
   end
 end
 ```
@@ -18,10 +18,7 @@ This will make sure that your database schema stays consistent with your codebas
 
 ```ruby
 # Change schema to version 3.
-Que.migrate! :version => 3
-
-# Update to whatever the latest schema version is.
-Que.migrate!
+Que.migrate! version: 3
 
 # Check your current schema version.
 Que.db_version #=> 3

data/docs/multiple_queues.md

@@ -1,27 +1,31 @@
 ## Multiple Queues
 
-Que supports the use of multiple queues in a single job table. This feature is intended to support the case where multiple applications (with distinct codebases) are sharing the same database. For instance, you might have a separate Ruby application that handles only processing credit cards. In that case, you can run that application's workers against a specific queue:
+Que supports the use of multiple queues in a single job table. Please note that this feature is intended to support the case where multiple codebases are sharing the same job queue - if you want to support jobs of differing priorities, the numeric priority system offers better flexibility and performance.
+
+For instance, you might have a separate Ruby application that handles only processing credit cards. In that case, you can run that application's workers against a specific queue:
 
 ```shell
 que --queue-name credit_cards
+# The -q flag is equivalent, and either can be passed multiple times.
+que -q default -q credit_cards
 ```
 
 Then you can set jobs to be enqueued in that queue specifically:
 
 ```ruby
-ProcessCreditCard.enqueue current_user.id, :queue => 'credit_cards'
+ProcessCreditCard.enqueue current_user.id, queue: 'credit_cards'
 
 # Or:
 
 class ProcessCreditCard < Que::Job
   # Set a default queue for this job class; this can be overridden by
   # passing the :queue parameter to enqueue like above.
-  @queue = 'credit_cards'
+  self.queue = 'credit_cards'
 end
 ```
 
 In some cases, the ProcessCreditCard class may not be defined in the application that is enqueueing the job. In that case, you can specify the job class as a string:
 
 ```ruby
-Que.enqueue current_user.id, :job_class => 'ProcessCreditCard', :queue => 'credit_cards'
+Que.enqueue current_user.id, job_class: 'ProcessCreditCard', queue: 'credit_cards'
 ```

data/docs/shutting_down_safely.md

@@ -2,6 +2,6 @@
 
 To ensure safe operation, Que needs to be very careful in how it shuts down. When a Ruby process ends normally, it calls Thread#kill on any threads that are still running - unfortunately, if a thread is in the middle of a transaction when this happens, there is a risk that it will be prematurely commited, resulting in data corruption. See [here](http://blog.headius.com/2008/02/ruby-threadraise-threadkill-timeoutrb.html) and [here](http://coderrr.wordpress.com/2011/05/03/beware-of-threadkill-or-your-activerecord-transactions-are-in-danger-of-being-partially-committed/) for more detail on this.
 
-To prevent this, Que will block a Ruby process from exiting until all jobs it is working have completed normally. Unfortunately, if you have long-running jobs, this may take a very long time (and if something goes wrong with a job's logic, it may never happen). The solution in this case is SIGKILL - luckily, Ruby processes that are killed via SIGKILL will end without using Thread#kill on its running threads. This is safer than exiting normally - when PostgreSQL loses the connection it will simply roll back the open transaction, if any, and unlock the job so it can be retried later by another worker. Be sure to read [Writing Reliable Jobs](https://github.com/chanks/que/blob/master/docs/writing_reliable_jobs.md) for information on how to design your jobs to fail safely.
+To prevent this, Que will block the worker process from exiting until all jobs it is working have completed normally. Unfortunately, if you have long-running jobs, this may take a very long time (and if something goes wrong with a job's logic, it may never happen). The solution in this case is SIGKILL - luckily, Ruby processes that are killed via SIGKILL will end without using Thread#kill on its running threads. This is safer than exiting normally - when PostgreSQL loses the connection it will simply roll back the open transaction, if any, and unlock the job so it can be retried later by another worker. Be sure to read [Writing Reliable Jobs](https://github.com/chanks/que/blob/master/docs/writing_reliable_jobs.md) for information on how to design your jobs to fail safely.
 
 So, be prepared to use SIGKILL on your Ruby processes if they run for too long. For example, Heroku takes a good approach to this - when Heroku's platform is shutting down a process, it sends SIGTERM, waits ten seconds, then sends SIGKILL if the process still hasn't exited. This is a nice compromise - it will give each of your currently running jobs ten seconds to complete, and any jobs that haven't finished by then will be interrupted and retried later.

data/docs/using_plain_connections.md

@@ -1,6 +1,10 @@
 ## Using Plain Postgres Connections
 
-If you're not using an ORM like ActiveRecord or Sequel, you can use one of two gems to manage a connection pool for your PG connections. The first is the ConnectionPool gem (be sure to add `gem 'connection_pool'` to your Gemfile):
+If you're not using an ORM like ActiveRecord or Sequel, you can use a distinct connection pool to manage your Postgres connections. Please be aware that if you **are** using ActiveRecord or Sequel, there's no reason for you to be using any of these methods - it's less efficient (unnecessary connections will waste memory on your database server) and you lose the reliability benefits of wrapping jobs in the same transactions as the rest of your data.
+
+## Using ConnectionPool or Pond
+
+Support for two connection pool gems is included in Que. The first is the ConnectionPool gem (be sure to add `gem 'connection_pool'` to your Gemfile):
 
 ```ruby
 require 'uri'
@@ -9,13 +13,14 @@ require 'connection_pool'
 
 uri = URI.parse(ENV['DATABASE_URL'])
 
-Que.connection = ConnectionPool.new :size => 10 do
-  PG::Connection.open :host     => uri.host,
-                      :user     => uri.user,
-                      :password => uri.password,
-                      :port     => uri.port || 5432,
-                      :dbname   => uri.path[1..-1]
-end
+Que.connection = ConnectionPool.new(size: 10) do
+  PG::Connection.open(
+    host:     uri.host,
+    user:     uri.user,
+    password: uri.password,
+    port:     uri.port || 5432,
+    dbname:   uri.path[1..-1]
+  )end
 ```
 
 Be sure to pick your pool size carefully - if you use 10 for the size, you'll incur the overhead of having 10 connections open to Postgres even if you never use more than a couple of them.
@@ -29,13 +34,32 @@ require 'pond'
 
 uri = URI.parse(ENV['DATABASE_URL'])
 
-Que.connection = Pond.new :maximum_size => 10 do
-  PG::Connection.open :host     => uri.host,
-                      :user     => uri.user,
-                      :password => uri.password,
-                      :port     => uri.port || 5432,
-                      :dbname   => uri.path[1..-1]
+Que.connection = Pond.new(maximum_size: 10) do
+  PG::Connection.open(
+    host:     uri.host,
+    user:     uri.user,
+    password: uri.password,
+    port:     uri.port || 5432,
+    dbname:   uri.path[1..-1]
+  )
+end
+```
+
+## Using Any Other Connection Pool
+
+You can use any other in-process connection pool by defining access to it in a proc that's passed to `Que.connection_proc = proc`. The proc you pass should accept a block and call it with a connection object. For instance, Que's built-in interface to Sequel's connection pool is basically implemented like:
+
+```ruby
+Que.connection_proc = proc do |&block|
+  DB.synchronize do |connection|
+    block.call(connection)
+  end
 end
 ```
 
-Please be aware that if you're using ActiveRecord or Sequel to manage your data, there's no reason for you to be using any of these methods - it's less efficient (unnecessary connections will waste memory on your database server) and you lose the reliability benefits of wrapping jobs in the same transactions as the rest of your data. In general, your app should probably be using a connection pool, and Que should probably hook into whatever connection pool you're already using.
+This proc must meet a few requirements:
+- The yielded object must be an instance of `PG::Connection`.
+- It must be reentrant - if it is called with a block, and then called again inside that block, it must return the same object. For example, in `proc.call{|conn1| proc.call{|conn2| conn1.object_id == conn2.object_id}}` the innermost condition must be true.
+- It must lock the connection object and prevent any other thread from accessing it for the duration of the block.
+
+If any of these conditions aren't met, Que will raise an error.

data/docs/using_sequel.md

@@ -11,7 +11,7 @@ Then you can safely use the same database object to transactionally protect your
 
 ```ruby
 class MyJob < Que::Job
-  def run
+  def run(user_id:)
     # Do stuff.
 
     DB.transaction do
@@ -23,9 +23,11 @@ class MyJob < Que::Job
   end
 end
 
-# In your controller action:
+# Or, in your controller action:
 DB.transaction do
   @user = User.create(params[:user])
-  MyJob.enqueue :user_id => @user.id
+  MyJob.enqueue user_id: @user.id
 end
 ```
+
+Sequel automatically wraps model persistance actions (create, update, destroy) in transactions, so you can simply call #enqueue methods from your models' callbacks, if you wish.

data/docs/writing_reliable_jobs.md

@@ -2,7 +2,7 @@
 
 Que does everything it can to ensure that jobs are worked exactly once, but if something bad happens when a job is halfway completed, there's no way around it - the job will need be repeated over again from the beginning, probably by a different worker. When you're writing jobs, you need to be prepared for this to happen.
 
-The safest type of job is one that reads in data, either from the database or from external APIs, then does some number crunching and writes the results to the database. These jobs are easy to make safe - simply write the results to the database inside a transaction, and also have the job destroy itself inside that transaction, like so:
+The safest type of job is one that reads in data, either from the database or from external APIs, then does some number crunching and writes the results to the database. These jobs are easy to make safe - simply write the results to the database inside a transaction, and also destroy the job inside that transaction, like so:
 
 ```ruby
 class UpdateWidgetPrice < Que::Job
@@ -12,9 +12,9 @@ class UpdateWidgetPrice < Que::Job
 
     ActiveRecord::Base.transaction do
       # Make changes to the database.
-      widget.update :price => price
+      widget.update price: price
 
-      # Destroy the job.
+      # Mark the job as destroyed, so it doesn't run again.
       destroy
     end
   end
@@ -28,10 +28,10 @@ The more difficult type of job is one that makes changes that can't be controlle
 ```ruby
 class ChargeCreditCard < Que::Job
   def run(user_id, credit_card_id)
-    CreditCardService.charge(credit_card_id, :amount => "$10.00")
+    CreditCardService.charge(credit_card_id, amount: "$10.00")
 
     ActiveRecord::Base.transaction do
-      User.where(:id => user_id).update_all :charged_at => Time.now
+      User.where(id: user_id).update_all charged_at: Time.now
       destroy
     end
   end
@@ -44,11 +44,11 @@ What if the process abruptly dies after we tell the provider to charge the credi
 class ChargeCreditCard < Que::Job
   def run(user_id, credit_card_id)
     unless CreditCardService.check_for_previous_charge(credit_card_id)
-      CreditCardService.charge(credit_card_id, :amount => "$10.00")
+      CreditCardService.charge(credit_card_id, amount: "$10.00")
     end
 
     ActiveRecord::Base.transaction do
-      User.where(:id => user_id).update_all :charged_at => Time.now
+      User.where(id: user_id).update_all charged_at: Time.now
       destroy
     end
   end
@@ -71,18 +71,14 @@ In this case, we don't have any no way to prevent the occasional double-sending
 
 ### Timeouts
 
-Long-running jobs aren't necessarily a problem in Que, since the overhead of an individual job isn't that big (just an open PG connection and an advisory lock held in memory). But jobs that hang indefinitely can tie up a worker and [block the Ruby process from exiting gracefully](https://github.com/chanks/que/blob/master/docs/shutting_down_safely.md), which is a pain.
+Long-running jobs aren't necessarily a problem for the database, since the overhead of an individual job is very small (just an advisory lock held in memory). But jobs that hang indefinitely can tie up a worker and [block the Ruby process from exiting gracefully](https://github.com/chanks/que/blob/master/docs/shutting_down_safely.md), which is a pain.
 
-Que doesn't offer a general way to kill jobs that have been running too long, because that currently can't be done safely in Ruby. Typically, one would use Ruby's Timeout module for this sort of thing, but wrapping a database transaction inside a timeout introduces a risk of premature commits, which can corrupt your data. See [here](http://blog.headius.com/2008/02/ruby-threadraise-threadkill-timeoutrb.html) and [here](http://coderrr.wordpress.com/2011/05/03/beware-of-threadkill-or-your-activerecord-transactions-are-in-danger-of-being-partially-committed/) for detail on why this is.
-
-However, if there's part of your job that is prone to hang (due to an API call or other HTTP request that never returns, for example), you can timeout those individual parts of your job relatively safely. For example, consider a job that needs to make an HTTP request and then write to the database:
+If there's part of your job that is prone to hang (due to an API call or other HTTP request that never returns, for example), you can (and should) timeout those parts of your job. For example, consider a job that needs to make an HTTP request and then write to the database:
 
 ```ruby
-require 'net/http'
-
 class ScrapeStuff < Que::Job
-  def run(domain_to_scrape, path_to_scrape)
-    result = Net::HTTP.get(domain_to_scrape, path_to_scrape)
+  def run(url_to_scrape)
+    result = YourHTTPLibrary.get(url_to_scrape)
 
     ActiveRecord::Base.transaction do
       # Insert result...
@@ -93,15 +89,12 @@ class ScrapeStuff < Que::Job
 end
 ```
 
-That request could take a very long time, or never return at all. Let's wrap it in a five-second timeout:
+That request could take a very long time, or never return at all. Let's use the timeout feature that almost all HTTP libraries offer some version of:
 
 ```ruby
-require 'net/http'
-require 'timeout'
-
 class ScrapeStuff < Que::Job
-  def run(domain_to_scrape, path_to_scrape)
-    result = Timeout.timeout(5){Net::HTTP.get(domain_to_scrape, path_to_scrape)}
+  def run(url_to_scrape)
+    result = YourHTTPLibrary.get(url_to_scrape, timeout: 5)
 
     ActiveRecord::Base.transaction do
       # Insert result...
@@ -112,6 +105,4 @@ class ScrapeStuff < Que::Job
 end
 ```
 
-Now, if the request takes more than five seconds, a `Timeout::Error` will be raised and Que will just retry the job later. This solution isn't perfect, since Timeout uses Thread#kill under the hood, which can lead to unpredictable behavior. But it's separate from our transaction, so there's no risk of losing data - even a catastrophic error that left Net::HTTP in a bad state would be fixable by restarting the process.
-
-Finally, remember that if you're using a library that offers its own timeout functionality, that's usually preferable to using the Timeout module.
+Now, if the request takes more than five seconds, an error will be raised (probably - check your library's documentation) and Que will just retry the job later.

data/lib/que.rb

@@ -1,200 +1,116 @@
 # frozen_string_literal: true
 
-require 'socket' # For hostname
-require 'json'
+require 'forwardable'
+require 'socket' # For Socket.gethostname
 
 module Que
-  autoload :Adapters,   'que/adapters/base'
-  autoload :Job,        'que/job'
-  autoload :Migrations, 'que/migrations'
-  autoload :SQL,        'que/sql'
-  autoload :Version,    'que/version'
-  autoload :Worker,     'que/worker'
-
-  HASH_DEFAULT_PROC = proc { |hash, key| hash[key.to_s] if Symbol === key }
-
-  INDIFFERENTIATOR = proc do |object|
-    case object
-    when Array
-      object.each(&INDIFFERENTIATOR)
-    when Hash
-      object.default_proc = HASH_DEFAULT_PROC
-      object.each { |key, value| object[key] = INDIFFERENTIATOR.call(value) }
-      object
-    else
-      object
-    end
-  end
-
-  SYMBOLIZER = proc do |object|
-    case object
-    when Hash
-      object.keys.each do |key|
-        object[key.to_sym] = SYMBOLIZER.call(object.delete(key))
-      end
-      object
-    when Array
-      object.map! { |e| SYMBOLIZER.call(e) }
-    else
-      object
-    end
-  end
+  CURRENT_HOSTNAME = Socket.gethostname.freeze
+  DEFAULT_QUEUE    = 'default'.freeze
+  TIME_REGEX       = /\A\d{4}\-\d{2}\-\d{2}T\d{2}:\d{2}:\d{2}.\d{6}Z\z/
+  CONFIG_MUTEX     = Mutex.new
+  MAXIMUM_PRIORITY = 32767
+
+  class Error < StandardError; end
+
+  # Store SQL strings frozen, with squashed whitespace so logs read better.
+  SQL = {}
+  def SQL.[]=(k,v); super(k, v.strip.gsub(/\s+/, ' ').freeze); end
+
+  # Load up modules that allow registration before modules that use it.
+  require_relative 'que/listener'
+
+  # Load utilities before main logic that will use them.
+  require_relative 'que/utils/assertions'
+  require_relative 'que/utils/constantization'
+  require_relative 'que/utils/error_notification'
+  require_relative 'que/utils/freeze'
+  require_relative 'que/utils/introspection'
+  require_relative 'que/utils/json_serialization'
+  require_relative 'que/utils/logging'
+  require_relative 'que/utils/middleware'
+  require_relative 'que/utils/queue_management'
+  require_relative 'que/utils/transactions'
+
+  require_relative 'que/connection'
+  require_relative 'que/connection_pool'
+  require_relative 'que/job_methods'
+  require_relative 'que/job'
+  require_relative 'que/job_cache'
+  require_relative 'que/locker'
+  require_relative 'que/metajob'
+  require_relative 'que/migrations'
+  require_relative 'que/poller'
+  require_relative 'que/result_queue'
+  require_relative 'que/version'
+  require_relative 'que/worker'
 
   class << self
-    attr_accessor :error_notifier
-    attr_writer :logger, :adapter, :log_formatter, :use_prepared_statements, :json_converter
-
-    def connection=(connection)
-      self.adapter =
-        if connection.to_s == 'ActiveRecord'
-          Adapters::ActiveRecord.new
+    include Utils::Assertions
+    include Utils::Constantization
+    include Utils::ErrorNotification
+    include Utils::Freeze
+    include Utils::Introspection
+    include Utils::JSONSerialization
+    include Utils::Logging
+    include Utils::Middleware
+    include Utils::QueueManagement
+    include Utils::Transactions
+
+    extend Forwardable
+
+    # Copy some commonly-used methods here, for convenience.
+    def_delegators :pool, :execute, :checkout, :in_transaction?
+    def_delegators Job, :enqueue, :run_synchronously, :run_synchronously=
+    def_delegators Migrations, :db_version, :migrate!
+
+    # Global configuration logic.
+    attr_accessor :use_prepared_statements
+    attr_writer :default_queue
+
+    def default_queue
+      @default_queue || DEFAULT_QUEUE
+    end
+
+    # Support simple integration with many common connection pools.
+    def connection=(conn)
+      self.connection_proc =
+        if conn.to_s == 'ActiveRecord'
+          # Load and setup AR compatibility.
+          require_relative 'que/active_record/connection'
+          m = Que::ActiveRecord::Connection::Middleware
+          middleware << m unless middleware.include?(m)
+          Que::ActiveRecord::Connection.method(:checkout)
         else
-          case connection.class.to_s
-          when 'Sequel::Postgres::Database' then Adapters::Sequel.new(connection)
-          when 'ConnectionPool'             then Adapters::ConnectionPool.new(connection)
-          when 'PG::Connection'             then Adapters::PG.new(connection)
-          when 'Pond'                       then Adapters::Pond.new(connection)
-          when 'NilClass'                   then connection
-          else raise "Que connection not recognized: #{connection.inspect}"
+          case conn.class.to_s
+          when 'Sequel::Postgres::Database' then conn.method(:synchronize)
+          when 'Pond'                       then conn.method(:checkout)
+          when 'ConnectionPool'             then conn.method(:with)
+          when 'NilClass'                   then conn
+          else raise Error, "Unsupported connection: #{conn.class}"
           end
         end
     end
 
-    def adapter
-      @adapter || raise("Que connection not established!")
-    end
-
-    def execute(*args)
-      adapter.execute(*args)
-    end
-
-    def clear!
-      execute "DELETE FROM que_jobs"
-    end
-
-    def job_stats
-      execute :job_stats
-    end
-
-    def worker_states
-      adapter.checkout do |conn|
-        if conn.server_version >= 90600
-          execute :worker_states_96
-        else
-          execute :worker_states_95
-        end
-      end
-    end
-
-    # Give us a cleaner interface when specifying a job_class as a string.
-    def enqueue(*args)
-      Job.enqueue(*args)
-    end
-
-    def db_version
-      Migrations.db_version
-    end
-
-    def migrate!(version = {:version => Migrations::CURRENT_VERSION})
-      Migrations.migrate!(version)
-    end
-
-    # Have to support create! and drop! in old migrations. They just created
-    # and dropped the bare table.
-    def create!
-      migrate! :version => 1
-    end
-
-    def drop!
-      migrate! :version => 0
+    # Integrate Que with any connection pool by passing it a reentrant block
+    # that locks and yields a Postgres connection.
+    def connection_proc=(connection_proc)
+      @pool = connection_proc && ConnectionPool.new(&connection_proc)
     end
 
-    def log(data)
-      level = data.delete(:level) || :info
-      data = {:lib => 'que', :hostname => Socket.gethostname, :pid => Process.pid, :thread => Thread.current.object_id}.merge(data)
-
-      if (l = logger) && output = log_formatter.call(data)
-        l.send level, output
-      end
-    end
-
-    def logger
-      @logger.respond_to?(:call) ? @logger.call : @logger
-    end
-
-    def log_formatter
-      @log_formatter ||= JSON.method(:dump)
-    end
-
-    def use_prepared_statements
-      setting = @use_prepared_statements
-      setting.nil? ? true : setting
-    end
-
-    def disable_prepared_statements
-      warn "Que.disable_prepared_statements has been deprecated, please update your code to invert the result of Que.disable_prepared_statements instead. This shim will be removed in Que version 1.0.0."
-      !use_prepared_statements
-    end
-
-    def disable_prepared_statements=(setting)
-      warn "Que.disable_prepared_statements= has been deprecated, please update your code to pass the inverted value to Que.use_prepared_statements= instead. This shim will be removed in Que version 1.0.0."
-      self.use_prepared_statements = !setting
-    end
-
-    def error_handler
-      warn "Que.error_handler has been renamed to Que.error_notifier, please update your code. This shim will be removed in Que version 1.0.0."
-      error_notifier
+    # How to actually access Que's established connection pool.
+    def pool
+      @pool || raise(Error, "Que connection not established!")
     end
 
-    def error_handler=(p)
-      warn "Que.error_handler= has been renamed to Que.error_notifier=, please update your code. This shim will be removed in Que version 1.0.0."
-      self.error_notifier = p
-    end
-
-    def constantize(camel_cased_word)
-      if camel_cased_word.respond_to?(:constantize)
-        # Use ActiveSupport's version if it exists.
-        camel_cased_word.constantize
-      else
-        camel_cased_word.split('::').inject(Object, &:const_get)
-      end
-    end
-
-    # A helper method to manage transactions, used mainly by the migration
-    # system. It's available for general use, but if you're using an ORM that
-    # provides its own transaction helper, be sure to use that instead, or the
-    # two may interfere with one another.
-    def transaction
-      adapter.checkout do
-        if adapter.in_transaction?
-          yield
-        else
-          begin
-            execute "BEGIN"
-            yield
-          rescue => error
-            raise
-          ensure
-            # Handle a raised error or a killed thread.
-            if error || Thread.current.status == 'aborting'
-              execute "ROLLBACK"
-            else
-              execute "COMMIT"
-            end
-          end
-        end
-      end
-    end
-
-    def json_converter
-      @json_converter ||= INDIFFERENTIATOR
-    end
-
-    # Copy some of the Worker class' config methods here for convenience.
-    [:mode, :mode=, :worker_count, :worker_count=, :wake_interval, :wake_interval=, :queue_name, :queue_name=, :wake!, :wake_all!].each do |meth|
-      define_method(meth) { |*args| Worker.send(meth, *args) }
-    end
+    # Set the current pool. Helpful for specs, but probably shouldn't be used
+    # generally.
+    attr_writer :pool
   end
+
+  # Set config defaults.
+  self.use_prepared_statements = true
 end
 
-require 'que/railtie' if defined? Rails::Railtie
+# Load Rails features as appropriate.
+require_relative 'que/rails/railtie'         if defined?(::Rails::Railtie)
+require_relative 'que/active_job/extensions' if defined?(::ActiveJob)