que 0.4.0 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 221284cc1195f925ca4ad320ddc6ea16e2441a5f
-  data.tar.gz: 6233e8871fb93b5e2f738653e9070c019ecf1baa
+  metadata.gz: 71cd78e2494e219cb26e3513ff4e41bbb335161c
+  data.tar.gz: 8d51de1cebe1eb810f1acf87e2a7b540a72aadfc
 SHA512:
-  metadata.gz: 9f85215c53bf87781cb503747738446878414dae0ac1afdb44536870f0df81a0d0761441b6a3863865fcc8d30a35ba52951399e0a656adfdfb54cdd1d0355ffb
-  data.tar.gz: 6d51c478c98f5fca91afd67c2c07779543fbae45cc7162cf041c94aeb8921e18b4df7a07238d42a3f20dd4d473c22c01727e92182d3b0bd5c332d08048920231
+  metadata.gz: 1a1d708935bfd81d8cf3c52732a5448a804dc7650741a0313f9879998f6eafd21fa5d2a32f7d9b9297041b4f783895c7f6305c0e742836dda894650fe8b01d7c
+  data.tar.gz: bac784c3fb91f11386c0b78cde93cd61754b0e66f03d00a646c3c3a9152f0509269595f55ffc7441aa62b7bc9926e39e672644d6f6dd2f39c88fd1bb7da17453

data/.travis.yml

@@ -0,0 +1,15 @@
+language: ruby
+rvm:
+  - "1.9.3"
+  - "2.0"
+  - "2.1"
+  - "rbx-2.1.1"
+  - "jruby-1.7.5"
+before_script:
+  - psql -c 'create database "que-test"' -U postgres
+  - bundle exec ruby -r sequel -r ./lib/que -e 'Que.connection=Sequel.connect("postgres://localhost/que-test"); Que.migrate!'
+
+script: "./spec/travis.rb"
+
+addons:
+  postgresql: 9.3

data/CHANGELOG.md

@@ -1,4 +1,26 @@
-### Unreleased
+### 0.5.0 (2014-01-14)
+
+*   When running a worker pool inside your web process on ActiveRecord, Que will now wake a worker once a transaction containing a queued job is committed.
+
+*   The `que:work` rake task now has a default wake_interval of 0.1 seconds, since it relies exclusively on polling to pick up jobs. You can set a QUE_WAKE_INTERVAL environment variable to change this. The environment variable to set a size for the worker pool in the rake task has also been changed from WORKER_COUNT to QUE_WORKER_COUNT.
+
+*   Officially support Ruby 1.9.3. Note that due to the Thread#kill problems (see "Remove Que.stop!" below) there's a danger of data corruption when running under 1.9, though.
+
+*   The default priority for jobs is now 100 (it was 1 before). Like always (and like delayed_job), a lower priority means it's more important. You can migrate the schema version to 2 to set the new default value on the que_jobs table, though it's only necessary if you're doing your own INSERTs - if you use `MyJob.queue`, it's already taken care of.
+
+*   Added a migration system to make it easier to change the schema when updating Que. You can now write, for example, `Que.migrate!(:version => 2)` in your migrations. Migrations are run transactionally.
+
+*   The logging format has changed to be more easily machine-readable. You can also now customize the logging format by assigning a callable to Que.log_formatter=. See the new doc on [logging](https://github.com/chanks/que/blob/master/docs/logging.md)) for details. The default logger level is INFO - for less critical information (such as when no jobs were found to be available or when a job-lock race condition has been detected and avoided) you can set the QUE_LOG_LEVEL environment variable to DEBUG.
+
+*   MultiJson is now a soft dependency. Que will use it if it is available, but it is not required.
+
+*   Remove Que.stop!.
+
+    Using Thread#raise to kill workers is a bad idea - the results are unpredictable and nearly impossible to spec reliably. Its purpose was to prevent premature commits in ActiveRecord/Sequel when a thread is killed during shutdown, but it's possible to detect that situation on Ruby 2.0+, so this is really better handled by the ORMs directly. See the pull requests for [Sequel](https://github.com/jeremyevans/sequel/pull/752) and [ActiveRecord](https://github.com/rails/rails/pull/13656).
+
+    Now, when a process exits, if the worker pool is running (whether in a rake task or in a web process) the exit will be stalled until all workers have finished their current jobs. If you have long-running jobs, this may take a long time. If you need the process to exit immediately, you can SIGKILL without any threat of commiting prematurely.
+
+### 0.4.0 (2014-01-05)
 
 *   Que.wake_all! was added, as a simple way to wake up all workers in the pool.
 

data/README.md

@@ -6,7 +6,7 @@ Que is a queue for Ruby and PostgreSQL that manages jobs using [advisory locks](
 
 * **Concurrency** - Workers don't block each other when trying to lock jobs, as often occurs with "SELECT FOR UPDATE"-style locking. This allows for very high throughput with a large number of workers.
 * **Efficiency** - Locks are held in memory, so locking a job doesn't incur a disk write. These first two points are what limit performance with other queues - all workers trying to lock jobs have to wait behind one that's persisting its UPDATE on a locked_at column to disk (and the disks of however many other servers your database is synchronously replicating to). Under heavy load, Que's bottleneck is CPU, not I/O.
-* **Safety** - If a Ruby process dies, the jobs it is working won't be lost, or left in a locked or ambiguous state - they immediately become available for any other worker to pick up.
+* **Safety** - If a Ruby process dies, the jobs it's working won't be lost, or left in a locked or ambiguous state - they immediately become available for any other worker to pick up.
 
 Additionally, there are the general benefits of storing jobs in Postgres, alongside the rest of your data, rather than in Redis or a dedicated queue:
 
@@ -14,13 +14,13 @@ Additionally, there are the general benefits of storing jobs in Postgres, alongs
 * **Atomic Backups** - Your jobs and data can be backed up together and restored as a snapshot. If your jobs relate to your data (and they usually do), there's no risk of jobs falling through the cracks during a recovery.
 * **Fewer Dependencies** - If you're already using Postgres (and you probably should be), a separate queue is another moving part that can break.
 
-Que's primary goal is reliability. When it's stable, you should be able to leave your application running indefinitely without worrying about jobs being lost due to a lack of transactional support, or left in limbo due to a crashing process. Que does everything it can to ensure that jobs you queue are performed exactly once (though the occasional repetition of a job can be impossible to avoid - see the docs on [how to write a reliable job](https://github.com/chanks/que/blob/master/docs/writing_reliable_jobs.md)).
+Que's primary goal is reliability. You should be able to leave your application running indefinitely without worrying about jobs being lost due to a lack of transactional support, or left in limbo due to a crashing process. Que does everything it can to ensure that jobs you queue are performed exactly once (though the occasional repetition of a job can be impossible to avoid - see the docs on [how to write a reliable job](https://github.com/chanks/que/blob/master/docs/writing_reliable_jobs.md)).
 
 Que's secondary goal is performance. It won't be able to match the speed or throughput of a dedicated queue, or maybe even a Redis-backed queue, but it should be fast enough for most use cases. In [benchmarks of RDBMS queues](https://github.com/chanks/queue-shootout) using PostgreSQL 9.3 on a AWS c3.8xlarge instance, Que approaches 10,000 jobs per second, or about twenty times the throughput of DelayedJob or QueueClassic. You are encouraged to try things out on your own production hardware, though.
 
 Que also includes a worker pool, so that multiple threads can process jobs in the same process. It can even do this in the background of your web process - if you're running on Heroku, for example, you don't need to run a separate worker dyno.
 
-*Please be careful when running Que in production. It's still very new compared to other RDBMS-backed queues, and there may be issues that haven't been ironed out yet. Bug reports are welcome.*
+*Please keep an eye out for problems when running Que in production. It's still new compared to other RDBMS-backed queues, and there may be issues that haven't been ironed out yet. Bug reports are welcome.*
 
 Que is tested on Ruby 2.0, Rubinius and JRuby (with the `jruby-pg` gem, which is [not yet functional with ActiveRecord](https://github.com/chanks/que/issues/4#issuecomment-29561356)). It requires Postgres 9.2+ for the JSON datatype.
 
@@ -40,7 +40,7 @@ Or install it yourself as:
 
 ## Usage
 
-The following assumes you're using Rails 4.0 and ActiveRecord. *Que hasn't been tested with versions of Rails before 4.0, and may or may not work with them.* For more information, or instructions on using Que outside of Rails or with Sequel or no ORM, see the [documentation](https://github.com/chanks/que/blob/master/docs).
+The following assumes you're using Rails 4.0 and ActiveRecord. *Que hasn't been tested with versions of Rails before 4.0, and may or may not work with them.* See the [/docs directory](https://github.com/chanks/que/blob/master/docs) for instructions on using Que [outside of Rails](https://github.com/chanks/que/blob/master/docs/advanced_setup.md), and with [Sequel](https://github.com/chanks/que/blob/master/docs/using_sequel.md) or [no ORM](https://github.com/chanks/que/blob/master/docs/using_plain_connections.md), among other things.
 
 First, generate and run a migration for the job table.
 
@@ -51,15 +51,19 @@ Create a class for each type of job you want to run:
 
     # app/jobs/charge_credit_card.rb
     class ChargeCreditCard < Que::Job
-      # Default options for this job. These may be omitted.
-      @default_priority = 3
+      # Default settings for this job. These are optional - without them, jobs
+      # will default to priority 1 and run immediately.
+      @default_priority = 10
       @default_run_at = proc { 1.minute.from_now }
 
-      def run(user_id, card_id, your_options = {})
+      def run(user_id, options)
         # Do stuff.
+        user = User[user_id]
+        card = CreditCard[options[:credit_card_id]]
 
         ActiveRecord::Base.transaction do
           # Write any changes you'd like to the database.
+          user.update_attributes :charged_at => Time.now
 
           # It's best to destroy the job in the same transaction as any other
           # changes you make. Que will destroy the job for you after the run
@@ -76,18 +80,18 @@ Queue your job. Again, it's best to do this in a transaction with other changes
     ActiveRecord::Base.transaction do
       # Persist credit card information
       card = CreditCard.create(params[:credit_card])
-      ChargeCreditCard.queue(current_user.id, card.id, :your_custom_option => 'whatever')
+      ChargeCreditCard.queue(current_user.id, :credit_card_id => card.id)
     end
 
-You can also schedule it to run at a specific time, or with a specific priority:
+You can also add options to run the job after a specific time, or with a specific priority:
 
-    # 1 is high priority, 5 is low priority.
-    ChargeCreditCard.queue current_user.id, card.id, :your_custom_option => 'whatever', :run_at => 1.day.from_now, :priority => 5
+    # The default priority is 100, and a lower number means a higher priority. 5 would be very important.
+    ChargeCreditCard.queue current_user.id, :credit_card_id => card.id, :run_at => 1.day.from_now, :priority => 5
 
 To determine what happens when a job is queued, you can set Que's mode in your application configuration. There are a few options for the mode:
 
-* `config.que.mode = :off` - In this mode, queueing a job will simply insert it into the database - the current process will make no effort to run it. You should use this if you want to use a dedicated process to work tasks (there's a rake task to do this, see below). This is the default when running `rails console` in the development or production environments.
-* `config.que.mode = :async` - In this mode, a pool of background workers is spun up, each running in their own thread. This is the default when running `rails server` in the development or production environments. See the docs for [more information on managing workers](https://github.com/chanks/que/blob/master/docs/managing_workers.md).
+* `config.que.mode = :off` - In this mode, queueing a job will simply insert it into the database - the current process will make no effort to run it. You should use this if you want to use a dedicated process to work tasks (there's a rake task to do this, see below). This is the default when running `rails console`.
+* `config.que.mode = :async` - In this mode, a pool of background workers is spun up, each running in their own thread. This is the default when running `rails server`. See the docs for [more information on managing workers](https://github.com/chanks/que/blob/master/docs/managing_workers.md).
 * `config.que.mode = :sync` - In this mode, any jobs you queue will be run in the same thread, synchronously (that is, `MyJob.queue` runs the job and won't return until it's completed). This makes your application's behavior easier to test, so it's the default in the test environment.
 
 ## Contributing

data/docs/advanced_setup.md

@@ -22,22 +22,24 @@ There are other docs to read if you're using [Sequel](https://github.com/chanks/
 
 After you've connected Que to the database, you can manage the jobs table:
 
-    # Create the jobs table:
-    Que.create!
+    # Create/update the jobs table to the latest schema version:
+    Que.migrate!
 
-    # Clear the jobs table of all jobs:
-    Que.clear!
+You'll want to migrate to a specific version if you're using migration files, to ensure that they work the same way even when you upgrade Que in the future:
 
-    # Drop the jobs table:
-    Que.drop!
+    # Update the schema to version #2.
+    Que.migrate! :version => 2
 
-### Other Setup
+    # To reverse the migration, drop the jobs table entirely:
+    Que.migrate! :version => 0
 
-You can give Que a logger to use if you like:
+There's also a helper method to clear the jobs table:
 
-    Que.logger = Logger.new(STDOUT)
+    Que.clear!
+
+### Other Setup
 
-You'll also need to set Que's mode manually:
+You'll need to set Que's mode manually:
 
     # Start the worker pool:
     Que.mode = :async
@@ -47,4 +49,4 @@ You'll also need to set Que's mode manually:
 
 Be sure to read the docs on [managing workers](https://github.com/chanks/que/blob/master/docs/managing_workers.md) for more information on using the worker pool.
 
-You may also want to set up an [error handler](https://github.com/chanks/que/blob/master/docs/error_handling.md) to track errors raised by jobs.
+You'll also want to set up [logging](https://github.com/chanks/que/blob/master/docs/logging.md) and an [error handler](https://github.com/chanks/que/blob/master/docs/error_handling.md) to track errors raised by jobs.

data/docs/customizing_que.md

@@ -0,0 +1,113 @@
+## Customizing Que
+
+One of Que's goals to be easily extensible and hackable (and if anyone has any suggestions on ways to accomplish that, please [open an issue](https://github.com/chanks/que/issues)). This document is meant to demonstrate some of the ways Que can be used to accomplish different tasks that it's not already designed for.
+
+### Recurring Jobs
+
+Que's support for scheduling jobs makes it easy to implement reliable recurring jobs. For example, suppose you want to run a job every hour that processes the users created in that time:
+
+    class Cron < Que::Job
+      def run
+        users = User.where(:created_at => @attrs[:run_at]...(@attrs[:run_at] + 1.hour))
+        # Do something with users.
+
+        ActiveRecord::Base.transaction do
+          destroy
+          self.class.queue :run_at => @attrs[:run_at] + 1.hour
+        end
+      end
+    end
+
+Note that instead of using Time.now in our database query, and requeueing the job at 1.hour.from_now, we use the run_at of the current job as our timestamp. This corrects for delays in running the job. Suppose that there's a backlog of priority jobs, or that the worker briefly goes down, and this job, which was supposed to run at 11:00 a.m. isn't run until 11:05 a.m. A lazier implementation would look for users created after 1.hour.ago, and miss those that signed up between 10:00 a.m. and 10:05 a.m.
+
+This also compensates for clock drift. `Time.now` on one of your application servers may not match `Time.now` on another application server may not match `now()` on your database server. The best way to stay reliable is have a single authoritative source on what the current time is, and your best source for authoritative information is always your database (this is why Que uses Postgres' `now()` function when locking jobs, by the way).
+
+Note also the use of the triple-dot range, which results in a query like `SELECT "users".* FROM "users" WHERE ("users"."created_at" >= '2014-01-08 10:00:00.000000' AND "users"."created_at" < '2014-01-08 11:00:00.000000')` instead of a BETWEEN condition. This ensures that a user created at 11:00 am exactly isn't processed twice, by the jobs starting at both 10 am and 11 am.
+
+### DelayedJob-style Jobs
+
+DelayedJob offers a simple API for delaying methods to objects:
+
+    @user.delay.activate!(@device)
+
+The API is pleasant, but implementing it requires storing marshalled Ruby objects in the database, which is both inefficient and prone to bugs - for example, if you deploy an update that changes the name of an instance variable (a contained, internal change that might seem completely innocuous), the marshalled objects in the database will retain the old instance variable name and will behave unexpectedly when unmarshalled into the new Ruby code.
+
+This is the danger of mixing the ephemeral state of a Ruby object in memory with the more permanent state of a database row. The advantage of Que's API is that, since your arguments are forced through a JSON serialization/deserialization process, it becomes your responsibility when designing a job class to establish an API for yourself (what the arguments to the job are and what they mean) that you will have to stick to in the future.
+
+That said, if you want to queue jobs in the DelayedJob style, that can be done relatively easily:
+
+    class Delayed < Que::Job
+      def run(receiver, method, args)
+        Marshal.load(receiver).send method, *Marshal.load(args)
+      end
+    end
+
+    class DelayedAction
+      def initialize(receiver)
+        @receiver = receiver
+      end
+
+      def method_missing(method, *args)
+        Delayed.queue Marshal.dump(@receiver), method, Marshal.dump(args)
+      end
+    end
+
+    class Object
+      def delay
+        DelayedAction.new(self)
+      end
+    end
+
+You can replace Marshal with YAML if you like.
+
+### QueueClassic-style Jobs
+
+You may find it a hassle to keep an individual class file for each type of job. QueueClassic has a simpler design, wherein you simply give it a class method to call, like:
+
+    QC.enqueue("Kernel.puts", "hello world")
+
+You can mimic this style with Que by using a simple job class:
+
+    class Command < Que::Job
+      def run(method, *args)
+        receiver, message = method.split('.')
+        Object.const_get(receiver).send(message, *args)
+      end
+    end
+
+    # Then:
+
+    Command.queue "Kernel.puts", "hello world"
+
+### Retaining Finished Jobs
+
+Que deletes jobs from the queue as they are worked, in order to keep the `que_jobs` table and index small and efficient. If you have a need to hold onto finished jobs, the recommended way to do this is to add a second table to hold them, and then insert them there as they are deleted from the queue. You can use Ruby's inheritance mechanics to do this cleanly:
+
+    Que.execute "CREATE TABLE finished_jobs AS SELECT * FROM que_jobs LIMIT 0"
+    # Or, better, use a proper CREATE TABLE with not-null constraints, and add whatever indexes you like.
+
+    class MyJobClass < Que::Job
+      def destroy
+        Que.execute "INSERT INTO finished_jobs SELECT * FROM que_jobs WHERE priority = $1::integer AND run_at = $2::timestamptz AND job_id = $3::bigint", @attrs.values_at(:priority, :run_at, :job_id)
+        super
+      end
+    end
+
+Then just have your job classes inherit from MyJobClass instead of Que::Job. If you need to query the jobs table and you want to include both finished and unfinished jobs, you might use:
+
+    Que.execute "CREATE VIEW all_jobs AS SELECT * FROM que_jobs UNION ALL SELECT * FROM finished_jobs"
+    Que.execute "SELECT * FROM all_jobs"
+
+Alternately, if you want a more foolproof solution and you're not scared of PostgreSQL, you can use a trigger:
+
+    CREATE FUNCTION please_save_my_job()
+    RETURNS trigger
+    LANGUAGE plpgsql
+    AS $$
+      BEGIN
+        INSERT INTO finished_jobs SELECT (OLD).*;
+        RETURN OLD;
+      END;
+    $$;
+
+    CREATE TRIGGER keep_all_my_old_jobs BEFORE DELETE ON que_jobs FOR EACH ROW EXECUTE PROCEDURE please_save_my_job();

data/docs/error_handling.md

@@ -2,7 +2,7 @@
 
 If an error is raised and left uncaught by your job, Que will save the error message and backtrace to the database and schedule the job to be retried later.
 
-If a given job fails repeatedly, Que will retry it at exponentially-increasing intervals equal to (failure_count[^4^] + 3) seconds. This means that a job will be retried 4 seconds after its first failure, 19 seconds after its second, 84 seconds after its third, 259 seconds after its fourth, and so on until it succeeds. This pattern is very similar to DelayedJob's.
+If a given job fails repeatedly, Que will retry it at exponentially-increasing intervals equal to (failure_count^4 + 3) seconds. This means that a job will be retried 4 seconds after its first failure, 19 seconds after its second, 84 seconds after its third, 259 seconds after its fourth, and so on until it succeeds. This pattern is very similar to DelayedJob's.
 
 Unlike DelayedJob, however, there is currently no maximum number of failures after which jobs will be deleted. Que's assumption is that if a job is erroring perpetually (and not just transiently), you will want to take action to get the job working properly rather than simply losing it silently.
 

data/docs/logging.md

@@ -0,0 +1,42 @@
+## Logging
+
+By default, Que logs important information in JSON to either Rails' logger (when running in a Rails web process) or STDOUT (when running as a rake task). So, your logs will look something like:
+
+    I, [2014-01-12T05:07:31.094201 #4687]  INFO -- : {"lib":"que","thread":104928,"event":"job_worked","elapsed":0.01045,"job":{"priority":"1","run_at":"2014-01-12 05:07:31.081877+00","job_id":"4","job_class":"MyJob","args":[],"error_count":"0"}}
+
+Of course you can have it log wherever you like:
+
+    Que.logger = Logger.new(...)
+
+    # Or, in your Rails configuration:
+
+    config.que.logger = Logger.new(...)
+
+You can use Que's logger in your jobs anywhere you like:
+
+    class MyJob
+      def run
+        Que.log :my_output => "my string"
+      end
+    end
+
+    #=> I, [2014-01-12T05:13:11.006776 #4914]  INFO -- : {"lib":"que","thread":24960,"my_output":"my string"}
+
+Que will always add a 'lib' key, so you can easily filter its output from that of other sources, and the object_id of the thread that emitted the log, so you can follow the actions of a particular worker if you wish. You can also pass a :level key to set the level of the output:
+
+    Que.log :level => :debug, :my_output => 'my string'
+    #=> D, [2014-01-12T05:16:15.221941 #5088] DEBUG -- : {"lib":"que","thread":24960,"my_output":"my string"}
+
+If you don't like JSON, you can also customize the format of the logging output by passing a callable object (such as a proc) to Que.log_formatter=. The proc should take a hash (the keys are symbols) and return a string. The keys and values are just as you would expect from the JSON output:
+
+    Que.log_formatter = proc do |data|
+      "Thread number #{data[:thread]} experienced a #{data[:event]}"
+    end
+
+If the log formatter returns nil or false, a nothing will be logged at all. You could use this to narrow down what you want to emit, for example:
+
+    Que.log_formatter = proc do |data|
+      if ['job_worked', 'job_unavailable'].include?(data[:event])
+        JSON.dump(data)
+      end
+    end

data/docs/managing_workers.md

@@ -65,3 +65,9 @@ Regardless of the `wake_interval` setting, you can always wake workers manually:
     Que.wake_all!
 
 `Que.wake_all!` is helpful if there are no jobs available and all your workers go to sleep, and then you queue a large number of jobs. Typically, it will take a little while for the entire pool of workers get going again - a new one will wake up every `wake_interval` seconds, but it will take up to `wake_interval * worker_count` seconds for all of them to get going. `Que.wake_all!` can get them all moving immediately.
+
+### Connection Pool Size
+
+For the job locking system to work properly, each worker thread needs to reserve a database connection from the connection pool for the period of time between when it locks a job and when it releases that lock (which won't happen until the job has been finished and deleted from the queue).
+
+So, for example, if you're running 6 workers in a rake task, you'll want to make sure that whatever connection pool Que is using (usually ActiveRecord's) has a maximum size of at least 6. If you're running those workers in a web process, you'll want the size to be at least 6 plus however many connections you expect your application to need for serving web requests (which may only be one if you're using Rails in single-threaded mode, or many more if you're running a threaded web server like Puma).

data/docs/using_sequel.md

@@ -23,5 +23,5 @@ Then you can safely use the same database object to transactionally protect your
     # In your controller action:
     DB.transaction do
       @user = User.create(params[:user])
-      SendRegistrationEmail.queue :user_id => @user.id
+      MyJob.queue :user_id => @user.id
     end

data/lib/generators/que/templates/add_que.rb

@@ -1,9 +1,11 @@
 class AddQue < ActiveRecord::Migration
   def self.up
-    Que.create!
+    # The current version as of this migration's creation.
+    Que.migrate! :version => 2
   end
 
   def self.down
-    Que.drop!
+    # Completely removes Que's job queue.
+    Que.migrate! :version => 0
   end
 end

data/lib/que.rb

@@ -1,13 +1,24 @@
+require 'time' # For Time#iso8601
+
 module Que
-  autoload :Adapters, 'que/adapters/base'
-  autoload :Job,      'que/job'
-  autoload :SQL,      'que/sql'
-  autoload :Version,  'que/version'
-  autoload :Worker,   'que/worker'
+  autoload :Adapters,   'que/adapters/base'
+  autoload :Job,        'que/job'
+  autoload :Migrations, 'que/migrations'
+  autoload :SQL,        'que/sql'
+  autoload :Version,    'que/version'
+  autoload :Worker,     'que/worker'
+
+  begin
+    require 'multi_json'
+    JSON_MODULE = MultiJson
+  rescue LoadError
+    require 'json'
+    JSON_MODULE = JSON
+  end
 
   class << self
     attr_accessor :logger, :error_handler
-    attr_writer :adapter
+    attr_writer :adapter, :log_formatter
 
     def adapter
       @adapter || raise("Que connection not established!")
@@ -27,12 +38,18 @@ module Que
       end
     end
 
+    # Have to support create! and drop! in old migrations. They just created
+    # and dropped the bare table.
     def create!
-      execute SQL[:create_table]
+      migrate! :version => 1
     end
 
     def drop!
-      execute "DROP TABLE que_jobs"
+      migrate! :version => 0
+    end
+
+    def migrate!(version = {:version => Migrations::CURRENT_VERSION})
+      Migrations.migrate!(version)
     end
 
     def clear!
@@ -54,8 +71,17 @@ module Que
                       end.to_a
     end
 
-    def log(level, text)
-      logger.send level, "[Que] #{text}" if logger
+    def log(data)
+      level = data.delete(:level) || :info
+      data = {:lib => 'que', :thread => Thread.current.object_id}.merge(data)
+
+      if logger && output = log_formatter.call(data)
+        logger.send level, output
+      end
+    end
+
+    def log_formatter
+      @log_formatter ||= JSON_MODULE.method(:dump)
     end
 
     # Helper for making hashes indifferently-accessible, even when nested
@@ -79,7 +105,7 @@ module Que
     end
 
     # Copy some of the Worker class' config methods here for convenience.
-    [:mode, :mode=, :worker_count, :worker_count=, :wake_interval, :wake_interval=, :stop!, :wake!, :wake_all!].each do |meth|
+    [:mode, :mode=, :worker_count, :worker_count=, :wake_interval, :wake_interval=, :wake!, :wake_all!].each do |meth|
       define_method(meth) { |*args| Worker.send(meth, *args) }
     end
   end