que 0.3.0 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: eca6d367104b9c0dcb8c5fb591641bebdc7c4a8f
-  data.tar.gz: a71620a15051793942059b7c423a2f78cc3b6f16
+  metadata.gz: 221284cc1195f925ca4ad320ddc6ea16e2441a5f
+  data.tar.gz: 6233e8871fb93b5e2f738653e9070c019ecf1baa
 SHA512:
-  metadata.gz: d59625b0b10cca06e54f9a6714bc5cb701d4895936ad9a5ff4a6500ab1f165f0ad5f3c293fded447979e3aa34379480591518e0819160c857a2a22a754c70d8b
-  data.tar.gz: 92741560f486b993a950538cb8187fa7183ef02822d386479b978401d5fc08d5750a93418b44c68fa66eb5ace6d90691fb9c9c86e472caad59e5650cab3b27cb
+  metadata.gz: 9f85215c53bf87781cb503747738446878414dae0ac1afdb44536870f0df81a0d0761441b6a3863865fcc8d30a35ba52951399e0a656adfdfb54cdd1d0355ffb
+  data.tar.gz: 6d51c478c98f5fca91afd67c2c07779543fbae45cc7162cf041c94aeb8921e18b4df7a07238d42a3f20dd4d473c22c01727e92182d3b0bd5c332d08048920231

data/CHANGELOG.md

@@ -1,3 +1,23 @@
+### Unreleased
+
+*   Que.wake_all! was added, as a simple way to wake up all workers in the pool.
+
+*   Que.sleep_period was renamed to the more descriptive Que.wake_interval.
+
+*   When queueing a job, Que will wait until the current transaction commits and then wake a background worker, if possible. This allows newly queued jobs to be started immediately instead of waiting for a worker to wake up and poll, which may be up to `Que.wake_interval` seconds.
+
+    This feature currently only works with Sequel, since there doesn't seem to be a clean way to do it on ActiveRecord (if anyone can figure one out, please let me know). Note that if you're using ActiveRecord, you can always manually trigger a single worker to wake up and check for work by manually calling Que.wake! after your transaction completes.
+
+*   Add Que.job_stats, which queries the database and returns statistics on the different job classes - for each class, how many are queued, how many are currently being worked, what is the highest error_count, and so on.
+
+*   Add Que.worker_states, which queries the database and returns all currently-locked jobs and info on their workers' connections - what and when was the last query they ran, are they waiting on locks, and so on.
+
+*   Have Que only clear advisory locks that it has taken when locking jobs, and not touch any that may have been taken by other code using the same connection.
+
+*   Add Que.worker_count, to retrieve the current number of workers in the pool of the current process.
+
+*   Much more internal cleanup.
+
 ### 0.3.0 (2013-12-21)
 
 *   Add Que.stop!, which immediately kills all jobs being worked in the process.

data/README.md

@@ -1,11 +1,11 @@
 # Que
 
-**TL;DR: Que is a high-performance alternative to DelayedJob or QueueClassic that improves the reliability of your application by helping you keep your jobs [consistent](https://en.wikipedia.org/wiki/ACID#Consistency) with the rest of your data.**
+**TL;DR: Que is a high-performance alternative to DelayedJob or QueueClassic that improves the reliability of your application by protecting your jobs with the same [ACID guarantees](https://en.wikipedia.org/wiki/ACID) as the rest of your data.**
 
 Que is a queue for Ruby and PostgreSQL that manages jobs using [advisory locks](http://www.postgresql.org/docs/current/static/explicit-locking.html#ADVISORY-LOCKS), which gives it several advantages over other RDBMS-backed queues:
 
 * **Concurrency** - Workers don't block each other when trying to lock jobs, as often occurs with "SELECT FOR UPDATE"-style locking. This allows for very high throughput with a large number of workers.
-* **Efficiency** - Locks are held in memory, so locking a job doesn't incur a disk write. These first two points are what limit performance with other queues - all workers trying to lock jobs have to wait behind one that's persisting its UPDATE on a locked_at column to disk (and the disks of however many other servers your database is replicating to). Under heavy load, Que's bottleneck is CPU, not I/O.
+* **Efficiency** - Locks are held in memory, so locking a job doesn't incur a disk write. These first two points are what limit performance with other queues - all workers trying to lock jobs have to wait behind one that's persisting its UPDATE on a locked_at column to disk (and the disks of however many other servers your database is synchronously replicating to). Under heavy load, Que's bottleneck is CPU, not I/O.
 * **Safety** - If a Ruby process dies, the jobs it is working won't be lost, or left in a locked or ambiguous state - they immediately become available for any other worker to pick up.
 
 Additionally, there are the general benefits of storing jobs in Postgres, alongside the rest of your data, rather than in Redis or a dedicated queue:
@@ -14,11 +14,11 @@ Additionally, there are the general benefits of storing jobs in Postgres, alongs
 * **Atomic Backups** - Your jobs and data can be backed up together and restored as a snapshot. If your jobs relate to your data (and they usually do), there's no risk of jobs falling through the cracks during a recovery.
 * **Fewer Dependencies** - If you're already using Postgres (and you probably should be), a separate queue is another moving part that can break.
 
-Que's primary goal is reliability. When it's stable, you should be able to leave your application running indefinitely without worrying about jobs being lost due to a lack of transactional support, or left in limbo due to a crashing process. Que does everything it can to ensure that jobs you queue are performed exactly once (though the occasional repetition of a job can be impossible to avoid - see the wiki page on [how to write a reliable job](https://github.com/chanks/que/wiki/Writing-Reliable-Jobs)).
+Que's primary goal is reliability. When it's stable, you should be able to leave your application running indefinitely without worrying about jobs being lost due to a lack of transactional support, or left in limbo due to a crashing process. Que does everything it can to ensure that jobs you queue are performed exactly once (though the occasional repetition of a job can be impossible to avoid - see the docs on [how to write a reliable job](https://github.com/chanks/que/blob/master/docs/writing_reliable_jobs.md)).
 
-Que's secondary goal is performance. It won't be able to match the speed or throughput of a dedicated queue, or maybe even a Redis-backed queue, but it should be fast enough for most use cases. In [benchmarks](https://github.com/chanks/queue-shootout) on an AWS c3.8xlarge instance, Que approaches 10,000 jobs per second, or about twenty times the throughput of DelayedJob or QueueClassic. You are encouraged to try things out on your own production hardware, though.
+Que's secondary goal is performance. It won't be able to match the speed or throughput of a dedicated queue, or maybe even a Redis-backed queue, but it should be fast enough for most use cases. In [benchmarks of RDBMS queues](https://github.com/chanks/queue-shootout) using PostgreSQL 9.3 on a AWS c3.8xlarge instance, Que approaches 10,000 jobs per second, or about twenty times the throughput of DelayedJob or QueueClassic. You are encouraged to try things out on your own production hardware, though.
 
-Que also includes a worker pool, so that multiple threads can process jobs in the same process. It can even do this in the background of your web process - if you're running on Heroku, for example, you won't need to run a separate worker dyno.
+Que also includes a worker pool, so that multiple threads can process jobs in the same process. It can even do this in the background of your web process - if you're running on Heroku, for example, you don't need to run a separate worker dyno.
 
 *Please be careful when running Que in production. It's still very new compared to other RDBMS-backed queues, and there may be issues that haven't been ironed out yet. Bug reports are welcome.*
 
@@ -40,9 +40,9 @@ Or install it yourself as:
 
 ## Usage
 
-The following is assuming you're using Rails 4.0. Que hasn't been tested with previous versions of Rails.
+The following assumes you're using Rails 4.0 and ActiveRecord. *Que hasn't been tested with versions of Rails before 4.0, and may or may not work with them.* For more information, or instructions on using Que outside of Rails or with Sequel or no ORM, see the [documentation](https://github.com/chanks/que/blob/master/docs).
 
-First, generate a migration for the job table.
+First, generate and run a migration for the job table.
 
     rails generate que:install
     rake db:migrate
@@ -51,7 +51,7 @@ Create a class for each type of job you want to run:
 
     # app/jobs/charge_credit_card.rb
     class ChargeCreditCard < Que::Job
-      # Custom job options.
+      # Default options for this job. These may be omitted.
       @default_priority = 3
       @default_run_at = proc { 1.minute.from_now }
 
@@ -84,30 +84,11 @@ You can also schedule it to run at a specific time, or with a specific priority:
     # 1 is high priority, 5 is low priority.
     ChargeCreditCard.queue current_user.id, card.id, :your_custom_option => 'whatever', :run_at => 1.day.from_now, :priority => 5
 
-To determine what happens when a job is queued, you can set Que's mode with `Que.mode = :off` or `config.que.mode = :off` in your application configuration. There are a few options for the mode:
+To determine what happens when a job is queued, you can set Que's mode in your application configuration. There are a few options for the mode:
 
-* `:off` - In this mode, queueing a job will simply insert it into the database - the current process will make no effort to run it. You should use this if you want to use a dedicated process to work tasks (there's a rake task to do this, see below). This is the default when running `rails console` in the development or production environments.
-* `:async` - In this mode, a pool of background workers is spun up, each running in their own thread. They will intermittently check for new jobs. This is the default when running `rails server` in the development or production environments. By default, there are 4 workers and they'll check for a new job every 5 seconds. You can modify these options with `Que.worker_count = 8` or `config.que.worker_count = 8` and `Que.sleep_period = 1` or `config.que.sleep_period = 1`.
-* `:sync` - In this mode, any jobs you queue will be run in the same thread, synchronously (that is, `MyJob.queue` runs the job and won't return until it's completed). This makes your application's behavior easier to test, so it's the default in the test environment.
-
-If you don't want to run workers in your web process, you can also work jobs in a rake task, similar to how other queueing systems work:
-
-    # Run a pool of 4 workers.
-    rake que:work
-
-    # Or configure the number of workers.
-    WORKER_COUNT=8 rake que:work
-
-    # If your app code isn't thread-safe, be sure to stick to one worker.
-    WORKER_COUNT=1 rake que:work
-
-If an error causes a job to fail, Que will repeat that job at exponentially-increasing intervals, similar to DelayedJob (the job will be retried at 4 seconds, 19 seconds, 84 seconds, 259 seconds...). You can also hook Que into whatever error notification system you're using:
-
-    config.que.error_handler = proc do |error|
-      # Do whatever you want with the error object.
-    end
-
-You can find more documentation on the [Github wiki](https://github.com/chanks/que/wiki).
+* `config.que.mode = :off` - In this mode, queueing a job will simply insert it into the database - the current process will make no effort to run it. You should use this if you want to use a dedicated process to work tasks (there's a rake task to do this, see below). This is the default when running `rails console` in the development or production environments.
+* `config.que.mode = :async` - In this mode, a pool of background workers is spun up, each running in their own thread. This is the default when running `rails server` in the development or production environments. See the docs for [more information on managing workers](https://github.com/chanks/que/blob/master/docs/managing_workers.md).
+* `config.que.mode = :sync` - In this mode, any jobs you queue will be run in the same thread, synchronously (that is, `MyJob.queue` runs the job and won't return until it's completed). This makes your application's behavior easier to test, so it's the default in the test environment.
 
 ## Contributing
 

data/docs/advanced_setup.md

@@ -0,0 +1,50 @@
+## Advanced Setup
+
+If you're using both Rails and ActiveRecord, the README describes how to get started with Que (which is pretty straightforward, since Que includes a Railtie that handles a lot of setup for you). Otherwise, you'll need to do some manual setup.
+
+If you're using ActiveRecord outside of Rails, you'll need to tell Que to piggyback on its connection pool after you've connected to the database:
+
+    ActiveRecord::Base.establish_connection(ENV['DATABASE_URL'])
+
+    require 'que'
+    Que.connection = ActiveRecord
+
+Then you can queue jobs just as you would in Rails:
+
+    ActiveRecord::Base.transaction do
+      @user = User.create(params[:user])
+      SendRegistrationEmail.queue :user_id => @user.id
+    end
+
+There are other docs to read if you're using [Sequel](https://github.com/chanks/que/blob/master/docs/using_sequel.md) or [plain Postgres connections](https://github.com/chanks/que/blob/master/docs/using_plain_connections.md) (with no ORM at all) instead of ActiveRecord.
+
+### Managing the Jobs Table
+
+After you've connected Que to the database, you can manage the jobs table:
+
+    # Create the jobs table:
+    Que.create!
+
+    # Clear the jobs table of all jobs:
+    Que.clear!
+
+    # Drop the jobs table:
+    Que.drop!
+
+### Other Setup
+
+You can give Que a logger to use if you like:
+
+    Que.logger = Logger.new(STDOUT)
+
+You'll also need to set Que's mode manually:
+
+    # Start the worker pool:
+    Que.mode = :async
+
+    # Or, when testing:
+    Que.mode = :sync
+
+Be sure to read the docs on [managing workers](https://github.com/chanks/que/blob/master/docs/managing_workers.md) for more information on using the worker pool.
+
+You may also want to set up an [error handler](https://github.com/chanks/que/blob/master/docs/error_handling.md) to track errors raised by jobs.

data/docs/error_handling.md

@@ -0,0 +1,17 @@
+## Error Handling
+
+If an error is raised and left uncaught by your job, Que will save the error message and backtrace to the database and schedule the job to be retried later.
+
+If a given job fails repeatedly, Que will retry it at exponentially-increasing intervals equal to (failure_count[^4^] + 3) seconds. This means that a job will be retried 4 seconds after its first failure, 19 seconds after its second, 84 seconds after its third, 259 seconds after its fourth, and so on until it succeeds. This pattern is very similar to DelayedJob's.
+
+Unlike DelayedJob, however, there is currently no maximum number of failures after which jobs will be deleted. Que's assumption is that if a job is erroring perpetually (and not just transiently), you will want to take action to get the job working properly rather than simply losing it silently.
+
+If you're using an error notification system (highly recommended, of course), you can hook Que into it by setting a callable as the error handler:
+
+    Que.error_handler = proc do |error|
+      # Do whatever you want with the error object.
+    end
+
+    # Or, in your Rails configuration:
+
+    config.que.error_handler = proc { |error| ... }

data/docs/inspecting_the_queue.md

@@ -0,0 +1,100 @@
+## Inspecting the Queue
+
+In order to remain simple and compatible with any ORM (or no ORM at all), Que is really just a very thin wrapper around some raw SQL. There are two methods available that query the jobs table and Postgres' system catalogs to retrieve information on the current state of the queue:
+
+### Job Stats
+
+You can call `Que.job_stats` to return some aggregate data on the types of jobs currently in the queue. Example output:
+
+    [
+      {
+        "job_class"=>"ChargeCreditCard",
+        "count"=>"10",
+        "count_working"=>"4",
+        "count_errored"=>"2",
+        "highest_error_count"=>"5",
+        "oldest_run_at"=>"2014-01-04 21:24:55.817129+00"
+      },
+      {
+        "job_class"=>"SendRegistrationEmail",
+        "count"=>"8",
+        "count_working"=>"0",
+        "count_errored"=>"0",
+        "highest_error_count"=>"0",
+        "oldest_run_at"=>"2014-01-04 22:24:55.81532+00"
+      }
+    ]
+
+This tells you that, for instance, there are ten ChargeCreditCard jobs in the queue, four of which are currently being worked, and two of which have experienced errors. One of them has started to process but experienced an error five times. The oldest_run_at is helpful for determining how long jobs have been sitting around, if you have backlog.
+
+### Worker States
+
+You can call `Que.worker_states` to return some information on every worker touching the queue (not just those in the current process). Example output:
+
+    [
+      {
+        "priority"=>"2",
+        "run_at"=>"2014-01-04 22:35:55.772324+00",
+        "job_id"=>"4592",
+        "job_class"=>"ChargeCreditCard",
+        "args"=>"[345,56]",
+        "error_count"=>"0",
+        "last_error"=>nil,
+        "pg_backend_pid"=>"1175",
+        "pg_state"=>"idle",
+        "pg_state_changed_at"=>"2014-01-04 22:35:55.777785+00",
+        "pg_last_query"=>"SELECT * FROM users",
+        "pg_last_query_started_at"=>"2014-01-04 22:35:55.777519+00",
+        "pg_transaction_started_at"=>nil,
+        "pg_waiting_on_lock"=>"f"
+      }
+    ]
+
+In this case, there is only one worker currently working the queue. The first seven fields are the attributes of the job it is currently running. The next seven fields are information about that worker's Postgres connection, and are taken from `pg_stat_activity` - see [Postgres' documentation](http://www.postgresql.org/docs/current/static/monitoring-stats.html#PG-STAT-ACTIVITY-VIEW) for more information on interpreting these fields.
+
+* `pg_backend_pid` - The pid of the Postgres process serving this worker. This is useful if you wanted to kill that worker's connection, for example, by running "SELECT pg_terminate_backend(1175)". This would free up the job to be attempted by another worker.
+* `pg_state` - The state of the Postgres backend. It may be "active" if the worker is currently running a query or "idle"/"idle in transaction" if it is not. It may also be in one of a few other less common states.
+* `pg_state_changed_at` - The timestamp for when the backend's state was last changed. If the backend is idle, this would reflect the time that the last query finished.
+* `pg_last_query` - The text of the current or most recent query that the worker sent to the database.
+* `pg_last_query_started_at` - The timestamp for when the last query began to run.
+* `pg_transaction_started_at` - The timestamp for when the worker's current transaction (if any) began.
+* `pg_waiting_on_lock` - Whether or not the worker is waiting for a lock in Postgres to be released.
+
+### Custom Queries
+
+If you want to query the jobs table yourself to see what's been queued or to check the state of various jobs, you can always use Que to execute whatever SQL you want:
+
+    Que.execute("select count(*) from que_jobs") #=> [{"count"=>"492"}]
+
+If you want to use ActiveRecord's features when querying, you can define your own model around Que's job table:
+
+    class QueJob < ActiveRecord::Base
+    end
+
+    # Or:
+
+    class MyJob < ActiveRecord::Base
+      self.table_name = :que_jobs
+    end
+
+Then you can query just as you would with any other model. Since the jobs table has a composite primary key, however, you probably won't be able to update or destroy jobs this way, though.
+
+If you're using Sequel, you can use the same technique:
+
+    class QueJob < Sequel::Model
+    end
+
+    # Or:
+
+    class MyJob < Sequel::Model(:que_jobs)
+    end
+
+And note that Sequel *does* support composite primary keys:
+
+    job = QueJob.where(:job_class => "ChargeCreditCard").first
+    job.priority = 1
+    job.save
+
+Or, you can just use Sequel's dataset methods:
+
+    DB[:que_jobs].where{priority > 3}.all

data/docs/managing_workers.md

@@ -0,0 +1,67 @@
+## Managing Workers
+
+Que provides a pool of workers to process jobs in a multithreaded fashion - this allows you to save memory by working many jobs simultaneously in the same process.
+
+When the worker pool is active (as it is by default when running `rails server`, or when you set Que.mode = :async), the default number of workers is 4. This is fine for most use cases, but the ideal number for your app will depend on your interpreter and what types of jobs you're running.
+
+Ruby MRI has a global interpreter lock (GIL), which prevents it from using more than one CPU core at a time. Having multiple workers running makes sense if your jobs tend to spend a lot of time in I/O (waiting on complex database queries, sending emails, making HTTP requests, etc.), as most jobs do. However, if your jobs are doing a lot of work in Ruby, they'll be spending a lot of time blocking each other, and having too many workers running will just slow everything down.
+
+JRuby and Rubinius, on the other hand, have no global interpreter lock, and so can make use of multiple CPU cores - you could potentially set the number of workers very high for them. You should experiment to find the best setting for your use case.
+
+You can change the number of workers in the pool whenever you like by setting the `worker_count` option:
+
+    Que.worker_count = 8
+
+    # Or, in your Rails configuration:
+    config.que.worker_count = 8
+
+### Working Jobs Via Rake Task
+
+If you don't want to burden your web processes with too much work and want to run workers in a background process instead, similar to how most other queues work, you can:
+
+    # Run a pool of 4 workers:
+    rake que:work
+
+    # Or configure the number of workers:
+    WORKER_COUNT=8 rake que:work
+
+### Thread-Unsafe Application Code
+
+If your application code is not thread-safe, you won't want any workers to be processing jobs while anything else is happening in the Ruby process. So, you'll want to turn the worker pool off by default:
+
+    Que.mode = :off
+
+    # Or, in your Rails configuration:
+    config.que.mode = :off
+
+This will prevent Que from trying to process jobs in the background of your web processes. In order to actually work jobs, you'll want to run a single worker at a time, and to do so via a separate rake task, like so:
+
+    WORKER_COUNT=1 rake que:work
+
+### The Wake Interval
+
+If a worker checks the job queue and finds no jobs ready for it to work, it will fall asleep. In order to make sure that newly-available jobs don't go unworked, a worker is awoken every so often to check for available work. By default, this happens every five seconds, but you can make it happen more or less often by setting a custom wake_interval:
+
+    Que.wake_interval = 2
+
+    # Or, in your Rails configuration:
+    config.que.wake_interval = 2   # 2.seconds also works fine.
+
+You can also choose to never let workers wake up on their own:
+
+    # Never wake up any workers:
+    Que.wake_interval = nil
+
+If you do this, though, you'll need to wake workers manually.
+
+### Manually Waking Workers
+
+Regardless of the `wake_interval` setting, you can always wake workers manually:
+
+    # Wake up a single worker to check the queue for work:
+    Que.wake!
+
+    # Wake up all workers in this process to check for work:
+    Que.wake_all!
+
+`Que.wake_all!` is helpful if there are no jobs available and all your workers go to sleep, and then you queue a large number of jobs. Typically, it will take a little while for the entire pool of workers get going again - a new one will wake up every `wake_interval` seconds, but it will take up to `wake_interval * worker_count` seconds for all of them to get going. `Que.wake_all!` can get them all moving immediately.

data/docs/using_plain_connections.md

@@ -0,0 +1,34 @@
+## Using Plain Postgres Connections
+
+If you're not using an ORM like ActiveRecord or Sequel, you can have Que access jobs using a plain Postgres connection:
+
+    require 'uri'
+    require 'pg'
+
+    uri = URI.parse(ENV['DATABASE_URL'])
+
+    Que.connection = PG::Connection.open :host     => uri.host,
+                                         :user     => uri.user,
+                                         :password => uri.password,
+                                         :port     => uri.port || 5432,
+                                         :dbname   => uri.path[1..-1]
+
+If you want to be able to use multithreading to run multiple jobs simultaneously in the same process, though, you'll need the ConnectionPool gem (be sure to add `gem 'connection_pool'` to your Gemfile):
+
+    require 'uri'
+    require 'pg'
+    require 'connection_pool'
+
+    uri = URI.parse(ENV['DATABASE_URL'])
+
+    Que.connection = ConnectionPool.new :size => 10 do
+      PG::Connection.open :host     => uri.host,
+                          :user     => uri.user,
+                          :password => uri.password,
+                          :port     => uri.port || 5432,
+                          :dbname   => uri.path[1..-1]
+    end
+
+Be sure to pick your pool size carefully - if you use 10 for the size, you'll incur the overhead of having 10 connections open to Postgres even if you never use more than a couple of them.
+
+Please be aware that if you're using ActiveRecord or Sequel to manage your data, there's no reason for you to be using either of these methods - it's less efficient (unnecessary connections will waste memory on your database server) and you lose the reliability benefits of wrapping jobs in the same transactions as the rest of your data.

data/docs/using_sequel.md

@@ -0,0 +1,27 @@
+## Using Sequel
+
+If you're using Sequel, with or without Rails, you'll need to give Que a specific database instance to use:
+
+    DB = Sequel.connect(ENV['DATABASE_URL'])
+    Que.connection = DB
+
+Then you can safely use the same database object to transactionally protect your jobs:
+
+    class MyJob < Que::Job
+      def run
+        # Do stuff.
+
+        DB.transaction do
+          # Make changes to the database.
+
+          # Destroying this job will be protected by the same transaction.
+          destroy
+        end
+      end
+    end
+
+    # In your controller action:
+    DB.transaction do
+      @user = User.create(params[:user])
+      SendRegistrationEmail.queue :user_id => @user.id
+    end

data/docs/writing_reliable_jobs.md

@@ -0,0 +1,62 @@
+## Writing Reliable Jobs
+
+Que does everything it can to ensure that jobs are worked exactly once, but if something bad happens when a job is halfway completed, there's no way around it - the job will need be repeated over again from the beginning, probably by a different worker. When you're writing jobs, you need to be prepared for this to happen.
+
+The safest type of job is one that reads in data, either from the database or from external APIs, then does some number crunching and writes the results to the database. These jobs are easy to make safe - simply write the results to the database inside a transaction, and also have the job destroy itself inside that transaction, like so:
+
+    class UpdateWidgetPrice < Que::Job
+      def run(widget_id)
+        widget = Widget[widget_id]
+        price  = ExternalService.get_widget_price(widget_id)
+
+        ActiveRecord::Base.transaction do
+          # Make changes to the database.
+          widget.update :price => price
+
+          # Destroy the job.
+          destroy
+        end
+      end
+    end
+
+Here, you're taking advantage of the guarantees of an [ACID](https://en.wikipedia.org/wiki/ACID) database. The job is destroyed along with the other changes, so either the write will succeed and the job will be run only once, or it will fail and the database will be left untouched. But even if it fails, the job can simply be retried, and there are no lingering effects from the first attempt, so no big deal.
+
+The more difficult type of job is one that makes changes that can't be controlled transactionally. For example, writing to an external service:
+
+    class ChargeCreditCard < Que::Job
+      def run(user_id, credit_card_id)
+        CreditCardService.charge(credit_card_id, :amount => "$10.00")
+
+        ActiveRecord::Base.transaction do
+          User.where(:id => user_id).update_all :charged_at => Time.now
+          destroy
+        end
+      end
+    end
+
+What if the process abruptly dies after we tell the provider to charge the credit card, but before we finish the transaction? Que will retry the job, but there's no way to tell where (or even if) it failed the first time. The credit card will be charged a second time, and then you've got an angry customer. The ideal solution in this case is to make the job [idempotent](https://en.wikipedia.org/wiki/Idempotence), meaning that it will have the same effect no matter how many times it is run:
+
+    class ChargeCreditCard < Que::Job
+      def run(user_id, credit_card_id)
+        unless CreditCardService.check_for_previous_charge(credit_card_id)
+          CreditCardService.charge(credit_card_id, :amount => "$10.00")
+        end
+
+        ActiveRecord::Base.transaction do
+          User.where(:id => user_id).update_all :charged_at => Time.now
+          destroy
+        end
+      end
+    end
+
+This makes the job slightly more complex, but reliable (or, at least, as reliable as your credit card service).
+
+Finally, there are some jobs where you won't want to write to the database at all:
+
+    class SendVerificationEmail < Que::Job
+      def run(email_address)
+        Mailer.verification_email(email_address).deliver
+      end
+    end
+
+In this case, we don't have any no way to prevent the occasional double-sending of an email. But, for ease of use, you can leave out the transaction and the `destroy` call entirely - Que will recognize that the job wasn't destroyed and will clean it up for you.