que 0.10.0 → 0.11.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: ea48a3d241852be82ac3310d9a9b77dd5c752c66
-  data.tar.gz: 9c959569e9b5eb520dffde4a21f08875d9efbbce
+  metadata.gz: 51b50d40120fb319646d1b216aa7f67931e9614b
+  data.tar.gz: c82fb2957389d088f3d6190311f9dae376fe82f6
 SHA512:
-  metadata.gz: 794d3b3a6e55beaa75a8142324a8eac2512505114b401aa1b306ef20e0e08f58913a8bb6685dd599a4bcadbdfec295c97637b22b6ddfdf8d2862225444e76ced
-  data.tar.gz: b186100af249195025f731e551155dcdcd9e4797a7164d0c71a9604842e5a3d604bd47f6b5331e0773bdfad1018332d21463266675251afeb5034407fc7dcae6
+  metadata.gz: 20c1f09a17e33aa0589ff01ef0983cf2000e9ae0be3594126a3c13aca10cca5d2b236f9296b17b8111d560fe2c62bc1d17924a3a00242ce31e768f95cdda39c8
+  data.tar.gz: e7f761b3622ae87e88f1ef85f81ebe8597ad82c4b1b07ce5aa00c0dc454cea08175ebf0fa2e01ebe0d7500220ee55f0035c8772945cd9ea95d63dfc5ca83c0f7

data/CHANGELOG.md

@@ -1,3 +1,17 @@
+### 0.11.0 (2015-09-04)
+
+*   A command-line program has been added that can be used to work jobs in a more flexible manner than the previous rake task. Run `que -h` for more information.
+
+*   The `rake que:work` rake task that was specific to Rails has been removed in favor of the CLI, and the various QUE_* environment variables no longer have any effect.
+
+*   The worker pool will no longer start automatically in the same process when running the rails server - this behavior was too prone to breakage. If you'd like to recreate the old behavior, you can manually set `Que.mode = :async` in your app whenever conditions are appropriate (classes have loaded, a database connection has been established, and the process will not be forking).
+
+*   Add a Que.disable_prepared_transactions= configuration option, to make it easier to use tools like pgbouncer. (#110)
+
+*   Add a Que.json_converter= option, to configure how arguments are transformed before being passed to the job. By default this is set to the `Que::INDIFFERENTIATOR` proc, which provides simple indifferent access (via strings or symbols) to args hashes. If you're using Rails, the default is to convert the args to HashWithIndifferentAccess instead. You can also pass it the Que::SYMBOLIZER proc, which will destructively convert all keys in the args hash to symbols (this will probably be the default in Que 1.0). If you want to define a custom converter, you will usually want to pass this option a proc, and you'll probably want it to be recursive. See the implementations of Que::INDIFFERENTIATOR and Que::SYMBOLIZER for examples. (#113)
+
+*   When using Que with ActiveRecord, workers now call `ActiveRecord::Base.clear_active_connections!` between jobs. This cleans up connections that ActiveRecord leaks when it is used to access mutliple databases. (#116)
+
 ### 0.10.0 (2015-03-18)
 
 *   When working jobs via the rake task, Rails applications are now eager-loaded if present, to avoid problems with multithreading and autoloading. (#96) (hmarr)

data/README.md

@@ -23,6 +23,8 @@ Que also includes a worker pool, so that multiple threads can process jobs in th
 
 Que is tested on Ruby 2.0, Rubinius and JRuby (with the `jruby-pg` gem, which is [not yet functional with ActiveRecord](https://github.com/chanks/que/issues/4#issuecomment-29561356)). It requires Postgres 9.2+ for the JSON datatype.
 
+**Please note** - Que's job table undergoes a lot of churn when it is under high load, and like any heavily-written table, is susceptible to bloat and slowness if Postgres isn't able to clean it up. The most common cause of this is long-running transactions, so it's recommended to try to keep all transactions against the database housing Que's job table as short as possible. This is good advice to remember for any high-activity database, but bears emphasizing when using tables that undergo a lot of writes.
+
 
 ## Installation
 
@@ -98,11 +100,12 @@ ChargeCreditCard.enqueue current_user.id, :credit_card_id => card.id, :run_at =>
 
 To determine what happens when a job is queued, you can set Que's mode. There are a few options for the mode:
 
-  * `Que.mode = :off` - In this mode, queueing a job will simply insert it into the database - the current process will make no effort to run it. You should use this if you want to use a dedicated process to work tasks (there's a rake task to do this, see below). This is the default when running `bin/rails console`.
+  * `Que.mode = :off` - In this mode, queueing a job will simply insert it into the database - the current process will make no effort to run it. You should use this if you want to use a dedicated process to work tasks (there's a rake task included that will do this, `rake que:work`). This is the default when running `bin/rails console`.
   * `Que.mode = :async` - In this mode, a pool of background workers is spun up, each running in their own thread. This is the default when running `bin/rails server`. See the docs for [more information on managing workers](https://github.com/chanks/que/blob/master/docs/managing_workers.md).
   * `Que.mode = :sync` - In this mode, any jobs you queue will be run in the same thread, synchronously (that is, `MyJob.enqueue` runs the job and won't return until it's completed). This makes your application's behavior easier to test, so it's the default in the test environment.
 
-If you're using ActiveRecord to dump your database's schema, you'll probably want to [set schema_format to :sql](http://guides.rubyonrails.org/migrations.html#types-of-schema-dumps) so that Que's table structure is managed correctly.
+**If you're using ActiveRecord to dump your database's schema, [set your schema_format to :sql](http://guides.rubyonrails.org/migrations.html#types-of-schema-dumps) so that Que's table structure is managed correctly.** (You can use schema_format as :ruby if you want but keep in mind this is highly advised against, as some parts of Que will not work.)
+
 
 ## Related Projects
 
@@ -112,15 +115,19 @@ If you're using ActiveRecord to dump your database's schema, you'll probably wan
 
 If you have a project that uses or relates to Que, feel free to submit a PR adding it to the list!
 
-## Contributing
 
-  1. Fork it
-  2. Create your feature branch (`git checkout -b my-new-feature`)
-  3. Commit your changes (`git commit -am 'Add some feature'`)
-  4. Push to the branch (`git push origin my-new-feature`)
-  5. Create new Pull Request
+## Community and Contributing
+
+  * For bugs in the library, please feel free to [open an issue](https://github.com/chanks/que/issues/new).
+  * For general discussion and questions/concerns that don't relate to obvious bugs, try posting on the [que-talk Google Group](https://groups.google.com/forum/#!forum/que-talk).
+  * For contributions, pull requests submitted via Github are welcome.
+
+Regarding contributions, one of the project's priorities is to keep Que as simple, lightweight and dependency-free as possible, and pull requests that change too much or wouldn't be useful to the majority of Que's users have a good chance of being rejected. If you're thinking of submitting a pull request that adds a new feature, consider starting a discussion in [que-talk](https://groups.google.com/forum/#!forum/que-talk) first about what it would do and how it would be implemented. If it's a sufficiently large feature, or if most of Que's users wouldn't find it useful, it may be best implemented as a standalone gem, like some of the related projects above.
+
+
+### Specs
 
-A note on running specs - Que's worker system is multithreaded and therefore prone to race conditions (especially on Rubinius). As such, if you've touched that code, a single spec run passing isn't a guarantee that any changes you've made haven't introduced bugs. One thing I like to do before pushing changes is rerun the specs many times and watching for hangs. You can do this from the command line with something like:
+A note on running specs - Que's worker system is multithreaded and therefore prone to race conditions (especially on interpreters without a global lock, like Rubinius or JRuby). As such, if you've touched that code, a single spec run passing isn't a guarantee that any changes you've made haven't introduced bugs. One thing I like to do before pushing changes is rerun the specs many times and watching for hangs. You can do this from the command line with something like:
 
     for i in {1..1000}; do rspec -b --seed $i; done
 

data/bin/que

@@ -0,0 +1,85 @@
+#!/usr/bin/env ruby
+
+require 'optparse'
+require 'ostruct'
+require 'logger'
+
+options = OpenStruct.new
+
+OptionParser.new do |opts|
+  opts.banner = 'usage: que [options] file/to/require ...'
+
+  opts.on('-w', '--worker-count [COUNT]', Integer, "Set number of workers in process (default: 4)") do |worker_count|
+    options.worker_count = worker_count
+  end
+
+  opts.on('-i', '--wake-interval [INTERVAL]', Float, "Set maximum interval between polls of the job queue (in seconds) (default: 0.1)") do |wake_interval|
+    options.wake_interval = wake_interval
+  end
+
+  opts.on('-l', '--log-level [LEVEL]', String, "Set level of Que's logger (debug, info, warn, error, fatal) (default: info)") do |log_level|
+    options.log_level = log_level
+  end
+
+  opts.on('-q', '--queue-name [NAME]', String, "Set the name of the queue to work jobs from (default: the default queue)") do |queue_name|
+    options.queue_name = queue_name
+  end
+
+  opts.on('-v', '--version', "Show Que version") do
+    $stdout.puts "Que version #{Version}"
+    exit 0
+  end
+
+  opts.on('-h', '--help', "Show help text") do
+    $stdout.puts opts
+    exit 0
+  end
+end.parse!(ARGV)
+
+if ARGV.length.zero?
+  $stdout.puts <<-OUTPUT
+You didn't include any Ruby files to require!
+Que needs to be able to load your application before it can process jobs.
+(Hint: If you're using Rails, try `que ./config/environment.rb`)
+(Or use `que -h` for a list of options)
+OUTPUT
+exit 1
+end
+
+ARGV.each do |file|
+  begin
+    require file
+  rescue LoadError
+    $stdout.puts "Could not load file '#{file}'"
+  end
+end
+
+Que.logger ||= Logger.new(STDOUT)
+
+begin
+  if log_level = options.log_level
+    Que.logger.level = Logger.const_get(log_level.upcase)
+  end
+rescue NameError
+  $stdout.puts "Bad logging level: #{log_level}"
+  exit 1
+end
+
+Que.queue_name    = options.queue_name    || Que.queue_name    || nil
+Que.worker_count  = options.worker_count  || Que.worker_count  || 4
+Que.wake_interval = options.wake_interval || Que.wake_interval || 0.1
+Que.mode          = :async
+
+stop = false
+%w(INT TERM).each { |signal| trap(signal) { stop = true } }
+
+loop do
+  sleep 0.01
+  break if stop
+end
+
+$stdout.puts
+$stdout.puts "Finishing Que's current jobs before exiting..."
+Que.worker_count = 0
+Que.mode = :off
+$stdout.puts "Que's jobs finished, exiting..."

data/docs/advanced_setup.md

@@ -4,17 +4,21 @@ If you're using both Rails and ActiveRecord, the README describes how to get sta
 
 If you're using ActiveRecord outside of Rails, you'll need to tell Que to piggyback on its connection pool after you've connected to the database:
 
-    ActiveRecord::Base.establish_connection(ENV['DATABASE_URL'])
+```ruby
+ActiveRecord::Base.establish_connection(ENV['DATABASE_URL'])
 
-    require 'que'
-    Que.connection = ActiveRecord
+require 'que'
+Que.connection = ActiveRecord
+```
 
 Then you can queue jobs just as you would in Rails:
 
-    ActiveRecord::Base.transaction do
-      @user = User.create(params[:user])
-      SendRegistrationEmail.enqueue :user_id => @user.id
-    end
+```ruby
+ActiveRecord::Base.transaction do
+  @user = User.create(params[:user])
+  SendRegistrationEmail.enqueue :user_id => @user.id
+end
+```
 
 There are other docs to read if you're using [Sequel](https://github.com/chanks/que/blob/master/docs/using_sequel.md) or [plain Postgres connections](https://github.com/chanks/que/blob/master/docs/using_plain_connections.md) (with no ORM at all) instead of ActiveRecord.
 
@@ -22,62 +26,75 @@ There are other docs to read if you're using [Sequel](https://github.com/chanks/
 
 If you want to run a worker pool in your web process and you're using a forking webserver like Phusion Passenger (in smart spawning mode), Unicorn or Puma in some configurations, you'll want to set `Que.mode = :off` in your application configuration and only start up the worker pool in the child processes after the DB connection has been reestablished. So, for Puma:
 
-    # config/puma.rb
-    on_worker_boot do
-      ActiveRecord::Base.establish_connection
+```ruby
+# config/puma.rb
+on_worker_boot do
+  ActiveRecord::Base.establish_connection
 
-      Que.mode = :async
-    end
+  Que.mode = :async
+end
+```
 
 And for Unicorn:
 
-    # config/unicorn.rb
-    after_fork do |server, worker|
-      ActiveRecord::Base.establish_connection
+```ruby
+# config/unicorn.rb
+after_fork do |server, worker|
+  ActiveRecord::Base.establish_connection
 
-      Que.mode = :async
-    end
+  Que.mode = :async
+end
+```
 
 And for Phusion Passenger:
 
-    # config.ru
-    if defined?(PhusionPassenger)
-      PhusionPassenger.on_event(:starting_worker_process) do |forked|
-        if forked
-          Que.mode = :async
-        end
-      end
+```ruby
+# config.ru
+if defined?(PhusionPassenger)
+  PhusionPassenger.on_event(:starting_worker_process) do |forked|
+    if forked
+      Que.mode = :async
     end
-
+  end
+end
+```
 
 ### Managing the Jobs Table
 
 After you've connected Que to the database, you can manage the jobs table:
 
-    # Create/update the jobs table to the latest schema version:
-    Que.migrate!
+```ruby
+# Create/update the jobs table to the latest schema version:
+Que.migrate!
+```
 
 You'll want to migrate to a specific version if you're using migration files, to ensure that they work the same way even when you upgrade Que in the future:
 
-    # Update the schema to version #3.
-    Que.migrate! :version => 3
+```ruby
+# Update the schema to version #3.
+Que.migrate! :version => 3
 
-    # To reverse the migration, drop the jobs table entirely:
-    Que.migrate! :version => 0
+# To reverse the migration, drop the jobs table entirely:
+Que.migrate! :version => 0
+```
 
 There's also a helper method to clear all jobs from the jobs table:
 
-    Que.clear!
+```ruby
+Que.clear!
+```
 
 ### Other Setup
 
 You'll need to set Que's mode manually:
 
-    # Start the worker pool:
-    Que.mode = :async
+```ruby
+# Start the worker pool:
+Que.mode = :async
 
-    # Or, when testing:
-    Que.mode = :sync
+# Or, when testing:
+Que.mode = :sync
+```
 
 Be sure to read the docs on [managing workers](https://github.com/chanks/que/blob/master/docs/managing_workers.md) for more information on using the worker pool.
 

data/docs/customizing_que.md

@@ -2,43 +2,47 @@
 
 One of Que's goals to be easily extensible and hackable (and if anyone has any suggestions on ways to accomplish that, please [open an issue](https://github.com/chanks/que/issues)). This document is meant to demonstrate some of the ways Que can be used to accomplish different tasks that it's not already designed for.
 
+Some of these features may be moved into core Que at some point, depending on how commonly useful they are.
+
 ### Recurring Jobs
 
 Que's support for scheduling jobs makes it easy to implement reliable recurring jobs. For example, suppose you want to run a job every hour that processes the users created in that time:
 
-      class CronJob < Que::Job
-        # Default repetition interval in seconds. Can be overridden in
-        # subclasses. Can use 1.minute if using Rails.
-        INTERVAL = 60
+```ruby
+class CronJob < Que::Job
+  # Default repetition interval in seconds. Can be overridden in
+  # subclasses. Can use 1.minute if using Rails.
+  INTERVAL = 60
 
-        attr_reader :start_at, :end_at, :run_again_at, :time_range
+  attr_reader :start_at, :end_at, :run_again_at, :time_range
 
-        def _run
-          args = attrs[:args].first
-          @start_at, @end_at = Time.at(args.delete('start_at')), Time.at(args.delete('end_at'))
-          @run_again_at = @end_at + self.class::INTERVAL
-          @time_range = @start_at...@end_at
+  def _run
+    args = attrs[:args].first
+    @start_at, @end_at = Time.at(args.delete('start_at')), Time.at(args.delete('end_at'))
+    @run_again_at = @end_at + self.class::INTERVAL
+    @time_range = @start_at...@end_at
 
-          super
+    super
 
-          args['start_at'] = @end_at.to_f
-          args['end_at']   = @run_again_at.to_f
-          self.class.enqueue(args, run_at: @run_again_at)
-        end
-      end
+    args['start_at'] = @end_at.to_f
+    args['end_at']   = @run_again_at.to_f
+    self.class.enqueue(args, run_at: @run_again_at)
+  end
+end
 
-      class MyCronJob < CronJob
-        INTERVAL = 3600
+class MyCronJob < CronJob
+  INTERVAL = 3600
 
-        def run(args)
-          User.where(created_at: time_range).each { ... }
-        end
-      end
+  def run(args)
+    User.where(created_at: time_range).each { ... }
+  end
+end
 
-      # To enqueue:
-      tf = Time.now
-      t0 = Time.now - 3600
-      MyCronJob.enqueue :start_at => t0.to_f, :end_at => tf.to_f
+# To enqueue:
+tf = Time.now
+t0 = Time.now - 3600
+MyCronJob.enqueue :start_at => t0.to_f, :end_at => tf.to_f
+```
 
 Note that instead of using Time.now in our database query, and requeueing the job at 1.hour.from_now, we use job arguments to track start and end times. This lets us correct for delays in running the job. Suppose that there's a backlog of priority jobs, or that the worker briefly goes down, and this job, which was supposed to run at 11:00 a.m. isn't run until 11:05 a.m. A lazier implementation would look for users created after 1.hour.ago, and miss those that signed up between 10:00 a.m. and 10:05 a.m.
 
@@ -52,7 +56,9 @@ Finally, by passing both the start and end times for the period to be processed,
 
 DelayedJob offers a simple API for delaying methods to objects:
 
-    @user.delay.activate!(@device)
+```ruby
+@user.delay.activate!(@device)
+```
 
 The API is pleasant, but implementing it requires storing marshalled Ruby objects in the database, which is both inefficient and prone to bugs - for example, if you deploy an update that changes the name of an instance variable (a contained, internal change that might seem completely innocuous), the marshalled objects in the database will retain the old instance variable name and will behave unexpectedly when unmarshalled into the new Ruby code.
 
@@ -60,27 +66,29 @@ This is the danger of mixing the ephemeral state of a Ruby object in memory with
 
 That said, if you want to queue jobs in the DelayedJob style, that can be done relatively easily:
 
-    class Delayed < Que::Job
-      def run(receiver, method, args)
-        Marshal.load(receiver).send method, *Marshal.load(args)
-      end
-    end
-
-    class DelayedAction
-      def initialize(receiver)
-        @receiver = receiver
-      end
-
-      def method_missing(method, *args)
-        Delayed.enqueue Marshal.dump(@receiver), method, Marshal.dump(args)
-      end
-    end
-
-    class Object
-      def delay
-        DelayedAction.new(self)
-      end
-    end
+```ruby
+class Delayed < Que::Job
+  def run(receiver, method, args)
+    Marshal.load(receiver).send method, *Marshal.load(args)
+  end
+end
+
+class DelayedAction
+  def initialize(receiver)
+    @receiver = receiver
+  end
+
+  def method_missing(method, *args)
+    Delayed.enqueue Marshal.dump(@receiver), method, Marshal.dump(args)
+  end
+end
+
+class Object
+  def delay
+    DelayedAction.new(self)
+  end
+end
+```
 
 You can replace Marshal with YAML if you like.
 
@@ -88,50 +96,105 @@ You can replace Marshal with YAML if you like.
 
 You may find it a hassle to keep an individual class file for each type of job. QueueClassic has a simpler design, wherein you simply give it a class method to call, like:
 
-    QC.enqueue("Kernel.puts", "hello world")
+```ruby
+QC.enqueue("Kernel.puts", "hello world")
+```
 
 You can mimic this style with Que by using a simple job class:
 
-    class Command < Que::Job
-      def run(method, *args)
-        receiver, message = method.split('.')
-        Object.const_get(receiver).send(message, *args)
-      end
-    end
+```ruby
+class Command < Que::Job
+  def run(method, *args)
+    receiver, message = method.split('.')
+    Object.const_get(receiver).send(message, *args)
+  end
+end
 
-    # Then:
+# Then:
 
-    Command.enqueue "Kernel.puts", "hello world"
+Command.enqueue "Kernel.puts", "hello world"
+```
 
 ### Retaining Finished Jobs
 
 Que deletes jobs from the queue as they are worked, in order to keep the `que_jobs` table and index small and efficient. If you have a need to hold onto finished jobs, the recommended way to do this is to add a second table to hold them, and then insert them there as they are deleted from the queue. You can use Ruby's inheritance mechanics to do this cleanly:
 
-    Que.execute "CREATE TABLE finished_jobs AS SELECT * FROM que_jobs LIMIT 0"
-    # Or, better, use a proper CREATE TABLE with not-null constraints, and add whatever indexes you like.
+```ruby
+Que.execute "CREATE TABLE finished_jobs AS SELECT * FROM que_jobs LIMIT 0"
+# Or, better, use a proper CREATE TABLE with not-null constraints, and add whatever indexes you like.
 
-    class MyJobClass < Que::Job
-      def destroy
-        Que.execute "INSERT INTO finished_jobs SELECT * FROM que_jobs WHERE queue = $1::text AND priority = $2::integer AND run_at = $3::timestamptz AND job_id = $4::bigint", @attrs.values_at(:queue, :priority, :run_at, :job_id)
-        super
-      end
-    end
+class MyJobClass < Que::Job
+  def destroy
+    Que.execute "INSERT INTO finished_jobs SELECT * FROM que_jobs WHERE queue = $1::text AND priority = $2::integer AND run_at = $3::timestamptz AND job_id = $4::bigint", @attrs.values_at(:queue, :priority, :run_at, :job_id)
+    super
+  end
+end
+```
 
 Then just have your job classes inherit from MyJobClass instead of Que::Job. If you need to query the jobs table and you want to include both finished and unfinished jobs, you might use:
 
-    Que.execute "CREATE VIEW all_jobs AS SELECT * FROM que_jobs UNION ALL SELECT * FROM finished_jobs"
-    Que.execute "SELECT * FROM all_jobs"
+```ruby
+Que.execute "CREATE VIEW all_jobs AS SELECT * FROM que_jobs UNION ALL SELECT * FROM finished_jobs"
+Que.execute "SELECT * FROM all_jobs"
+```
 
 Alternately, if you want a more foolproof solution and you're not scared of PostgreSQL, you can use a trigger:
 
-    CREATE FUNCTION please_save_my_job()
-    RETURNS trigger
-    LANGUAGE plpgsql
-    AS $$
-      BEGIN
-        INSERT INTO finished_jobs SELECT (OLD).*;
-        RETURN OLD;
-      END;
-    $$;
-
-    CREATE TRIGGER keep_all_my_old_jobs BEFORE DELETE ON que_jobs FOR EACH ROW EXECUTE PROCEDURE please_save_my_job();
+```sql
+CREATE FUNCTION please_save_my_job()
+RETURNS trigger
+LANGUAGE plpgsql
+AS $$
+  BEGIN
+    INSERT INTO finished_jobs SELECT (OLD).*;
+    RETURN OLD;
+  END;
+$$;
+
+CREATE TRIGGER keep_all_my_old_jobs BEFORE DELETE ON que_jobs FOR EACH ROW EXECUTE PROCEDURE please_save_my_job();
+```
+
+### Not Retrying Certain Failed Jobs
+
+By default, when jobs fail, Que reschedules them to be retried later. If instead you'd like certain jobs to not be retried, and instead move them elsewhere to be examined later, you can accomplish that easily. First, we need a place for the failed jobs to be stored:
+
+```sql
+CREATE TABLE failed_jobs AS SELECT * FROM que_jobs LIMIT 0
+```
+
+Then, create a module that you can use in the jobs you don't want to retry:
+
+```ruby
+module SkipRetries
+  def run(*args)
+    super
+  rescue
+    sql = <<-SQL
+      WITH failed AS (
+        DELETE
+        FROM   que_jobs
+        WHERE  queue    = $1::text
+        AND    priority = $2::smallint
+        AND    run_at   = $3::timestamptz
+        AND    job_id   = $4::bigint
+        RETURNING *
+      )
+      INSERT INTO failed_jobs
+        SELECT * FROM failed;
+    SQL
+
+    Que.execute sql, @attrs.values_at(:queue, :priority, :run_at, :job_id)
+
+    raise # Reraises caught error.
+  end
+end
+
+class RunOnceJob < Que::Job
+  prepend SkipRetries
+
+  def run(*args)
+    # Do something - if this job runs an error it'll be moved to the
+    # failed_jobs table and not retried.
+  end
+end
+```

data/docs/error_handling.md

@@ -4,40 +4,44 @@ If an error is raised and left uncaught by your job, Que will save the error mes
 
 If a given job fails repeatedly, Que will retry it at exponentially-increasing intervals equal to (failure_count^4 + 3) seconds. This means that a job will be retried 4 seconds after its first failure, 19 seconds after its second, 84 seconds after its third, 259 seconds after its fourth, and so on until it succeeds. This pattern is very similar to DelayedJob's. Alternately, you can define your own retry logic by setting an interval to delay each time, or a callable that accepts the number of failures and returns an interval:
 
-    class MyJob < Que::Job
-      # Just retry a failed job every 5 seconds:
-      @retry_interval = 5
+```ruby
+class MyJob < Que::Job
+  # Just retry a failed job every 5 seconds:
+  @retry_interval = 5
 
-      # Always retry this job immediately (not recommended, or transient
-      # errors will spam your error reporting):
-      @retry_interval = 0
+  # Always retry this job immediately (not recommended, or transient
+  # errors will spam your error reporting):
+  @retry_interval = 0
 
-      # Increase the delay by 30 seconds every time this job fails:
-      @retry_interval = proc { |count| count * 30 }
-    end
+  # Increase the delay by 30 seconds every time this job fails:
+  @retry_interval = proc { |count| count * 30 }
+end
+```
 
 Unlike DelayedJob, however, there is currently no maximum number of failures after which jobs will be deleted. Que's assumption is that if a job is erroring perpetually (and not just transiently), you will want to take action to get the job working properly rather than simply losing it silently.
 
 If you're using an error notification system (highly recommended, of course), you can hook Que into it by setting a callable as the error handler:
 
-    Que.error_handler = proc do |error, job|
-      # Do whatever you want with the error object or job row here.
-
-      # Note that the job passed is not the actual job object, but the hash
-      # representing the job row in the database, which looks like:
-
-      # {
-      #   "queue" => "my_queue",
-      #   "priority" => 100,
-      #   "run_at" => 2015-03-06 11:07:08 -0500,
-      #   "job_id" => 65,
-      #   "job_class" => "MyJob",
-      #   "args" => ['argument', 78],
-      #   "error_count" => 0
-      # }
-
-      # This is done because the job may not have been able to be deserialized
-      # properly, if the name of the job class was changed or the job is being
-      # retrieved and worked by the wrong app. The job argument may also be
-      # nil, if there was a connection failure or something similar.
-    end
+```ruby
+Que.error_handler = proc do |error, job|
+  # Do whatever you want with the error object or job row here.
+
+  # Note that the job passed is not the actual job object, but the hash
+  # representing the job row in the database, which looks like:
+
+  # {
+  #   "queue" => "my_queue",
+  #   "priority" => 100,
+  #   "run_at" => 2015-03-06 11:07:08 -0500,
+  #   "job_id" => 65,
+  #   "job_class" => "MyJob",
+  #   "args" => ['argument', 78],
+  #   "error_count" => 0
+  # }
+
+  # This is done because the job may not have been able to be deserialized
+  # properly, if the name of the job class was changed or the job is being
+  # retrieved and worked by the wrong app. The job argument may also be
+  # nil, if there was a connection failure or something similar.
+end
+```