que 0.10.0 → 0.11.0

data/docs/inspecting_the_queue.md

@@ -6,24 +6,26 @@ In order to remain simple and compatible with any ORM (or no ORM at all), Que is
 
 You can call `Que.job_stats` to return some aggregate data on the types of jobs currently in the queue. Example output:
 
-    [
-      {
-        "job_class"=>"ChargeCreditCard",
-        "count"=>"10",
-        "count_working"=>"4",
-        "count_errored"=>"2",
-        "highest_error_count"=>"5",
-        "oldest_run_at"=>"2014-01-04 21:24:55.817129+00"
-      },
-      {
-        "job_class"=>"SendRegistrationEmail",
-        "count"=>"8",
-        "count_working"=>"0",
-        "count_errored"=>"0",
-        "highest_error_count"=>"0",
-        "oldest_run_at"=>"2014-01-04 22:24:55.81532+00"
-      }
-    ]
+```ruby
+[
+  {
+    "job_class"=>"ChargeCreditCard",
+    "count"=>"10",
+    "count_working"=>"4",
+    "count_errored"=>"2",
+    "highest_error_count"=>"5",
+    "oldest_run_at"=>"2014-01-04 21:24:55.817129+00"
+  },
+  {
+    "job_class"=>"SendRegistrationEmail",
+    "count"=>"8",
+    "count_working"=>"0",
+    "count_errored"=>"0",
+    "highest_error_count"=>"0",
+    "oldest_run_at"=>"2014-01-04 22:24:55.81532+00"
+  }
+]
+```
 
 This tells you that, for instance, there are ten ChargeCreditCard jobs in the queue, four of which are currently being worked, and two of which have experienced errors. One of them has started to process but experienced an error five times. The oldest_run_at is helpful for determining how long jobs have been sitting around, if you have backlog.
 
@@ -31,24 +33,26 @@ This tells you that, for instance, there are ten ChargeCreditCard jobs in the qu
 
 You can call `Que.worker_states` to return some information on every worker touching the queue (not just those in the current process). Example output:
 
-    [
-      {
-        "priority"=>"2",
-        "run_at"=>"2014-01-04 22:35:55.772324+00",
-        "job_id"=>"4592",
-        "job_class"=>"ChargeCreditCard",
-        "args"=>"[345,56]",
-        "error_count"=>"0",
-        "last_error"=>nil,
-        "pg_backend_pid"=>"1175",
-        "pg_state"=>"idle",
-        "pg_state_changed_at"=>"2014-01-04 22:35:55.777785+00",
-        "pg_last_query"=>"SELECT * FROM users",
-        "pg_last_query_started_at"=>"2014-01-04 22:35:55.777519+00",
-        "pg_transaction_started_at"=>nil,
-        "pg_waiting_on_lock"=>"f"
-      }
-    ]
+```ruby
+[
+  {
+    "priority"=>"2",
+    "run_at"=>"2014-01-04 22:35:55.772324+00",
+    "job_id"=>"4592",
+    "job_class"=>"ChargeCreditCard",
+    "args"=>"[345,56]",
+    "error_count"=>"0",
+    "last_error"=>nil,
+    "pg_backend_pid"=>"1175",
+    "pg_state"=>"idle",
+    "pg_state_changed_at"=>"2014-01-04 22:35:55.777785+00",
+    "pg_last_query"=>"SELECT * FROM users",
+    "pg_last_query_started_at"=>"2014-01-04 22:35:55.777519+00",
+    "pg_transaction_started_at"=>nil,
+    "pg_waiting_on_lock"=>"f"
+  }
+]
+```
 
 In this case, there is only one worker currently working the queue. The first seven fields are the attributes of the job it is currently running. The next seven fields are information about that worker's Postgres connection, and are taken from `pg_stat_activity` - see [Postgres' documentation](http://www.postgresql.org/docs/current/static/monitoring-stats.html#PG-STAT-ACTIVITY-VIEW) for more information on interpreting these fields.
 
@@ -64,37 +68,47 @@ In this case, there is only one worker currently working the queue. The first se
 
 If you want to query the jobs table yourself to see what's been queued or to check the state of various jobs, you can always use Que to execute whatever SQL you want:
 
-    Que.execute("select count(*) from que_jobs") #=> [{"count"=>"492"}]
+```ruby
+Que.execute("select count(*) from que_jobs") #=> [{"count"=>"492"}]
+```
 
 If you want to use ActiveRecord's features when querying, you can define your own model around Que's job table:
 
-    class QueJob < ActiveRecord::Base
-    end
+```ruby
+class QueJob < ActiveRecord::Base
+end
 
-    # Or:
+# Or:
 
-    class MyJob < ActiveRecord::Base
-      self.table_name = :que_jobs
-    end
+class MyJob < ActiveRecord::Base
+  self.table_name = :que_jobs
+end
+```
 
 Then you can query just as you would with any other model. Since the jobs table has a composite primary key, however, you probably won't be able to update or destroy jobs this way, though.
 
 If you're using Sequel, you can use the same technique:
 
-    class QueJob < Sequel::Model
-    end
+```ruby
+class QueJob < Sequel::Model
+end
 
-    # Or:
+# Or:
 
-    class MyJob < Sequel::Model(:que_jobs)
-    end
+class MyJob < Sequel::Model(:que_jobs)
+end
+```
 
 And note that Sequel *does* support composite primary keys:
 
-    job = QueJob.where(:job_class => "ChargeCreditCard").first
-    job.priority = 1
-    job.save
+```ruby
+job = QueJob.where(:job_class => "ChargeCreditCard").first
+job.priority = 1
+job.save
+```
 
 Or, you can just use Sequel's dataset methods:
 
-    DB[:que_jobs].where{priority > 3}.all
+```ruby
+DB[:que_jobs].where{priority > 3}.all
+```

data/docs/logging.md

@@ -2,37 +2,49 @@
 
 By default, Que logs important information in JSON to either Rails' logger (when running in a Rails web process) or STDOUT (when running as a rake task). So, your logs will look something like:
 
-    I, [2014-01-12T05:07:31.094201 #4687]  INFO -- : {"lib":"que","thread":104928,"event":"job_worked","elapsed":0.01045,"job":{"priority":"1","run_at":"2014-01-12 05:07:31.081877+00","job_id":"4","job_class":"MyJob","args":[],"error_count":"0"}}
+```
+I, [2014-01-12T05:07:31.094201 #4687]  INFO -- : {"lib":"que","thread":104928,"event":"job_worked","elapsed":0.01045,"job":{"priority":"1","run_at":"2014-01-12 05:07:31.081877+00","job_id":"4","job_class":"MyJob","args":[],"error_count":"0"}}
+```
 
 Of course you can have it log wherever you like:
 
-    Que.logger = Logger.new(...)
+```ruby
+Que.logger = Logger.new(...)
+```
 
 You can use Que's logger in your jobs anywhere you like:
 
-    class MyJob
-      def run
-        Que.log :my_output => "my string"
-      end
-    end
+```ruby
+class MyJob
+  def run
+    Que.log :my_output => "my string"
+  end
+end
 
-    #=> I, [2014-01-12T05:13:11.006776 #4914]  INFO -- : {"lib":"que","thread":24960,"my_output":"my string"}
+#=> I, [2014-01-12T05:13:11.006776 #4914]  INFO -- : {"lib":"que","thread":24960,"my_output":"my string"}
+```
 
 Que will always add a 'lib' key, so you can easily filter its output from that of other sources, and the object_id of the thread that emitted the log, so you can follow the actions of a particular worker if you wish. You can also pass a :level key to set the level of the output:
 
-    Que.log :level => :debug, :my_output => 'my string'
-    #=> D, [2014-01-12T05:16:15.221941 #5088] DEBUG -- : {"lib":"que","thread":24960,"my_output":"my string"}
+```ruby
+Que.log :level => :debug, :my_output => 'my string'
+#=> D, [2014-01-12T05:16:15.221941 #5088] DEBUG -- : {"lib":"que","thread":24960,"my_output":"my string"}
+```
 
 If you don't like JSON, you can also customize the format of the logging output by passing a callable object (such as a proc) to Que.log_formatter=. The proc should take a hash (the keys are symbols) and return a string. The keys and values are just as you would expect from the JSON output:
 
-    Que.log_formatter = proc do |data|
-      "Thread number #{data[:thread]} experienced a #{data[:event]}"
-    end
+```ruby
+Que.log_formatter = proc do |data|
+  "Thread number #{data[:thread]} experienced a #{data[:event]}"
+end
+```
 
 If the log formatter returns nil or false, a nothing will be logged at all. You could use this to narrow down what you want to emit, for example:
 
-    Que.log_formatter = proc do |data|
-      if ['job_worked', 'job_unavailable'].include?(data[:event])
-        JSON.dump(data)
-      end
-    end
+```ruby
+Que.log_formatter = proc do |data|
+  if [:job_worked, :job_unavailable].include?(data[:event])
+    JSON.dump(data)
+  end
+end
+```

data/docs/managing_workers.md

@@ -2,7 +2,7 @@
 
 Que provides a pool of workers to process jobs in a multithreaded fashion - this allows you to save memory by working many jobs simultaneously in the same process.
 
-When the worker pool is active (as it is by default when running `rails server`, or when you set Que.mode = :async), the default number of workers is 4. This is fine for most use cases, but the ideal number for your app will depend on your interpreter and what types of jobs you're running.
+When the worker pool is active (as it is by default when running `rails server`, or when you set `Que.mode = :async`), the default number of workers is 4. This is fine for most use cases, but the ideal number for your app will depend on your interpreter and what types of jobs you're running.
 
 Ruby MRI has a global interpreter lock (GIL), which prevents it from using more than one CPU core at a time. Having multiple workers running makes sense if your jobs tend to spend a lot of time in I/O (waiting on complex database queries, sending emails, making HTTP requests, etc.), as most jobs do. However, if your jobs are doing a lot of work in Ruby, they'll be spending a lot of time blocking each other, and having too many workers running will just slow everything down.
 
@@ -10,42 +10,56 @@ JRuby and Rubinius, on the other hand, have no global interpreter lock, and so c
 
 You can change the number of workers in the pool whenever you like by setting the `worker_count` option:
 
-    Que.worker_count = 8
+```ruby
+Que.worker_count = 8
+```
 
 ### Working Jobs Via Rake Task
 
 If you don't want to burden your web processes with too much work and want to run workers in a background process instead, similar to how most other queues work, you can:
 
-    # Run a pool of 4 workers:
-    rake que:work
+```shell
+# Run a pool of 4 workers:
+rake que:work
 
-    # Or configure the number of workers:
-    QUE_WORKER_COUNT=8 rake que:work
+# Or configure the number of workers:
+QUE_WORKER_COUNT=8 rake que:work
+```
 
 Other options available via environment variables are `QUE_QUEUE` to determine which named queue jobs are pulled from, and `QUE_WAKE_INTERVAL` to determine how long workers will wait to poll again when there are no jobs available. For example, to run 2 workers that run jobs from the "other_queue" queue and wait a half-second between polls, you could do:
 
-    QUE_QUEUE=other_queue QUE_WORKER_COUNT=2 QUE_WAKE_INTERVAL=0.5 rake que:work
+```shell
+QUE_QUEUE=other_queue QUE_WORKER_COUNT=2 QUE_WAKE_INTERVAL=0.5 rake que:work
+```
 
 ### Thread-Unsafe Application Code
 
 If your application code is not thread-safe, you won't want any workers to be processing jobs while anything else is happening in the Ruby process. So, you'll want to turn the worker pool off by default:
 
-    Que.mode = :off
+```ruby
+Que.mode = :off
+```
 
 This will prevent Que from trying to process jobs in the background of your web processes. In order to actually work jobs, you'll want to run a single worker at a time, and to do so via a separate rake task, like so:
 
-    QUE_WORKER_COUNT=1 rake que:work
+```shell
+QUE_WORKER_COUNT=1 rake que:work
+```
 
 ### The Wake Interval
 
 If a worker checks the job queue and finds no jobs ready for it to work, it will fall asleep. In order to make sure that newly-available jobs don't go unworked, a worker is awoken every so often to check for available work. By default, this happens every five seconds, but you can make it happen more or less often by setting a custom wake_interval:
 
-    Que.wake_interval = 2 # In Rails, 2.seconds also works fine.
+```ruby
+Que.wake_interval = 2 # In Rails, 2.seconds also works fine.
+```
 
 You can also choose to never let workers wake up on their own:
 
-    # Never wake up any workers:
-    Que.wake_interval = nil
+```ruby
+# Never wake up any workers:
+Que.wake_interval = nil
+```
 
 If you do this, though, you'll need to wake workers manually.
 
@@ -53,11 +67,13 @@ If you do this, though, you'll need to wake workers manually.
 
 Regardless of the `wake_interval` setting, you can always wake workers manually:
 
-    # Wake up a single worker to check the queue for work:
-    Que.wake!
+```ruby
+# Wake up a single worker to check the queue for work:
+Que.wake!
 
-    # Wake up all workers in this process to check for work:
-    Que.wake_all!
+# Wake up all workers in this process to check for work:
+Que.wake_all!
+```
 
 `Que.wake_all!` is helpful if there are no jobs available and all your workers go to sleep, and then you queue a large number of jobs. Typically, it will take a little while for the entire pool of workers get going again - a new one will wake up every `wake_interval` seconds, but it will take up to `wake_interval * worker_count` seconds for all of them to get going. `Que.wake_all!` can get them all moving immediately.
 

data/docs/migrating.md

@@ -2,25 +2,29 @@
 
 Some new releases of Que may require updates to the database schema. It's recommended that you integrate these updates alongside your other database migrations. For example, when Que released version 0.6.0, the schema version was updated from 2 to 3. If you're running ActiveRecord, you could make a migration to perform this upgrade like so:
 
-    class UpdateQue < ActiveRecord::Migration
-      def self.up
-        Que.migrate! :version => 3
-      end
+```ruby
+class UpdateQue < ActiveRecord::Migration
+  def self.up
+    Que.migrate! :version => 3
+  end
 
-      def self.down
-        Que.migrate! :version => 2
-      end
-    end
+  def self.down
+    Que.migrate! :version => 2
+  end
+end
+```
 
 This will make sure that your database schema stays consistent with your codebase. If you're looking for something quicker and dirtier, you can always manually migrate in a console session:
 
-    # Change schema to version 3.
-    Que.migrate! :version => 3
+```ruby
+# Change schema to version 3.
+Que.migrate! :version => 3
 
-    # Update to whatever the latest schema version is.
-    Que.migrate!
+# Update to whatever the latest schema version is.
+Que.migrate!
 
-    # Check your current schema version.
-    Que.db_version #=> 3
+# Check your current schema version.
+Que.db_version #=> 3
+```
 
 Note that you can remove Que from your database completely by migrating to version 0.

data/docs/multiple_queues.md

@@ -2,20 +2,26 @@
 
 Que supports the use of multiple queues in a single job table. This feature is intended to support the case where multiple applications (with distinct codebases) are sharing the same database. For instance, you might have a separate Ruby application that handles only processing credit cards. In that case, you can run that application's workers against a specific queue:
 
-    QUE_QUEUE=credit_cards rake que:work
+```shell
+QUE_QUEUE=credit_cards rake que:work
+```
 
 Then you can set jobs to be enqueued in that queue specifically:
 
-    ProcessCreditCard.enqueue current_user.id, :queue => 'credit_cards'
+```ruby
+ProcessCreditCard.enqueue current_user.id, :queue => 'credit_cards'
 
-    # Or:
+# Or:
 
-    class ProcessCreditCard < Que::Job
-      # Set a default queue for this job class; this can be overridden by
-      # passing the :queue parameter to enqueue like above.
-      @queue = 'credit_cards'
-    end
+class ProcessCreditCard < Que::Job
+  # Set a default queue for this job class; this can be overridden by
+  # passing the :queue parameter to enqueue like above.
+  @queue = 'credit_cards'
+end
+```
 
 In some cases, the ProcessCreditCard class may not be defined in the application that is enqueueing the job. In that case, you can specify the job class as a string:
 
-    Que.enqueue current_user.id, :job_class => 'ProcessCreditCard', :queue => 'credit_cards'
+```ruby
+Que.enqueue current_user.id, :job_class => 'ProcessCreditCard', :queue => 'credit_cards'
+```

data/docs/using_plain_connections.md

@@ -2,49 +2,55 @@
 
 If you're not using an ORM like ActiveRecord or Sequel, you can have Que access jobs using a plain Postgres connection:
 
-    require 'uri'
-    require 'pg'
+```ruby
+require 'uri'
+require 'pg'
 
-    uri = URI.parse(ENV['DATABASE_URL'])
+uri = URI.parse(ENV['DATABASE_URL'])
 
-    Que.connection = PG::Connection.open :host     => uri.host,
-                                         :user     => uri.user,
-                                         :password => uri.password,
-                                         :port     => uri.port || 5432,
-                                         :dbname   => uri.path[1..-1]
+Que.connection = PG::Connection.open :host     => uri.host,
+                                     :user     => uri.user,
+                                     :password => uri.password,
+                                     :port     => uri.port || 5432,
+                                     :dbname   => uri.path[1..-1]
+```
 
 If you want to be able to use multithreading to run multiple jobs simultaneously in the same process, though, you'll need the ConnectionPool gem (be sure to add `gem 'connection_pool'` to your Gemfile):
 
-    require 'uri'
-    require 'pg'
-    require 'connection_pool'
+```ruby
+require 'uri'
+require 'pg'
+require 'connection_pool'
 
-    uri = URI.parse(ENV['DATABASE_URL'])
+uri = URI.parse(ENV['DATABASE_URL'])
 
-    Que.connection = ConnectionPool.new :size => 10 do
-      PG::Connection.open :host     => uri.host,
-                          :user     => uri.user,
-                          :password => uri.password,
-                          :port     => uri.port || 5432,
-                          :dbname   => uri.path[1..-1]
-    end
+Que.connection = ConnectionPool.new :size => 10 do
+  PG::Connection.open :host     => uri.host,
+                      :user     => uri.user,
+                      :password => uri.password,
+                      :port     => uri.port || 5432,
+                      :dbname   => uri.path[1..-1]
+end
+```
 
 Be sure to pick your pool size carefully - if you use 10 for the size, you'll incur the overhead of having 10 connections open to Postgres even if you never use more than a couple of them.
 
 The Pond gem doesn't have this drawback - it is very similar to ConnectionPool, but establishes connections lazily (add `gem 'pond'` to your Gemfile):
 
-    require 'uri'
-    require 'pg'
-    require 'pond'
-
-    uri = URI.parse(ENV['DATABASE_URL'])
-
-    Que.connection = Pond.new :maximum_size => 10 do
-      PG::Connection.open :host     => uri.host,
-                          :user     => uri.user,
-                          :password => uri.password,
-                          :port     => uri.port || 5432,
-                          :dbname   => uri.path[1..-1]
-    end
+```ruby
+require 'uri'
+require 'pg'
+require 'pond'
+
+uri = URI.parse(ENV['DATABASE_URL'])
+
+Que.connection = Pond.new :maximum_size => 10 do
+  PG::Connection.open :host     => uri.host,
+                      :user     => uri.user,
+                      :password => uri.password,
+                      :port     => uri.port || 5432,
+                      :dbname   => uri.path[1..-1]
+end
+```
 
 Please be aware that if you're using ActiveRecord or Sequel to manage your data, there's no reason for you to be using any of these methods - it's less efficient (unnecessary connections will waste memory on your database server) and you lose the reliability benefits of wrapping jobs in the same transactions as the rest of your data.

data/docs/using_sequel.md

@@ -2,26 +2,30 @@
 
 If you're using Sequel, with or without Rails, you'll need to give Que a specific database instance to use:
 
-    DB = Sequel.connect(ENV['DATABASE_URL'])
-    Que.connection = DB
+```ruby
+DB = Sequel.connect(ENV['DATABASE_URL'])
+Que.connection = DB
+```
 
 Then you can safely use the same database object to transactionally protect your jobs:
 
-    class MyJob < Que::Job
-      def run
-        # Do stuff.
+```ruby
+class MyJob < Que::Job
+  def run
+    # Do stuff.
 
-        DB.transaction do
-          # Make changes to the database.
+    DB.transaction do
+      # Make changes to the database.
 
-          # Destroying this job will be protected by the same transaction.
-          destroy
-        end
-      end
+      # Destroying this job will be protected by the same transaction.
+      destroy
     end
+  end
+end
 
-    # In your controller action:
-    DB.transaction do
-      @user = User.create(params[:user])
-      MyJob.enqueue :user_id => @user.id
-    end
+# In your controller action:
+DB.transaction do
+  @user = User.create(params[:user])
+  MyJob.enqueue :user_id => @user.id
+end
+```