que 0.5.0 → 0.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 71cd78e2494e219cb26e3513ff4e41bbb335161c
-  data.tar.gz: 8d51de1cebe1eb810f1acf87e2a7b540a72aadfc
+  metadata.gz: f698e236aff233a785bc44075257a17c1ae35e3b
+  data.tar.gz: 0916c6cbb864a9c148bb43e9765cd569330abdf9
 SHA512:
-  metadata.gz: 1a1d708935bfd81d8cf3c52732a5448a804dc7650741a0313f9879998f6eafd21fa5d2a32f7d9b9297041b4f783895c7f6305c0e742836dda894650fe8b01d7c
-  data.tar.gz: bac784c3fb91f11386c0b78cde93cd61754b0e66f03d00a646c3c3a9152f0509269595f55ffc7441aa62b7bc9926e39e672644d6f6dd2f39c88fd1bb7da17453
+  metadata.gz: 4bad7602663617ced6acf5e76cd041c8f6ec7ed0339b7706ca2121e305cb61bb4a55b2bdc1752c7f5f14ad2147e6e65df7e6d1a0192ca5bb25d12dd02f123b90
+  data.tar.gz: e7719dc683f696209c4f5dcdb07495693bb891d2a6dbace5ec7be9954037234024b0d246c8394a62f75272dc03acdcbbfea43f517acd9669467a352672a410bb

data/.gitignore

@@ -3,7 +3,7 @@
 .bundle
 .config
 .yardoc
-Gemfile.lock
+Gemfile*.lock
 InstalledFiles
 _yardoc
 coverage

data/.travis.yml

@@ -3,7 +3,7 @@ rvm:
   - "1.9.3"
   - "2.0"
   - "2.1"
-  - "rbx-2.1.1"
+  - "rbx-2"
   - "jruby-1.7.5"
 before_script:
   - psql -c 'create database "que-test"' -U postgres

data/CHANGELOG.md

@@ -1,6 +1,26 @@
+### 0.6.0 (2014-02-04)
+
+*   **A schema upgrade to version 3 is required for this release.** See [the migration doc](https://github.com/chanks/que/blob/master/docs/migrating.md) for information if you're upgrading from a previous release.
+
+*   You can now run a job's logic directly (without enqueueing it) like `MyJob.run(arg1, arg2, :other_arg => arg3)`. This is useful when a job class encapsulates logic that you want to invoke without involving the entire queue.
+
+*   You can now check the current version of Que's database schema with `Que.db_version`.
+
+*   The method for enqueuing a job has been renamed from `MyJob.queue` to `MyJob.enqueue`, since we were beginning to use the word 'queue' in a LOT of places. `MyJob.queue` still works, but it may be removed at some point.
+
+*   The variables for setting the defaults for a given job class have been changed from `@default_priority` to `@priority` and `@default_run_at` to `@run_at`. The old variables still work, but like `Job.queue`, they may be removed at some point.
+
+*   Log lines now include the machine's hostname, since a pid alone may not uniquely identify a process.
+
+*   Multiple queues are now supported. See [the docs](https://github.com/chanks/que/blob/master/docs/multiple_queues.md) for details. (chanks, joevandyk)
+
+*   Rubinius 2.2 is now supported. (brixen)
+
+*   Job classes may now define their own logic for determining the retry interval when a job raises an error. See [error handling](https://github.com/chanks/que/blob/master/docs/error_handling.md) for more information.
+
 ### 0.5.0 (2014-01-14)
 
-*   When running a worker pool inside your web process on ActiveRecord, Que will now wake a worker once a transaction containing a queued job is committed.
+*   When running a worker pool inside your web process on ActiveRecord, Que will now wake a worker once a transaction containing a queued job is committed. (joevandyk, chanks)
 
 *   The `que:work` rake task now has a default wake_interval of 0.1 seconds, since it relies exclusively on polling to pick up jobs. You can set a QUE_WAKE_INTERVAL environment variable to change this. The environment variable to set a size for the worker pool in the rake task has also been changed from WORKER_COUNT to QUE_WORKER_COUNT.
 

data/Gemfile

@@ -15,4 +15,9 @@ group :test do
   gem 'pry'
 end
 
+platforms :rbx do
+  gem 'rubysl', '~> 2.0'
+  gem 'json', '~> 1.8'
+end
+
 gemspec

data/README.md

@@ -13,6 +13,7 @@ Additionally, there are the general benefits of storing jobs in Postgres, alongs
 * **Transactional Control** - Queue a job along with other changes to your database, and it'll commit or rollback with everything else. If you're using ActiveRecord or Sequel, Que can piggyback on their connections, so setup is simple and jobs are protected by the transactions you're already using.
 * **Atomic Backups** - Your jobs and data can be backed up together and restored as a snapshot. If your jobs relate to your data (and they usually do), there's no risk of jobs falling through the cracks during a recovery.
 * **Fewer Dependencies** - If you're already using Postgres (and you probably should be), a separate queue is another moving part that can break.
+* **Security** - Postgres' support for SSL connections keeps your data safe in transport, for added protection when you're running workers on cloud platforms that you can't completely control.
 
 Que's primary goal is reliability. You should be able to leave your application running indefinitely without worrying about jobs being lost due to a lack of transactional support, or left in limbo due to a crashing process. Que does everything it can to ensure that jobs you queue are performed exactly once (though the occasional repetition of a job can be impossible to avoid - see the docs on [how to write a reliable job](https://github.com/chanks/que/blob/master/docs/writing_reliable_jobs.md)).
 
@@ -52,9 +53,9 @@ Create a class for each type of job you want to run:
     # app/jobs/charge_credit_card.rb
     class ChargeCreditCard < Que::Job
       # Default settings for this job. These are optional - without them, jobs
-      # will default to priority 1 and run immediately.
-      @default_priority = 10
-      @default_run_at = proc { 1.minute.from_now }
+      # will default to priority 100 and run immediately.
+      @priority = 10
+      @run_at = proc { 1.minute.from_now }
 
       def run(user_id, options)
         # Do stuff.
@@ -80,19 +81,19 @@ Queue your job. Again, it's best to do this in a transaction with other changes
     ActiveRecord::Base.transaction do
       # Persist credit card information
       card = CreditCard.create(params[:credit_card])
-      ChargeCreditCard.queue(current_user.id, :credit_card_id => card.id)
+      ChargeCreditCard.enqueue(current_user.id, :credit_card_id => card.id)
     end
 
 You can also add options to run the job after a specific time, or with a specific priority:
 
     # The default priority is 100, and a lower number means a higher priority. 5 would be very important.
-    ChargeCreditCard.queue current_user.id, :credit_card_id => card.id, :run_at => 1.day.from_now, :priority => 5
+    ChargeCreditCard.enqueue current_user.id, :credit_card_id => card.id, :run_at => 1.day.from_now, :priority => 5
 
 To determine what happens when a job is queued, you can set Que's mode in your application configuration. There are a few options for the mode:
 
 * `config.que.mode = :off` - In this mode, queueing a job will simply insert it into the database - the current process will make no effort to run it. You should use this if you want to use a dedicated process to work tasks (there's a rake task to do this, see below). This is the default when running `rails console`.
 * `config.que.mode = :async` - In this mode, a pool of background workers is spun up, each running in their own thread. This is the default when running `rails server`. See the docs for [more information on managing workers](https://github.com/chanks/que/blob/master/docs/managing_workers.md).
-* `config.que.mode = :sync` - In this mode, any jobs you queue will be run in the same thread, synchronously (that is, `MyJob.queue` runs the job and won't return until it's completed). This makes your application's behavior easier to test, so it's the default in the test environment.
+* `config.que.mode = :sync` - In this mode, any jobs you queue will be run in the same thread, synchronously (that is, `MyJob.enqueue` runs the job and won't return until it's completed). This makes your application's behavior easier to test, so it's the default in the test environment.
 
 ## Contributing
 

data/docs/advanced_setup.md

@@ -13,11 +13,21 @@ Then you can queue jobs just as you would in Rails:
 
     ActiveRecord::Base.transaction do
       @user = User.create(params[:user])
-      SendRegistrationEmail.queue :user_id => @user.id
+      SendRegistrationEmail.enqueue :user_id => @user.id
     end
 
 There are other docs to read if you're using [Sequel](https://github.com/chanks/que/blob/master/docs/using_sequel.md) or [plain Postgres connections](https://github.com/chanks/que/blob/master/docs/using_plain_connections.md) (with no ORM at all) instead of ActiveRecord.
 
+### Forking Servers
+
+If you want to run a worker pool in your web process and you're using a forking webserver like Unicorn or Puma in some configurations, you'll want to set `Que.mode = :off` in your application configuration and only start up the worker pool in the child processes. So, for Puma:
+
+    # config/puma.rb
+    on_worker_boot do
+      # Reestablish your database connection, etc...
+      Que.mode = :async
+    end
+
 ### Managing the Jobs Table
 
 After you've connected Que to the database, you can manage the jobs table:
@@ -27,13 +37,13 @@ After you've connected Que to the database, you can manage the jobs table:
 
 You'll want to migrate to a specific version if you're using migration files, to ensure that they work the same way even when you upgrade Que in the future:
 
-    # Update the schema to version #2.
-    Que.migrate! :version => 2
+    # Update the schema to version #3.
+    Que.migrate! :version => 3
 
     # To reverse the migration, drop the jobs table entirely:
     Que.migrate! :version => 0
 
-There's also a helper method to clear the jobs table:
+There's also a helper method to clear all jobs from the jobs table:
 
     Que.clear!
 

data/docs/customizing_que.md

@@ -13,7 +13,7 @@ Que's support for scheduling jobs makes it easy to implement reliable recurring
 
         ActiveRecord::Base.transaction do
           destroy
-          self.class.queue :run_at => @attrs[:run_at] + 1.hour
+          self.class.enqueue :run_at => @attrs[:run_at] + 1.hour
         end
       end
     end
@@ -48,7 +48,7 @@ That said, if you want to queue jobs in the DelayedJob style, that can be done r
       end
 
       def method_missing(method, *args)
-        Delayed.queue Marshal.dump(@receiver), method, Marshal.dump(args)
+        Delayed.enqueue Marshal.dump(@receiver), method, Marshal.dump(args)
       end
     end
 
@@ -77,7 +77,7 @@ You can mimic this style with Que by using a simple job class:
 
     # Then:
 
-    Command.queue "Kernel.puts", "hello world"
+    Command.enqueue "Kernel.puts", "hello world"
 
 ### Retaining Finished Jobs
 
@@ -88,7 +88,7 @@ Que deletes jobs from the queue as they are worked, in order to keep the `que_jo
 
     class MyJobClass < Que::Job
       def destroy
-        Que.execute "INSERT INTO finished_jobs SELECT * FROM que_jobs WHERE priority = $1::integer AND run_at = $2::timestamptz AND job_id = $3::bigint", @attrs.values_at(:priority, :run_at, :job_id)
+        Que.execute "INSERT INTO finished_jobs SELECT * FROM que_jobs WHERE queue = $1::text AND priority = $2::integer AND run_at = $3::timestamptz AND job_id = $4::bigint", @attrs.values_at(:queue, :priority, :run_at, :job_id)
         super
       end
     end

data/docs/error_handling.md

@@ -2,7 +2,19 @@
 
 If an error is raised and left uncaught by your job, Que will save the error message and backtrace to the database and schedule the job to be retried later.
 
-If a given job fails repeatedly, Que will retry it at exponentially-increasing intervals equal to (failure_count^4 + 3) seconds. This means that a job will be retried 4 seconds after its first failure, 19 seconds after its second, 84 seconds after its third, 259 seconds after its fourth, and so on until it succeeds. This pattern is very similar to DelayedJob's.
+If a given job fails repeatedly, Que will retry it at exponentially-increasing intervals equal to (failure_count^4 + 3) seconds. This means that a job will be retried 4 seconds after its first failure, 19 seconds after its second, 84 seconds after its third, 259 seconds after its fourth, and so on until it succeeds. This pattern is very similar to DelayedJob's. Alternately, you can define your own retry logic by setting an interval to delay each time, or a callable that accepts the number of failures and returns an interval:
+
+    class MyJob < Que::Job
+      # Just retry a failed job every 5 seconds:
+      @retry_interval = 5
+
+      # Always retry this job immediately (not recommended, or transient
+      # errors will spam your error reporting):
+      @retry_interval = 0
+
+      # Increase the delay by 30 seconds every time this job fails:
+      @retry_interval = proc { |count| count * 30 }
+    end
 
 Unlike DelayedJob, however, there is currently no maximum number of failures after which jobs will be deleted. Que's assumption is that if a job is erroring perpetually (and not just transiently), you will want to take action to get the job working properly rather than simply losing it silently.
 

data/docs/managing_workers.md

@@ -23,7 +23,7 @@ If you don't want to burden your web processes with too much work and want to ru
     rake que:work
 
     # Or configure the number of workers:
-    WORKER_COUNT=8 rake que:work
+    QUE_WORKER_COUNT=8 rake que:work
 
 ### Thread-Unsafe Application Code
 
@@ -36,7 +36,7 @@ If your application code is not thread-safe, you won't want any workers to be pr
 
 This will prevent Que from trying to process jobs in the background of your web processes. In order to actually work jobs, you'll want to run a single worker at a time, and to do so via a separate rake task, like so:
 
-    WORKER_COUNT=1 rake que:work
+    QUE_WORKER_COUNT=1 rake que:work
 
 ### The Wake Interval
 

data/docs/migrating.md

@@ -0,0 +1,26 @@
+## Migrating
+
+Some new releases of Que may require updates to the database schema. It's recommended that you integrate these updates alongside your other database migrations. For example, when Que released version 0.6.0, the schema version was updated from 2 to 3. If you're running ActiveRecord, you could make a migration to perform this upgrade like so:
+
+    class UpdateQue < ActiveRecord::Migration
+      def self.up
+        Que.migrate! :version => 3
+      end
+
+      def self.down
+        Que.migrate! :version => 2
+      end
+    end
+
+This will make sure that your database schema stays consistent with your codebase. If you're looking for something quicker and dirtier, you can always manually migrate in a console session:
+
+    # Change schema to version 3.
+    Que.migrate! :version => 3
+
+    # Update to whatever the latest schema version is.
+    Que.migrate!
+
+    # Check your current schema version.
+    Que.db_version #=> 3
+
+Note that you can remove Que from your database completely by migrating to version 0.

data/docs/multiple_queues.md

@@ -0,0 +1,13 @@
+## Multiple Queues
+
+Que supports the use of multiple queues in a single job table. This feature is intended to support the case where multiple applications (with distinct codebases) are sharing the same database. For instance, you might have a separate Ruby application that handles only processing credit cards. In that case, you can run that application's workers against a specific queue:
+
+    QUE_QUEUE=credit_cards rake que:work
+
+Then you can set jobs to be enqueued in that queue specifically:
+
+    ProcessCreditCard.enqueue current_user.id, :queue => 'credit_cards'
+
+In some cases, the ProcessCreditCard class may not be defined in the application that is enqueueing the job. In that case, you can specify the job class as a string:
+
+    Que.enqueue current_user.id, :job_class => 'ProcessCreditCard', :queue => 'credit_cards'

data/docs/shutting_down_safely.md

@@ -0,0 +1,7 @@
+## Shutting Down Safely
+
+To ensure safe operation, Que needs to be very careful in how it shuts down. When a Ruby process ends normally, it calls Thread#kill on any threads that are still running - unfortunately, if a thread is in the middle of a transaction when this happens, there is a risk that it will be prematurely commited, resulting in data corruption. See [here](http://blog.headius.com/2008/02/ruby-threadraise-threadkill-timeoutrb.html) and [here](http://coderrr.wordpress.com/2011/05/03/beware-of-threadkill-or-your-activerecord-transactions-are-in-danger-of-being-partially-committed/) for more detail on this.
+
+To prevent this, Que will block a Ruby process from exiting until all jobs it is working have completed normally. Unfortunately, if you have long-running jobs, this may take a very long time (and if something goes wrong with a job's logic, it may never happen). The solution in this case is SIGKILL - luckily, Ruby processes that are killed via SIGKILL will end without using Thread#kill on its running threads. This is safer than exiting normally - when PostgreSQL loses the connection it will simply roll back the open transaction, if any, and unlock the job so it can be retried later by another worker. Be sure to read [Writing Reliable Jobs](https://github.com/chanks/que/blob/master/docs/writing_reliable_jobs.md) for information on how to design your jobs to fail safely.
+
+So, be prepared to use SIGKILL on your Ruby processes if they run for too long. For example, Heroku takes a good approach to this - when Heroku's platform is shutting down a process, it sends SIGTERM, waits ten seconds, then sends SIGKILL if the process still hasn't exited. This is a nice compromise - it will give each of your currently running jobs ten seconds to complete, and any jobs that haven't finished by then will be interrupted and retried later.

data/docs/writing_reliable_jobs.md

@@ -60,3 +60,46 @@ Finally, there are some jobs where you won't want to write to the database at al
     end
 
 In this case, we don't have any no way to prevent the occasional double-sending of an email. But, for ease of use, you can leave out the transaction and the `destroy` call entirely - Que will recognize that the job wasn't destroyed and will clean it up for you.
+
+### Timeouts
+
+Long-running jobs aren't necessarily a problem in Que, since the overhead of an individual job isn't that big (just an open PG connection and an advisory lock held in memory). But jobs that hang indefinitely can tie up a worker and [block the Ruby process from exiting gracefully](https://github.com/chanks/que/blob/master/docs/shutting_down_safely.md), which is a pain.
+
+Que doesn't offer a general way to kill jobs that have been running too long, because that currently can't be done safely in Ruby. Typically, one would use Ruby's Timeout module for this sort of thing, but wrapping a database transaction inside a timeout introduces a risk of premature commits, which can corrupt your data. See [here](http://blog.headius.com/2008/02/ruby-threadraise-threadkill-timeoutrb.html) and [here](http://coderrr.wordpress.com/2011/05/03/beware-of-threadkill-or-your-activerecord-transactions-are-in-danger-of-being-partially-committed/) for detail on why this is.
+
+However, if there's part of your job that is prone to hang (due to an API call or other HTTP request that never returns, for example), you can timeout those individual parts of your job relatively safely. For example, consider a job that needs to make an HTTP request and then write to the database:
+
+    require 'net/http'
+
+    class ScrapeStuff < Que::Job
+      def run(domain_to_scrape, path_to_scrape)
+        result = Net::HTTP.get(domain_to_scrape, path_to_scrape)
+
+        ActiveRecord::Base.transaction do
+          # Insert result...
+
+          destroy
+        end
+      end
+    end
+
+That request could take a very long time, or never return at all. Let's wrap it in a five-second timeout:
+
+    require 'net/http'
+    require 'timeout'
+
+    class ScrapeStuff < Que::Job
+      def run(domain_to_scrape, path_to_scrape)
+        result = Timeout.timeout(5){Net::HTTP.get(domain_to_scrape, path_to_scrape)}
+
+        ActiveRecord::Base.transaction do
+          # Insert result...
+
+          destroy
+        end
+      end
+    end
+
+Now, if the request takes more than five seconds, a `Timeout::Error` will be raised and Que will just retry the job later. This solution isn't perfect, since Timeout uses Thread#kill under the hood, which can lead to unpredictable behavior. But it's separate from our transaction, so there's no risk of losing data - even a catastrophic error that left Net::HTTP in a bad state would be fixable by restarting the process.
+
+Finally, remember that if you're using a library that offers its own timeout functionality, that's usually preferable to using the Timeout module.

data/lib/generators/que/templates/add_que.rb

@@ -1,7 +1,7 @@
 class AddQue < ActiveRecord::Migration
   def self.up
     # The current version as of this migration's creation.
-    Que.migrate! :version => 2
+    Que.migrate! :version => 3
   end
 
   def self.down

data/lib/que.rb

@@ -1,4 +1,4 @@
-require 'time' # For Time#iso8601
+require 'socket' # For hostname
 
 module Que
   autoload :Adapters,   'que/adapters/base'
@@ -20,10 +20,6 @@ module Que
     attr_accessor :logger, :error_handler
     attr_writer :adapter, :log_formatter
 
-    def adapter
-      @adapter || raise("Que connection not established!")
-    end
-
     def connection=(connection)
       self.adapter = if connection.to_s == 'ActiveRecord'
         Adapters::ActiveRecord.new
@@ -38,18 +34,12 @@ module Que
       end
     end
 
-    # Have to support create! and drop! in old migrations. They just created
-    # and dropped the bare table.
-    def create!
-      migrate! :version => 1
-    end
-
-    def drop!
-      migrate! :version => 0
+    def adapter
+      @adapter || raise("Que connection not established!")
     end
 
-    def migrate!(version = {:version => Migrations::CURRENT_VERSION})
-      Migrations.migrate!(version)
+    def execute(*args)
+      adapter.execute(*args)
     end
 
     def clear!
@@ -64,16 +54,32 @@ module Que
       execute :worker_states
     end
 
-    def execute(command, *args)
-      indifferentiate case command
-                        when Symbol then adapter.execute_prepared(command, *args)
-                        when String then adapter.execute(command, *args)
-                      end.to_a
+    # Give us a cleaner interface when specifying a job_class as a string.
+    def enqueue(*args)
+      Job.enqueue(*args)
+    end
+
+    def db_version
+      Migrations.db_version
+    end
+
+    def migrate!(version = {:version => Migrations::CURRENT_VERSION})
+      Migrations.migrate!(version)
+    end
+
+    # Have to support create! and drop! in old migrations. They just created
+    # and dropped the bare table.
+    def create!
+      migrate! :version => 1
+    end
+
+    def drop!
+      migrate! :version => 0
     end
 
     def log(data)
       level = data.delete(:level) || :info
-      data = {:lib => 'que', :thread => Thread.current.object_id}.merge(data)
+      data = {:lib => 'que', :hostname => Socket.gethostname, :thread => Thread.current.object_id}.merge(data)
 
       if logger && output = log_formatter.call(data)
         logger.send level, output
@@ -84,26 +90,6 @@ module Que
       @log_formatter ||= JSON_MODULE.method(:dump)
     end
 
-    # Helper for making hashes indifferently-accessible, even when nested
-    # within each other and within arrays.
-    def indifferentiate(object)
-      case object
-      when Hash
-        h = if {}.respond_to?(:with_indifferent_access) # Better support for Rails.
-              {}.with_indifferent_access
-            else
-              Hash.new { |hash, key| hash[key.to_s] if Symbol === key }
-            end
-
-        object.each { |k, v| h[k] = indifferentiate(v) }
-        h
-      when Array
-        object.map { |v| indifferentiate(v) }
-      else
-        object
-      end
-    end
-
     # Copy some of the Worker class' config methods here for convenience.
     [:mode, :mode=, :worker_count, :worker_count=, :wake_interval, :wake_interval=, :wake!, :wake_all!].each do |meth|
       define_method(meth) { |*args| Worker.send(meth, *args) }