que 0.8.2 → 0.9.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 8029277244c5b5f8e4d838c9e16b2ebddd402919
-  data.tar.gz: 06fe39d7feb085098468d4b7f019aa0a40433016
+  metadata.gz: 6e9631191c30bd35b86706f09611faebc0f99e07
+  data.tar.gz: b53b3005e278471f4e4ee7e55a41637db002a2be
 SHA512:
-  metadata.gz: 3564b06d93c6001367f7d728e6ba22600abe53609a6551bc73aac2dda6cd6ab55629e12b002adb9998981960358617f8b0fd045d5ee9367ef52e1fd371dfb654
-  data.tar.gz: 2f7731e0b01ffed874937273c86364f7a19bd6de67a573f8c82aedf1dcc5918e6c841ae1b78dbbd90c085f9f5112c5440551f72f4738081ec9c24a1b8f49aae7
+  metadata.gz: 7a8da4e60a2d3e9bd85be9891cf7d3c736621ed5e72d07d2f6145e2430ca7af15d678441c2d3bd9358aee1aab84fde5a73d14be0af1a87c2425a3ec51de652de
+  data.tar.gz: 1245796b0d471b8ab3ccc528f016cb244dad07fc80282fcae82e5bd1203281df01e40bd012e04cc1eade1a63a8424a3488043a81ddf68d02886ca6b8256c5067

data/CHANGELOG.md

@@ -1,3 +1,7 @@
+### 0.9.0 (2014-12-16)
+
+*   The error_handler callable is now be passed two objects, the error and the job that raised it. If your current error_handler is a proc, as recommended in the docs, you shouldn't need to make any code changes, unless you want to use the job in your error handling. If your error_handler is a lambda, or another callable with a strict arity requirement, you'll want to change it before upgrading. (#69) (statianzo)
+
 ### 0.8.2 (2014-10-12)
 
 *   Fix errors raised during rollbacks in the ActiveRecord adapter, which remained silent until Rails 4.2. (#64, #65) (Strech)

data/README.md

@@ -2,7 +2,7 @@
 
 **TL;DR: Que is a high-performance alternative to DelayedJob or QueueClassic that improves the reliability of your application by protecting your jobs with the same [ACID guarantees](https://en.wikipedia.org/wiki/ACID) as the rest of your data.**
 
-Que is a queue for Ruby and PostgreSQL that manages jobs using [advisory locks](http://www.postgresql.org/docs/current/static/explicit-locking.html#ADVISORY-LOCKS), which gives it several advantages over other RDBMS-backed queues:
+Que ("keɪ", or "kay") is a queue for Ruby and PostgreSQL that manages jobs using [advisory locks](http://www.postgresql.org/docs/current/static/explicit-locking.html#ADVISORY-LOCKS), which gives it several advantages over other RDBMS-backed queues:
 
   * **Concurrency** - Workers don't block each other when trying to lock jobs, as often occurs with "SELECT FOR UPDATE"-style locking. This allows for very high throughput with a large number of workers.
   * **Efficiency** - Locks are held in memory, so locking a job doesn't incur a disk write. These first two points are what limit performance with other queues - all workers trying to lock jobs have to wait behind one that's persisting its UPDATE on a locked_at column to disk (and the disks of however many other servers your database is synchronously replicating to). Under heavy load, Que's bottleneck is CPU, not I/O.
@@ -104,6 +104,13 @@ To determine what happens when a job is queued, you can set Que's mode in your a
 
 If you're using ActiveRecord to dump your database's schema, you'll probably want to [set schema_format to :sql](http://guides.rubyonrails.org/migrations.html#types-of-schema-dumps) so that Que's table structure is managed correctly.
 
+## Related Gems
+
+  * [que-web](https://github.com/statianzo/que-web) is a Sinatra-based UI for inspecting your job queue.
+  * [que-testing](https://github.com/statianzo/que-testing) allows making assertions on enqueued jobs.
+
+If you have a gem that uses or relates to Que, feel free to submit a PR adding it to the list!
+
 ## Contributing
 
   1. Fork it

data/docs/customizing_que.md

@@ -6,24 +6,48 @@ One of Que's goals to be easily extensible and hackable (and if anyone has any s
 
 Que's support for scheduling jobs makes it easy to implement reliable recurring jobs. For example, suppose you want to run a job every hour that processes the users created in that time:
 
-    class Cron < Que::Job
-      def run
-        users = User.where(:created_at => @attrs[:run_at]...(@attrs[:run_at] + 1.hour))
-        # Do something with users.
-
-        ActiveRecord::Base.transaction do
-          destroy
-          self.class.enqueue :run_at => @attrs[:run_at] + 1.hour
+      class CronJob < Que::Job
+        # Default repetition interval in seconds. Can be overridden in
+        # subclasses. Can use 1.minute if using Rails.
+        INTERVAL = 60
+
+        attr_reader :start_at, :end_at, :run_again_at, :time_range
+
+        def _run
+          args = attrs[:args].first
+          @start_at, @end_at = Time.at(args.delete('start_at')), Time.at(args.delete('end_at'))
+          @run_again_at = @end_at + self.class::INTERVAL
+          @time_range = @start_at...@end_at
+
+          super
+
+          args['start_at'] = @end_at.to_f
+          args['end_at']   = @run_again_at.to_f
+          self.class.enqueue(args, run_at: @run_again_at)
         end
       end
-    end
 
-Note that instead of using Time.now in our database query, and requeueing the job at 1.hour.from_now, we use the run_at of the current job as our timestamp. This corrects for delays in running the job. Suppose that there's a backlog of priority jobs, or that the worker briefly goes down, and this job, which was supposed to run at 11:00 a.m. isn't run until 11:05 a.m. A lazier implementation would look for users created after 1.hour.ago, and miss those that signed up between 10:00 a.m. and 10:05 a.m.
+      class MyCronJob < CronJob
+        INTERVAL = 3600
+
+        def run(args)
+          User.where(created_at: time_range).each { ... }
+        end
+      end
+
+      # To enqueue:
+      tf = Time.now
+      t0 = Time.now - 3600
+      MyCronJob.enqueue :start_at => t0.to_f, :end_at => tf.to_f
+
+Note that instead of using Time.now in our database query, and requeueing the job at 1.hour.from_now, we use job arguments to track start and end times. This lets us correct for delays in running the job. Suppose that there's a backlog of priority jobs, or that the worker briefly goes down, and this job, which was supposed to run at 11:00 a.m. isn't run until 11:05 a.m. A lazier implementation would look for users created after 1.hour.ago, and miss those that signed up between 10:00 a.m. and 10:05 a.m.
 
 This also compensates for clock drift. `Time.now` on one of your application servers may not match `Time.now` on another application server may not match `now()` on your database server. The best way to stay reliable is have a single authoritative source on what the current time is, and your best source for authoritative information is always your database (this is why Que uses Postgres' `now()` function when locking jobs, by the way).
 
 Note also the use of the triple-dot range, which results in a query like `SELECT "users".* FROM "users" WHERE ("users"."created_at" >= '2014-01-08 10:00:00.000000' AND "users"."created_at" < '2014-01-08 11:00:00.000000')` instead of a BETWEEN condition. This ensures that a user created at 11:00 am exactly isn't processed twice, by the jobs starting at both 10 am and 11 am.
 
+Finally, by passing both the start and end times for the period to be processed, and only using the interval to calculate the period for the following job, we make it easy to change the interval at which the job runs, without the risk of missing or double-processing any users.
+
 ### DelayedJob-style Jobs
 
 DelayedJob offers a simple API for delaying methods to objects:

data/lib/que/job.rb

@@ -118,7 +118,7 @@ module Que
 
             if Que.error_handler
               # Similarly, protect the work loop from a failure of the error handler.
-              Que.error_handler.call(error) rescue nil
+              Que.error_handler.call(error, job) rescue nil
             end
 
             return {:event => :job_errored, :error => error, :job => job}

data/lib/que/version.rb

@@ -1,3 +1,3 @@
 module Que
-  Version = '0.8.2'
+  Version = '0.9.0'
 end

data/spec/unit/customization_spec.rb

@@ -4,20 +4,60 @@ require 'spec_helper'
 # stay functional.
 describe "Customizing Que" do
   it "Cron should allow for easy recurring jobs" do
-    class Cron < Que::Job
-      def run
-        destroy
-        self.class.enqueue :run_at => @attrs[:run_at] + 3600
+    begin
+      class CronJob < Que::Job
+        # Default repetition interval in seconds. Can be overridden in
+        # subclasses. Can use 1.minute if using Rails.
+        INTERVAL = 60
+
+        attr_reader :start_at, :end_at, :run_again_at, :time_range
+
+        def _run
+          args = attrs[:args].first
+          @start_at, @end_at = Time.at(args.delete('start_at')), Time.at(args.delete('end_at'))
+          @run_again_at = @end_at + self.class::INTERVAL
+          @time_range = @start_at...@end_at
+
+          super
+
+          args['start_at'] = @end_at.to_f
+          args['end_at']   = @run_again_at.to_f
+          self.class.enqueue(args, run_at: @run_again_at)
+        end
       end
-    end
 
-    Cron.enqueue
+      class MyCronJob < CronJob
+        INTERVAL = 1.5
 
-    run_at = DB[:que_jobs].get(:run_at).to_f
+        def run(args)
+          $args       = args.dup
+          $time_range = time_range
+        end
+      end
 
-    Que::Job.work
+      t = (Time.now - 1000).to_f.round(6)
+      MyCronJob.enqueue :start_at => t, :end_at => t + 1.5, :arg => 4
 
-    DB[:que_jobs].get(:run_at).to_f.should be_within(0.000001).of(run_at + 3600)
+      $args.should be nil
+      $time_range.should be nil
+
+      Que::Job.work
+
+      $args.should == {'arg' => 4}
+      $time_range.begin.to_f.round(6).should be_within(0.000001).of t
+      $time_range.end.to_f.round(6).should be_within(0.000001).of t + 1.5
+      $time_range.exclude_end?.should be true
+
+      DB[:que_jobs].get(:run_at).to_f.round(6).should be_within(0.000001).of(t + 3.0)
+      args = JSON.parse(DB[:que_jobs].get(:args)).first
+      args.keys.should == ['arg', 'start_at', 'end_at']
+      args['arg'].should == 4
+      args['start_at'].should be_within(0.000001).of(t + 1.5)
+      args['end_at'].should be_within(0.000001).of(t + 3.0)
+    ensure
+      $args       = nil
+      $time_range = nil
+    end
   end
 
   it "Object#delay should allow for simpler job enqueueing" do

data/spec/unit/work_spec.rb

@@ -323,6 +323,22 @@ describe Que::Job, '.work' do
       end
     end
 
+    it "should pass job to an error handler, if one is defined" do
+      begin
+        jobs = []
+        Que.error_handler = proc { |error, job| jobs << job }
+
+        ErrorJob.enqueue
+        result = Que::Job.work
+
+        jobs.count.should be 1
+        job = jobs[0]
+        job.should be result[:job]
+      ensure
+        Que.error_handler = nil
+      end
+    end
+
     it "should not do anything if the error handler itelf throws an error" do
       begin
         Que.error_handler = proc { |error| raise "Another error!" }

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: que
 version: !ruby/object:Gem::Version
-  version: 0.8.2
+  version: 0.9.0
 platform: ruby
 authors:
 - Chris Hanks
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2014-10-12 00:00:00.000000000 Z
+date: 2014-12-16 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler