que 0.14.3 → 1.0.0.beta

data/.rspec

@@ -1,2 +0,0 @@
---color
---order random

data/.travis.yml

@@ -1,64 +0,0 @@
-language: ruby
-cache: bundler
-
-rvm:
-  - 2.2
-  - 2.3
-  - 2.4
-  - 2.5
-  - ruby-head
-  # The Rubinii aren't installing properly on Travis :/
-  # - rbx-2
-  # - rbx
-
-matrix:
-  allow_failures:
-    # Ruby head failures aren't disastrous, because we won't support it until
-    # it's released, but it's good to be aware of.
-    - rvm: ruby-head
-
-gemfile:
-  - spec/gemfiles/Gemfile.current
-  - spec/gemfiles/Gemfile.old
-  - spec/gemfiles/Gemfile.older
-  - spec/gemfiles/Gemfile.oldest
-
-env:
-  - PG_VERSION=9.3
-  - PG_VERSION=9.4
-  - PG_VERSION=9.5
-  - PG_VERSION=9.6
-
-before_install:
-  # Stop all running Postgreses, so we don't get port conflicts when installing
-  # a new version.
-  - sudo /etc/init.d/postgresql stop
-  # Install whatever version we're using.
-  - sudo apt-get install postgresql-$PG_VERSION
-  # If we just installed Postgres 9.6 it won't have a proper pg_hba set up, so...
-  - sudo mkdir -p /etc/postgresql/9.6/main
-  - sudo cp -v /etc/postgresql/9.{5,6}/main/pg_hba.conf
-  # Hook up the Postgres version we care about to the right port.
-  - sudo sed -i "s/port = ..../port = 5432/g" /etc/postgresql/$PG_VERSION/main/postgresql.conf
-  # If we just installed a new Postgres version it'll be running, so make sure
-  # they're all stopped, again.
-  - sudo /etc/init.d/postgresql stop
-  # Start up the one we care about.
-  - sudo /etc/init.d/postgresql start $PG_VERSION
-  # A newly-installed Postgres won't have a travis user, so create one. But, if
-  # one already exists this will fail, so drop it first. Kinda stupid.
-  - sudo -u postgres dropuser --if-exists -p 5432 travis &>/dev/null
-  - sudo -u postgres createuser -p 5432 travis &>/dev/null
-
-before_script:
-  - psql -c 'create database "que-test"' -U postgres
-
-script:
-  # Run the complete test suite:
-  - bundle exec rspec -fd -b -P ./spec/**/*_spec.rb
-  # Run the test suite without adapters/ActiveRecord, to make sure the codebase
-  # doesn't accidentally rely on any ActiveSupport-isms.
-  - bundle exec rspec -fd -b -P ./spec/unit/*_spec.rb
-
-notifications:
-  email: false

data/Gemfile

@@ -1,24 +0,0 @@
-source 'https://rubygems.org'
-
-group :development, :test do
-  gem 'rake', '< 11.0'
-
-  gem 'activerecord',    :require => nil
-  gem 'sequel',          :require => nil
-  gem 'connection_pool', :require => nil
-  gem 'pond',            :require => nil
-  gem 'pg',              :require => nil, :platform => :ruby
-  gem 'pg_jruby',        :require => nil, :platform => :jruby
-end
-
-group :test do
-  gem 'rspec', '~> 2.14.1'
-  gem 'pry'
-end
-
-platforms :rbx do
-  gem 'rubysl', '~> 2.0'
-  gem 'json', '~> 1.8'
-end
-
-gemspec

data/docs/customizing_que.md

@@ -1,200 +0,0 @@
-## Customizing Que
-
-One of Que's goals to be easily extensible and hackable (and if anyone has any suggestions on ways to accomplish that, please [open an issue](https://github.com/chanks/que/issues)). This document is meant to demonstrate some of the ways Que can be used to accomplish different tasks that it's not already designed for.
-
-Some of these features may be moved into core Que at some point, depending on how commonly useful they are.
-
-### Recurring Jobs
-
-Que's support for scheduling jobs makes it easy to implement reliable recurring jobs. For example, suppose you want to run a job every hour that processes the users created in that time:
-
-```ruby
-class CronJob < Que::Job
-  # Default repetition interval in seconds. Can be overridden in
-  # subclasses. Can use 1.minute if using Rails.
-  INTERVAL = 60
-
-  attr_reader :start_at, :end_at, :run_again_at, :time_range
-
-  def _run
-    args = attrs[:args].first
-    @start_at, @end_at = Time.at(args.delete('start_at')), Time.at(args.delete('end_at'))
-    @run_again_at = @end_at + self.class::INTERVAL
-    @time_range = @start_at...@end_at
-
-    super
-
-    args['start_at'] = @end_at.to_f
-    args['end_at']   = @run_again_at.to_f
-    self.class.enqueue(args, run_at: @run_again_at)
-  end
-end
-
-class MyCronJob < CronJob
-  INTERVAL = 3600
-
-  def run(args)
-    User.where(created_at: time_range).each { ... }
-  end
-end
-
-# To enqueue:
-tf = Time.now
-t0 = Time.now - 3600
-MyCronJob.enqueue :start_at => t0.to_f, :end_at => tf.to_f
-```
-
-Note that instead of using Time.now in our database query, and requeueing the job at 1.hour.from_now, we use job arguments to track start and end times. This lets us correct for delays in running the job. Suppose that there's a backlog of priority jobs, or that the worker briefly goes down, and this job, which was supposed to run at 11:00 a.m. isn't run until 11:05 a.m. A lazier implementation would look for users created after 1.hour.ago, and miss those that signed up between 10:00 a.m. and 10:05 a.m.
-
-This also compensates for clock drift. `Time.now` on one of your application servers may not match `Time.now` on another application server may not match `now()` on your database server. The best way to stay reliable is have a single authoritative source on what the current time is, and your best source for authoritative information is always your database (this is why Que uses Postgres' `now()` function when locking jobs, by the way).
-
-Note also the use of the triple-dot range, which results in a query like `SELECT "users".* FROM "users" WHERE ("users"."created_at" >= '2014-01-08 10:00:00.000000' AND "users"."created_at" < '2014-01-08 11:00:00.000000')` instead of a BETWEEN condition. This ensures that a user created at 11:00 am exactly isn't processed twice, by the jobs starting at both 10 am and 11 am.
-
-Finally, by passing both the start and end times for the period to be processed, and only using the interval to calculate the period for the following job, we make it easy to change the interval at which the job runs, without the risk of missing or double-processing any users.
-
-### DelayedJob-style Jobs
-
-DelayedJob offers a simple API for delaying methods to objects:
-
-```ruby
-@user.delay.activate!(@device)
-```
-
-The API is pleasant, but implementing it requires storing marshalled Ruby objects in the database, which is both inefficient and prone to bugs - for example, if you deploy an update that changes the name of an instance variable (a contained, internal change that might seem completely innocuous), the marshalled objects in the database will retain the old instance variable name and will behave unexpectedly when unmarshalled into the new Ruby code.
-
-This is the danger of mixing the ephemeral state of a Ruby object in memory with the more permanent state of a database row. The advantage of Que's API is that, since your arguments are forced through a JSON serialization/deserialization process, it becomes your responsibility when designing a job class to establish an API for yourself (what the arguments to the job are and what they mean) that you will have to stick to in the future.
-
-That said, if you want to queue jobs in the DelayedJob style, that can be done relatively easily:
-
-```ruby
-class Delayed < Que::Job
-  def run(receiver, method, args)
-    Marshal.load(receiver).send method, *Marshal.load(args)
-  end
-end
-
-class DelayedAction
-  def initialize(receiver)
-    @receiver = receiver
-  end
-
-  def method_missing(method, *args)
-    Delayed.enqueue Marshal.dump(@receiver), method, Marshal.dump(args)
-  end
-end
-
-class Object
-  def delay
-    DelayedAction.new(self)
-  end
-end
-```
-
-You can replace Marshal with YAML if you like.
-
-### QueueClassic-style Jobs
-
-You may find it a hassle to keep an individual class file for each type of job. QueueClassic has a simpler design, wherein you simply give it a class method to call, like:
-
-```ruby
-QC.enqueue("Kernel.puts", "hello world")
-```
-
-You can mimic this style with Que by using a simple job class:
-
-```ruby
-class Command < Que::Job
-  def run(method, *args)
-    receiver, message = method.split('.')
-    Object.const_get(receiver).send(message, *args)
-  end
-end
-
-# Then:
-
-Command.enqueue "Kernel.puts", "hello world"
-```
-
-### Retaining Finished Jobs
-
-Que deletes jobs from the queue as they are worked, in order to keep the `que_jobs` table and index small and efficient. If you have a need to hold onto finished jobs, the recommended way to do this is to add a second table to hold them, and then insert them there as they are deleted from the queue. You can use Ruby's inheritance mechanics to do this cleanly:
-
-```ruby
-Que.execute "CREATE TABLE finished_jobs AS SELECT * FROM que_jobs LIMIT 0"
-# Or, better, use a proper CREATE TABLE with not-null constraints, and add whatever indexes you like.
-
-class MyJobClass < Que::Job
-  def destroy
-    Que.execute "INSERT INTO finished_jobs SELECT * FROM que_jobs WHERE queue = $1::text AND priority = $2::integer AND run_at = $3::timestamptz AND job_id = $4::bigint", @attrs.values_at(:queue, :priority, :run_at, :job_id)
-    super
-  end
-end
-```
-
-Then just have your job classes inherit from MyJobClass instead of Que::Job. If you need to query the jobs table and you want to include both finished and unfinished jobs, you might use:
-
-```ruby
-Que.execute "CREATE VIEW all_jobs AS SELECT * FROM que_jobs UNION ALL SELECT * FROM finished_jobs"
-Que.execute "SELECT * FROM all_jobs"
-```
-
-Alternately, if you want a more foolproof solution and you're not scared of PostgreSQL, you can use a trigger:
-
-```sql
-CREATE FUNCTION please_save_my_job()
-RETURNS trigger
-LANGUAGE plpgsql
-AS $$
-  BEGIN
-    INSERT INTO finished_jobs SELECT (OLD).*;
-    RETURN OLD;
-  END;
-$$;
-
-CREATE TRIGGER keep_all_my_old_jobs BEFORE DELETE ON que_jobs FOR EACH ROW EXECUTE PROCEDURE please_save_my_job();
-```
-
-### Not Retrying Certain Failed Jobs
-
-By default, when jobs fail, Que reschedules them to be retried later. If instead you'd like certain jobs to not be retried, and instead move them elsewhere to be examined later, you can accomplish that easily. First, we need a place for the failed jobs to be stored:
-
-```sql
-CREATE TABLE failed_jobs AS SELECT * FROM que_jobs LIMIT 0
-```
-
-Then, create a module that you can use in the jobs you don't want to retry:
-
-```ruby
-module SkipRetries
-  def run(*args)
-    super
-  rescue
-    sql = <<-SQL
-      WITH failed AS (
-        DELETE
-        FROM   que_jobs
-        WHERE  queue    = $1::text
-        AND    priority = $2::smallint
-        AND    run_at   = $3::timestamptz
-        AND    job_id   = $4::bigint
-        RETURNING *
-      )
-      INSERT INTO failed_jobs
-        SELECT * FROM failed;
-    SQL
-
-    Que.execute sql, @attrs.values_at(:queue, :priority, :run_at, :job_id)
-
-    raise # Reraises caught error.
-  end
-end
-
-class RunOnceJob < Que::Job
-  prepend SkipRetries
-
-  def run(*args)
-    # Do something - if this job runs an error it'll be moved to the
-    # failed_jobs table and not retried.
-  end
-end
-```

data/lib/generators/que/install_generator.rb

@@ -1,24 +0,0 @@
-# frozen_string_literal: true
-
-require 'rails/generators'
-require 'rails/generators/migration'
-require 'active_record'
-
-module Que
-  class InstallGenerator < Rails::Generators::Base
-    include Rails::Generators::Migration
-
-    namespace 'que:install'
-    self.source_paths << File.join(File.dirname(__FILE__), 'templates')
-    desc "Generates a migration to add Que's job table."
-
-    def self.next_migration_number(dirname)
-      next_migration_number = current_migration_number(dirname) + 1
-      ActiveRecord::Migration.next_migration_number(next_migration_number)
-    end
-
-    def create_migration_file
-      migration_template 'add_que.rb', 'db/migrate/add_que.rb'
-    end
-  end
-end

data/lib/generators/que/templates/add_que.rb

@@ -1,13 +0,0 @@
-# frozen_string_literal: true
-
-class AddQue < ActiveRecord::Migration[4.2]
-  def self.up
-    # The current version as of this migration's creation.
-    Que.migrate! :version => 3
-  end
-
-  def self.down
-    # Completely removes Que's job queue.
-    Que.migrate! :version => 0
-  end
-end

data/lib/que/adapters/active_record.rb

@@ -1,40 +0,0 @@
-# frozen_string_literal: true
-
-module Que
-  module Adapters
-    class ActiveRecord < Base
-      def checkout
-        checkout_activerecord_adapter { |conn| yield conn.raw_connection }
-      end
-
-      def cleanup!
-        # ActiveRecord will check out connections to the current thread when
-        # queries are executed and not return them to the pool until
-        # explicitly requested to. The wisdom of this API is questionable, and
-        # it doesn't pose a problem for the typical case of workers using a
-        # single PG connection (since we ensure that connection is checked in
-        # and checked out responsibly), but since ActiveRecord supports
-        # connections to multiple databases, it's easy for people using that
-        # feature to unknowingly leak connections to other databases. So, take
-        # the additional step of telling ActiveRecord to check in all of the
-        # current thread's connections between jobs.
-        ::ActiveRecord::Base.clear_active_connections!
-      end
-
-      private
-
-      def checkout_activerecord_adapter(&block)
-        # Use Rails' executor (if present) to make sure that the connection
-        # we're using isn't taken from us while the block runs. See
-        # https://github.com/chanks/que/issues/166#issuecomment-274218910
-        if defined?(Rails.application.executor)
-          Rails.application.executor.wrap do
-            ::ActiveRecord::Base.connection_pool.with_connection(&block)
-          end
-        else
-          ::ActiveRecord::Base.connection_pool.with_connection(&block)
-        end
-      end
-    end
-  end
-end

data/lib/que/adapters/base.rb

@@ -1,133 +0,0 @@
-# frozen_string_literal: true
-
-require 'time' # For Time.parse.
-
-module Que
-  module Adapters
-    autoload :ActiveRecord,   'que/adapters/active_record'
-    autoload :ConnectionPool, 'que/adapters/connection_pool'
-    autoload :PG,             'que/adapters/pg'
-    autoload :Pond,           'que/adapters/pond'
-    autoload :Sequel,         'que/adapters/sequel'
-
-    class Base
-      def initialize(thing = nil)
-        @prepared_statements = {}
-      end
-
-      # The only method that adapters really need to implement. Should lock a
-      # PG::Connection (or something that acts like a PG::Connection) so that
-      # no other threads are using it and yield it to the block. Should also
-      # be re-entrant.
-      def checkout(&block)
-        raise NotImplementedError
-      end
-
-      # Called after Que has returned its connection to whatever pool it's
-      # using.
-      def cleanup!
-      end
-
-      # Called after a job is queued in async mode, to prompt a worker to
-      # wake up after the current transaction commits. Not all adapters will
-      # implement this.
-      def wake_worker_after_commit
-        false
-      end
-
-      def execute(command, params = [])
-        params = params.map do |param|
-          case param
-            # The pg gem unfortunately doesn't convert fractions of time instances, so cast them to a string.
-            when Time then param.strftime("%Y-%m-%d %H:%M:%S.%6N %z")
-            when Array, Hash then JSON.dump(param)
-            else param
-          end
-        end
-
-        cast_result \
-          case command
-            when Symbol then execute_prepared(command, params)
-            when String then execute_sql(command, params)
-          end
-      end
-
-      def in_transaction?
-        checkout { |conn| conn.transaction_status != ::PG::PQTRANS_IDLE }
-      end
-
-      private
-
-      def execute_sql(sql, params)
-        args = params.empty? ? [sql] : [sql, params]
-        checkout { |conn| conn.async_exec(*args) }
-      end
-
-      def execute_prepared(name, params)
-        checkout do |conn|
-          # Prepared statement errors have the potential to foul up the entire
-          # transaction, so if we're in one, err on the side of safety.
-          return execute_sql(SQL[name], params) if !Que.use_prepared_statements || in_transaction?
-
-          statements = @prepared_statements[conn] ||= {}
-
-          begin
-            unless statements[name]
-              conn.prepare("que_#{name}", SQL[name])
-              prepared_just_now = statements[name] = true
-            end
-
-            conn.exec_prepared("que_#{name}", params)
-          rescue ::PG::InvalidSqlStatementName => error
-            # Reconnections on ActiveRecord can cause the same connection
-            # objects to refer to new backends, so recover as well as we can.
-
-            unless prepared_just_now
-              Que.log :level => 'warn', :event => "reprepare_statement", :name => name
-              statements[name] = false
-              retry
-            end
-
-            raise error
-          end
-        end
-      end
-
-      CAST_PROCS = {}
-
-      # Integer, bigint, smallint:
-      CAST_PROCS[23] = CAST_PROCS[20] = CAST_PROCS[21] = proc(&:to_i)
-
-      # Timestamp with time zone.
-      CAST_PROCS[1184] = Time.method(:parse)
-
-      # JSON.
-      CAST_PROCS[114] = -> (value) { JSON.parse(value, create_additions: false) }
-
-      # Boolean:
-      CAST_PROCS[16] = -> (value) {
-        case value
-        when String then value == 't'
-        when TrueClass, FalseClass then value
-        else raise "Unexpected boolean value: #{value.inspect} (#{value.class})"
-        end
-      }
-
-      def cast_result(result)
-        output = result.to_a
-
-        result.fields.each_with_index do |field, index|
-          if converter = CAST_PROCS[result.ftype(index)]
-            output.each do |hash|
-              unless (value = hash[field]).nil?
-                hash[field] = converter.call(value)
-              end
-            end
-          end
-        end
-
-        output.map!(&Que.json_converter)
-      end
-    end
-  end
-end