que 0.11.3 → 2.2.0

data/docs/using_plain_connections.md

@@ -1,41 +0,0 @@
-## Using Plain Postgres Connections
-
-If you're not using an ORM like ActiveRecord or Sequel, you can use one of two gems to manage a connection pool for your PG connections. The first is the ConnectionPool gem (be sure to add `gem 'connection_pool'` to your Gemfile):
-
-```ruby
-require 'uri'
-require 'pg'
-require 'connection_pool'
-
-uri = URI.parse(ENV['DATABASE_URL'])
-
-Que.connection = ConnectionPool.new :size => 10 do
-  PG::Connection.open :host     => uri.host,
-                      :user     => uri.user,
-                      :password => uri.password,
-                      :port     => uri.port || 5432,
-                      :dbname   => uri.path[1..-1]
-end
-```
-
-Be sure to pick your pool size carefully - if you use 10 for the size, you'll incur the overhead of having 10 connections open to Postgres even if you never use more than a couple of them.
-
-The Pond gem doesn't have this drawback - it is very similar to ConnectionPool, but establishes connections lazily (add `gem 'pond'` to your Gemfile):
-
-```ruby
-require 'uri'
-require 'pg'
-require 'pond'
-
-uri = URI.parse(ENV['DATABASE_URL'])
-
-Que.connection = Pond.new :maximum_size => 10 do
-  PG::Connection.open :host     => uri.host,
-                      :user     => uri.user,
-                      :password => uri.password,
-                      :port     => uri.port || 5432,
-                      :dbname   => uri.path[1..-1]
-end
-```
-
-Please be aware that if you're using ActiveRecord or Sequel to manage your data, there's no reason for you to be using any of these methods - it's less efficient (unnecessary connections will waste memory on your database server) and you lose the reliability benefits of wrapping jobs in the same transactions as the rest of your data. In general, your app should probably be using a connection pool, and Que should probably hook into whatever connection pool you're already using.

data/docs/using_sequel.md

@@ -1,31 +0,0 @@
-## Using Sequel
-
-If you're using Sequel, with or without Rails, you'll need to give Que a specific database instance to use:
-
-```ruby
-DB = Sequel.connect(ENV['DATABASE_URL'])
-Que.connection = DB
-```
-
-Then you can safely use the same database object to transactionally protect your jobs:
-
-```ruby
-class MyJob < Que::Job
-  def run
-    # Do stuff.
-
-    DB.transaction do
-      # Make changes to the database.
-
-      # Destroying this job will be protected by the same transaction.
-      destroy
-    end
-  end
-end
-
-# In your controller action:
-DB.transaction do
-  @user = User.create(params[:user])
-  MyJob.enqueue :user_id => @user.id
-end
-```

data/docs/writing_reliable_jobs.md

@@ -1,117 +0,0 @@
-## Writing Reliable Jobs
-
-Que does everything it can to ensure that jobs are worked exactly once, but if something bad happens when a job is halfway completed, there's no way around it - the job will need be repeated over again from the beginning, probably by a different worker. When you're writing jobs, you need to be prepared for this to happen.
-
-The safest type of job is one that reads in data, either from the database or from external APIs, then does some number crunching and writes the results to the database. These jobs are easy to make safe - simply write the results to the database inside a transaction, and also have the job destroy itself inside that transaction, like so:
-
-```ruby
-class UpdateWidgetPrice < Que::Job
-  def run(widget_id)
-    widget = Widget[widget_id]
-    price  = ExternalService.get_widget_price(widget_id)
-
-    ActiveRecord::Base.transaction do
-      # Make changes to the database.
-      widget.update :price => price
-
-      # Destroy the job.
-      destroy
-    end
-  end
-end
-```
-
-Here, you're taking advantage of the guarantees of an [ACID](https://en.wikipedia.org/wiki/ACID) database. The job is destroyed along with the other changes, so either the write will succeed and the job will be run only once, or it will fail and the database will be left untouched. But even if it fails, the job can simply be retried, and there are no lingering effects from the first attempt, so no big deal.
-
-The more difficult type of job is one that makes changes that can't be controlled transactionally. For example, writing to an external service:
-
-```ruby
-class ChargeCreditCard < Que::Job
-  def run(user_id, credit_card_id)
-    CreditCardService.charge(credit_card_id, :amount => "$10.00")
-
-    ActiveRecord::Base.transaction do
-      User.where(:id => user_id).update_all :charged_at => Time.now
-      destroy
-    end
-  end
-end
-```
-
-What if the process abruptly dies after we tell the provider to charge the credit card, but before we finish the transaction? Que will retry the job, but there's no way to tell where (or even if) it failed the first time. The credit card will be charged a second time, and then you've got an angry customer. The ideal solution in this case is to make the job [idempotent](https://en.wikipedia.org/wiki/Idempotence), meaning that it will have the same effect no matter how many times it is run:
-
-```ruby
-class ChargeCreditCard < Que::Job
-  def run(user_id, credit_card_id)
-    unless CreditCardService.check_for_previous_charge(credit_card_id)
-      CreditCardService.charge(credit_card_id, :amount => "$10.00")
-    end
-
-    ActiveRecord::Base.transaction do
-      User.where(:id => user_id).update_all :charged_at => Time.now
-      destroy
-    end
-  end
-end
-```
-
-This makes the job slightly more complex, but reliable (or, at least, as reliable as your credit card service).
-
-Finally, there are some jobs where you won't want to write to the database at all:
-
-```ruby
-class SendVerificationEmail < Que::Job
-  def run(email_address)
-    Mailer.verification_email(email_address).deliver
-  end
-end
-```
-
-In this case, we don't have any no way to prevent the occasional double-sending of an email. But, for ease of use, you can leave out the transaction and the `destroy` call entirely - Que will recognize that the job wasn't destroyed and will clean it up for you.
-
-### Timeouts
-
-Long-running jobs aren't necessarily a problem in Que, since the overhead of an individual job isn't that big (just an open PG connection and an advisory lock held in memory). But jobs that hang indefinitely can tie up a worker and [block the Ruby process from exiting gracefully](https://github.com/chanks/que/blob/master/docs/shutting_down_safely.md), which is a pain.
-
-Que doesn't offer a general way to kill jobs that have been running too long, because that currently can't be done safely in Ruby. Typically, one would use Ruby's Timeout module for this sort of thing, but wrapping a database transaction inside a timeout introduces a risk of premature commits, which can corrupt your data. See [here](http://blog.headius.com/2008/02/ruby-threadraise-threadkill-timeoutrb.html) and [here](http://coderrr.wordpress.com/2011/05/03/beware-of-threadkill-or-your-activerecord-transactions-are-in-danger-of-being-partially-committed/) for detail on why this is.
-
-However, if there's part of your job that is prone to hang (due to an API call or other HTTP request that never returns, for example), you can timeout those individual parts of your job relatively safely. For example, consider a job that needs to make an HTTP request and then write to the database:
-
-```ruby
-require 'net/http'
-
-class ScrapeStuff < Que::Job
-  def run(domain_to_scrape, path_to_scrape)
-    result = Net::HTTP.get(domain_to_scrape, path_to_scrape)
-
-    ActiveRecord::Base.transaction do
-      # Insert result...
-
-      destroy
-    end
-  end
-end
-```
-
-That request could take a very long time, or never return at all. Let's wrap it in a five-second timeout:
-
-```ruby
-require 'net/http'
-require 'timeout'
-
-class ScrapeStuff < Que::Job
-  def run(domain_to_scrape, path_to_scrape)
-    result = Timeout.timeout(5){Net::HTTP.get(domain_to_scrape, path_to_scrape)}
-
-    ActiveRecord::Base.transaction do
-      # Insert result...
-
-      destroy
-    end
-  end
-end
-```
-
-Now, if the request takes more than five seconds, a `Timeout::Error` will be raised and Que will just retry the job later. This solution isn't perfect, since Timeout uses Thread#kill under the hood, which can lead to unpredictable behavior. But it's separate from our transaction, so there's no risk of losing data - even a catastrophic error that left Net::HTTP in a bad state would be fixable by restarting the process.
-
-Finally, remember that if you're using a library that offers its own timeout functionality, that's usually preferable to using the Timeout module.

data/lib/generators/que/install_generator.rb

@@ -1,24 +0,0 @@
-# frozen_string_literal: true
-
-require 'rails/generators'
-require 'rails/generators/migration'
-require 'active_record'
-
-module Que
-  class InstallGenerator < Rails::Generators::Base
-    include Rails::Generators::Migration
-
-    namespace 'que:install'
-    self.source_paths << File.join(File.dirname(__FILE__), 'templates')
-    desc "Generates a migration to add Que's job table."
-
-    def self.next_migration_number(dirname)
-      next_migration_number = current_migration_number(dirname) + 1
-      ActiveRecord::Migration.next_migration_number(next_migration_number)
-    end
-
-    def create_migration_file
-      migration_template 'add_que.rb', 'db/migrate/add_que.rb'
-    end
-  end
-end

data/lib/generators/que/templates/add_que.rb

@@ -1,13 +0,0 @@
-# frozen_string_literal: true
-
-class AddQue < ActiveRecord::Migration
-  def self.up
-    # The current version as of this migration's creation.
-    Que.migrate! :version => 3
-  end
-
-  def self.down
-    # Completely removes Que's job queue.
-    Que.migrate! :version => 0
-  end
-end

data/lib/que/adapters/active_record.rb

@@ -1,54 +0,0 @@
-# frozen_string_literal: true
-
-module Que
-  module Adapters
-    class ActiveRecord < Base
-      def checkout
-        checkout_activerecord_adapter { |conn| yield conn.raw_connection }
-      end
-
-      def wake_worker_after_commit
-        # Works with ActiveRecord 3.2 and 4 (possibly earlier, didn't check)
-        if in_transaction?
-          checkout_activerecord_adapter { |adapter| adapter.add_transaction_record(TransactionCallback.new) }
-        else
-          Que.wake!
-        end
-      end
-
-      def cleanup!
-        # ActiveRecord will check out connections to the current thread when
-        # queries are executed and not return them to the pool until
-        # explicitly requested to. The wisdom of this API is questionable, and
-        # it doesn't pose a problem for the typical case of workers using a
-        # single PG connection (since we ensure that connection is checked in
-        # and checked out responsibly), but since ActiveRecord supports
-        # connections to multiple databases, it's easy for people using that
-        # feature to unknowingly leak connections to other databases. So, take
-        # the additional step of telling ActiveRecord to check in all of the
-        # current thread's connections between jobs.
-        ::ActiveRecord::Base.clear_active_connections!
-      end
-
-      class TransactionCallback
-        def has_transactional_callbacks?
-          true
-        end
-
-        def rolledback!(force_restore_state = false, should_run_callbacks = true)
-          # no-op
-        end
-
-        def committed!(should_run_callbacks = true)
-          Que.wake!
-        end
-      end
-
-      private
-
-      def checkout_activerecord_adapter(&block)
-        ::ActiveRecord::Base.connection_pool.with_connection(&block)
-      end
-    end
-  end
-end

data/lib/que/adapters/base.rb

@@ -1,127 +0,0 @@
-# frozen_string_literal: true
-
-require 'time' # For Time.parse.
-
-module Que
-  module Adapters
-    autoload :ActiveRecord,   'que/adapters/active_record'
-    autoload :ConnectionPool, 'que/adapters/connection_pool'
-    autoload :PG,             'que/adapters/pg'
-    autoload :Pond,           'que/adapters/pond'
-    autoload :Sequel,         'que/adapters/sequel'
-
-    class Base
-      def initialize(thing = nil)
-        @prepared_statements = {}
-      end
-
-      # The only method that adapters really need to implement. Should lock a
-      # PG::Connection (or something that acts like a PG::Connection) so that
-      # no other threads are using it and yield it to the block. Should also
-      # be re-entrant.
-      def checkout(&block)
-        raise NotImplementedError
-      end
-
-      # Called after Que has returned its connection to whatever pool it's
-      # using.
-      def cleanup!
-      end
-
-      # Called after a job is queued in async mode, to prompt a worker to
-      # wake up after the current transaction commits. Not all adapters will
-      # implement this.
-      def wake_worker_after_commit
-        false
-      end
-
-      def execute(command, params = [])
-        params = params.map do |param|
-          case param
-            # The pg gem unfortunately doesn't convert fractions of time instances, so cast them to a string.
-            when Time then param.strftime("%Y-%m-%d %H:%M:%S.%6N %z")
-            when Array, Hash then JSON_MODULE.dump(param)
-            else param
-          end
-        end
-
-        cast_result \
-          case command
-            when Symbol then execute_prepared(command, params)
-            when String then execute_sql(command, params)
-          end
-      end
-
-      def in_transaction?
-        checkout { |conn| conn.transaction_status != ::PG::PQTRANS_IDLE }
-      end
-
-      private
-
-      def execute_sql(sql, params)
-        args = params.empty? ? [sql] : [sql, params]
-        checkout { |conn| conn.async_exec(*args) }
-      end
-
-      def execute_prepared(name, params)
-        checkout do |conn|
-          # Prepared statement errors have the potential to foul up the entire
-          # transaction, so if we're in one, err on the side of safety.
-          return execute_sql(SQL[name], params) if Que.disable_prepared_statements || in_transaction?
-
-          statements = @prepared_statements[conn] ||= {}
-
-          begin
-            unless statements[name]
-              conn.prepare("que_#{name}", SQL[name])
-              prepared_just_now = statements[name] = true
-            end
-
-            conn.exec_prepared("que_#{name}", params)
-          rescue ::PG::InvalidSqlStatementName => error
-            # Reconnections on ActiveRecord can cause the same connection
-            # objects to refer to new backends, so recover as well as we can.
-
-            unless prepared_just_now
-              Que.log :level => 'warn', :event => "reprepare_statement", :name => name
-              statements[name] = false
-              retry
-            end
-
-            raise error
-          end
-        end
-      end
-
-      CAST_PROCS = {}
-
-      # Integer, bigint, smallint:
-      CAST_PROCS[23] = CAST_PROCS[20] = CAST_PROCS[21] = proc(&:to_i)
-
-      # Timestamp with time zone.
-      CAST_PROCS[1184] = Time.method(:parse)
-
-      # JSON.
-      CAST_PROCS[114] = JSON_MODULE.method(:load)
-
-      # Boolean:
-      CAST_PROCS[16] = 't'.method(:==)
-
-      def cast_result(result)
-        output = result.to_a
-
-        result.fields.each_with_index do |field, index|
-          if converter = CAST_PROCS[result.ftype(index)]
-            output.each do |hash|
-              unless (value = hash[field]).nil?
-                hash[field] = converter.call(value)
-              end
-            end
-          end
-        end
-
-        output.map!(&Que.json_converter)
-      end
-    end
-  end
-end

data/lib/que/adapters/connection_pool.rb

@@ -1,16 +0,0 @@
-# frozen_string_literal: true
-
-module Que
-  module Adapters
-    class ConnectionPool < Base
-      def initialize(pool)
-        @pool = pool
-        super
-      end
-
-      def checkout(&block)
-        @pool.with(&block)
-      end
-    end
-  end
-end

data/lib/que/adapters/pg.rb

@@ -1,21 +0,0 @@
-# frozen_string_literal: true
-
-require 'monitor'
-
-module Que
-  module Adapters
-    class PG < Base
-      attr_reader :lock
-
-      def initialize(pg)
-        @pg   = pg
-        @lock = Monitor.new # Must be re-entrant.
-        super
-      end
-
-      def checkout
-        @lock.synchronize { yield @pg }
-      end
-    end
-  end
-end

data/lib/que/adapters/pond.rb

@@ -1,16 +0,0 @@
-# frozen_string_literal: true
-
-module Que
-  module Adapters
-    class Pond < Base
-      def initialize(pond)
-        @pond = pond
-        super
-      end
-
-      def checkout(&block)
-        @pond.checkout(&block)
-      end
-    end
-  end
-end

data/lib/que/adapters/sequel.rb

@@ -1,20 +0,0 @@
-# frozen_string_literal: true
-
-module Que
-  module Adapters
-    class Sequel < Base
-      def initialize(db)
-        @db = db
-        super
-      end
-
-      def checkout(&block)
-        @db.synchronize(&block)
-      end
-
-      def wake_worker_after_commit
-        @db.after_commit { Que.wake! }
-      end
-    end
-  end
-end

data/lib/que/railtie.rb

@@ -1,16 +0,0 @@
-# frozen_string_literal: true
-
-module Que
-  class Railtie < Rails::Railtie
-    config.que = Que
-
-    Que.logger         = proc { Rails.logger }
-    Que.mode           = :sync if Rails.env.test?
-    Que.connection     = ::ActiveRecord if defined? ::ActiveRecord
-    Que.json_converter = :with_indifferent_access.to_proc
-
-    rake_tasks do
-      load 'que/rake_tasks.rb'
-    end
-  end
-end

data/lib/que/rake_tasks.rb

@@ -1,59 +0,0 @@
-# frozen_string_literal: true
-
-namespace :que do
-  desc "Process Que's jobs using a worker pool"
-  task :work => :environment do
-    $stdout.sync = true
-
-    $stdout.puts "The que:work rake task has been deprecated and will be removed in Que 1.0. Please transition to the que command line interface instead."
-
-    if defined?(::Rails) && Rails.respond_to?(:application)
-      # ActiveSupport's dependency autoloading isn't threadsafe, and Que uses
-      # multiple threads, which means that eager loading is necessary. Rails
-      # explicitly prevents eager loading when the environment task is invoked,
-      # so we need to manually eager load the app here.
-      Rails.application.eager_load!
-    end
-
-    Que.logger.level  = Logger.const_get((ENV['QUE_LOG_LEVEL'] || 'INFO').upcase)
-    Que.worker_count  = (ENV['QUE_WORKER_COUNT'] || 4).to_i
-    Que.wake_interval = (ENV['QUE_WAKE_INTERVAL'] || 0.1).to_f
-    Que.queue_name    = ENV['QUE_QUEUE'] if ENV['QUE_QUEUE']
-    Que.mode          = :async
-
-    # When changing how signals are caught, be sure to test the behavior with
-    # the rake task in tasks/safe_shutdown.rb.
-
-    stop = false
-    %w( INT TERM ).each do |signal|
-      trap(signal) {stop = true}
-    end
-
-    at_exit do
-      $stdout.puts "Finishing Que's current jobs before exiting..."
-      Que.worker_count = 0
-      Que.mode = :off
-      $stdout.puts "Que's jobs finished, exiting..."
-    end
-
-    loop do
-      sleep 0.01
-      break if stop
-    end
-  end
-
-  desc "Migrate Que's job table to the most recent version (creating it if it doesn't exist)"
-  task :migrate => :environment do
-    Que.migrate!
-  end
-
-  desc "Drop Que's job table"
-  task :drop => :environment do
-    Que.drop!
-  end
-
-  desc "Clear Que's job table"
-  task :clear => :environment do
-    Que.clear!
-  end
-end