queue_classic 3.2.0.RC1 → 4.0.0

data/README.md

@@ -1,41 +1,54 @@
 # queue_classic
+A simple, efficient worker queue for Ruby & PostgreSQL.
 
-<p align="center">
-  <b>Simple, efficient worker queue for Ruby & PostgreSQL</b>
-  <br />
-  <a href="https://travis-ci.org/QueueClassic/queue_classic"><img src="http://img.shields.io/travis/QueueClassic/queue_classic/master.svg?style=flat" /></a>
+Why this over something like Resque. Two reasons:
 
-  <a href="https://codeclimate.com/github/QueueClassic/queue_classic"><img src="http://img.shields.io/codeclimate/github/QueueClassic/queue_classic.svg?style=flat" /></a>
+1. Your jobs can be equeued in the same transaction as other modifications to the database, and will only be processed when everything is commited. This is a hard pattern to develop around for queues done outside your database
+2. Less things to run, if you don't already have Redis or a dedicated queue in your stack.
 
-  <a href="http://badge.fury.io/rb/queue_classic"><img src="http://img.shields.io/gem/v/queue_classic.svg?style=flat" alt="Gem Version" height="18"></a>
-</p>
+![master](https://github.com/QueueClassic/queue_classic/actions/workflows/main.yaml/badge.svg?branch=master)
 
+[![Gem Version](http://img.shields.io/gem/v/queue_classic.svg?style=flat)](http://badge.fury.io/rb/queue_classic)
 
-**IMPORTANT NOTE REGARDING VERSIONS**
+**IMPORTANT NOTE: This README is representing the current work for queue_classic, which is generally the pending next version.**
 
-**This README is representing the current work for queue_classic edge [unstable]. You can find the README for other versions:**
+You can always find the latest and previous releases here:
 
-- current release candidate: [v3.1.X](https://github.com/QueueClassic/queue_classic/tree/3-1-stable)
-- latest stable can be found: [v3.0.X](https://github.com/QueueClassic/queue_classic/tree/3-0-stable)
-- older stable: [v2.2.3](https://github.com/QueueClassic/queue_classic/tree/v2.2.3)
+https://github.com/QueueClassic/queue_classic/releases
 
+## Other related projects
+If you're interested in this project, you might also want to checkout:
 
-## What is queue_classic?
+ * [Que](https://github.com/que-rb/que)
+ * [GoodJob](https://github.com/bensheldon/good_job)
+ * [Delayed Job](https://github.com/collectiveidea/delayed_job)
+
+For a list of other queues (which may or may not be Postgres backed), checkout - https://edgeapi.rubyonrails.org/classes/ActiveJob/QueueAdapters.html
 
+## What is queue_classic?
 queue_classic provides a simple interface to a PostgreSQL-backed message queue. queue_classic specializes in concurrent locking and minimizing database load while providing a simple, intuitive developer experience. queue_classic assumes that you are already using PostgreSQL in your production environment and that adding another dependency (e.g. redis, beanstalkd, 0mq) is undesirable.
 
-## Features
+A major benefit is the ability to enqueue inside transactions, ensuring things are done only when your changes are commited.
 
-* Leverage of PostgreSQL's listen/notify & row locking.
+## Other related projects
+* [Queue Classic Plus](https://github.com/rainforestapp/queue_classic_plus) - adds support for retrying with specific exceptions, transaction processing of jobs, metric collection, etc
+* [Queue Classic Admin](https://github.com/QueueClassic/queue_classic_admin) - Admin interface for queue_classic
+* [Queue Classic Matchers](https://github.com/rainforestapp/queue_classic_matchers) - RSpec matchers for queue_classic
+
+## Features
+* Leverage of PostgreSQL's listen/notify, skip locked, and row locking.
 * Support for multiple queues with heterogeneous workers.
 * JSON data format.
-* Forking workers.
 * Workers can work multiple queues.
-* Reduced row contention using a [relaxed FIFO](http://www.cs.tau.ac.il/~shanir/nir-pubs-web/Papers/Lock_Free.pdf) technique.
 
-## Table of content
+### Requirements
+For this version, the requirements are as follows:
+* Ruby 2.6, 2.7, 3.0, 3.1 - i.e. currently supported Ruby versions
+* Postgres ~> 9.6
+* Rubygem: pg ~> 1.1
 
-* [Documentation](http://rubydoc.info/gems/queue_classic/2.2.3/frames)
+## Table of contents
+* [Documentation](https://www.rubydoc.info/gems/queue_classic/)
 * [Usage](#usage)
 * [Setup](#setup)
 * [Upgrade from earlier versions to V3.1](#upgrade-from-earlier-versions)
@@ -47,15 +60,14 @@ queue_classic provides a simple interface to a PostgreSQL-backed message queue.
 * [License](#license)
 
 ## Usage
-
-There are 2 ways to use queue_classic.
+There are 2 ways to use queue_classic:
 
 * Producing Jobs
 * Working Jobs
 
 ### Producing Jobs
-
-The first argument is a string which represents a ruby object and a method name. The second argument(s) will be passed along as arguments to the method invocation defined by the first argument. The set of arguments will be encoded as JSON in the database.
+#### Simple Enqueue
+The first argument is a string which represents a ruby object and a method name. The second argument(s) will be passed along as arguments to the method defined by the first argument. The set of arguments will be encoded as JSON and stored in the database.
 
 ```ruby
 # This method has no arguments.
@@ -78,47 +90,50 @@ p_queue = QC::Queue.new("priority_queue")
 p_queue.enqueue("Kernel.puts", ["hello", "world"])
 ```
 
-There is also the possibility to schedule a job at a specified time in the future. It will not be worked off before that specified time.
+#### Scheduling for later
+There is also the ability to schedule a job to run at a specified time in the future. The job will become processable after the specified time, and will be processed as-soon-as-possible.
 
 ```ruby
-# Specifying the job execution time exactly.
+# Specify the job execution time exactly
 QC.enqueue_at(Time.new(2024,01,02,10,00), "Kernel.puts", "hello future")
 
-# Specifying the job execution time as an offset in seconds.
+# Specify the job execution time as an offset in seconds
 QC.enqueue_in(60, "Kernel.puts", "hello from 1 minute later")
 ```
 
 ### Working Jobs
-
-There are two ways to work jobs. The first approach is to use the Rake task. The second approach is to use a custom executable.
+There are two ways to work/process jobs. The first approach is to use the Rake task. The second approach is to use a custom executable.
 
 #### Rake Task
-
-Require queue_classic in your Rakefile.
+Require queue_classic in your Rakefile:
 
 ```ruby
 require 'queue_classic'
 require 'queue_classic/tasks'
 ```
 
-Start the worker via the Rakefile.
+##### Work all queues
+Start the worker via the Rakefile:
+
 ```bash
-$ bundle exec rake qc:work
+bundle exec rake qc:work
 ```
 
-Setup a worker to work a non-default queue.
+##### Work a single specific queue
+Setup a worker to work only a specific, non-default queue:
+
 ```bash
-$ QUEUE="priority_queue" bundle exec rake qc:work
+QUEUE="priority_queue" bundle exec rake qc:work
 ```
 
-Setup a worker to work multiple queues.
+##### Work multiple queues
+In this scenario, on each iteration of the worker's loop, it will look for jobs in the first queue prior to looking at the second queue. This means that the first queue must be empty before the worker will look at the second queue.
+
 ```bash
-$ QUEUES="priority_queue,secondary_queue" bundle exec rake qc:work
+QUEUES="priority_queue,secondary_queue" bundle exec rake qc:work
 ```
-In this scenario, on each iteration of the worker's loop, it will look for jobs in the first queue prior to looking at the second queue. This means that the first queue must be empty before the worker will look at the second queue.
 
 #### Custom Worker
-
 This example is probably not production ready; however, it serves as an example of how to leverage the code in the Worker class to fit your non-default requirements.
 
 ```ruby
@@ -128,7 +143,6 @@ require 'queue_classic'
 FailedQueue = QC::Queue.new("failed_jobs")
 
 class MyWorker < QC::Worker
-
   # A job is a Hash containing these attributes:
   # :id Integer, the job id
   # :method String, containing the object and method
@@ -141,7 +155,7 @@ class MyWorker < QC::Worker
      # Do something with the job
      ...
   end
-  
+
   # This method will be called when an exception
   # is raised during the execution of the job.
   # First argument is the job that failed.
@@ -149,7 +163,6 @@ class MyWorker < QC::Worker
   def handle_failure(job, e)
     FailedQueue.enqueue(job[:method], *job[:args])
   end
-    
 end
 
 worker = MyWorker.new
@@ -158,164 +171,128 @@ trap('INT') { exit }
 trap('TERM') { worker.stop }
 
 loop do
-  job = worker.lock_job
-  Timeout::timeout(5) { worker.process(job) }
+  queue, job = worker.lock_job
+  Timeout::timeout(5) { worker.process(queue, job) }
 end
 ```
 
-The `qc:work` rake task uses `QC::Worker` by default. However, it's easy to
-inject your own worker class:
+The `qc:work` rake task uses `QC::Worker` by default. However, it's easy to inject your own worker class:
 
 ```ruby
 QC.default_worker_class = MyWorker
 ```
 
 ## Setup
-
-In addition to installing the rubygem, you will need to prepare your database. Database preparation includes creating a table and loading PL/pgSQL functions. You can issue the database preparation commands using `PSQL(1)` or use a database migration script.
+In addition to installing the rubygem, you will need to prepare your database. Database preparation includes creating a table and loading PL/pgSQL functions. You can issue the database preparation commands using `psql` or use a database migration script.
 
 ### Quick Start
-
-```bash
-$ gem install queue_classic
-$ createdb queue_classic_test
-$ export QC_DATABASE_URL="postgres://username:password@localhost/queue_classic_test"
-$ ruby -r queue_classic -e "QC::Setup.create"
-$ ruby -r queue_classic -e "QC.enqueue('Kernel.puts', 'hello world')"
-$ ruby -r queue_classic -e "QC::Worker.new.work"
+```bash{:copy}
+gem install queue_classic
+createdb queue_classic_test
+export QC_DATABASE_URL="postgres://username:password@localhost/queue_classic_test"
+ruby -r queue_classic -e "QC::Setup.create"
+ruby -r queue_classic -e "QC.enqueue('Kernel.puts', 'hello world')"
+ruby -r queue_classic -e "QC::Worker.new.work"
 ```
 
 ### Ruby on Rails Setup
+Declare dependencies in Gemfile:
 
-Declare dependencies in Gemfile.
-```ruby
-source "https://rubygems.org"
-gem "queue_classic", "~> 3.0.0"
+```ruby{:copy}
+source 'https://rubygems.org' do
+  gem 'queue_classic'
+end
 ```
 
-Add the database tables and stored procedures.
+Install queue_classic, which adds the needed migrations for the database tables and stored procedures:
 
-```
+```bash
 rails generate queue_classic:install
 bundle exec rake db:migrate
 ```
 
-#### Active Job
+#### Database connection
+Starting with with queue_classic 3.1, Rails is automatically detected and its connection is used. If you don't want to use the automatic database connection, set this environment variable to false: `export QC_RAILS_DATABASE=false`. 
 
-If you use Rails 4.2+, all you need to do is to set `config.active_job.queue_adapter = :queue_classic` in your `application.rb`. Everything else will be taken care for you. You can now use the Active Job functionality from now.
+> **Note:** If you do not share the connection, you cannot enqueue in the same transaction as whatever you're doing in Rails.
 
-Just for your information, queue_classic detects your database connection and uses it.
+**Note on using ActiveRecord migrations:** If you use the migration, and you wish to use commands that reset the database from the stored schema (e.g. `rake db:reset`), your application must be configured with `config.active_record.schema_format = :sql` in `config/application.rb`. If you don't do this, the PL/pgSQL function that queue_classic creates will be lost when you reset the database.
 
-### Rake Task Setup
+#### Active Job
+If you use Rails 4.2+ and want to use Active Job, all you need to do is to set `config.active_job.queue_adapter = :queue_classic` in your `application.rb`. Everything else will be taken care for you. You can now use the Active Job functionality from now.
 
-Alternatively, you can use the Rake task to prepare your database.
+### Plain Ruby Setup
+If you're not using Rails, you can use the Rake task to prepare your database:
 
-```bash
+```bash{:copy}
 # Creating the table and functions
-$ bundle exec rake qc:create
+bundle exec rake qc:create
 
 # Dropping the table and functions
-$ bundle exec rake qc:drop
+bundle exec rake qc:drop
 ```
 
-### Database connection
+#### Database connection
+By default, queue_classic will use the `QC_DATABASE_URL` falling back on `DATABASE_URL`. The URL must be in the following format: `postgres://username:password@localhost/database_name`. If you use Heroku's PostgreSQL service, this will already be set. If you don't want to set this variable, you can set the connection in an initializer. **QueueClassic will maintain its own connection to the database.** This may double the number of connections to your database.
 
-#### Ruby on Rails
-
-Starting with with queue_classic 3.1, Rails is automatically detected and its connection is used.
-
-If you don't want to use the automatic database connection, set this environment variable to false: `export QC_RAILS_DATABASE=false`
-
-**Note on using ActiveRecord migrations:** If you use the migration, and you wish to use commands that reset the database from the stored schema (e.g. `rake db:reset`), your application must be configured with `config.active_record.schema_format = :sql` in `config/application.rb`.  If you don't do this, the PL/pgSQL function that queue_classic creates will be lost when you reset the database.
-
-
-#### Other Ruby apps
-
-By default, queue_classic will use the QC_DATABASE_URL falling back on DATABASE_URL. The URL must be in the following format: `postgres://username:password@localhost/database_name`.  If you use Heroku's PostgreSQL service, this will already be set. If you don't want to set this variable, you can set the connection in an initializer. **QueueClassic will maintain its own connection to the database.** This may double the number of connections to your database.
-
-## Upgrade from earlier versions
+## Upgrading from earlier versions
 If you are upgrading from a previous version of queue_classic, you might need some new database columns and/or functions. Luckily enough for you, it is easy to do so.
 
 ### Ruby on Rails
+These two commands will add the newer migrations:
 
-You just need to run those lines, which will copy the new required migrations:
-
-```
+```bash{:copy}
 rails generate queue_classic:install
 bundle exec rake db:migrate
 ```
+
 ### Rake Task
+This rake task will update you to the latest version:
 
-This rake task will get you covered:
-```bash
+```bash{:copy}
 # Updating the table and functions
-$ bundle exec rake qc:update
+bundle exec rake qc:update
 ```
 
 ## Configuration
-
 All configuration takes place in the form of environment vars. See [queue_classic.rb](https://github.com/QueueClassic/queue_classic/blob/master/lib/queue_classic.rb#L23-62) for a list of options.
 
-## JSON
-
-If you are running PostgreSQL 9.4 or higher, queue_classic will use the [jsonb](http://www.postgresql.org/docs/9.4/static/datatype-json.html) datatype for new tables. Versions 9.2 and 9.3 will use the `json` data type and versions 9.1 and lower will use the `text` data type.
-If you are updating queue_classic and are running PostgreSQL >= 9.4, run the following to switch to `jsonb`:
-```
-alter table queue_classic_jobs alter column args type jsonb using (args::jsonb);
-```
-
 ## Logging
-
-By default queue_classic does not talk very much.
-If you find yourself in a situation where you need to know what's happening inside QC, there are two different kind of logging you can enable: DEBUG and MEASURE.
+By default queue_classic does not talk very much. If you find yourself in a situation where you need to know what's happening inside QC, there are two different kind of logging you can enable: `DEBUG` and `MEASURE`.
 
 ### Measure
-This will output the time to process and that kind of thing. To enable it, set the `QC_MEASURE`:
+This will output the time to process and some more statistics. To enable it, set the `QC_MEASURE`:
 
-```
+```bash{:copy}
 export QC_MEASURE="true"
 ```
 
 ### Debug
 You can enable the debug output by setting the `DEBUG` environment variable:
 
-```
+```bash{:copy}
 export DEBUG="true"
 ```
 
 ## Support
-
 If you think you have found a bug, feel free to open an issue. Use the following template for the new issue:
 
-1. List Versions: Ruby, PostgreSQL, queue_classic.
+1. List your versions: Ruby, PostgreSQL, queue_classic.
 2. Define what you would have expected to happen.
 3. List what actually happened.
 4. Provide sample codes & commands which will reproduce the problem.
 
-If you have general questions about how to use queue_classic, send a message to the mailing list:
-
-https://groups.google.com/d/forum/queue_classic
-
 ## Hacking on queue_classic
-
-### Dependencies
-
-* Ruby 1.9.2
-* Postgres ~> 9.0
-* Rubygem: pg ~> 0.11.0
-* For JRuby, see [queue_classic_java](https://github.com/bdon/queue_classic_java)
-
 ### Running Tests
-
-```bash
-$ bundle
-$ createdb queue_classic_test
-$ export QC_DATABASE_URL="postgres://username:pass@localhost/queue_classic_test"
-$ rake
+```bash{:copy}
+bundle
+createdb queue_classic_test
+export QC_DATABASE_URL="postgres://username:pass@localhost/queue_classic_test"
+bundle exec rake                      # run all tests
+bundle exec ruby test/queue_test.rb   # run a single test
 ```
 
 ## License
-
 Copyright (C) 2010 Ryan Smith
 
 Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

data/Rakefile

@@ -1,3 +1,5 @@
+# frozen_string_literal: true
+
 $:.unshift("lib")
 
 require "bundler/gem_tasks"

data/lib/generators/queue_classic/install_generator.rb

@@ -1,3 +1,5 @@
+# frozen_string_literal: true
+
 require 'rails/generators'
 require 'rails/generators/migration'
 require 'active_record'
@@ -31,6 +33,10 @@ module QC
       if self.class.migration_exists?('db/migrate', 'update_queue_classic_3_1_0').nil?
         migration_template 'update_queue_classic_3_1_0.rb', 'db/migrate/update_queue_classic_3_1_0.rb'
       end
+
+      if self.class.migration_exists?('db/migrate', 'update_queue_classic_4_0_0').nil?
+        migration_template 'update_queue_classic_4_0_0.rb', 'db/migrate/update_queue_classic_4_0_0.rb'
+      end
     end
   end
 end

data/lib/generators/queue_classic/templates/add_queue_classic.rb

@@ -1,4 +1,6 @@
-class AddQueueClassic < ActiveRecord::Migration
+# frozen_string_literal: true
+
+class AddQueueClassic < ActiveRecord::Migration[4.2]
   def self.up
     QC::Setup.create
   end

data/lib/generators/queue_classic/templates/update_queue_classic_3_0_0.rb

@@ -1,4 +1,6 @@
-class UpdateQueueClassic300 < ActiveRecord::Migration
+# frozen_string_literal: true
+
+class UpdateQueueClassic300 < ActiveRecord::Migration[4.2]
   def self.up
     QC::Setup.update_to_3_0_0
   end

data/lib/generators/queue_classic/templates/update_queue_classic_3_0_2.rb

@@ -1,4 +1,6 @@
-class UpdateQueueClassic302 < ActiveRecord::Migration
+# frozen_string_literal: true
+
+class UpdateQueueClassic302 < ActiveRecord::Migration[4.2]
   def self.up
     QC::Setup.update_to_3_0_0
   end

data/lib/generators/queue_classic/templates/update_queue_classic_3_1_0.rb

@@ -1,4 +1,6 @@
-class UpdateQueueClassic310 < ActiveRecord::Migration
+# frozen_string_literal: true
+
+class UpdateQueueClassic310 < ActiveRecord::Migration[4.2]
   def self.up
     QC::Setup.update_to_3_1_0
   end

data/lib/generators/queue_classic/templates/update_queue_classic_4_0_0.rb

@@ -0,0 +1,11 @@
+# frozen_string_literal: true
+
+class UpdateQueueClassic400 < ActiveRecord::Migration[4.2]
+  def self.up
+    QC::Setup.update_to_4_0_0
+  end
+
+  def self.down
+    QC::Setup.downgrade_from_4_0_0
+  end
+end

data/lib/queue_classic/config.rb

@@ -1,3 +1,5 @@
+# frozen_string_literal: true
+
 module QC
   module Config
     # You can use the APP_NAME to query for
@@ -58,7 +60,6 @@ module QC
 
     # The worker class instantiated by QC's rake tasks.
     def default_worker_class
-
       @worker_class ||= (ENV["QC_DEFAULT_WORKER_CLASS"] && Kernel.const_get(ENV["QC_DEFAULT_WORKER_CLASS"]) ||
                          QC::Worker)
 

data/lib/queue_classic/conn_adapter.rb

@@ -1,27 +1,37 @@
+# frozen_string_literal: true
+
 require 'uri'
 require 'pg'
 
 module QC
   class ConnAdapter
 
-    attr_accessor :connection
-    def initialize(c=nil)
-      @connection = c.nil? ? establish_new : validate!(c)
+    def initialize(args={})
+      @active_record_connection_share = args[:active_record_connection_share]
+      @_connection = args[:connection]
       @mutex = Mutex.new
     end
 
+    def connection
+      if @active_record_connection_share && Object.const_defined?('ActiveRecord')
+        ActiveRecord::Base.connection.raw_connection
+      else
+        @_connection ||= establish_new
+      end
+    end
+
     def execute(stmt, *params)
       @mutex.synchronize do
         QC.log(:at => "exec_sql", :sql => stmt.inspect)
         begin
           params = nil if params.empty?
-          r = @connection.exec(stmt, params)
+          r = connection.exec(stmt, params)
           result = []
           r.each {|t| result << t}
           result.length > 1 ? result : result.pop
-        rescue PGError => e
+        rescue PG::Error => e
           QC.log(:error => e.inspect)
-          @connection.reset
+          connection.reset
           raise
         end
       end
@@ -30,10 +40,10 @@ module QC
     def wait(time, *channels)
       @mutex.synchronize do
         listen_cmds = channels.map {|c| 'LISTEN "' + c.to_s + '"'}
-        @connection.exec(listen_cmds.join(';'))
+        connection.exec(listen_cmds.join(';'))
         wait_for_notify(time)
         unlisten_cmds = channels.map {|c| 'UNLISTEN "' + c.to_s + '"'}
-        @connection.exec(unlisten_cmds.join(';'))
+        connection.exec(unlisten_cmds.join(';'))
         drain_notify
       end
     end
@@ -41,7 +51,7 @@ module QC
     def disconnect
       @mutex.synchronize do
         begin
-          @connection.close
+          connection.close
         rescue => e
           QC.log(:at => 'disconnect', :error => e.message)
         end
@@ -59,12 +69,12 @@ module QC
 
     def wait_for_notify(t)
       Array.new.tap do |msgs|
-        @connection.wait_for_notify(t) {|event, pid, msg| msgs << msg}
+        connection.wait_for_notify(t) {|event, pid, msg| msgs << msg}
       end
     end
 
     def drain_notify
-      until @connection.notifies.nil?
+      until connection.notifies.nil?
         QC.log(:at => "drain_notifications")
       end
     end
@@ -77,10 +87,15 @@ module QC
 
     def establish_new
       QC.log(:at => "establish_conn")
-      conn = PGconn.connect(*normalize_db_url(db_url))
-      if conn.status != PGconn::CONNECTION_OK
+      conn = PG.connect(*normalize_db_url(db_url))
+      if conn.status != PG::CONNECTION_OK
         QC.log(:error => conn.error)
       end
+
+      if conn.server_version < 90600
+        raise "This version of Queue Classic does not support Postgres older than 9.6 (90600). This version is #{conn.server_version}. If you need that support, please use an older version."
+      end
+
       conn.exec("SET application_name = '#{QC.app_name}'")
       conn
     end
@@ -92,7 +107,7 @@ module QC
       [
        host, # host or percent-encoded socket path
        url.port || 5432,
-       nil, '', #opts, tty
+       nil, nil, #opts, tty
        url.path.gsub("/",""), # database name
        url.user,
        url.password
@@ -106,6 +121,5 @@ module QC
             raise(ArgumentError, "missing QC_DATABASE_URL or DATABASE_URL")
       @db_url = URI.parse(url)
     end
-
   end
 end