que 1.0.0.beta4 → 1.0.0.beta5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f2493b1dd8d4a7042ead34f78d19b83320c65f7eccd08a14336bbcfd8f349766
-  data.tar.gz: e3142695bd3ba1ad21aee57aeeeee47c260ed15ef86e93c6b942fe9e61745ed3
+  metadata.gz: 90ae5c7c65d93f4ea3833d80351c2e0d3ed3f706ba00411937e8aba2137a41d7
+  data.tar.gz: 58aba934ee2735522e602dd5777de666c6acbee1cb531d9918ce6f2b75f3b693
 SHA512:
-  metadata.gz: 8c2cadc33fc259cacc64cb0bdc7532451e9afbb344959b152d9477c75002ce1581858978ac69ac3a399968fd6f853ff93d41e3ac44edf09f67d8c547f97a92cb
-  data.tar.gz: 9319db034deccd32abdf97b69e5bcde5fe25d5b6cc424c9b07c26935b4b8c70e083314bf57ca3829f74bbecfdf895b2a92aec9386c17db81adf64127dfc3e4c7
+  metadata.gz: 1cbfa5f3c6d13868897ac9f1fcf079aaeaa29c9eadfd231044a484878fbf8e99f4b159a4458690027b181b29c5da3abafcc1c9de2d33b3c487b1fac4a15c7e50
+  data.tar.gz: 1ed4b71f51cd72e3968de6ec745e2de592dc021e9d334526630b3011d3f5526388b8562da4bf513066cb68d4dee033043115027ddfc0e9bef5e816ed2c35ede5

data/.github/workflows/{ruby.yml → tests.yml}

@@ -1,18 +1,22 @@
-name: Ruby
+name: tests
 
-on: [pull_request]
+on: [push, pull_request]
 
 jobs:
   test:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        ruby_version: [2.5.x, 2.6.x]
+        ruby_version: [2.5.x, 2.6.x, 2.7.x]
         gemfile: ["4.2", "5.2", "6.0"]
-        postgres_version: [9, 10, 11]
+        postgres_version: [9, 10, 11, 12]
+        exclude:
+          - { gemfile: "4.2", ruby_version: "2.7.x" }
     services:
       db:
         image: postgres:${{ matrix.postgres_version }}
+        env:
+          POSTGRES_HOST_AUTH_METHOD: trust
         ports: ['5432:5432']
         options: >-
           --health-cmd pg_isready

data/README.md

@@ -1,4 +1,4 @@
-# Que
+# Que ![tests](https://github.com/que-rb/que/workflows/tests/badge.svg)
 
 **This README and the rest of the docs on the master branch all refer to Que 1.0, which is currently in beta. If you're using version 0.x, please refer to the docs on [the 0.x branch](https://github.com/que-rb/que/tree/0.x).**
 
@@ -17,7 +17,7 @@ Additionally, there are the general benefits of storing jobs in Postgres, alongs
   * **Fewer Dependencies** - If you're already using Postgres (and you probably should be), a separate queue is another moving part that can break.
   * **Security** - Postgres' support for SSL connections keeps your data safe in transport, for added protection when you're running workers on cloud platforms that you can't completely control.
 
-Que's primary goal is reliability. You should be able to leave your application running indefinitely without worrying about jobs being lost due to a lack of transactional support, or left in limbo due to a crashing process. Que does everything it can to ensure that jobs you queue are performed exactly once (though the occasional repetition of a job can be impossible to avoid - see the docs on [how to write a reliable job](https://github.com/que-rb/que/blob/master/docs/writing_reliable_jobs.md)).
+Que's primary goal is reliability. You should be able to leave your application running indefinitely without worrying about jobs being lost due to a lack of transactional support, or left in limbo due to a crashing process. Que does everything it can to ensure that jobs you queue are performed exactly once (though the occasional repetition of a job can be impossible to avoid - see the docs on [how to write a reliable job](/docs/README.md#writing-reliable-jobs)).
 
 Que's secondary goal is performance. The worker process is multithreaded, so that a single process can run many jobs simultaneously.
 
@@ -114,8 +114,14 @@ You can also add options to run the job after a specific time, or with a specifi
 ``` ruby
 ChargeCreditCard.enqueue card.id, user_id: current_user.id, run_at: 1.day.from_now, priority: 5
 ```
+## Running the Que Worker
+In order to process jobs, you must start a separate worker process outside of your main server. 
 
-Finally, you can work jobs using the included `que` CLI. Try running `que -h` to get a list of runtime options:
+```
+bundle exec que
+```
+
+Try running `que -h` to get a list of runtime options:
 ```
 $ que -h
 usage: que [options] [file/to/require] ...
@@ -130,7 +136,21 @@ You may need to pass que a file path to require so that it can load your app. Qu
 
 If you're using ActiveRecord to dump your database's schema, please [set your schema_format to :sql](http://guides.rubyonrails.org/migrations.html#types-of-schema-dumps) so that Que's table structure is managed correctly. This is a good idea regardless, as the `:ruby` schema format doesn't support many of PostgreSQL's advanced features.
 
-Pre-1.0, the default queue name needed to be configured in order for Que to work out of the box with Rails. In 1.0 the default queue name is now 'default', as Rails expects, but when Rails enqueues some types of jobs it may try to use another queue name that isn't worked by default - in particular, ActionMailer uses a queue named 'mailers' by default, so in your app config you'll also need to set `config.action_mailer.deliver_later_queue_name = 'default'` if you're using ActionMailer.
+Pre-1.0, the default queue name needed to be configured in order for Que to work out of the box with Rails. In 1.0 the default queue name is now 'default', as Rails expects, but when Rails enqueues some types of jobs it may try to use another queue name that isn't worked by default. You can either:
+
+* [Configure Rails](https://guides.rubyonrails.org/configuring.html) to send all internal job types to the 'default' queue by adding the following to `config/application.rb`:
+```ruby
+config.action_mailer.deliver_later_queue_name = :default
+config.action_mailbox.queues.incineration = :default
+config.action_mailbox.queues.routing = :default
+config.active_storage.queues.analysis = :default
+config.active_storage.queues.purge = :default
+```
+
+* [Tell que](/docs#multiple-queues) to work all of these queues (less efficient because it requires polling all of them):
+```
+que -q default -q mailers -q action_mailbox_incineration -q action_mailbox_routing -q active_storage_analysis -q active_storage_purge
+```
 
 Also, if you would like to integrate Que with Active Job, you can do it by setting the adapter in `config/application.rb` or in a specific environment by setting it in `config/environments/production.rb`, for example:
 ```ruby
@@ -147,12 +167,17 @@ If you later decide to switch a job from Active Job to Que to have transactional
 
 There are a couple ways to do testing. You may want to set `Que::Job.run_synchronously = true`, which will cause JobClass.enqueue to simply execute the job's logic synchronously, as if you'd run JobClass.run(*your_args). Or, you may want to leave it disabled so you can assert on the job state once they are stored in the database.
 
+## Documentation
+
+**For full documentation, see [here](docs/README.md)**.
+
 ## Related Projects
 
 These projects are tested to be compatible with Que 1.x:
 
 - [que-web](https://github.com/statianzo/que-web) is a Sinatra-based UI for inspecting your job queue.
 - [que-scheduler](https://github.com/hlascelles/que-scheduler) lets you schedule tasks using a cron style config file.
+- [que-locks](https://github.com/airhorns/que-locks) lets you lock around job execution for so only one job runs at once for a set of arguments.
 
 If you have a project that uses or relates to Que, feel free to submit a PR adding it to the list!
 
@@ -164,7 +189,6 @@ If you have a project that uses or relates to Que, feel free to submit a PR addi
 
 Regarding contributions, one of the project's priorities is to keep Que as simple, lightweight and dependency-free as possible, and pull requests that change too much or wouldn't be useful to the majority of Que's users have a good chance of being rejected. If you're thinking of submitting a pull request that adds a new feature, consider starting a discussion in [que-talk](https://groups.google.com/forum/#!forum/que-talk) first about what it would do and how it would be implemented. If it's a sufficiently large feature, or if most of Que's users wouldn't find it useful, it may be best implemented as a standalone gem, like some of the related projects above.
 
-
 ### Specs
 
 A note on running specs - Que's worker system is multithreaded and therefore prone to race conditions. As such, if you've touched that code, a single spec run passing isn't a guarantee that any changes you've made haven't introduced bugs. One thing I like to do before pushing changes is rerun the specs many times and watching for hangs. You can do this from the command line with something like:

data/bin/command_line_interface.rb

@@ -183,8 +183,8 @@ OUTPUT
         args.each do |file|
           begin
             require file
-          rescue LoadError
-            output.puts "Could not load file '#{file}'"
+          rescue LoadError => e
+            output.puts "Could not load file '#{file}': #{e}"
             return 1
           end
         end

data/docs/README.md

@@ -37,9 +37,9 @@ Docs Index
   * [destroy](#destroy)
   * [finish](#finish)
   * [expire](#expire)
-  * [retry_in](#retry-in)
-  * [error_count](#error-count)
-  * [default_resolve_action](#default-resolve-action)
+  * [retry_in](#retry_in)
+  * [error_count](#error_count)
+  * [default_resolve_action](#default_resolve_action)
 - [Writing Reliable Jobs](#writing-reliable-jobs)
   * [Timeouts](#timeouts)
 - [Middleware](#middleware)
@@ -122,7 +122,7 @@ ActiveRecord::Base.transaction do
 end
 ```
 
-There are other docs to read if you're using [Sequel](#using-sequel) or [plain Postgres connections](#using-plain-connections) (with no ORM at all) instead of ActiveRecord.
+There are other docs to read if you're using [Sequel](#using-sequel) or [plain Postgres connections](#using-plain-postgres-connections) (with no ORM at all) instead of ActiveRecord.
 
 ### Managing the Jobs Table
 
@@ -463,7 +463,7 @@ Que.enqueue current_user.id, job_class: 'ProcessCreditCard', queue: 'credit_card
 
 To ensure safe operation, Que needs to be very careful in how it shuts down. When a Ruby process ends normally, it calls Thread#kill on any threads that are still running - unfortunately, if a thread is in the middle of a transaction when this happens, there is a risk that it will be prematurely commited, resulting in data corruption. See [here](http://blog.headius.com/2008/02/ruby-threadraise-threadkill-timeoutrb.html) and [here](http://coderrr.wordpress.com/2011/05/03/beware-of-threadkill-or-your-activerecord-transactions-are-in-danger-of-being-partially-committed/) for more detail on this.
 
-To prevent this, Que will block the worker process from exiting until all jobs it is working have completed normally. Unfortunately, if you have long-running jobs, this may take a very long time (and if something goes wrong with a job's logic, it may never happen). The solution in this case is SIGKILL - luckily, Ruby processes that are killed via SIGKILL will end without using Thread#kill on its running threads. This is safer than exiting normally - when PostgreSQL loses the connection it will simply roll back the open transaction, if any, and unlock the job so it can be retried later by another worker. Be sure to read [Writing Reliable Jobs](#writing-reliable-jobs.md) for information on how to design your jobs to fail safely.
+To prevent this, Que will block the worker process from exiting until all jobs it is working have completed normally. Unfortunately, if you have long-running jobs, this may take a very long time (and if something goes wrong with a job's logic, it may never happen). The solution in this case is SIGKILL - luckily, Ruby processes that are killed via SIGKILL will end without using Thread#kill on its running threads. This is safer than exiting normally - when PostgreSQL loses the connection it will simply roll back the open transaction, if any, and unlock the job so it can be retried later by another worker. Be sure to read [Writing Reliable Jobs](#writing-reliable-jobs) for information on how to design your jobs to fail safely.
 
 So, be prepared to use SIGKILL on your Ruby processes if they run for too long. For example, Heroku takes a good approach to this - when Heroku's platform is shutting down a process, it sends SIGTERM, waits ten seconds, then sends SIGKILL if the process still hasn't exited. This is a nice compromise - it will give each of your currently running jobs ten seconds to complete, and any jobs that haven't finished by then will be interrupted and retried later.
 
@@ -593,7 +593,7 @@ Additionally, including `Que::ActiveJob::JobExtensions` lets you define a run()
 
 ## Job Helper Methods
 
-There are a number of instance methods on Que::Job that you can use in your jobs, preferably in transactions. See [Writing Reliable Jobs](/writing_reliable_jobs.md) for more information on where to use these methods.
+There are a number of instance methods on Que::Job that you can use in your jobs, preferably in transactions. See [Writing Reliable Jobs](#writing-reliable-jobs) for more information on where to use these methods.
 
 ### destroy
 

data/lib/que/active_record/model.rb

@@ -3,7 +3,7 @@
 module Que
   module ActiveRecord
     class Model < ::ActiveRecord::Base
-      self.table_name = :que_jobs
+      self.table_name = 'public.que_jobs'
 
       t = arel_table
 

data/lib/que/connection.rb

@@ -119,6 +119,10 @@ module Que
       loop { break if next_notification.nil? }
     end
 
+    def server_version
+      wrapped_connection.server_version
+    end
+
     def in_transaction?
       wrapped_connection.transaction_status != ::PG::PQTRANS_IDLE
     end

data/lib/que/job_buffer.rb

@@ -59,10 +59,8 @@ module Que
 
         # Relying on the hash's contents being sorted, here.
         priority_queues.reverse_each do |_, pq|
-          pq.waiting_count.times do
-            job = _shift_job(pq.priority)
-            break if job.nil? # False would mean we're stopping.
-            pq.push(job)
+          pq.populate do
+            _shift_job(pq.priority)
           end
         end
 
@@ -75,7 +73,7 @@ module Que
 
     def shift(priority = nil)
       queue = priority_queues.fetch(priority) { raise Error, "not a permitted priority! #{priority}" }
-      queue.pop
+      queue.pop || shift_job(priority)
     end
 
     def shift_job(priority = nil)
@@ -158,6 +156,10 @@ module Que
       sync { _stopping? }
     end
 
+    def job_available?(priority)
+      (job = @array.first) && job.priority_sufficient?(priority)
+    end
+
     private
 
     def _buffer_space
@@ -210,15 +212,14 @@ module Que
       def pop
         sync do
           loop do
-            return false if @stopping
-
-            if item = @items.pop
+            if @stopping
+              return false
+            elsif item = @items.pop
               return item
+            elsif job_buffer.job_available?(priority)
+              return false
             end
 
-            job = job_buffer.shift_job(priority)
-            return job unless job.nil? # False means we're stopping.
-
             @waiting += 1
             @cv.wait(mutex)
             @waiting -= 1
@@ -226,18 +227,20 @@ module Que
         end
       end
 
-      def push(item)
+      def stop
         sync do
-          Que.assert(waiting_count > 0)
-          @items << item
-          @cv.signal
+          @stopping = true
+          @cv.broadcast
         end
       end
 
-      def stop
+      def populate
         sync do
-          @stopping = true
-          @cv.broadcast
+          waiting_count.times do
+            job = yield
+            break if job.nil? # False would mean we're stopping.
+            _push(job)
+          end
         end
       end
 
@@ -250,6 +253,12 @@ module Que
       def sync(&block)
         mutex.synchronize(&block)
       end
+
+      def _push(item)
+        Que.assert(waiting_count > 0)
+        @items << item
+        @cv.signal
+      end
     end
   end
 end

data/lib/que/locker.rb

@@ -393,10 +393,12 @@ module Que
         }
       end
 
+      materalize_cte = connection.server_version >= 12_00_00
+
       jobs =
         connection.execute \
           <<-SQL
-            WITH jobs AS (SELECT * FROM que_jobs WHERE id IN (#{ids.join(', ')}))
+            WITH jobs AS #{materalize_cte ? 'MATERIALIZED' : ''} (SELECT * FROM que_jobs WHERE id IN (#{ids.join(', ')}))
             SELECT * FROM jobs WHERE pg_try_advisory_lock(id)
           SQL
 

data/lib/que/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Que
-  VERSION = '1.0.0.beta4'
+  VERSION = '1.0.0.beta5'
 end

data/lib/que/worker.rb

@@ -54,10 +54,17 @@ module Que
     private
 
     def work_loop
-      # Blocks until a job of the appropriate priority is available. If the
-      # queue is shutting down this will return nil, which breaks the loop and
+      # Blocks until a job of the appropriate priority is available.
+      # `fetch_next_metajob` normally returns a job to be processed.
+      # If the queue is shutting down it will return false, which breaks the loop and
       # lets the thread finish.
-      while metajob = fetch_next_metajob
+      while (metajob = fetch_next_metajob) != false
+        # If metajob is nil instead of false, we've hit a rare race condition where
+        # there was a job in the buffer when the worker code checked, but the job was
+        # picked up by the time we got around to shifting it off the buffer.
+        # Letting this case go unhandled leads to worker threads exiting pre-maturely, so
+        # we check explicitly and continue the loop.
+        next if metajob.nil?
         id = metajob.id
 
         Que.internal_log(:worker_received_job, self) { {id: id} }

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: que
 version: !ruby/object:Gem::Version
-  version: 1.0.0.beta4
+  version: 1.0.0.beta5
 platform: ruby
 authors:
 - Chris Hanks
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2020-01-17 00:00:00.000000000 Z
+date: 2021-12-23 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -32,7 +32,7 @@ executables:
 extensions: []
 extra_rdoc_files: []
 files:
-- ".github/workflows/ruby.yml"
+- ".github/workflows/tests.yml"
 - ".gitignore"
 - CHANGELOG.1.0.beta.md
 - CHANGELOG.md
@@ -99,7 +99,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: 1.3.1
 requirements: []
-rubygems_version: 3.0.3
+rubygems_version: 3.1.6
 signing_key: 
 specification_version: 4
 summary: A PostgreSQL-based Job Queue