que-scheduler 3.2.7 → 3.4.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1085b028076fd5380d046d26f49dbc42004e0a0c9a0576e8e92cff93726e980f
-  data.tar.gz: 61c315d358848616d9bdc74a6704a011e0ca6165ebd5fce104eeac7a1ad830ec
+  metadata.gz: a3dcdf3dc34be0c572a49822ddad6b35464ee30f4424e0823c04008ae0b4facb
+  data.tar.gz: d46bdd71340462e52ea918aea603cecf27786e9e014c49c25c1459bc6181b383
 SHA512:
-  metadata.gz: b11700cb20b528d2cac6b00960736e1107277163d80b22f68eb498fe00b5530260c6f89e1f6c6981f982af33b73bf6d7ebf8fa9273e35fd6495fb8cb0f516aa1
-  data.tar.gz: aa6b0ca104cbe5daf013c748d854df1e9755a065a7183ef9182783d155707272fbd27d89b58c8097b16ba7368b42f3c68f5802756f4f012851c2392bc36a8640
+  metadata.gz: e1bf5bde9c9811a6675a7c3ac5ee983e47131ab9dda803f3961bbd272b64c4f7a79e601ffd5db671709ef36eb77e10552ad7c91b3719b0b51ac056d34565a23f
+  data.tar.gz: c60d779f6b2bc7e0f27d6ad8f09549d9a8767aca647ae0330f5e59f95b6a6d013537b34ed2bf1e290e2485843d9aa0a4018871deadf1352b57571b2380ce6629

data/README.md

@@ -22,12 +22,13 @@ needs to be run, enqueueing those jobs, then enqueueing itself to check again la
 look for it is `config/que_schedule.yml`. They are essentially the same as resque-scheduler
 files, but with additional features.
 
-1. Add a migration to start the job scheduler and prepare the audit table.
+1. Add a migration to start the job scheduler and prepare the audit table. Note that this migration 
+   will fail if Que is set to execute jobs synchronously, i.e. `Que::Job.run_synchronously = true`.
 
     ```ruby
     class CreateQueSchedulerSchema < ActiveRecord::Migration
       def change
-        Que::Scheduler::Migrations.migrate!(version: 4)
+        Que::Scheduler::Migrations.migrate!(version: 5)
       end
     end
     ```
@@ -65,10 +66,21 @@ BatchOrders:
   cron: "0 0 * * *"
   priority: 25
   
-# Specify job arguments
+# Specify array job arguments
 SendOrders:
   cron: "0 0 * * *"
   args: ['open']
+
+# Specify hash job arguments. Note, this appears as a single hash to `run`, not as kwargs.
+SendPreorders:
+  cron: "0 0 * * *"
+  args:
+    order_type: special
+  
+# Specify a single nil argument
+SendPostorders:
+  cron: "0 0 * * *"
+  args: ~ # See https://stackoverflow.com/a/51990876/1267203
   
 # Use simpler cron syntax
 SendBilling:
@@ -136,8 +148,54 @@ end
 
 ## Scheduler Audit
 
-An audit table _que_scheduler_audit_ is written to by the scheduler to keep a history of what jobs 
-were enqueued when. It is created by the included migration tasks.
+An audit table `que_scheduler_audit` is written to by the scheduler to keep a history of when the 
+scheduler ran to calculate what was necessary to run (if anything). It is created by the included 
+migration tasks.
+
+Additionally, there is the audit table `que_scheduler_audit_enqueued`. This logs every job that 
+the scheduler enqueues.
+
+When there is a major version (breaking) change, a migration should be run in. The version of the 
+migration proceeds at a faster rate than the version of the gem. To run in all the migrations required
+up to a number, just migrate to that number with one line, and it will perform all the intermediary steps. 
+
+ie, This will perform all migrations necessary up to the latest version, skipping any already 
+performed.
+
+```ruby
+class CreateQueSchedulerSchema < ActiveRecord::Migration
+  def change
+    Que::Scheduler::Migrations.migrate!(version: 5)
+  end
+end
+```
+
+The changes in past migrations were: 
+
+| Version | Changes                                                                         |
+|:-------:|---------------------------------------------------------------------------------|
+|    1    | Enqueued the main Que::Scheduler. This is the job that performs the scheduling. |
+|    2    | Added the audit table `que_scheduler_audit`.                                    |
+|    3    | Added the audit table `que_scheduler_audit_enqueued`.                           |
+|    4    | Updated the the audit tables to use bigints                                     |
+|    5    | Dropped an unnecessary index                                                    |
+
+## Built in optional job for audit clear down
+
+que-scheduler comes with the `QueSchedulerAuditClearDownJob` job built in that you can optionally
+schedule to clear down audit rows if you don't need to retain them indefinitely. You should add this
+to your own scheduler config yaml.
+
+For example:
+
+```yaml
+# This will clear down the oldest que-scheduler audit rows. Since que-scheduler
+# runs approximately every minute, 129600 is 90 days.
+Que::Scheduler::Jobs::QueSchedulerAuditClearDownJob:
+  cron: "0 0 * * *"
+  args:
+    retain_row_count: 129600
+```
 
 ## HA Redundancy and DB restores
 
@@ -172,23 +230,6 @@ then reschedules itself. The flow is as follows:
 1. After a deploy that changes the schedule, the job notices any new jobs to schedule, and knows which
    ones to forget. It does not need to be re-enqueued or restarted.
    
-## DB Migrations
-
-When there is a major version (breaking) change, a migration should be run in. The version of the 
-migration proceeds at a faster rate than the version of the gem. To run in all the migrations required
-up to a number, just migrate to that number with one line, and it will perform all the intermediary steps. 
-
-ie, `Que::Scheduler::Migrations.migrate!(version: 4)` will perform all migrations necessary to 
-reach migration version `4`.
-
-As of migration `4`, two elements are added to the DB for que-scheduler to run. 
-
-1. The first is the scheduler job itself, which runs forever, re-enqueuing itself to performs its 
-   duties.
-1. The second part comprises the audit table `que_scheduler_audit` and the "enqueued" table 
-  `que_scheduler_audit_enqueued`. The first tracks when the scheduler calculated what was necessary to run 
-  (if anything). The second then logs every job that the scheduler enqueues. 
-
 ## Testing Configuration
 
 You can add tests to validate your configuration during the spec phase. This will perform a variety 
@@ -234,3 +275,6 @@ This gem was inspired by the makers of the excellent [Que](https://github.com/ch
 
 * @jish
 * @joehorsnell
+* @bnauta
+* @papodaca
+* @krzyzak

data/lib/que-scheduler.rb

@@ -1,3 +1,3 @@
 # rubocop:disable Naming/FileName
-require 'que/scheduler'
+require "que/scheduler"
 # rubocop:enable Naming/FileName

data/lib/que/scheduler.rb

@@ -1,7 +1,8 @@
-require 'que/scheduler/version'
-require 'que/scheduler/version_support'
-require 'que/scheduler/config'
-require 'que/scheduler/scheduler_job'
-require 'que/scheduler/db'
-require 'que/scheduler/audit'
-require 'que/scheduler/migrations'
+require "que/scheduler/version"
+require "que/scheduler/version_support"
+require "que/scheduler/config"
+require "que/scheduler/scheduler_job"
+require "que/scheduler/db"
+require "que/scheduler/audit"
+require "que/scheduler/migrations"
+require "que/scheduler/jobs/que_scheduler_audit_clear_down_job"

data/lib/que/scheduler/audit.rb

@@ -1,10 +1,12 @@
 # frozen_string_literal: true
 
+require_relative "to_enqueue"
+
 module Que
   module Scheduler
     module Audit
-      TABLE_NAME = 'que_scheduler_audit'
-      ENQUEUED_TABLE_NAME = 'que_scheduler_audit_enqueued'
+      TABLE_NAME = "que_scheduler_audit"
+      ENQUEUED_TABLE_NAME = "que_scheduler_audit_enqueued"
       INSERT_AUDIT = %{
         INSERT INTO #{TABLE_NAME} (scheduler_job_id, executed_at)
         VALUES ($1::bigint, $2::timestamptz)
@@ -24,11 +26,10 @@ module Que
         def append(scheduler_job_id, executed_at, enqueued_jobs)
           ::Que::Scheduler::VersionSupport.execute(INSERT_AUDIT, [scheduler_job_id, executed_at])
           enqueued_jobs.each do |j|
-            attrs = Que::Scheduler::VersionSupport.job_attributes(j)
             inserted = ::Que::Scheduler::VersionSupport.execute(
               INSERT_AUDIT_ENQUEUED,
               [scheduler_job_id] +
-                attrs.values_at(:job_class, :queue, :priority, :args, :job_id, :run_at)
+                j.values_at(:job_class, :queue, :priority, :args, :job_id, :run_at)
             )
             raise "Cannot save audit row #{scheduler_job_id} #{executed_at} #{j}" if inserted.empty?
           end

data/lib/que/scheduler/config.rb

@@ -1,5 +1,5 @@
-require 'que'
-require_relative 'version_support'
+require "que"
+require_relative "version_support"
 
 module Que
   module Scheduler
@@ -21,7 +21,7 @@ module Que
 end
 
 Que::Scheduler.configure do |config|
-  config.schedule_location = ENV.fetch('QUE_SCHEDULER_CONFIG_LOCATION', 'config/que_schedule.yml')
+  config.schedule_location = ENV.fetch("QUE_SCHEDULER_CONFIG_LOCATION", "config/que_schedule.yml")
   config.transaction_adapter = ::Que.method(:transaction)
   config.que_scheduler_queue = Que::Scheduler::VersionSupport.default_scheduler_queue
 end

data/lib/que/scheduler/db.rb

@@ -1,13 +1,13 @@
 # frozen_string_literal: true
 
-require_relative 'config'
+require_relative "config"
 
 module Que
   module Scheduler
     module Db
       SCHEDULER_COUNT_SQL =
         "SELECT COUNT(*) FROM que_jobs WHERE job_class = 'Que::Scheduler::SchedulerJob'"
-      NOW_SQL = 'SELECT now()'
+      NOW_SQL = "SELECT now()"
 
       class << self
         def count_schedulers

data/lib/que/scheduler/defined_job.rb

@@ -1,6 +1,5 @@
-require 'hashie'
-require 'fugit'
-require 'backports/2.4.0/hash/compact'
+require "hashie"
+require "fugit"
 
 # This is the definition of one scheduleable job in the que-scheduler config yml file.
 module Que
@@ -10,23 +9,23 @@ module Que
 
       DEFINED_JOB_TYPES = [
         DEFINED_JOB_TYPE_DEFAULT = :default,
-        DEFINED_JOB_TYPE_EVERY_EVENT = :every_event
+        DEFINED_JOB_TYPE_EVERY_EVENT = :every_event,
       ].freeze
 
-      property :name, required: true
-      property :job_class, required: true, transform_with: lambda { |v|
-        job_class = Object.const_get(v)
-        job_class < Que::Job ? job_class : err_field(:job_class, v)
-      }
-      property :cron, required: true, transform_with: lambda { |v|
-        Fugit::Cron.parse(v) || err_field(:cron, v)
-      }
-      property :queue, transform_with: ->(v) { v.is_a?(String) ? v : err_field(:queue, v) }
-      property :priority, transform_with: ->(v) { v.is_a?(Integer) ? v : err_field(:priority, v) }
-      property :args
-      property :schedule_type, default: DEFINED_JOB_TYPE_DEFAULT, transform_with: lambda { |v|
-        v.to_sym.tap { |vs| DEFINED_JOB_TYPES.include?(vs) || err_field(:schedule_type, v) }
-      }
+      property :name
+      property :job_class, transform_with: ->(v) { Object.const_get(v) }
+      property :cron, transform_with: ->(v) { Fugit::Cron.parse(v) }
+      property :queue
+      property :priority
+      property :args_array
+      property :schedule_type, default: DEFINED_JOB_TYPE_DEFAULT
+
+      class << self
+        def create(options)
+          defined_job = new(options.compact)
+          defined_job.freeze.tap { |dj| dj.validate(options) }
+        end
+      end
 
       # Given a "last time", return the next Time the event will occur, or nil if it
       # is after "to".
@@ -49,36 +48,80 @@ module Que
         generate_to_enqueue_list(missed_times)
       end
 
-      class << self
-        private
+      def validate(options)
+        validate_fields_presence(options)
+        validate_fields_types(options)
+        validate_job_class_related(options)
+      end
+
+      private
 
-        def err_field(field, value)
-          schedule = Que::Scheduler.configuration.schedule_location
-          raise "Invalid #{field} '#{value}' in que-scheduler schedule #{schedule}"
+      # rubocop:disable Style/GuardClause This reads better as a conditional
+      def validate_fields_types(options)
+        unless queue.nil? || queue.is_a?(String)
+          err_field(:queue, options, "queue must be a string")
+        end
+        unless priority.nil? || priority.is_a?(Integer)
+          err_field(:priority, options, "priority must be an integer")
+        end
+        unless DEFINED_JOB_TYPES.include?(schedule_type)
+          err_field(:schedule_type, options, "Not in #{DEFINED_JOB_TYPES}")
         end
       end
+      # rubocop:enable Style/GuardClause
 
-      private
+      def validate_fields_presence(options)
+        err_field(:name, options, "name must be present") if name.nil?
+        err_field(:job_class, options, "job_class must be present") if job_class.nil?
+        # An invalid cron is nil
+        err_field(:cron, options, "cron must be present") if cron.nil?
+      end
+
+      def validate_job_class_related(options)
+        # Only support known job engines
+        unless Que::Scheduler::ToEnqueue.valid_job_class?(job_class)
+          err_field(:job_class, options, "Job #{job_class} was not a supported job type")
+        end
+
+        # queue name is only supported for a subrange of ActiveJob versions. Print this out as a
+        # warning.
+        if queue &&
+           Que::Scheduler::ToEnqueue.active_job_sufficient_version? &&
+           job_class < ::ActiveJob::Base &&
+           Que::Scheduler::ToEnqueue.active_job_version < Gem::Version.create("6.0.3")
+          puts <<-ERR
+            WARNING from que-scheduler....
+            Between versions 4.2.3 and 6.0.2 (inclusive) Rails did not support setting queue names
+            on que jobs with ActiveJob, so que-scheduler cannot support it.
+            See removed in Rails 4.2.3
+              https://github.com/rails/rails/pull/19498
+            And readded in Rails 6.0.3
+              https://github.com/rails/rails/pull/38635
+
+            Please remove all "queue" keys from ActiveJobs defined in the que-scheduler.yml config.
+            Specifically #{queue} for job #{name}.
+          ERR
+        end
+      end
 
-      class ToEnqueue < Hashie::Dash
-        property :args, required: true, default: []
-        property :queue
-        property :priority
-        property :job_class, required: true
+      def err_field(field, options, reason = "")
+        schedule = Que::Scheduler.configuration.schedule_location
+        value = options[field]
+        raise "Invalid #{field} '#{value}' for '#{name}' in que-scheduler schedule #{schedule}.\n" \
+              "#{reason}"
       end
 
       def generate_to_enqueue_list(missed_times)
         return [] if missed_times.empty?
 
         options = to_h.slice(:args, :queue, :priority, :job_class).compact
-        args_array = args.is_a?(Array) ? args : Array(args)
 
         if schedule_type == DefinedJob::DEFINED_JOB_TYPE_EVERY_EVENT
           missed_times.map do |time_missed|
-            ToEnqueue.new(options.merge(args: [time_missed] + args_array))
+            ToEnqueue.create(options.merge(args: [time_missed] + args_array))
           end
         else
-          [ToEnqueue.new(options.merge(args: args_array))]
+          [ToEnqueue.create(options.merge(args: args_array))]
         end
       end
     end

data/lib/que/scheduler/enqueueing_calculator.rb

@@ -1,4 +1,4 @@
-require 'fugit'
+require "fugit"
 
 module Que
   module Scheduler

data/lib/que/scheduler/jobs/que_scheduler_audit_clear_down_job.rb

@@ -0,0 +1,45 @@
+require "que"
+
+# This job can optionally be scheduled to clear down the que-scheduler audit log if it
+# isn't required in the long term.
+module Que
+  module Scheduler
+    module Jobs
+      class QueSchedulerAuditClearDownJob < Que::Job
+        class << self
+          def build_sql(table_name)
+            <<-SQL
+              WITH deleted AS (
+                DELETE FROM #{table_name}
+                WHERE scheduler_job_id <= (
+                  SELECT scheduler_job_id FROM que_scheduler_audit
+                  ORDER BY scheduler_job_id DESC
+                  LIMIT 1 OFFSET $1
+                ) RETURNING *
+              ) SELECT count(*) FROM deleted;
+            SQL
+          end
+        end
+
+        DELETE_AUDIT_ENQUEUED_SQL = build_sql("que_scheduler_audit_enqueued").freeze
+        DELETE_AUDIT_SQL = build_sql("que_scheduler_audit").freeze
+
+        # Very low priority
+        Que::Scheduler::VersionSupport.set_priority(self, 100)
+
+        def run(options)
+          retain_row_count = options.fetch(:retain_row_count)
+          Que::Scheduler::Db.transaction do
+            # This may delete zero or more than `retain_row_count` depending on if anything was
+            # scheduled in each of the past schedule runs
+            Que::Scheduler::VersionSupport.execute(DELETE_AUDIT_ENQUEUED_SQL, [retain_row_count])
+            # This will delete all but `retain_row_count` oldest rows
+            count = Que::Scheduler::VersionSupport.execute(DELETE_AUDIT_SQL, [retain_row_count])
+            log = "#{self.class} cleared down #{count.first.fetch(:count)} rows"
+            ::Que.log(event: "que-scheduler".to_sym, message: log)
+          end
+        end
+      end
+    end
+  end
+end

data/lib/que/scheduler/migrations/5/down.sql

@@ -0,0 +1 @@
+CREATE UNIQUE INDEX index_que_scheduler_audit_on_scheduler_job_id ON que_scheduler_audit USING btree (scheduler_job_id);

data/lib/que/scheduler/migrations/5/up.sql

@@ -0,0 +1 @@
+DROP INDEX index_que_scheduler_audit_on_scheduler_job_id;

data/lib/que/scheduler/schedule.rb

@@ -1,4 +1,4 @@
-require_relative 'defined_job'
+require_relative "defined_job"
 
 module Que
   module Scheduler
@@ -24,20 +24,36 @@ module Que
           end.to_h
         end
 
-        private
-
         def hash_item_to_defined_job(name, defined_job_hash)
-          Que::Scheduler::DefinedJob.new(
-            {
-              name: name,
-              job_class: defined_job_hash['class'] || name,
-              queue: defined_job_hash['queue'],
-              args: defined_job_hash['args'],
-              priority: defined_job_hash['priority'],
-              cron: defined_job_hash['cron'],
-              schedule_type: defined_job_hash['schedule_type'],
-            }.compact
-          ).freeze
+          # Que stores arguments as a json array. If the args we have to provide are already an
+          # array we can can simply pass them through. If it is a single non-nil value, then we make
+          # an array with one item which is that value (this includes if it is a hash). It could
+          # also be a single nil value.
+          args_array =
+            if !defined_job_hash.key?("args")
+              # No args were requested
+              []
+            else
+              args = defined_job_hash["args"]
+              if args.is_a?(Array)
+                # An array of args was requested
+                args
+              else
+                # A single value, a nil, or a hash was requested. que expects this to
+                # be enqueued as an array of 1 item
+                [args]
+              end
+            end
+
+          Que::Scheduler::DefinedJob.create(
+            name: name,
+            job_class: defined_job_hash["class"] || name,
+            queue: defined_job_hash["queue"],
+            args_array: args_array,
+            priority: defined_job_hash["priority"],
+            cron: defined_job_hash["cron"],
+            schedule_type: defined_job_hash["schedule_type"]&.to_sym
+          )
         end
       end
     end

data/lib/que/scheduler/scheduler_job.rb

@@ -1,10 +1,13 @@
-require 'que'
+require "que"
+require "active_support"
+require "active_support/core_ext/time"
 
-require_relative 'schedule'
-require_relative 'enqueueing_calculator'
-require_relative 'scheduler_job_args'
-require_relative 'state_checks'
-require_relative 'version_support'
+require_relative "schedule"
+require_relative "enqueueing_calculator"
+require_relative "scheduler_job_args"
+require_relative "state_checks"
+require_relative "to_enqueue"
+require_relative "version_support"
 
 # The main job that runs every minute, determining what needs to be enqueued, enqueues the required
 # jobs, then re-enqueues itself.
@@ -13,8 +16,8 @@ module Que
     class SchedulerJob < Que::Job
       SCHEDULER_FREQUENCY = 60
 
-      # Always highest possible priority.
       Que::Scheduler::VersionSupport.set_priority(self, 0)
+      Que::Scheduler::VersionSupport.apply_retry_semantics(self)
 
       def run(options = nil)
         Que::Scheduler::Db.transaction do
@@ -22,46 +25,36 @@ module Que
 
           scheduler_job_args = SchedulerJobArgs.build(options)
           logs = ["que-scheduler last ran at #{scheduler_job_args.last_run_time}."]
-
           result = EnqueueingCalculator.parse(Scheduler.schedule.values, scheduler_job_args)
           enqueued_jobs = enqueue_required_jobs(result, logs)
           enqueue_self_again(
             scheduler_job_args, scheduler_job_args.as_time, result.job_dictionary, enqueued_jobs
           )
-
           # Only now we're sure nothing errored, log the results
-          logs.each { |str| ::Que.log(event: 'que-scheduler'.to_sym, message: str) }
+          logs.each { |str| ::Que.log(event: "que-scheduler".to_sym, message: str) }
           destroy
         end
       end
 
-      def enqueue_required_jobs(result, logs)
-        result.missed_jobs.map do |to_enqueue|
-          job_class = to_enqueue.job_class
-          args = to_enqueue.args
-          remaining_hash = to_enqueue.except(:job_class, :args)
-          enqueued_job =
-            if args.is_a?(Hash)
-              job_class.enqueue(args.merge(remaining_hash))
-            else
-              job_class.enqueue(*args, remaining_hash)
-            end
-          check_enqueued_job(enqueued_job, job_class, args, logs)
+      def enqueue_required_jobs(calculator_result, logs)
+        calculator_result.missed_jobs.map do |to_enqueue|
+          to_enqueue.enqueue.tap do |enqueued_job|
+            check_enqueued_job(to_enqueue, enqueued_job, logs)
+          end
         end.compact
       end
 
       private
 
-      def check_enqueued_job(enqueued_job, job_class, args, logs)
-        if enqueued_job.is_a?(Que::Job)
-          job_id = Que::Scheduler::VersionSupport.job_attributes(enqueued_job).fetch(:job_id)
-          logs << "que-scheduler enqueueing #{job_class} #{job_id} with args: #{args}"
-          enqueued_job
-        else
-          # This can happen if a middleware nixes the enqueue call
-          logs << "que-scheduler called enqueue on #{job_class} but did not receive a #{Que::Job}"
-          nil
-        end
+      def check_enqueued_job(to_enqueue, enqueued_job, logs)
+        logs << if enqueued_job.present?
+                  "que-scheduler enqueueing #{enqueued_job.job_class} " \
+                              "#{enqueued_job.job_id} with args: #{enqueued_job.args}"
+                else
+                  # This can happen if a middleware nixes the enqueue call
+                  "que-scheduler called enqueue on #{to_enqueue.job_class} " \
+                              "but it reported no job was scheduled. Has `enqueue` been overridden?"
+                end
       end
 
       def enqueue_self_again(scheduler_job_args, last_full_execution, job_dictionary, enqueued_jobs)
@@ -79,8 +72,8 @@ module Que
         )
 
         # rubocop:disable Style/GuardClause This reads better as a conditional
-        unless check_enqueued_job(enqueued_job, SchedulerJob, {}, [])
-          raise 'SchedulerJob could not self-schedule. Has `.enqueue` been monkey patched?'
+        unless Que::Scheduler::VersionSupport.job_attributes(enqueued_job).fetch(:job_id)
+          raise "SchedulerJob could not self-schedule. Has `.enqueue` been monkey patched?"
         end
         # rubocop:enable Style/GuardClause
       end

data/lib/que/scheduler/scheduler_job_args.rb

@@ -1,6 +1,6 @@
-require 'hashie'
-require 'active_support'
-require 'active_support/time_with_zone'
+require "hashie"
+require "active_support"
+require "active_support/time_with_zone"
 
 # These are the args that are used for this particular run of the scheduler.
 module Que

data/lib/que/scheduler/state_checks.rb

@@ -1,6 +1,6 @@
-require_relative 'audit'
-require_relative 'db'
-require_relative 'migrations'
+require_relative "audit"
+require_relative "db"
+require_relative "migrations"
 
 module Que
   module Scheduler
@@ -17,9 +17,22 @@ module Que
           db_version = Que::Scheduler::Migrations.db_version
           return if db_version == Que::Scheduler::Migrations::MAX_VERSION
 
+          sync_err =
+            if Que::Scheduler::VersionSupport.running_synchronously? && db_version.zero?
+              code = Que::Scheduler::VersionSupport.running_synchronously_code?
+              <<-ERR_SYNC
+                You currently have Que to run in synchronous mode using
+                #{code}, so it is most likely this error
+                has happened during an initial migration. You should disable synchronous mode and
+                try again. Note, que-scheduler uses "forward time" scheduled jobs, so will not work
+                in synchronous mode.
+
+              ERR_SYNC
+            end
+
           raise(<<-ERR)
             The que-scheduler db migration state was found to be #{db_version}. It should be #{Que::Scheduler::Migrations::MAX_VERSION}.
-
+            #{sync_err}
             que-scheduler adds some tables to the DB to provide an audit history of what was
             enqueued when, and with what options and arguments. The structure of these tables is
             versioned, and should match that version required by the gem.
@@ -42,6 +55,9 @@ module Que
                 Que::Scheduler::Migrations.migrate!(version: #{Que::Scheduler::Migrations::MAX_VERSION})
               end
             end
+
+            It is also possible that you are running a migration with Que set up to execute jobs
+            synchronously. This will fail as que-scheduler needs the above tables to work.
           ERR
         end
 

data/lib/que/scheduler/to_enqueue.rb

@@ -0,0 +1,157 @@
+require "que"
+
+# This module uses polymorphic dispatch to centralise the differences between supporting Que::Job
+# and other job systems.
+module Que
+  module Scheduler
+    class ToEnqueue < Hashie::Dash
+      property :args, required: true, default: []
+      property :queue
+      property :priority
+      property :run_at, required: true
+      property :job_class, required: true
+
+      class << self
+        def create(options)
+          type_from_job_class(options.fetch(:job_class)).new(
+            options.merge(run_at: Que::Scheduler::Db.now)
+          )
+        end
+
+        def valid_job_class?(job_class)
+          type_from_job_class(job_class).present?
+        end
+
+        def active_job_version
+          Gem.loaded_specs["activejob"]&.version
+        end
+
+        def active_job_sufficient_version?
+          # ActiveJob 4.x does not support job_ids correctly
+          # https://github.com/rails/rails/pull/20056/files
+          active_job_version && active_job_version > Gem::Version.create("5")
+        end
+
+        def active_job_version_supports_queues?
+          # Supporting queue name in ActiveJob was removed in Rails 4.2.3
+          # https://github.com/rails/rails/pull/19498
+          # and readded in Rails 6.0.3
+          # https://github.com/rails/rails/pull/38635
+          ToEnqueue.active_job_version && ToEnqueue.active_job_version >=
+            Gem::Version.create("6.0.3")
+        end
+
+        private
+
+        def type_from_job_class(job_class)
+          types.each do |type, implementation|
+            return implementation if job_class < type
+          end
+          nil
+        end
+
+        def types
+          @types ||=
+            begin
+              hash = {
+                ::Que::Job => QueJobType,
+              }
+              hash[::ActiveJob::Base] = ActiveJobType if ToEnqueue.active_job_sufficient_version?
+              hash
+            end
+        end
+      end
+    end
+
+    # For jobs of type Que::Job
+    class QueJobType < ToEnqueue
+      def enqueue
+        job_settings = to_h.slice(:queue, :priority, :run_at).compact
+        job =
+          if args.is_a?(Hash)
+            job_class.enqueue(**args.merge(job_settings))
+          else
+            job_class.enqueue(*args, **job_settings)
+          end
+
+        return nil if job.nil? || !job # nil in Rails < 6.1, false after.
+
+        # Now read the just inserted job back out of the DB to get the actual values that will
+        # be used when the job is worked.
+        values = Que::Scheduler::VersionSupport.job_attributes(job).slice(
+          :args, :queue, :priority, :run_at, :job_class, :job_id
+        )
+        EnqueuedJobType.new(values)
+      end
+    end
+
+    # For jobs of type ActiveJob
+    class ActiveJobType < ToEnqueue
+      def enqueue
+        job = enqueue_active_job
+
+        return nil if job.nil? || !job # nil in Rails < 6.1, false after.
+
+        enqueued_values = calculate_enqueued_values(job)
+        EnqueuedJobType.new(enqueued_values)
+      end
+
+      def calculate_enqueued_values(job)
+        # Now read the just inserted job back out of the DB to get the actual values that will
+        # be used when the job is worked.
+        data = JSON.parse(job.to_json, symbolize_names: true)
+
+        # ActiveJob scheduled_at is returned as a float, where we want a Time for consistency
+        scheduled_at =
+          begin
+            scheduled_at_float = data[:scheduled_at]
+            scheduled_at_float ? Time.zone.at(scheduled_at_float) : nil
+          end
+
+        # Rails didn't support queues for ActiveJob for a while
+        used_queue = data[:queue_name] if ToEnqueue.active_job_version_supports_queues?
+
+        # We can't get the priority out of the DB, as the returned `job` doesn't give us access
+        # to the underlying ActiveJob that was scheduled. We have no option but to assume
+        # it was what we told it to use. If no priority was specified, we must assume it was
+        # the Que default, which is 100 t.ly/1jRK5
+        assume_used_priority = priority.nil? ? 100 : priority
+
+        {
+          args: data.fetch(:arguments),
+          queue: used_queue,
+          priority: assume_used_priority,
+          run_at: scheduled_at,
+          job_class: job_class.to_s,
+          job_id: data.fetch(:provider_job_id),
+        }
+      end
+
+      def enqueue_active_job
+        job_settings = {
+          priority: priority,
+          wait_until: run_at,
+          queue: queue || Que::Scheduler::VersionSupport.default_scheduler_queue,
+        }.compact
+
+        job_class_set = job_class.set(**job_settings)
+        if args.is_a?(Hash)
+          job_class_set.perform_later(**args)
+        else
+          job_class_set.perform_later(*args)
+        end
+      end
+    end
+
+    # A value object returned after a job has been enqueued. This is necessary as Que (normal) and
+    # ActiveJob return very different objects from the `enqueue` call.
+    class EnqueuedJobType < Hashie::Dash
+      property :args
+      property :queue
+      property :priority
+      property :run_at, required: true
+      property :job_class, required: true
+      property :job_id, required: true
+    end
+  end
+end

data/lib/que/scheduler/version.rb

@@ -1,5 +1,5 @@
 module Que
   module Scheduler
-    VERSION = '3.2.7'.freeze
+    VERSION = "3.4.2".freeze
   end
 end

data/lib/que/scheduler/version_support.rb

@@ -1,24 +1,41 @@
-require 'que'
+require "que"
 
 # The purpose of this module is to centralise the differences when supporting both que 0.x and
 # 1.x with the same gem.
 module Que
   module Scheduler
     module VersionSupport
+      RETRY_PROC = proc { |count|
+        # Maximum one hour, otherwise use the default backoff
+        count > 7 ? (60 * 60) : ((count**4) + 3)
+      }
+
       class << self
+        # Ensure que-scheduler runs at the highest priority. This is because its priority is a
+        # the top of all jobs it enqueues.
         def set_priority(context, priority)
           if zero_major?
-            context.instance_variable_set('@priority', priority)
+            context.instance_variable_set("@priority", priority)
           else
             context.priority = priority
           end
         end
 
+        # Ensure the job runs at least once an hour when it is backing off due to errors
+        def apply_retry_semantics(context)
+          if zero_major?
+            context.instance_variable_set("@retry_interval", RETRY_PROC)
+          else
+            context.maximum_retry_count = 1 << 128 # Heat death of universe
+            context.retry_interval = RETRY_PROC
+          end
+        end
+
         def job_attributes(enqueued_job)
           if zero_major?
-            enqueued_job.attrs.transform_keys(&:to_sym)
+            enqueued_job.attrs.to_h.transform_keys(&:to_sym)
           else
-            enqueued_job.que_attrs.transform_keys(&:to_sym).tap do |hash|
+            enqueued_job.que_attrs.to_h.transform_keys(&:to_sym).tap do |hash|
               hash[:job_id] = hash.delete(:id)
             end
           end
@@ -31,22 +48,26 @@ module Que
         end
 
         def default_scheduler_queue
-          if zero_major?
-            ''
-          else
-            Que::DEFAULT_QUEUE
-          end
+          zero_major? ? "" : Que::DEFAULT_QUEUE
+        end
+
+        def running_synchronously?
+          zero_major? ? (Que.mode == :sync) : Que.run_synchronously
+        end
+
+        def running_synchronously_code?
+          zero_major? ? "Que.mode == :sync" : "Que.run_synchronously = true"
         end
 
         def zero_major?
           # This is the only way to handle beta releases too
-          @zero_major ||= Gem.loaded_specs['que'].version.to_s.split('.').first.to_i.zero?
+          @zero_major ||= Gem.loaded_specs["que"].version.to_s.split(".").first.to_i.zero?
         end
 
         private
 
         def normalise_array_of_hashes(array)
-          array.map { |row| row.transform_keys(&:to_sym) }
+          array.map { |row| row.to_h.transform_keys(&:to_sym) }
         end
       end
     end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: que-scheduler
 version: !ruby/object:Gem::Version
-  version: 3.2.7
+  version: 3.4.2
 platform: ruby
 authors:
 - Harry Lascelles
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2019-10-19 00:00:00.000000000 Z
+date: 2020-09-17 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activesupport
@@ -24,20 +24,6 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '4.0'
-- !ruby/object:Gem::Dependency
-  name: backports
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '3.10'
-  type: :runtime
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '3.10'
 - !ruby/object:Gem::Dependency
   name: fugit
   requirement: !ruby/object:Gem::Requirement
@@ -62,16 +48,22 @@ dependencies:
   name: hashie
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
         version: '3'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '5'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
         version: '3'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '5'
 - !ruby/object:Gem::Dependency
   name: que
   requirement: !ruby/object:Gem::Requirement
@@ -81,7 +73,7 @@ dependencies:
         version: '0.12'
     - - "<="
       - !ruby/object:Gem::Version
-        version: 1.0.0.beta3
+        version: 1.0.0.beta4
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
@@ -91,7 +83,7 @@ dependencies:
         version: '0.12'
     - - "<="
       - !ruby/object:Gem::Version
-        version: 1.0.0.beta3
+        version: 1.0.0.beta4
 - !ruby/object:Gem::Dependency
   name: activerecord
   requirement: !ruby/object:Gem::Requirement
@@ -222,16 +214,16 @@ dependencies:
   name: reek
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - '='
+    - - ">="
       - !ruby/object:Gem::Version
-        version: 5.0.2
+        version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - '='
+    - - ">="
       - !ruby/object:Gem::Version
-        version: 5.0.2
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: rspec
   requirement: !ruby/object:Gem::Requirement
@@ -250,16 +242,30 @@ dependencies:
   name: rubocop
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - '='
       - !ruby/object:Gem::Version
-        version: 0.68.1
+        version: 0.84.0
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - '='
+      - !ruby/object:Gem::Version
+        version: 0.84.0
+- !ruby/object:Gem::Dependency
+  name: rubocop-rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
       - !ruby/object:Gem::Version
-        version: 0.68.1
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: sqlite3
   requirement: !ruby/object:Gem::Requirement
@@ -317,6 +323,7 @@ files:
 - lib/que/scheduler/db.rb
 - lib/que/scheduler/defined_job.rb
 - lib/que/scheduler/enqueueing_calculator.rb
+- lib/que/scheduler/jobs/que_scheduler_audit_clear_down_job.rb
 - lib/que/scheduler/migrations.rb
 - lib/que/scheduler/migrations/1/down.sql
 - lib/que/scheduler/migrations/1/up.sql
@@ -326,10 +333,13 @@ files:
 - lib/que/scheduler/migrations/3/up.sql
 - lib/que/scheduler/migrations/4/down.sql
 - lib/que/scheduler/migrations/4/up.sql
+- lib/que/scheduler/migrations/5/down.sql
+- lib/que/scheduler/migrations/5/up.sql
 - lib/que/scheduler/schedule.rb
 - lib/que/scheduler/scheduler_job.rb
 - lib/que/scheduler/scheduler_job_args.rb
 - lib/que/scheduler/state_checks.rb
+- lib/que/scheduler/to_enqueue.rb
 - lib/que/scheduler/version.rb
 - lib/que/scheduler/version_support.rb
 homepage: https://github.com/hlascelles/que-scheduler
@@ -356,8 +366,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project: 
-rubygems_version: 2.7.6.2
+rubygems_version: 3.0.3
 signing_key: 
 specification_version: 4
 summary: A cron scheduler for Que