que-scheduler 3.3.0 → 3.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5707036df23117e995d696388d17bfc0488eef9c7ba30211b1a3700e6f0fb94f
-  data.tar.gz: 3133e78529f3658c9509e0a1240189424caefc8219b7da035530e68fdbfb38c6
+  metadata.gz: 53cadc9fafa321124d4f52caa66503d8590cb1eaa11ee2ac86e0208c5d6a0fef
+  data.tar.gz: 2a5731c305089b58c3841f51b67c2dc698718e123a9b68395e92e1dfff37a693
 SHA512:
-  metadata.gz: 4609167d5adfc2e7b9c583d3cbece89dd34e8624512b0c8c257804c05934964a343336b192944ce2790fde4f4b6029173df2f76d28fe7daee72e0a1ae473bad2
-  data.tar.gz: d55658a7923c50010a8a38713dc25c0f08d182004cb5156a336921f55b35e91631edb042f5e76f194d58ed04245ea8f08c78dd64386c2479ee4bdb8ba9a4512a
+  metadata.gz: '087a32c4e9245c3bbaa26fb0e891f2b232c2617bad351d354eb502c0edb853a32ec786601a71e796fe5baa1b13f3691f1dc1c17f55d1b1973648101cd60602c7'
+  data.tar.gz: 1ba223766fa1d52b79b7ebcdcd534da366636cc306f538e6bc067830a2999b57913c1ade0f5a381287883222aedadc4aa380f9fb479223838468f7df3dfa5d73

data/README.md

@@ -22,12 +22,13 @@ needs to be run, enqueueing those jobs, then enqueueing itself to check again la
 look for it is `config/que_schedule.yml`. They are essentially the same as resque-scheduler
 files, but with additional features.
 
-1. Add a migration to start the job scheduler and prepare the audit table. Note that this migration will fail if Que is set to execute jobs synchronously, i.e. `Que::Job.run_synchronously = true`.
+1. Add a migration to start the job scheduler and prepare the audit table. Note that this migration 
+   will fail if Que is set to execute jobs synchronously, i.e. `Que::Job.run_synchronously = true`.
 
     ```ruby
     class CreateQueSchedulerSchema < ActiveRecord::Migration
       def change
-        Que::Scheduler::Migrations.migrate!(version: 4)
+        Que::Scheduler::Migrations.migrate!(version: 5)
       end
     end
     ```
@@ -65,10 +66,21 @@ BatchOrders:
   cron: "0 0 * * *"
   priority: 25
   
-# Specify job arguments
+# Specify array job arguments
 SendOrders:
   cron: "0 0 * * *"
   args: ['open']
+
+# Specify hash job arguments. Note, this appears as a single hash to `run`, not as kwargs.
+SendPreorders:
+  cron: "0 0 * * *"
+  args:
+    order_type: special
+  
+# Specify a single nil argument
+SendPostorders:
+  cron: "0 0 * * *"
+  args: ~ # See https://stackoverflow.com/a/51990876/1267203
   
 # Use simpler cron syntax
 SendBilling:
@@ -136,8 +148,54 @@ end
 
 ## Scheduler Audit
 
-An audit table _que_scheduler_audit_ is written to by the scheduler to keep a history of what jobs 
-were enqueued when. It is created by the included migration tasks.
+An audit table `que_scheduler_audit` is written to by the scheduler to keep a history of when the 
+scheduler ran to calculate what was necessary to run (if anything). It is created by the included 
+migration tasks.
+
+Additionally, there is the audit table `que_scheduler_audit_enqueued`. This logs every job that 
+the scheduler enqueues.
+
+When there is a major version (breaking) change, a migration should be run in. The version of the 
+migration proceeds at a faster rate than the version of the gem. To run in all the migrations required
+up to a number, just migrate to that number with one line, and it will perform all the intermediary steps. 
+
+ie, This will perform all migrations necessary up to the latest version, skipping any already 
+performed.
+
+    ```ruby
+    class CreateQueSchedulerSchema < ActiveRecord::Migration
+      def change
+        Que::Scheduler::Migrations.migrate!(version: 5)
+      end
+    end
+    ```
+
+The changes in past migrations were: 
+
+| Version | Changes                                                                         |
+|:-------:|---------------------------------------------------------------------------------|
+|    1    | Enqueued the main Que::Scheduler. This is the job that performs the scheduling. |
+|    2    | Added the audit table `que_scheduler_audit`.                                    |
+|    3    | Added the audit table `que_scheduler_audit_enqueued`.                           |
+|    4    | Updated the the audit tables to use bigints                                     |
+|    5    | Dropped an unnecessary index                                                    |
+
+## Built in optional job for audit clear down
+
+que-scheduler comes with the `QueSchedulerAuditClearDownJob` job built in that you can optionally
+schedule to clear down audit rows if you don't need to retain them indefinitely. You should add this
+to your own scheduler config yaml.
+
+For example:
+
+```yaml
+# This will clear down the oldest que-scheduler audit rows. Since que-scheduler
+# runs approximately every minute, 129600 is 90 days.
+Que::Scheduler::Jobs::QueSchedulerAuditClearDownJob:
+  cron: "0 0 * * *"
+  args:
+    retain_row_count: 129600
+```
 
 ## HA Redundancy and DB restores
 
@@ -172,23 +230,6 @@ then reschedules itself. The flow is as follows:
 1. After a deploy that changes the schedule, the job notices any new jobs to schedule, and knows which
    ones to forget. It does not need to be re-enqueued or restarted.
    
-## DB Migrations
-
-When there is a major version (breaking) change, a migration should be run in. The version of the 
-migration proceeds at a faster rate than the version of the gem. To run in all the migrations required
-up to a number, just migrate to that number with one line, and it will perform all the intermediary steps. 
-
-ie, `Que::Scheduler::Migrations.migrate!(version: 4)` will perform all migrations necessary to 
-reach migration version `4`.
-
-As of migration `4`, two elements are added to the DB for que-scheduler to run. 
-
-1. The first is the scheduler job itself, which runs forever, re-enqueuing itself to performs its 
-   duties.
-1. The second part comprises the audit table `que_scheduler_audit` and the "enqueued" table 
-  `que_scheduler_audit_enqueued`. The first tracks when the scheduler calculated what was necessary to run 
-  (if anything). The second then logs every job that the scheduler enqueues. 
-
 ## Testing Configuration
 
 You can add tests to validate your configuration during the spec phase. This will perform a variety 
@@ -236,3 +277,4 @@ This gem was inspired by the makers of the excellent [Que](https://github.com/ch
 * @joehorsnell
 * @bnauta
 * @papodaca
+* @krzyzak

data/lib/que/scheduler.rb

@@ -5,3 +5,4 @@ require 'que/scheduler/scheduler_job'
 require 'que/scheduler/db'
 require 'que/scheduler/audit'
 require 'que/scheduler/migrations'
+require 'que/scheduler/jobs/que_scheduler_audit_clear_down_job'

data/lib/que/scheduler/defined_job.rb

@@ -1,6 +1,5 @@
 require 'hashie'
 require 'fugit'
-require 'backports/2.4.0/hash/compact'
 
 # This is the definition of one scheduleable job in the que-scheduler config yml file.
 module Que
@@ -18,7 +17,7 @@ module Que
       property :cron, transform_with: ->(v) { Fugit::Cron.parse(v) }
       property :queue
       property :priority
-      property :args
+      property :args_array
       property :schedule_type, default: DEFINED_JOB_TYPE_DEFAULT
 
       class << self
@@ -116,7 +115,6 @@ module Que
         return [] if missed_times.empty?
 
         options = to_h.slice(:args, :queue, :priority, :job_class).compact
-        args_array = args.is_a?(Array) ? args : Array(args)
 
         if schedule_type == DefinedJob::DEFINED_JOB_TYPE_EVERY_EVENT
           missed_times.map do |time_missed|

data/lib/que/scheduler/jobs/que_scheduler_audit_clear_down_job.rb

@@ -0,0 +1,45 @@
+require 'que'
+
+# This job can optionally be scheduled to clear down the que-scheduler audit log if it
+# isn't required in the long term.
+module Que
+  module Scheduler
+    module Jobs
+      class QueSchedulerAuditClearDownJob < Que::Job
+        class << self
+          def build_sql(table_name)
+            <<-SQL
+              WITH deleted AS (
+                DELETE FROM #{table_name}
+                WHERE scheduler_job_id <= (
+                  SELECT scheduler_job_id FROM que_scheduler_audit
+                  ORDER BY scheduler_job_id DESC
+                  LIMIT 1 OFFSET $1
+                ) RETURNING *
+              ) SELECT count(*) FROM deleted;
+            SQL
+          end
+        end
+
+        DELETE_AUDIT_ENQUEUED_SQL = build_sql('que_scheduler_audit_enqueued').freeze
+        DELETE_AUDIT_SQL = build_sql('que_scheduler_audit').freeze
+
+        # Very low priority
+        Que::Scheduler::VersionSupport.set_priority(self, 100)
+
+        def run(options)
+          retain_row_count = options.fetch(:retain_row_count)
+          Que::Scheduler::Db.transaction do
+            # This may delete zero or more than `retain_row_count` depending on if anything was
+            # scheduled in each of the past schedule runs
+            Que::Scheduler::VersionSupport.execute(DELETE_AUDIT_ENQUEUED_SQL, [retain_row_count])
+            # This will delete all but `retain_row_count` oldest rows
+            count = Que::Scheduler::VersionSupport.execute(DELETE_AUDIT_SQL, [retain_row_count])
+            log = "#{self.class} cleared down #{count.first.fetch(:count)} rows"
+            ::Que.log(event: 'que-scheduler'.to_sym, message: log)
+          end
+        end
+      end
+    end
+  end
+end

data/lib/que/scheduler/migrations/5/down.sql

@@ -0,0 +1 @@
+CREATE UNIQUE INDEX index_que_scheduler_audit_on_scheduler_job_id ON que_scheduler_audit USING btree (scheduler_job_id);

data/lib/que/scheduler/migrations/5/up.sql

@@ -0,0 +1 @@
+DROP INDEX index_que_scheduler_audit_on_scheduler_job_id;

data/lib/que/scheduler/schedule.rb

@@ -24,14 +24,32 @@ module Que
           end.to_h
         end
 
-        private
-
         def hash_item_to_defined_job(name, defined_job_hash)
+          # Que stores arguments as a json array. If the args we have to provide are already an
+          # array we can can simply pass them through. If it is a single non-nil value, then we make
+          # an array with one item which is that value (this includes if it is a hash). It could
+          # also be a single nil value.
+          args_array =
+            if !defined_job_hash.key?('args')
+              # No args were requested
+              []
+            else
+              args = defined_job_hash['args']
+              if args.is_a?(Array)
+                # An array of args was requested
+                args
+              else
+                # A single value, a nil, or a hash was requested. que expects this to
+                # be enqueued as an array of 1 item
+                [args]
+              end
+            end
+
           Que::Scheduler::DefinedJob.create(
             name: name,
             job_class: defined_job_hash['class'] || name,
             queue: defined_job_hash['queue'],
-            args: defined_job_hash['args'],
+            args_array: args_array,
             priority: defined_job_hash['priority'],
             cron: defined_job_hash['cron'],
             schedule_type: defined_job_hash['schedule_type']&.to_sym

data/lib/que/scheduler/scheduler_job.rb

@@ -34,8 +34,8 @@ module Que
         end
       end
 
-      def enqueue_required_jobs(result, logs)
-        result.missed_jobs.map do |to_enqueue|
+      def enqueue_required_jobs(calculator_result, logs)
+        calculator_result.missed_jobs.map do |to_enqueue|
           to_enqueue.enqueue.tap do |enqueued_job|
             check_enqueued_job(to_enqueue, enqueued_job, logs)
           end

data/lib/que/scheduler/to_enqueue.rb

@@ -32,6 +32,15 @@ module Que
           active_job_version && active_job_version > Gem::Version.create('5')
         end
 
+        def active_job_version_supports_queues?
+          # Supporting queue name in ActiveJob was removed in Rails 4.2.3
+          # https://github.com/rails/rails/pull/19498
+          # and readded in Rails 6.0.3
+          # https://github.com/rails/rails/pull/38635
+          ToEnqueue.active_job_version && ToEnqueue.active_job_version >=
+            Gem::Version.create('6.0.3')
+        end
+
         private
 
         def type_from_job_class(job_class)
@@ -87,8 +96,6 @@ module Que
         EnqueuedJobType.new(enqueued_values)
       end
 
-      private
-
       def calculate_enqueued_values(job)
         # Now read the just inserted job back out of the DB to get the actual values that will
         # be used when the job is worked.
@@ -101,9 +108,8 @@ module Que
             scheduled_at_float ? Time.zone.at(scheduled_at_float) : nil
           end
 
-        # Rails doesn't support queues for ActiveJob
-        # https://github.com/rails/rails/pull/19498
-        used_queue = nil
+        # Rails didn't support queues for ActiveJob for a while
+        used_queue = data[:queue_name] if ToEnqueue.active_job_version_supports_queues?
 
         # We can't get the priority out of the DB, as the returned `job` doesn't give us access
         # to the underlying ActiveJob that was scheduled. We have no option but to assume
@@ -125,6 +131,7 @@ module Que
         job_settings = {
           priority: priority,
           wait_until: run_at,
+          queue: queue || Que::Scheduler::VersionSupport.default_scheduler_queue,
         }.compact
 
         job_class_set = job_class.set(**job_settings)

data/lib/que/scheduler/version.rb

@@ -1,5 +1,5 @@
 module Que
   module Scheduler
-    VERSION = '3.3.0'.freeze
+    VERSION = '3.4.0'.freeze
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: que-scheduler
 version: !ruby/object:Gem::Version
-  version: 3.3.0
+  version: 3.4.0
 platform: ruby
 authors:
 - Harry Lascelles
-autorequire: 
+autorequire:
 bindir: bin
 cert_chain: []
-date: 2020-04-04 00:00:00.000000000 Z
+date: 2020-06-07 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activesupport
@@ -24,20 +24,6 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '4.0'
-- !ruby/object:Gem::Dependency
-  name: backports
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '3.10'
-  type: :runtime
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '3.10'
 - !ruby/object:Gem::Dependency
   name: fugit
   requirement: !ruby/object:Gem::Requirement
@@ -258,14 +244,28 @@ dependencies:
     requirements:
     - - '='
       - !ruby/object:Gem::Version
-        version: 0.80.0
+        version: 0.84.0
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - '='
       - !ruby/object:Gem::Version
-        version: 0.80.0
+        version: 0.84.0
+- !ruby/object:Gem::Dependency
+  name: rubocop-rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: sqlite3
   requirement: !ruby/object:Gem::Requirement
@@ -323,6 +323,7 @@ files:
 - lib/que/scheduler/db.rb
 - lib/que/scheduler/defined_job.rb
 - lib/que/scheduler/enqueueing_calculator.rb
+- lib/que/scheduler/jobs/que_scheduler_audit_clear_down_job.rb
 - lib/que/scheduler/migrations.rb
 - lib/que/scheduler/migrations/1/down.sql
 - lib/que/scheduler/migrations/1/up.sql
@@ -332,6 +333,8 @@ files:
 - lib/que/scheduler/migrations/3/up.sql
 - lib/que/scheduler/migrations/4/down.sql
 - lib/que/scheduler/migrations/4/up.sql
+- lib/que/scheduler/migrations/5/down.sql
+- lib/que/scheduler/migrations/5/up.sql
 - lib/que/scheduler/schedule.rb
 - lib/que/scheduler/scheduler_job.rb
 - lib/que/scheduler/scheduler_job_args.rb
@@ -348,7 +351,7 @@ metadata:
   changelog_uri: https://github.com/hlascelles/que-scheduler/blob/master/CHANGELOG.md
   source_code_uri: https://github.com/hlascelles/que-scheduler/
   bug_tracker_uri: https://github.com/hlascelles/que-scheduler/issues
-post_install_message: 
+post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -364,7 +367,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubygems_version: 3.0.3
-signing_key: 
+signing_key:
 specification_version: 4
 summary: A cron scheduler for Que
 test_files: []