que-scheduler 3.4.1 → 4.0.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: cf1ee6c62677a8c215d48d990da257e0f4854c8e6e354e1eae3e90fc0685b601
-  data.tar.gz: dd948edfb90841bc54fc0bb3fc9e725bf4127f1ca31c53654a209ed6d5332e42
+  metadata.gz: 198960bfd5797bdc85e0b320170350bb3a752e5c4ed90db31b86cf040b5dbd92
+  data.tar.gz: 3f7128c8d159d5007ae42e72eca65059fd5e3f7cc9d60706dd72d7ea5002e686
 SHA512:
-  metadata.gz: 7ae4812d01e7ea24300afb6b76a5194f3a25a912852723e30cbc8a1405d1822e8fb543dee955049f87e3063a9ae8b322675e6bbb836c0ab9fe0cc46c119b180c
-  data.tar.gz: 7246c2709a90c3e2ac7e14a9a8ee2f91792f3cccf0f516ad2dc6f119ddcf025b9eb9324fcf4e3eb6f7b279646066a02693f2f1364773a34c3aed011e041012da
+  metadata.gz: db16c38be02a29d413687a3faaffd7749a88ca141bfcee724b0f1c410ddb4094ca6b92c73c878250d088cbaec5242dd8b755d3d3bf44dacd80dd921b710eb9bd
+  data.tar.gz: fd46b785a759e3435491e54015d7ffff61bcf8ad5b0bd782a505838c148e9c778d6c994aaad71697cbe815bd47dffa22dcc3e35de8dc312eebbcc85a2752e26a

data/README.md

@@ -18,9 +18,9 @@ needs to be run, enqueueing those jobs, then enqueueing itself to check again la
     ```ruby
     gem 'que-scheduler'
     ```
-1. Specify a schedule in a yml file (see below). The default location that que-scheduler will 
-look for it is `config/que_schedule.yml`. They are essentially the same as resque-scheduler
-files, but with additional features.
+1. Specify a schedule in a yml file or programmatically (see below). The default location that 
+que-scheduler will look for it is `config/que_schedule.yml`. The format is essentially the same as
+resque-scheduler files, but with additional features.
 
 1. Add a migration to start the job scheduler and prepare the audit table. Note that this migration 
    will fail if Que is set to execute jobs synchronously, i.e. `Que::Job.run_synchronously = true`.
@@ -28,14 +28,18 @@ files, but with additional features.
     ```ruby
     class CreateQueSchedulerSchema < ActiveRecord::Migration
       def change
-        Que::Scheduler::Migrations.migrate!(version: 5)
+        Que::Scheduler::Migrations.migrate!(version: 6)
       end
     end
     ```
     
 ## Schedule configuration
 
-The schedule file is a list of que job classes with arguments and a schedule frequency (in crontab 
+The schedule file should be placed here: `config/que_schedule.yml`. Alternatively if you
+wish to generate the configuration dynamically, you can set it directly using an initializer
+(see "Gem configuration" below).
+
+The file is a list of que job classes with arguments and a schedule frequency (in crontab 
 syntax). The format is similar to the resque-scheduler format, though priorities must be supplied as
 integers, and job classes must be migrated from Resque to Que. Cron syntax can be anything
 understood by [fugit](https://github.com/floraison/fugit#fugitcron).
@@ -131,19 +135,34 @@ A job can have a `schedule_type` assigned to it. Valid values are:
 
 ## Gem configuration
 
-You can configure some aspects of the gem with an initializer. The default is given below.
+You can configure some aspects of the gem with a config block (eg in a Rails initializer). 
+The default is given below. You can omit any configuration sections you are not intending to change.
+It is quite likely you won't have to create this config at all.
 
 ```ruby
 Que::Scheduler.configure do |config|
   # The location of the schedule yaml file.
-  config.schedule_location = ENV.fetch('QUE_SCHEDULER_CONFIG_LOCATION', 'config/que_schedule.yml')
+  config.schedule_location = ENV.fetch("QUE_SCHEDULER_CONFIG_LOCATION", "config/que_schedule.yml")
+
+  # The schedule as a hash. You can use this if you want to build the schedule yourself at runtime.
+  # This will override the above value if provided.
+  config.schedule = {
+    SpecifiedByHashTestJob: {
+      cron: "02 11 * * *"
+    }
+  }
   
-  # Specify a transaction block adapter. By default, que-scheduler uses the one supplied by que.
-  # However, if, for example you rely on listeners to ActiveRecord's exact `transaction` method, or 
+  # The transaction block adapter. By default, que-scheduler uses the one supplied by que.
+  # However if, for example, you rely on listeners to ActiveRecord's exact `transaction` method, or 
   # Sequel's DB.after_commit helper, then you can supply it here.
   config.transaction_adapter = ::Que.method(:transaction)
-end
 
+  # Which queue name the que-scheduler job should self-schedule on. Typically this is the default
+  # queue of que, which has a different name in Que 0.x ("") and 1.x ("default").
+  # It *must* be the "highest throughput" queue - do not work the scheduler on a "long 
+  # running jobs" queue. It is very unlikely you will want to change this. 
+  config.que_scheduler_queue = ENV.fetch("QUE_SCHEDULER_QUEUE", "" or "default")
+end
 ```
 
 ## Scheduler Audit
@@ -155,20 +174,40 @@ migration tasks.
 Additionally, there is the audit table `que_scheduler_audit_enqueued`. This logs every job that 
 the scheduler enqueues.
 
+que-scheduler comes with the `QueSchedulerAuditClearDownJob` job built in that you can optionally
+schedule to clear down audit rows if you don't need to retain them indefinitely. You should add this
+to your own scheduler config yaml.
+
+For example:
+
+```yaml
+# This will clear down the oldest que-scheduler audit rows. Since que-scheduler
+# runs approximately every minute, 129600 is 90 days.
+Que::Scheduler::Jobs::QueSchedulerAuditClearDownJob:
+  cron: "0 0 * * *"
+  args:
+    retain_row_count: 129600
+```
+
+## Required migrations
+
 When there is a major version (breaking) change, a migration should be run in. The version of the 
-migration proceeds at a faster rate than the version of the gem. To run in all the migrations required
-up to a number, just migrate to that number with one line, and it will perform all the intermediary steps. 
+latest migration proceeds at a faster rate than the version of the gem. eg If the gem is on version
+3 then the migrations may be on version 6). 
+
+To run in all the migrations required up to a number, just migrate to that number with one line, and
+it will perform all the intermediary steps. 
 
 ie, This will perform all migrations necessary up to the latest version, skipping any already 
 performed.
 
-    ```ruby
-    class CreateQueSchedulerSchema < ActiveRecord::Migration
-      def change
-        Que::Scheduler::Migrations.migrate!(version: 5)
-      end
-    end
-    ```
+```ruby
+class CreateQueSchedulerSchema < ActiveRecord::Migration
+  def change
+    Que::Scheduler::Migrations.migrate!(version: 6)
+  end
+end
+```
 
 The changes in past migrations were: 
 
@@ -179,22 +218,15 @@ The changes in past migrations were:
 |    3    | Added the audit table `que_scheduler_audit_enqueued`.                           |
 |    4    | Updated the the audit tables to use bigints                                     |
 |    5    | Dropped an unnecessary index                                                    |
+|    6    | Enforced single scheduler job at the trigger level                              |
 
-## Built in optional job for audit clear down
+The changes to the DB ([DDL](https://en.wikipedia.org/wiki/Data_definition_language)) are all 
+captured in the structure.sql so will be re-run in correctly if squashed - except for the actual 
+scheduling of the job itself (as that is [DML](https://en.wikipedia.org/wiki/Data_manipulation_language)).
+If you squash your migrations make sure this is added as the final line:
 
-que-scheduler comes with the `QueSchedulerAuditClearDownJob` job built in that you can optionally
-schedule to clear down audit rows if you don't need to retain them indefinitely. You should add this
-to your own scheduler config yaml.
-
-For example:
-
-```yaml
-# This will clear down the oldest que-scheduler audit rows. Since que-scheduler
-# runs approximately every minute, 129600 is 90 days.
-Que::Scheduler::Jobs::QueSchedulerAuditClearDownJob:
-  cron: "0 0 * * *"
-  args:
-    retain_row_count: 129600
+```ruby
+Que::Scheduler::Migrations.reenqueue_scheduler_if_missing
 ```
 
 ## HA Redundancy and DB restores
@@ -210,8 +242,8 @@ in a coherent state with the rest of your database.
 ## Concurrent scheduler detection
 
 No matter how many tasks you have defined in your schedule, you will only ever need one que-scheduler
-job enqueued. que-scheduler knows this, and it will check before performing any operations that 
-there is only one of itself present.
+job enqueued. que-scheduler knows this, and there are DB constraints in place to ensure there is
+only ever exactly one scheduler job.
 
 It also follows que job design [best practices](https://github.com/chanks/que/blob/master/docs/writing_reliable_jobs.md),
 using ACID guarantees, to ensure that it will never run multiple times. If the scheduler crashes for any reason,
@@ -263,6 +295,10 @@ The scheduler will then continue to retry indefinitely.
 que-scheduler uses [semantic versioning](https://semver.org/), so major version changes will usually 
 require additional actions to be taken upgrading from one major version to another. 
 
+## Changelog
+
+A full changelog can be found here: [CHANGELOG.md](https://github.com/hlascelles/que-scheduler/blob/master/CHANGELOG.md)
+
 ## System requirements
 
 Your [postgres](https://www.postgresql.org/) database must be at least version 9.4.0.
@@ -273,8 +309,9 @@ This gem was inspired by the makers of the excellent [Que](https://github.com/ch
 
 ## Contributors
 
+* @bnauta
+* @JackDanger
 * @jish
 * @joehorsnell
-* @bnauta
-* @papodaca
 * @krzyzak
+* @papodaca

data/lib/que/scheduler.rb

@@ -1,6 +1,6 @@
 require "que/scheduler/version"
 require "que/scheduler/version_support"
-require "que/scheduler/config"
+require "que/scheduler/configuration"
 require "que/scheduler/scheduler_job"
 require "que/scheduler/db"
 require "que/scheduler/audit"

data/lib/que/scheduler/configuration.rb

@@ -0,0 +1,35 @@
+require "que"
+require_relative "version_support"
+
+module Que
+  module Scheduler
+    class Configuration
+      attr_accessor :schedule_location
+      attr_accessor :schedule
+      attr_accessor :transaction_adapter
+      attr_accessor :que_scheduler_queue
+    end
+
+    class << self
+      attr_accessor :configuration
+
+      def configure
+        self.configuration ||= Configuration.new
+        yield(configuration)
+      end
+
+      def apply_defaults
+        configure do |config|
+          config.schedule_location =
+            ENV.fetch("QUE_SCHEDULER_CONFIG_LOCATION", "config/que_schedule.yml")
+          config.transaction_adapter = ::Que.method(:transaction)
+          config.que_scheduler_queue =
+            ENV.fetch("QUE_SCHEDULER_QUEUE", Que::Scheduler::VersionSupport.default_scheduler_queue)
+          config.schedule = nil
+        end
+      end
+    end
+  end
+end
+
+Que::Scheduler.apply_defaults

data/lib/que/scheduler/db.rb

@@ -1,6 +1,6 @@
 # frozen_string_literal: true
 
-require_relative "config"
+require_relative "configuration"
 
 module Que
   module Scheduler

data/lib/que/scheduler/migrations.rb

@@ -41,6 +41,11 @@ module Que
           result.any?
         end
 
+        # This method is only intended for use in squashed migrations
+        def reenqueue_scheduler_if_missing
+          Que::Scheduler::SchedulerJob.enqueue if Que::Scheduler::Db.count_schedulers.zero?
+        end
+
         private
 
         def migrate_up(current, version)

data/lib/que/scheduler/migrations/6/down.sql

@@ -0,0 +1,7 @@
+DROP TRIGGER que_scheduler_prevent_job_deletion_trigger ON que_jobs;
+
+DROP FUNCTION que_scheduler_prevent_job_deletion();
+
+DROP FUNCTION que_scheduler_check_job_exists();
+
+DROP INDEX que_scheduler_job_in_que_jobs_unique_index;

data/lib/que/scheduler/migrations/6/up.sql

@@ -0,0 +1,26 @@
+-- Ensure there is no more than one scheduler
+CREATE UNIQUE INDEX que_scheduler_job_in_que_jobs_unique_index ON que_jobs(job_class)
+WHERE job_class = 'Que::Scheduler::SchedulerJob';
+
+-- Ensure there is at least one scheduler
+CREATE OR REPLACE FUNCTION que_scheduler_check_job_exists() RETURNS bool AS $$
+SELECT EXISTS(SELECT * FROM que_jobs WHERE job_class = 'Que::Scheduler::SchedulerJob');
+$$ LANGUAGE SQL;
+
+CREATE OR REPLACE FUNCTION que_scheduler_prevent_job_deletion() RETURNS TRIGGER AS
+$BODY$
+DECLARE
+BEGIN
+    IF OLD.job_class = 'Que::Scheduler::SchedulerJob' THEN
+        IF NOT que_scheduler_check_job_exists() THEN
+            raise exception 'Deletion of que_scheduler job % prevented. Deleting the que_scheduler job is almost certainly a mistake.', OLD.job_id;
+        END IF;
+    END IF;
+    RETURN OLD;
+END;
+$BODY$
+LANGUAGE 'plpgsql';
+
+CREATE CONSTRAINT TRIGGER que_scheduler_prevent_job_deletion_trigger AFTER UPDATE OR DELETE ON que_jobs
+DEFERRABLE INITIALLY DEFERRED
+FOR EACH ROW EXECUTE PROCEDURE que_scheduler_prevent_job_deletion();

data/lib/que/scheduler/schedule.rb

@@ -1,20 +1,37 @@
+require_relative "configuration"
 require_relative "defined_job"
 
 module Que
   module Scheduler
     class Schedule
       class << self
+        # The main method for determining the schedule. It has to evaluate the schedule as late as
+        # possible (ie just as it is about to be used) as we cannot guarantee we are in a Rails
+        # app with initializers. In a future release this may change to "fast fail" in Rails by
+        # checking the config up front.
         def schedule
           @schedule ||=
             begin
-              location = Que::Scheduler.configuration.schedule_location
-              from_file(location)
+              configuration = Que::Scheduler.configuration
+              if !configuration.schedule.nil?
+                # If an explicit schedule as a hash has been defined, use that.
+                from_hash(configuration.schedule)
+              elsif File.exist?(configuration.schedule_location)
+                # If the schedule is defined as a file location, then load it and return it.
+                from_file(configuration.schedule_location)
+              else
+                raise "No que-scheduler config set, or file found " \
+                      "at #{configuration.schedule_location}"
+              end
             end
         end
 
         def from_file(location)
-          yml = IO.read(location)
-          config_hash = YAML.safe_load(yml)
+          from_yaml(IO.read(location))
+        end
+
+        def from_yaml(config)
+          config_hash = YAML.safe_load(config)
           from_hash(config_hash)
         end
 

data/lib/que/scheduler/scheduler_job.rb

@@ -1,4 +1,6 @@
 require "que"
+require "active_support"
+require "active_support/core_ext/time"
 
 require_relative "schedule"
 require_relative "enqueueing_calculator"
@@ -14,8 +16,8 @@ module Que
     class SchedulerJob < Que::Job
       SCHEDULER_FREQUENCY = 60
 
-      Que::Scheduler::VersionSupport.set_priority(self, 0)
-      Que::Scheduler::VersionSupport.apply_retry_semantics(self)
+      VersionSupport.set_priority(self, 0)
+      VersionSupport.apply_retry_semantics(self)
 
       def run(options = nil)
         Que::Scheduler::Db.transaction do
@@ -25,12 +27,13 @@ module Que
           logs = ["que-scheduler last ran at #{scheduler_job_args.last_run_time}."]
           result = EnqueueingCalculator.parse(Scheduler.schedule.values, scheduler_job_args)
           enqueued_jobs = enqueue_required_jobs(result, logs)
+          # Remove this job and schedule self again
+          destroy
           enqueue_self_again(
             scheduler_job_args, scheduler_job_args.as_time, result.job_dictionary, enqueued_jobs
           )
           # Only now we're sure nothing errored, log the results
           logs.each { |str| ::Que.log(event: "que-scheduler".to_sym, message: str) }
-          destroy
         end
       end
 
@@ -57,7 +60,7 @@ module Que
 
       def enqueue_self_again(scheduler_job_args, last_full_execution, job_dictionary, enqueued_jobs)
         # Log last run...
-        job_id = Que::Scheduler::VersionSupport.job_attributes(self).fetch(:job_id)
+        job_id = VersionSupport.job_attributes(self).fetch(:job_id)
         Audit.append(job_id, scheduler_job_args.as_time, enqueued_jobs)
 
         # And rerun...
@@ -70,7 +73,7 @@ module Que
         )
 
         # rubocop:disable Style/GuardClause This reads better as a conditional
-        unless Que::Scheduler::VersionSupport.job_attributes(enqueued_job).fetch(:job_id)
+        unless enqueued_job && VersionSupport.job_attributes(enqueued_job).fetch(:job_id)
           raise "SchedulerJob could not self-schedule. Has `.enqueue` been monkey patched?"
         end
         # rubocop:enable Style/GuardClause

data/lib/que/scheduler/state_checks.rb

@@ -8,7 +8,6 @@ module Que
       class << self
         def check
           assert_db_migrated
-          assert_one_scheduler_job
         end
 
         private
@@ -60,21 +59,6 @@ module Que
             synchronously. This will fail as que-scheduler needs the above tables to work.
           ERR
         end
-
-        def assert_one_scheduler_job
-          schedulers = Que::Scheduler::Db.count_schedulers
-          return if schedulers == 1
-
-          raise(<<-ERR)
-            Only one #{Que::Scheduler::SchedulerJob.name} should be enqueued. #{schedulers} were found.
-
-            que-scheduler works by running a self-enqueueing version of itself that determines which
-            jobs should be enqueued based on the provided config. If two or more que-schedulers were
-            to run at once, then duplicate jobs would occur.
-
-            To resolve this problem, please remove any duplicate scheduler jobs from the que_jobs table.
-          ERR
-        end
       end
     end
   end

data/lib/que/scheduler/version.rb

@@ -1,5 +1,5 @@
 module Que
   module Scheduler
-    VERSION = "3.4.1".freeze
+    VERSION = "4.0.2".freeze
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: que-scheduler
 version: !ruby/object:Gem::Version
-  version: 3.4.1
+  version: 4.0.2
 platform: ruby
 authors:
 - Harry Lascelles
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2020-07-11 00:00:00.000000000 Z
+date: 2020-11-29 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activesupport
@@ -112,6 +112,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: climate_control
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: combustion
   requirement: !ruby/object:Gem::Requirement
@@ -319,7 +333,7 @@ files:
 - lib/que-scheduler.rb
 - lib/que/scheduler.rb
 - lib/que/scheduler/audit.rb
-- lib/que/scheduler/config.rb
+- lib/que/scheduler/configuration.rb
 - lib/que/scheduler/db.rb
 - lib/que/scheduler/defined_job.rb
 - lib/que/scheduler/enqueueing_calculator.rb
@@ -335,6 +349,8 @@ files:
 - lib/que/scheduler/migrations/4/up.sql
 - lib/que/scheduler/migrations/5/down.sql
 - lib/que/scheduler/migrations/5/up.sql
+- lib/que/scheduler/migrations/6/down.sql
+- lib/que/scheduler/migrations/6/up.sql
 - lib/que/scheduler/schedule.rb
 - lib/que/scheduler/scheduler_job.rb
 - lib/que/scheduler/scheduler_job_args.rb

data/lib/que/scheduler/config.rb

@@ -1,27 +0,0 @@
-require "que"
-require_relative "version_support"
-
-module Que
-  module Scheduler
-    class << self
-      attr_accessor :configuration
-    end
-
-    def self.configure
-      self.configuration ||= Configuration.new
-      yield(configuration)
-    end
-
-    class Configuration
-      attr_accessor :schedule_location
-      attr_accessor :transaction_adapter
-      attr_accessor :que_scheduler_queue
-    end
-  end
-end
-
-Que::Scheduler.configure do |config|
-  config.schedule_location = ENV.fetch("QUE_SCHEDULER_CONFIG_LOCATION", "config/que_schedule.yml")
-  config.transaction_adapter = ::Que.method(:transaction)
-  config.que_scheduler_queue = Que::Scheduler::VersionSupport.default_scheduler_queue
-end