que-scheduler 3.0.0 → 3.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f10811f1ba2ecc81c7d0fea63a94dd31f85e383d819de9bae939b24751817ee8
-  data.tar.gz: 7d461f16288e3161845f1c5ea55dbd422ec23fda41ce86394dc0532d2f563ba8
+  metadata.gz: b10ffa2b0fda7c713a51d8780d9f6477e7579fcd0ab47f9fa65084a9f31b5780
+  data.tar.gz: 32ed325e8c054851c61be53f1bb18b79f22c5e41f9f4fa2baa20070fbe078110
 SHA512:
-  metadata.gz: ec49442550c653717fd8a943a8d46b4ada8431c78098841e9e773b214c89097d0e174936396c05c79cc91d42ebb9c5d5bf8b6bf677d05ae07ba049c011d405f3
-  data.tar.gz: b32792dd60a8895b84cba9b26a6cc01602eae2eacac097e7c4593db2aa2897a4e21eae9a4d626aef1f1ead58f9f13d01cacd588e962446f5ddbfecf6faf40fa5
+  metadata.gz: d26040a8470cee8985e62436c55d013700412c6f8928a1952c37c90e13e91bbf7820b62f55118c238108b0020cc627f27cd209051076048003146839e52960d4
+  data.tar.gz: a58efd112aeb2f304394ca7578e10019d0a1c28b7f1796e18a68a6d49ac23f40e50db49b8e31f3258d8a1034cf547ff4a69b5a60a980c67574776eef54bb2419

data/README.md

@@ -18,11 +18,11 @@ needs to be run, enqueueing those jobs, then enqueueing itself to check again la
     ```ruby
     gem 'que-scheduler'
     ```
-1. Specify a schedule config in a yml file (see below). The default location that que-scheduler will 
-look for it is `config/que_schedule.yml`. They are essentially the same as resque-scheduler config
+1. Specify a schedule in a yml file (see below). The default location that que-scheduler will 
+look for it is `config/que_schedule.yml`. They are essentially the same as resque-scheduler
 files, but with additional features.
 
-1. Add a migration to start the job scheduler.
+1. Add a migration to start the job scheduler and prepare the audit table.
 
     ```ruby
     class CreateQueSchedulerSchema < ActiveRecord::Migration
@@ -35,7 +35,7 @@ files, but with additional features.
 ## Schedule configuration
 
 The schedule file is a list of que job classes with arguments and a schedule frequency (in crontab 
-syntax). The format is similar to the resque-scheduler config format, though priorities must be supplied as
+syntax). The format is similar to the resque-scheduler format, though priorities must be supplied as
 integers, and job classes must be migrated from Resque to Que. Cron syntax can be anything
 understood by [fugit](https://github.com/floraison/fugit#fugitcron).
 
@@ -111,11 +111,22 @@ A job can have a `schedule_type` assigned to it. Valid values are:
    This feature ensures that jobs which *must run* for a certain cron match will always eventually 
   execute, even after a total system crash, or even a DB backup restore.
 
-## Environment Variables
+## Gem configuration
 
-You can configure some aspects of the gem with environment variables.
+You can configure some aspects of the gem with an initializer. The default is given below.
 
-* `QUE_SCHEDULER_CONFIG_LOCATION` - The location of the schedule configuration (default `config/que_schedule.yml`)
+```ruby
+Que::Scheduler.configure do |config|
+  # The location of the schedule yaml file.
+  config.schedule_location = ENV.fetch('QUE_SCHEDULER_CONFIG_LOCATION', 'config/que_schedule.yml')
+  
+  # Specify a transaction block adapter. By default, que-scheduler uses the one supplied by que.
+  # However, if, for example you rely on listeners to ActiveRecord's exact `transaction` method, or 
+  # Sequel's DB.after_commit helper, then you can supply it here.
+  config.transaction_adapter = ::Que.method(:transaction)
+end
+
+```
 
 ## Scheduler Audit
 
@@ -134,7 +145,7 @@ in a coherent state with the rest of your database.
 
 ## Concurrent scheduler detection
 
-No matter how many tasks you have defined in your config, you will only ever need one que-scheduler
+No matter how many tasks you have defined in your schedule, you will only ever need one que-scheduler
 job enqueued. que-scheduler knows this, and it will check before performing any operations that 
 there is only one of itself present.
 
@@ -144,17 +155,59 @@ it will rollback correctly and try again. It won't schedule jobs twice for a cro
 
 ## How it works
 
-que-scheduler is a job that reads a config file, enqueues any jobs it determines that need to be run,
+que-scheduler is a job that reads a schedule file, enqueues any jobs it determines that need to be run,
 then reschedules itself. The flow is as follows:
 
 1. The que-scheduler job runs for the very first time.
-1. que-scheduler loads the schedule config file. It will not schedule any other jobs, except itself, 
+1. que-scheduler loads the schedule file. It will not schedule any other jobs, except itself, 
    as it has never run before.
 1. Some time later it runs again. It knows what jobs it should be monitoring, and notices that some 
    have are due. It enqueues those jobs and then itself. Repeat.
-1. After a deploy that changes the config, the job notices any new jobs to schedule, and knows which
+1. After a deploy that changes the schedule, the job notices any new jobs to schedule, and knows which
    ones to forget. It does not need to be re-enqueued or restarted.
    
+## DB Migrations
+
+When there is a major version (breaking) change, a migration should be run in. The version of the 
+migration proceeds at a faster rate than the version of the gem. To run in all the migrations required
+up to a number, just migrate to that number with one line, and it will perform all the intermediary steps. 
+
+ie, `Que::Scheduler::Migrations.migrate!(version: 4)` will perform all migrations necessary to 
+reach migration version `4`.
+
+As of migration `4`, two elements are added to the DB for que-scheduler to run. 
+
+1. The first is the scheduler job itself, which runs forever, re-enqueuing itself to performs its 
+   duties.
+1. The second part comprises the audit table `que_scheduler_audit` and the "enqueued" table 
+  `que_scheduler_audit`. The first tracks when the scheduler calculated what was necessary to run 
+  (if anything). The second then logs every job that the scheduler enqueues. 
+
+## Upgrading
+
+que-scheduler uses [semantic versioning](https://semver.org/), so major version changes will usually 
+require additional actions to be taken upgrading from one major version to another. 
+
+Major feature changes are listed below. The full 
+[CHANGELOG](https://github.com/hlascelles/que-scheduler/blob/master/CHANGELOG.md) can be found in 
+the root of the project. 
+
+#### Versions 3.x 
+  - Addition of a config initializer.
+  - Addition of numerous extra columns to the audit table.
+  - Required cumulative migration: `Que::Scheduler::Migrations.migrate!(version: 4)`
+#### Versions 2.x 
+  - Introduction of the audit table.
+  - Support for older versions of postgres
+  - Required cumulative migration: `Que::Scheduler::Migrations.migrate!(version: 3)`
+#### Versions 1.x
+  - Sequel support
+  - Config specified Timezone support
+  - Required migration adding the initial job: `Que::Scheduler::SchedulerJob.enqueue`
+#### Versions 0.x
+  - The first public release. 
+  - Required migration adding the initial job: `Que::Scheduler::SchedulerJob.enqueue`
+   
 ## System requirements
 
 Your [postgres](https://www.postgresql.org/) database must be at least version 9.4.0.

data/lib/que-scheduler.rb

@@ -0,0 +1,3 @@
+# rubocop:disable Naming/FileName
+require 'que/scheduler'
+# rubocop:enable Naming/FileName

data/lib/que/scheduler.rb

@@ -1,4 +1,5 @@
 require 'que/scheduler/version'
+require 'que/scheduler/config'
 require 'que/scheduler/scheduler_job'
 require 'que/scheduler/db'
 require 'que/scheduler/audit'

data/lib/que/scheduler/config.rb

@@ -0,0 +1,24 @@
+require 'que'
+
+module Que
+  module Scheduler
+    class << self
+      attr_accessor :configuration
+    end
+
+    def self.configure
+      self.configuration ||= Configuration.new
+      yield(configuration)
+    end
+
+    class Configuration
+      attr_accessor :schedule_location
+      attr_accessor :transaction_adapter
+    end
+  end
+end
+
+Que::Scheduler.configure do |config|
+  config.schedule_location = ENV.fetch('QUE_SCHEDULER_CONFIG_LOCATION', 'config/que_schedule.yml')
+  config.transaction_adapter = ::Que.method(:transaction)
+end

data/lib/que/scheduler/db.rb

@@ -15,6 +15,10 @@ module Que
         def now
           Que.execute(NOW_SQL).first.values.first
         end
+
+        def transaction
+          Que::Scheduler.configuration.transaction_adapter.call { yield }
+        end
       end
     end
   end

data/lib/que/scheduler/defined_job.rb

@@ -6,10 +6,6 @@ require 'backports/2.4.0/hash/compact'
 module Que
   module Scheduler
     class DefinedJob < Hashie::Dash
-      QUE_SCHEDULER_CONFIG_LOCATION =
-        ENV.fetch('QUE_SCHEDULER_CONFIG_LOCATION', 'config/que_schedule.yml')
-      ERROR_MESSAGE_SUFFIX = "in que-scheduler config #{QUE_SCHEDULER_CONFIG_LOCATION}".freeze
-
       include Hashie::Extensions::Dash::PropertyTranslation
 
       SCHEDULE_TYPES = [
@@ -55,7 +51,8 @@ module Que
 
       class << self
         def defined_jobs
-          @defined_jobs ||= YAML.safe_load(IO.read(QUE_SCHEDULER_CONFIG_LOCATION)).map do |k, v|
+          schedule_string = IO.read(Que::Scheduler.configuration.schedule_location)
+          @defined_jobs ||= YAML.safe_load(schedule_string).map do |k, v|
             Que::Scheduler::DefinedJob.new(
               {
                 name: k,
@@ -73,7 +70,8 @@ module Que
         private
 
         def err_field(field, value)
-          raise "Invalid #{field} '#{value}' #{ERROR_MESSAGE_SUFFIX}"
+          schedule = Que::Scheduler.configuration.schedule_location
+          raise "Invalid #{field} '#{value}' in que-scheduler schedule #{schedule}"
         end
       end
 

data/lib/que/scheduler/migrations.rb

@@ -13,7 +13,7 @@ module Que
 
       class << self
         def migrate!(version:)
-          ::Que.transaction do
+          Que::Scheduler::Db.transaction do
             current = db_version
             if current < version
               migrate_up(current, version)

data/lib/que/scheduler/scheduler_job.rb

@@ -15,7 +15,7 @@ module Que
       @priority = 0
 
       def run(options = nil)
-        ::Que.transaction do
+        Que::Scheduler::Db.transaction do
           assert_one_scheduler_job
           scheduler_job_args = SchedulerJobArgs.build(options)
           logs = ["que-scheduler last ran at #{scheduler_job_args.last_run_time}."]
@@ -43,14 +43,24 @@ module Que
             else
               job_class.enqueue(*args, remaining_hash)
             end
+          check_enqueued_job(enqueued_job, job_class, args, logs)
+        end.compact
+      end
+
+      private
+
+      def check_enqueued_job(enqueued_job, job_class, args, logs)
+        if enqueued_job.is_a?(Que::Job)
           job_id = enqueued_job.attrs.fetch('job_id')
-          logs << "que-scheduler enqueueing #{job_class} #{job_id} with: #{to_enqueue}"
+          logs << "que-scheduler enqueueing #{job_class} #{job_id} with args: #{args}"
           enqueued_job
+        else
+          # This can happen if a middleware nixes the enqueue call
+          logs << "que-scheduler called enqueue on #{job_class} but did not receive a #{Que::Job}"
+          nil
         end
       end
 
-      private
-
       def assert_one_scheduler_job
         schedulers = Que::Scheduler::Db.count_schedulers
         return if schedulers == 1

data/lib/que/scheduler/version.rb

@@ -1,5 +1,5 @@
 module Que
   module Scheduler
-    VERSION = '3.0.0'.freeze
+    VERSION = '3.1.0'.freeze
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: que-scheduler
 version: !ruby/object:Gem::Version
-  version: 3.0.0
+  version: 3.1.0
 platform: ruby
 authors:
 - Harry Lascelles
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2018-05-23 00:00:00.000000000 Z
+date: 2018-06-01 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activesupport
@@ -136,6 +136,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: database_cleaner
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: fasterer
   requirement: !ruby/object:Gem::Requirement
@@ -298,8 +312,10 @@ extensions: []
 extra_rdoc_files: []
 files:
 - README.md
+- lib/que-scheduler.rb
 - lib/que/scheduler.rb
 - lib/que/scheduler/audit.rb
+- lib/que/scheduler/config.rb
 - lib/que/scheduler/db.rb
 - lib/que/scheduler/defined_job.rb
 - lib/que/scheduler/engine.rb