que-scheduler 3.2.8 → 3.4.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: c2a35cace72621c452055873ea88a13aba5d00a63933df91b2d4d60840f11337
4
- data.tar.gz: 1b2d3bbb599260b37f8ca8c10a12bd03992b08ed74b0c75f47905b1f8238dfd5
3
+ metadata.gz: 4ba02202915db229bcbd1c2e859f29299803507dbf61bd460008400abca3611c
4
+ data.tar.gz: 6293d7b6e6766d0e75fe65df01850f356b44b030c8b8ae539eb4c71059d9d15d
5
5
  SHA512:
6
- metadata.gz: ab5ad0c46554887d523d7b9f90c8336da989489e76cc69a0bfe5465b66b34e7a7c3d0f2fc6c0dc026660926b51e4fff0fde029600b93e52386147565a4c10a2a
7
- data.tar.gz: 7359bfdc4db605aea48c8bb87b1f02a8203dc103fa6e6c9ba5d6f33d051026c7e4abc7d4e5a503660d522d99f89c720455879a7b631929bd4e591961388afa75
6
+ metadata.gz: d8d18550867d7d54cfb5c03aaa68876d423876a870b972cde0fbc4e8399fb616d327ef0410ebbc8e3d9021a34d73a960774eabb808f2b33a526e11ea2fb579fe
7
+ data.tar.gz: bfc0e4be1f64d2b070dacb0863cc710ce13cf58cb30924fcf9f301d2e535be4ef9530740949bdb5abb90c0173924e71f7f151e4622405caf8c14a9d7d6f0469d
data/README.md CHANGED
@@ -18,23 +18,28 @@ needs to be run, enqueueing those jobs, then enqueueing itself to check again la
18
18
  ```ruby
19
19
  gem 'que-scheduler'
20
20
  ```
21
- 1. Specify a schedule in a yml file (see below). The default location that que-scheduler will
22
- look for it is `config/que_schedule.yml`. They are essentially the same as resque-scheduler
23
- files, but with additional features.
21
+ 1. Specify a schedule in a yml file or programmatically (see below). The default location that
22
+ que-scheduler will look for it is `config/que_schedule.yml`. The format is essentially the same as
23
+ resque-scheduler files, but with additional features.
24
24
 
25
- 1. Add a migration to start the job scheduler and prepare the audit table. Note that this migration will fail if Que is set to execute jobs synchronously, i.e. `Que::Job.run_synchronously = true`.
25
+ 1. Add a migration to start the job scheduler and prepare the audit table. Note that this migration
26
+ will fail if Que is set to execute jobs synchronously, i.e. `Que::Job.run_synchronously = true`.
26
27
 
27
28
  ```ruby
28
29
  class CreateQueSchedulerSchema < ActiveRecord::Migration
29
30
  def change
30
- Que::Scheduler::Migrations.migrate!(version: 4)
31
+ Que::Scheduler::Migrations.migrate!(version: 5)
31
32
  end
32
33
  end
33
34
  ```
34
35
 
35
36
  ## Schedule configuration
36
37
 
37
- The schedule file is a list of que job classes with arguments and a schedule frequency (in crontab
38
+ The schedule file should be placed here: `config/que_schedule.yml`. Alternatively if you
39
+ wish to generate the configuration dynamically, you can set it directly with
40
+ `Que::Scheduler.schedule = some_hash`.
41
+
42
+ The file is a list of que job classes with arguments and a schedule frequency (in crontab
38
43
  syntax). The format is similar to the resque-scheduler format, though priorities must be supplied as
39
44
  integers, and job classes must be migrated from Resque to Que. Cron syntax can be anything
40
45
  understood by [fugit](https://github.com/floraison/fugit#fugitcron).
@@ -65,10 +70,21 @@ BatchOrders:
65
70
  cron: "0 0 * * *"
66
71
  priority: 25
67
72
 
68
- # Specify job arguments
73
+ # Specify array job arguments
69
74
  SendOrders:
70
75
  cron: "0 0 * * *"
71
76
  args: ['open']
77
+
78
+ # Specify hash job arguments. Note, this appears as a single hash to `run`, not as kwargs.
79
+ SendPreorders:
80
+ cron: "0 0 * * *"
81
+ args:
82
+ order_type: special
83
+
84
+ # Specify a single nil argument
85
+ SendPostorders:
86
+ cron: "0 0 * * *"
87
+ args: ~ # See https://stackoverflow.com/a/51990876/1267203
72
88
 
73
89
  # Use simpler cron syntax
74
90
  SendBilling:
@@ -136,8 +152,54 @@ end
136
152
 
137
153
  ## Scheduler Audit
138
154
 
139
- An audit table _que_scheduler_audit_ is written to by the scheduler to keep a history of what jobs
140
- were enqueued when. It is created by the included migration tasks.
155
+ An audit table `que_scheduler_audit` is written to by the scheduler to keep a history of when the
156
+ scheduler ran to calculate what was necessary to run (if anything). It is created by the included
157
+ migration tasks.
158
+
159
+ Additionally, there is the audit table `que_scheduler_audit_enqueued`. This logs every job that
160
+ the scheduler enqueues.
161
+
162
+ When there is a major version (breaking) change, a migration should be run in. The version of the
163
+ migration proceeds at a faster rate than the version of the gem. To run in all the migrations required
164
+ up to a number, just migrate to that number with one line, and it will perform all the intermediary steps.
165
+
166
+ ie, This will perform all migrations necessary up to the latest version, skipping any already
167
+ performed.
168
+
169
+ ```ruby
170
+ class CreateQueSchedulerSchema < ActiveRecord::Migration
171
+ def change
172
+ Que::Scheduler::Migrations.migrate!(version: 5)
173
+ end
174
+ end
175
+ ```
176
+
177
+ The changes in past migrations were:
178
+
179
+ | Version | Changes |
180
+ |:-------:|---------------------------------------------------------------------------------|
181
+ | 1 | Enqueued the main Que::Scheduler. This is the job that performs the scheduling. |
182
+ | 2 | Added the audit table `que_scheduler_audit`. |
183
+ | 3 | Added the audit table `que_scheduler_audit_enqueued`. |
184
+ | 4 | Updated the the audit tables to use bigints |
185
+ | 5 | Dropped an unnecessary index |
186
+
187
+ ## Built in optional job for audit clear down
188
+
189
+ que-scheduler comes with the `QueSchedulerAuditClearDownJob` job built in that you can optionally
190
+ schedule to clear down audit rows if you don't need to retain them indefinitely. You should add this
191
+ to your own scheduler config yaml.
192
+
193
+ For example:
194
+
195
+ ```yaml
196
+ # This will clear down the oldest que-scheduler audit rows. Since que-scheduler
197
+ # runs approximately every minute, 129600 is 90 days.
198
+ Que::Scheduler::Jobs::QueSchedulerAuditClearDownJob:
199
+ cron: "0 0 * * *"
200
+ args:
201
+ retain_row_count: 129600
202
+ ```
141
203
 
142
204
  ## HA Redundancy and DB restores
143
205
 
@@ -172,23 +234,6 @@ then reschedules itself. The flow is as follows:
172
234
  1. After a deploy that changes the schedule, the job notices any new jobs to schedule, and knows which
173
235
  ones to forget. It does not need to be re-enqueued or restarted.
174
236
 
175
- ## DB Migrations
176
-
177
- When there is a major version (breaking) change, a migration should be run in. The version of the
178
- migration proceeds at a faster rate than the version of the gem. To run in all the migrations required
179
- up to a number, just migrate to that number with one line, and it will perform all the intermediary steps.
180
-
181
- ie, `Que::Scheduler::Migrations.migrate!(version: 4)` will perform all migrations necessary to
182
- reach migration version `4`.
183
-
184
- As of migration `4`, two elements are added to the DB for que-scheduler to run.
185
-
186
- 1. The first is the scheduler job itself, which runs forever, re-enqueuing itself to performs its
187
- duties.
188
- 1. The second part comprises the audit table `que_scheduler_audit` and the "enqueued" table
189
- `que_scheduler_audit_enqueued`. The first tracks when the scheduler calculated what was necessary to run
190
- (if anything). The second then logs every job that the scheduler enqueues.
191
-
192
237
  ## Testing Configuration
193
238
 
194
239
  You can add tests to validate your configuration during the spec phase. This will perform a variety
@@ -235,3 +280,6 @@ This gem was inspired by the makers of the excellent [Que](https://github.com/ch
235
280
  * @jish
236
281
  * @joehorsnell
237
282
  * @bnauta
283
+ * @papodaca
284
+ * @krzyzak
285
+ * @JackDanger
@@ -1,3 +1,3 @@
1
1
  # rubocop:disable Naming/FileName
2
- require 'que/scheduler'
2
+ require "que/scheduler"
3
3
  # rubocop:enable Naming/FileName
@@ -1,7 +1,8 @@
1
- require 'que/scheduler/version'
2
- require 'que/scheduler/version_support'
3
- require 'que/scheduler/config'
4
- require 'que/scheduler/scheduler_job'
5
- require 'que/scheduler/db'
6
- require 'que/scheduler/audit'
7
- require 'que/scheduler/migrations'
1
+ require "que/scheduler/version"
2
+ require "que/scheduler/version_support"
3
+ require "que/scheduler/config"
4
+ require "que/scheduler/scheduler_job"
5
+ require "que/scheduler/db"
6
+ require "que/scheduler/audit"
7
+ require "que/scheduler/migrations"
8
+ require "que/scheduler/jobs/que_scheduler_audit_clear_down_job"
@@ -1,10 +1,12 @@
1
1
  # frozen_string_literal: true
2
2
 
3
+ require_relative "to_enqueue"
4
+
3
5
  module Que
4
6
  module Scheduler
5
7
  module Audit
6
- TABLE_NAME = 'que_scheduler_audit'
7
- ENQUEUED_TABLE_NAME = 'que_scheduler_audit_enqueued'
8
+ TABLE_NAME = "que_scheduler_audit"
9
+ ENQUEUED_TABLE_NAME = "que_scheduler_audit_enqueued"
8
10
  INSERT_AUDIT = %{
9
11
  INSERT INTO #{TABLE_NAME} (scheduler_job_id, executed_at)
10
12
  VALUES ($1::bigint, $2::timestamptz)
@@ -24,11 +26,10 @@ module Que
24
26
  def append(scheduler_job_id, executed_at, enqueued_jobs)
25
27
  ::Que::Scheduler::VersionSupport.execute(INSERT_AUDIT, [scheduler_job_id, executed_at])
26
28
  enqueued_jobs.each do |j|
27
- attrs = Que::Scheduler::VersionSupport.job_attributes(j)
28
29
  inserted = ::Que::Scheduler::VersionSupport.execute(
29
30
  INSERT_AUDIT_ENQUEUED,
30
31
  [scheduler_job_id] +
31
- attrs.values_at(:job_class, :queue, :priority, :args, :job_id, :run_at)
32
+ j.values_at(:job_class, :queue, :priority, :args, :job_id, :run_at)
32
33
  )
33
34
  raise "Cannot save audit row #{scheduler_job_id} #{executed_at} #{j}" if inserted.empty?
34
35
  end
@@ -1,5 +1,5 @@
1
- require 'que'
2
- require_relative 'version_support'
1
+ require "que"
2
+ require_relative "version_support"
3
3
 
4
4
  module Que
5
5
  module Scheduler
@@ -21,7 +21,7 @@ module Que
21
21
  end
22
22
 
23
23
  Que::Scheduler.configure do |config|
24
- config.schedule_location = ENV.fetch('QUE_SCHEDULER_CONFIG_LOCATION', 'config/que_schedule.yml')
24
+ config.schedule_location = ENV.fetch("QUE_SCHEDULER_CONFIG_LOCATION", "config/que_schedule.yml")
25
25
  config.transaction_adapter = ::Que.method(:transaction)
26
26
  config.que_scheduler_queue = Que::Scheduler::VersionSupport.default_scheduler_queue
27
27
  end
@@ -1,13 +1,13 @@
1
1
  # frozen_string_literal: true
2
2
 
3
- require_relative 'config'
3
+ require_relative "config"
4
4
 
5
5
  module Que
6
6
  module Scheduler
7
7
  module Db
8
8
  SCHEDULER_COUNT_SQL =
9
9
  "SELECT COUNT(*) FROM que_jobs WHERE job_class = 'Que::Scheduler::SchedulerJob'"
10
- NOW_SQL = 'SELECT now()'
10
+ NOW_SQL = "SELECT now()"
11
11
 
12
12
  class << self
13
13
  def count_schedulers
@@ -1,6 +1,5 @@
1
- require 'hashie'
2
- require 'fugit'
3
- require 'backports/2.4.0/hash/compact'
1
+ require "hashie"
2
+ require "fugit"
4
3
 
5
4
  # This is the definition of one scheduleable job in the que-scheduler config yml file.
6
5
  module Que
@@ -10,23 +9,23 @@ module Que
10
9
 
11
10
  DEFINED_JOB_TYPES = [
12
11
  DEFINED_JOB_TYPE_DEFAULT = :default,
13
- DEFINED_JOB_TYPE_EVERY_EVENT = :every_event
12
+ DEFINED_JOB_TYPE_EVERY_EVENT = :every_event,
14
13
  ].freeze
15
14
 
16
- property :name, required: true
17
- property :job_class, required: true, transform_with: lambda { |v|
18
- job_class = Object.const_get(v)
19
- job_class < Que::Job ? job_class : err_field(:job_class, v)
20
- }
21
- property :cron, required: true, transform_with: lambda { |v|
22
- Fugit::Cron.parse(v) || err_field(:cron, v)
23
- }
24
- property :queue, transform_with: ->(v) { v.is_a?(String) ? v : err_field(:queue, v) }
25
- property :priority, transform_with: ->(v) { v.is_a?(Integer) ? v : err_field(:priority, v) }
26
- property :args
27
- property :schedule_type, default: DEFINED_JOB_TYPE_DEFAULT, transform_with: lambda { |v|
28
- v.to_sym.tap { |vs| DEFINED_JOB_TYPES.include?(vs) || err_field(:schedule_type, v) }
29
- }
15
+ property :name
16
+ property :job_class, transform_with: ->(v) { Object.const_get(v) }
17
+ property :cron, transform_with: ->(v) { Fugit::Cron.parse(v) }
18
+ property :queue
19
+ property :priority
20
+ property :args_array
21
+ property :schedule_type, default: DEFINED_JOB_TYPE_DEFAULT
22
+
23
+ class << self
24
+ def create(options)
25
+ defined_job = new(options.compact)
26
+ defined_job.freeze.tap { |dj| dj.validate(options) }
27
+ end
28
+ end
30
29
 
31
30
  # Given a "last time", return the next Time the event will occur, or nil if it
32
31
  # is after "to".
@@ -49,36 +48,80 @@ module Que
49
48
  generate_to_enqueue_list(missed_times)
50
49
  end
51
50
 
52
- class << self
53
- private
51
+ def validate(options)
52
+ validate_fields_presence(options)
53
+ validate_fields_types(options)
54
+ validate_job_class_related(options)
55
+ end
56
+
57
+ private
54
58
 
55
- def err_field(field, value)
56
- schedule = Que::Scheduler.configuration.schedule_location
57
- raise "Invalid #{field} '#{value}' in que-scheduler schedule #{schedule}"
59
+ # rubocop:disable Style/GuardClause This reads better as a conditional
60
+ def validate_fields_types(options)
61
+ unless queue.nil? || queue.is_a?(String)
62
+ err_field(:queue, options, "queue must be a string")
63
+ end
64
+ unless priority.nil? || priority.is_a?(Integer)
65
+ err_field(:priority, options, "priority must be an integer")
66
+ end
67
+ unless DEFINED_JOB_TYPES.include?(schedule_type)
68
+ err_field(:schedule_type, options, "Not in #{DEFINED_JOB_TYPES}")
58
69
  end
59
70
  end
71
+ # rubocop:enable Style/GuardClause
60
72
 
61
- private
73
+ def validate_fields_presence(options)
74
+ err_field(:name, options, "name must be present") if name.nil?
75
+ err_field(:job_class, options, "job_class must be present") if job_class.nil?
76
+ # An invalid cron is nil
77
+ err_field(:cron, options, "cron must be present") if cron.nil?
78
+ end
79
+
80
+ def validate_job_class_related(options)
81
+ # Only support known job engines
82
+ unless Que::Scheduler::ToEnqueue.valid_job_class?(job_class)
83
+ err_field(:job_class, options, "Job #{job_class} was not a supported job type")
84
+ end
85
+
86
+ # queue name is only supported for a subrange of ActiveJob versions. Print this out as a
87
+ # warning.
88
+ if queue &&
89
+ Que::Scheduler::ToEnqueue.active_job_sufficient_version? &&
90
+ job_class < ::ActiveJob::Base &&
91
+ Que::Scheduler::ToEnqueue.active_job_version < Gem::Version.create("6.0.3")
92
+ puts <<-ERR
93
+ WARNING from que-scheduler....
94
+ Between versions 4.2.3 and 6.0.2 (inclusive) Rails did not support setting queue names
95
+ on que jobs with ActiveJob, so que-scheduler cannot support it.
96
+ See removed in Rails 4.2.3
97
+ https://github.com/rails/rails/pull/19498
98
+ And readded in Rails 6.0.3
99
+ https://github.com/rails/rails/pull/38635
100
+
101
+ Please remove all "queue" keys from ActiveJobs defined in the que-scheduler.yml config.
102
+ Specifically #{queue} for job #{name}.
103
+ ERR
104
+ end
105
+ end
62
106
 
63
- class ToEnqueue < Hashie::Dash
64
- property :args, required: true, default: []
65
- property :queue
66
- property :priority
67
- property :job_class, required: true
107
+ def err_field(field, options, reason = "")
108
+ schedule = Que::Scheduler.configuration.schedule_location
109
+ value = options[field]
110
+ raise "Invalid #{field} '#{value}' for '#{name}' in que-scheduler schedule #{schedule}.\n" \
111
+ "#{reason}"
68
112
  end
69
113
 
70
114
  def generate_to_enqueue_list(missed_times)
71
115
  return [] if missed_times.empty?
72
116
 
73
117
  options = to_h.slice(:args, :queue, :priority, :job_class).compact
74
- args_array = args.is_a?(Array) ? args : Array(args)
75
118
 
76
119
  if schedule_type == DefinedJob::DEFINED_JOB_TYPE_EVERY_EVENT
77
120
  missed_times.map do |time_missed|
78
- ToEnqueue.new(options.merge(args: [time_missed] + args_array))
121
+ ToEnqueue.create(options.merge(args: [time_missed] + args_array))
79
122
  end
80
123
  else
81
- [ToEnqueue.new(options.merge(args: args_array))]
124
+ [ToEnqueue.create(options.merge(args: args_array))]
82
125
  end
83
126
  end
84
127
  end
@@ -1,4 +1,4 @@
1
- require 'fugit'
1
+ require "fugit"
2
2
 
3
3
  module Que
4
4
  module Scheduler
@@ -0,0 +1,45 @@
1
+ require "que"
2
+
3
+ # This job can optionally be scheduled to clear down the que-scheduler audit log if it
4
+ # isn't required in the long term.
5
+ module Que
6
+ module Scheduler
7
+ module Jobs
8
+ class QueSchedulerAuditClearDownJob < Que::Job
9
+ class << self
10
+ def build_sql(table_name)
11
+ <<-SQL
12
+ WITH deleted AS (
13
+ DELETE FROM #{table_name}
14
+ WHERE scheduler_job_id <= (
15
+ SELECT scheduler_job_id FROM que_scheduler_audit
16
+ ORDER BY scheduler_job_id DESC
17
+ LIMIT 1 OFFSET $1
18
+ ) RETURNING *
19
+ ) SELECT count(*) FROM deleted;
20
+ SQL
21
+ end
22
+ end
23
+
24
+ DELETE_AUDIT_ENQUEUED_SQL = build_sql("que_scheduler_audit_enqueued").freeze
25
+ DELETE_AUDIT_SQL = build_sql("que_scheduler_audit").freeze
26
+
27
+ # Very low priority
28
+ Que::Scheduler::VersionSupport.set_priority(self, 100)
29
+
30
+ def run(options)
31
+ retain_row_count = options.fetch(:retain_row_count)
32
+ Que::Scheduler::Db.transaction do
33
+ # This may delete zero or more than `retain_row_count` depending on if anything was
34
+ # scheduled in each of the past schedule runs
35
+ Que::Scheduler::VersionSupport.execute(DELETE_AUDIT_ENQUEUED_SQL, [retain_row_count])
36
+ # This will delete all but `retain_row_count` oldest rows
37
+ count = Que::Scheduler::VersionSupport.execute(DELETE_AUDIT_SQL, [retain_row_count])
38
+ log = "#{self.class} cleared down #{count.first.fetch(:count)} rows"
39
+ ::Que.log(event: "que-scheduler".to_sym, message: log)
40
+ end
41
+ end
42
+ end
43
+ end
44
+ end
45
+ end
@@ -0,0 +1 @@
1
+ CREATE UNIQUE INDEX index_que_scheduler_audit_on_scheduler_job_id ON que_scheduler_audit USING btree (scheduler_job_id);
@@ -0,0 +1 @@
1
+ DROP INDEX index_que_scheduler_audit_on_scheduler_job_id;
@@ -1,4 +1,4 @@
1
- require_relative 'defined_job'
1
+ require_relative "defined_job"
2
2
 
3
3
  module Que
4
4
  module Scheduler
@@ -12,9 +12,16 @@ module Que
12
12
  end
13
13
  end
14
14
 
15
+ def schedule=(schedule_config)
16
+ @schedule = schedule_config.nil? ? nil : from_yaml(schedule_config)
17
+ end
18
+
15
19
  def from_file(location)
16
- yml = IO.read(location)
17
- config_hash = YAML.safe_load(yml)
20
+ from_yaml(IO.read(location))
21
+ end
22
+
23
+ def from_yaml(config)
24
+ config_hash = YAML.safe_load(config)
18
25
  from_hash(config_hash)
19
26
  end
20
27
 
@@ -24,20 +31,36 @@ module Que
24
31
  end.to_h
25
32
  end
26
33
 
27
- private
28
-
29
34
  def hash_item_to_defined_job(name, defined_job_hash)
30
- Que::Scheduler::DefinedJob.new(
31
- {
32
- name: name,
33
- job_class: defined_job_hash['class'] || name,
34
- queue: defined_job_hash['queue'],
35
- args: defined_job_hash['args'],
36
- priority: defined_job_hash['priority'],
37
- cron: defined_job_hash['cron'],
38
- schedule_type: defined_job_hash['schedule_type'],
39
- }.compact
40
- ).freeze
35
+ # Que stores arguments as a json array. If the args we have to provide are already an
36
+ # array we can can simply pass them through. If it is a single non-nil value, then we make
37
+ # an array with one item which is that value (this includes if it is a hash). It could
38
+ # also be a single nil value.
39
+ args_array =
40
+ if !defined_job_hash.key?("args")
41
+ # No args were requested
42
+ []
43
+ else
44
+ args = defined_job_hash["args"]
45
+ if args.is_a?(Array)
46
+ # An array of args was requested
47
+ args
48
+ else
49
+ # A single value, a nil, or a hash was requested. que expects this to
50
+ # be enqueued as an array of 1 item
51
+ [args]
52
+ end
53
+ end
54
+
55
+ Que::Scheduler::DefinedJob.create(
56
+ name: name,
57
+ job_class: defined_job_hash["class"] || name,
58
+ queue: defined_job_hash["queue"],
59
+ args_array: args_array,
60
+ priority: defined_job_hash["priority"],
61
+ cron: defined_job_hash["cron"],
62
+ schedule_type: defined_job_hash["schedule_type"]&.to_sym
63
+ )
41
64
  end
42
65
  end
43
66
  end
@@ -46,6 +69,10 @@ module Que
46
69
  def schedule
47
70
  Schedule.schedule
48
71
  end
72
+
73
+ def schedule=(value)
74
+ Schedule.schedule = value
75
+ end
49
76
  end
50
77
  end
51
78
  end
@@ -1,10 +1,13 @@
1
- require 'que'
1
+ require "que"
2
+ require "active_support"
3
+ require "active_support/core_ext/time"
2
4
 
3
- require_relative 'schedule'
4
- require_relative 'enqueueing_calculator'
5
- require_relative 'scheduler_job_args'
6
- require_relative 'state_checks'
7
- require_relative 'version_support'
5
+ require_relative "schedule"
6
+ require_relative "enqueueing_calculator"
7
+ require_relative "scheduler_job_args"
8
+ require_relative "state_checks"
9
+ require_relative "to_enqueue"
10
+ require_relative "version_support"
8
11
 
9
12
  # The main job that runs every minute, determining what needs to be enqueued, enqueues the required
10
13
  # jobs, then re-enqueues itself.
@@ -22,46 +25,36 @@ module Que
22
25
 
23
26
  scheduler_job_args = SchedulerJobArgs.build(options)
24
27
  logs = ["que-scheduler last ran at #{scheduler_job_args.last_run_time}."]
25
-
26
28
  result = EnqueueingCalculator.parse(Scheduler.schedule.values, scheduler_job_args)
27
29
  enqueued_jobs = enqueue_required_jobs(result, logs)
28
30
  enqueue_self_again(
29
31
  scheduler_job_args, scheduler_job_args.as_time, result.job_dictionary, enqueued_jobs
30
32
  )
31
-
32
33
  # Only now we're sure nothing errored, log the results
33
- logs.each { |str| ::Que.log(event: 'que-scheduler'.to_sym, message: str) }
34
+ logs.each { |str| ::Que.log(event: "que-scheduler".to_sym, message: str) }
34
35
  destroy
35
36
  end
36
37
  end
37
38
 
38
- def enqueue_required_jobs(result, logs)
39
- result.missed_jobs.map do |to_enqueue|
40
- job_class = to_enqueue.job_class
41
- args = to_enqueue.args
42
- remaining_hash = to_enqueue.except(:job_class, :args)
43
- enqueued_job =
44
- if args.is_a?(Hash)
45
- job_class.enqueue(args.merge(remaining_hash))
46
- else
47
- job_class.enqueue(*args, remaining_hash)
48
- end
49
- check_enqueued_job(enqueued_job, job_class, args, logs)
39
+ def enqueue_required_jobs(calculator_result, logs)
40
+ calculator_result.missed_jobs.map do |to_enqueue|
41
+ to_enqueue.enqueue.tap do |enqueued_job|
42
+ check_enqueued_job(to_enqueue, enqueued_job, logs)
43
+ end
50
44
  end.compact
51
45
  end
52
46
 
53
47
  private
54
48
 
55
- def check_enqueued_job(enqueued_job, job_class, args, logs)
56
- if enqueued_job.is_a?(Que::Job)
57
- job_id = Que::Scheduler::VersionSupport.job_attributes(enqueued_job).fetch(:job_id)
58
- logs << "que-scheduler enqueueing #{job_class} #{job_id} with args: #{args}"
59
- enqueued_job
60
- else
61
- # This can happen if a middleware nixes the enqueue call
62
- logs << "que-scheduler called enqueue on #{job_class} but did not receive a #{Que::Job}"
63
- nil
64
- end
49
+ def check_enqueued_job(to_enqueue, enqueued_job, logs)
50
+ logs << if enqueued_job.present?
51
+ "que-scheduler enqueueing #{enqueued_job.job_class} " \
52
+ "#{enqueued_job.job_id} with args: #{enqueued_job.args}"
53
+ else
54
+ # This can happen if a middleware nixes the enqueue call
55
+ "que-scheduler called enqueue on #{to_enqueue.job_class} " \
56
+ "but it reported no job was scheduled. Has `enqueue` been overridden?"
57
+ end
65
58
  end
66
59
 
67
60
  def enqueue_self_again(scheduler_job_args, last_full_execution, job_dictionary, enqueued_jobs)
@@ -79,8 +72,8 @@ module Que
79
72
  )
80
73
 
81
74
  # rubocop:disable Style/GuardClause This reads better as a conditional
82
- unless check_enqueued_job(enqueued_job, SchedulerJob, {}, [])
83
- raise 'SchedulerJob could not self-schedule. Has `.enqueue` been monkey patched?'
75
+ unless Que::Scheduler::VersionSupport.job_attributes(enqueued_job).fetch(:job_id)
76
+ raise "SchedulerJob could not self-schedule. Has `.enqueue` been monkey patched?"
84
77
  end
85
78
  # rubocop:enable Style/GuardClause
86
79
  end
@@ -1,6 +1,6 @@
1
- require 'hashie'
2
- require 'active_support'
3
- require 'active_support/time_with_zone'
1
+ require "hashie"
2
+ require "active_support"
3
+ require "active_support/time_with_zone"
4
4
 
5
5
  # These are the args that are used for this particular run of the scheduler.
6
6
  module Que
@@ -1,6 +1,6 @@
1
- require_relative 'audit'
2
- require_relative 'db'
3
- require_relative 'migrations'
1
+ require_relative "audit"
2
+ require_relative "db"
3
+ require_relative "migrations"
4
4
 
5
5
  module Que
6
6
  module Scheduler
@@ -0,0 +1,157 @@
1
+ require "que"
2
+
3
+ # This module uses polymorphic dispatch to centralise the differences between supporting Que::Job
4
+ # and other job systems.
5
+ module Que
6
+ module Scheduler
7
+ class ToEnqueue < Hashie::Dash
8
+ property :args, required: true, default: []
9
+ property :queue
10
+ property :priority
11
+ property :run_at, required: true
12
+ property :job_class, required: true
13
+
14
+ class << self
15
+ def create(options)
16
+ type_from_job_class(options.fetch(:job_class)).new(
17
+ options.merge(run_at: Que::Scheduler::Db.now)
18
+ )
19
+ end
20
+
21
+ def valid_job_class?(job_class)
22
+ type_from_job_class(job_class).present?
23
+ end
24
+
25
+ def active_job_version
26
+ Gem.loaded_specs["activejob"]&.version
27
+ end
28
+
29
+ def active_job_sufficient_version?
30
+ # ActiveJob 4.x does not support job_ids correctly
31
+ # https://github.com/rails/rails/pull/20056/files
32
+ active_job_version && active_job_version > Gem::Version.create("5")
33
+ end
34
+
35
+ def active_job_version_supports_queues?
36
+ # Supporting queue name in ActiveJob was removed in Rails 4.2.3
37
+ # https://github.com/rails/rails/pull/19498
38
+ # and readded in Rails 6.0.3
39
+ # https://github.com/rails/rails/pull/38635
40
+ ToEnqueue.active_job_version && ToEnqueue.active_job_version >=
41
+ Gem::Version.create("6.0.3")
42
+ end
43
+
44
+ private
45
+
46
+ def type_from_job_class(job_class)
47
+ types.each do |type, implementation|
48
+ return implementation if job_class < type
49
+ end
50
+ nil
51
+ end
52
+
53
+ def types
54
+ @types ||=
55
+ begin
56
+ hash = {
57
+ ::Que::Job => QueJobType,
58
+ }
59
+ hash[::ActiveJob::Base] = ActiveJobType if ToEnqueue.active_job_sufficient_version?
60
+ hash
61
+ end
62
+ end
63
+ end
64
+ end
65
+
66
+ # For jobs of type Que::Job
67
+ class QueJobType < ToEnqueue
68
+ def enqueue
69
+ job_settings = to_h.slice(:queue, :priority, :run_at).compact
70
+ job =
71
+ if args.is_a?(Hash)
72
+ job_class.enqueue(**args.merge(job_settings))
73
+ else
74
+ job_class.enqueue(*args, **job_settings)
75
+ end
76
+
77
+ return nil if job.nil? || !job # nil in Rails < 6.1, false after.
78
+
79
+ # Now read the just inserted job back out of the DB to get the actual values that will
80
+ # be used when the job is worked.
81
+ values = Que::Scheduler::VersionSupport.job_attributes(job).slice(
82
+ :args, :queue, :priority, :run_at, :job_class, :job_id
83
+ )
84
+ EnqueuedJobType.new(values)
85
+ end
86
+ end
87
+
88
+ # For jobs of type ActiveJob
89
+ class ActiveJobType < ToEnqueue
90
+ def enqueue
91
+ job = enqueue_active_job
92
+
93
+ return nil if job.nil? || !job # nil in Rails < 6.1, false after.
94
+
95
+ enqueued_values = calculate_enqueued_values(job)
96
+ EnqueuedJobType.new(enqueued_values)
97
+ end
98
+
99
+ def calculate_enqueued_values(job)
100
+ # Now read the just inserted job back out of the DB to get the actual values that will
101
+ # be used when the job is worked.
102
+ data = JSON.parse(job.to_json, symbolize_names: true)
103
+
104
+ # ActiveJob scheduled_at is returned as a float, where we want a Time for consistency
105
+ scheduled_at =
106
+ begin
107
+ scheduled_at_float = data[:scheduled_at]
108
+ scheduled_at_float ? Time.zone.at(scheduled_at_float) : nil
109
+ end
110
+
111
+ # Rails didn't support queues for ActiveJob for a while
112
+ used_queue = data[:queue_name] if ToEnqueue.active_job_version_supports_queues?
113
+
114
+ # We can't get the priority out of the DB, as the returned `job` doesn't give us access
115
+ # to the underlying ActiveJob that was scheduled. We have no option but to assume
116
+ # it was what we told it to use. If no priority was specified, we must assume it was
117
+ # the Que default, which is 100 t.ly/1jRK5
118
+ assume_used_priority = priority.nil? ? 100 : priority
119
+
120
+ {
121
+ args: data.fetch(:arguments),
122
+ queue: used_queue,
123
+ priority: assume_used_priority,
124
+ run_at: scheduled_at,
125
+ job_class: job_class.to_s,
126
+ job_id: data.fetch(:provider_job_id),
127
+ }
128
+ end
129
+
130
+ def enqueue_active_job
131
+ job_settings = {
132
+ priority: priority,
133
+ wait_until: run_at,
134
+ queue: queue || Que::Scheduler::VersionSupport.default_scheduler_queue,
135
+ }.compact
136
+
137
+ job_class_set = job_class.set(**job_settings)
138
+ if args.is_a?(Hash)
139
+ job_class_set.perform_later(**args)
140
+ else
141
+ job_class_set.perform_later(*args)
142
+ end
143
+ end
144
+ end
145
+
146
+ # A value object returned after a job has been enqueued. This is necessary as Que (normal) and
147
+ # ActiveJob return very different objects from the `enqueue` call.
148
+ class EnqueuedJobType < Hashie::Dash
149
+ property :args
150
+ property :queue
151
+ property :priority
152
+ property :run_at, required: true
153
+ property :job_class, required: true
154
+ property :job_id, required: true
155
+ end
156
+ end
157
+ end
@@ -1,5 +1,5 @@
1
1
  module Que
2
2
  module Scheduler
3
- VERSION = '3.2.8'.freeze
3
+ VERSION = "3.4.3".freeze
4
4
  end
5
5
  end
@@ -1,4 +1,4 @@
1
- require 'que'
1
+ require "que"
2
2
 
3
3
  # The purpose of this module is to centralise the differences when supporting both que 0.x and
4
4
  # 1.x with the same gem.
@@ -15,7 +15,7 @@ module Que
15
15
  # the top of all jobs it enqueues.
16
16
  def set_priority(context, priority)
17
17
  if zero_major?
18
- context.instance_variable_set('@priority', priority)
18
+ context.instance_variable_set("@priority", priority)
19
19
  else
20
20
  context.priority = priority
21
21
  end
@@ -24,7 +24,7 @@ module Que
24
24
  # Ensure the job runs at least once an hour when it is backing off due to errors
25
25
  def apply_retry_semantics(context)
26
26
  if zero_major?
27
- context.instance_variable_set('@retry_interval', RETRY_PROC)
27
+ context.instance_variable_set("@retry_interval", RETRY_PROC)
28
28
  else
29
29
  context.maximum_retry_count = 1 << 128 # Heat death of universe
30
30
  context.retry_interval = RETRY_PROC
@@ -33,9 +33,9 @@ module Que
33
33
 
34
34
  def job_attributes(enqueued_job)
35
35
  if zero_major?
36
- enqueued_job.attrs.transform_keys(&:to_sym)
36
+ enqueued_job.attrs.to_h.transform_keys(&:to_sym)
37
37
  else
38
- enqueued_job.que_attrs.transform_keys(&:to_sym).tap do |hash|
38
+ enqueued_job.que_attrs.to_h.transform_keys(&:to_sym).tap do |hash|
39
39
  hash[:job_id] = hash.delete(:id)
40
40
  end
41
41
  end
@@ -48,7 +48,7 @@ module Que
48
48
  end
49
49
 
50
50
  def default_scheduler_queue
51
- zero_major? ? '' : Que::DEFAULT_QUEUE
51
+ zero_major? ? "" : Que::DEFAULT_QUEUE
52
52
  end
53
53
 
54
54
  def running_synchronously?
@@ -56,18 +56,18 @@ module Que
56
56
  end
57
57
 
58
58
  def running_synchronously_code?
59
- zero_major? ? 'Que.mode == :sync' : 'Que.run_synchronously = true'
59
+ zero_major? ? "Que.mode == :sync" : "Que.run_synchronously = true"
60
60
  end
61
61
 
62
62
  def zero_major?
63
63
  # This is the only way to handle beta releases too
64
- @zero_major ||= Gem.loaded_specs['que'].version.to_s.split('.').first.to_i.zero?
64
+ @zero_major ||= Gem.loaded_specs["que"].version.to_s.split(".").first.to_i.zero?
65
65
  end
66
66
 
67
67
  private
68
68
 
69
69
  def normalise_array_of_hashes(array)
70
- array.map { |row| row.transform_keys(&:to_sym) }
70
+ array.map { |row| row.to_h.transform_keys(&:to_sym) }
71
71
  end
72
72
  end
73
73
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: que-scheduler
3
3
  version: !ruby/object:Gem::Version
4
- version: 3.2.8
4
+ version: 3.4.3
5
5
  platform: ruby
6
6
  authors:
7
7
  - Harry Lascelles
8
- autorequire:
8
+ autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2020-02-09 00:00:00.000000000 Z
11
+ date: 2020-10-28 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: activesupport
@@ -24,20 +24,6 @@ dependencies:
24
24
  - - ">="
25
25
  - !ruby/object:Gem::Version
26
26
  version: '4.0'
27
- - !ruby/object:Gem::Dependency
28
- name: backports
29
- requirement: !ruby/object:Gem::Requirement
30
- requirements:
31
- - - "~>"
32
- - !ruby/object:Gem::Version
33
- version: '3.10'
34
- type: :runtime
35
- prerelease: false
36
- version_requirements: !ruby/object:Gem::Requirement
37
- requirements:
38
- - - "~>"
39
- - !ruby/object:Gem::Version
40
- version: '3.10'
41
27
  - !ruby/object:Gem::Dependency
42
28
  name: fugit
43
29
  requirement: !ruby/object:Gem::Requirement
@@ -228,16 +214,16 @@ dependencies:
228
214
  name: reek
229
215
  requirement: !ruby/object:Gem::Requirement
230
216
  requirements:
231
- - - '='
217
+ - - ">="
232
218
  - !ruby/object:Gem::Version
233
- version: 5.0.2
219
+ version: '0'
234
220
  type: :development
235
221
  prerelease: false
236
222
  version_requirements: !ruby/object:Gem::Requirement
237
223
  requirements:
238
- - - '='
224
+ - - ">="
239
225
  - !ruby/object:Gem::Version
240
- version: 5.0.2
226
+ version: '0'
241
227
  - !ruby/object:Gem::Dependency
242
228
  name: rspec
243
229
  requirement: !ruby/object:Gem::Requirement
@@ -256,16 +242,30 @@ dependencies:
256
242
  name: rubocop
257
243
  requirement: !ruby/object:Gem::Requirement
258
244
  requirements:
259
- - - "~>"
245
+ - - '='
260
246
  - !ruby/object:Gem::Version
261
- version: 0.68.1
247
+ version: 0.84.0
262
248
  type: :development
263
249
  prerelease: false
264
250
  version_requirements: !ruby/object:Gem::Requirement
265
251
  requirements:
266
- - - "~>"
252
+ - - '='
253
+ - !ruby/object:Gem::Version
254
+ version: 0.84.0
255
+ - !ruby/object:Gem::Dependency
256
+ name: rubocop-rspec
257
+ requirement: !ruby/object:Gem::Requirement
258
+ requirements:
259
+ - - ">="
267
260
  - !ruby/object:Gem::Version
268
- version: 0.68.1
261
+ version: '0'
262
+ type: :development
263
+ prerelease: false
264
+ version_requirements: !ruby/object:Gem::Requirement
265
+ requirements:
266
+ - - ">="
267
+ - !ruby/object:Gem::Version
268
+ version: '0'
269
269
  - !ruby/object:Gem::Dependency
270
270
  name: sqlite3
271
271
  requirement: !ruby/object:Gem::Requirement
@@ -323,6 +323,7 @@ files:
323
323
  - lib/que/scheduler/db.rb
324
324
  - lib/que/scheduler/defined_job.rb
325
325
  - lib/que/scheduler/enqueueing_calculator.rb
326
+ - lib/que/scheduler/jobs/que_scheduler_audit_clear_down_job.rb
326
327
  - lib/que/scheduler/migrations.rb
327
328
  - lib/que/scheduler/migrations/1/down.sql
328
329
  - lib/que/scheduler/migrations/1/up.sql
@@ -332,10 +333,13 @@ files:
332
333
  - lib/que/scheduler/migrations/3/up.sql
333
334
  - lib/que/scheduler/migrations/4/down.sql
334
335
  - lib/que/scheduler/migrations/4/up.sql
336
+ - lib/que/scheduler/migrations/5/down.sql
337
+ - lib/que/scheduler/migrations/5/up.sql
335
338
  - lib/que/scheduler/schedule.rb
336
339
  - lib/que/scheduler/scheduler_job.rb
337
340
  - lib/que/scheduler/scheduler_job_args.rb
338
341
  - lib/que/scheduler/state_checks.rb
342
+ - lib/que/scheduler/to_enqueue.rb
339
343
  - lib/que/scheduler/version.rb
340
344
  - lib/que/scheduler/version_support.rb
341
345
  homepage: https://github.com/hlascelles/que-scheduler
@@ -347,7 +351,7 @@ metadata:
347
351
  changelog_uri: https://github.com/hlascelles/que-scheduler/blob/master/CHANGELOG.md
348
352
  source_code_uri: https://github.com/hlascelles/que-scheduler/
349
353
  bug_tracker_uri: https://github.com/hlascelles/que-scheduler/issues
350
- post_install_message:
354
+ post_install_message:
351
355
  rdoc_options: []
352
356
  require_paths:
353
357
  - lib
@@ -362,9 +366,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
362
366
  - !ruby/object:Gem::Version
363
367
  version: '0'
364
368
  requirements: []
365
- rubyforge_project:
366
- rubygems_version: 2.7.6.2
367
- signing_key:
369
+ rubygems_version: 3.0.3
370
+ signing_key:
368
371
  specification_version: 4
369
372
  summary: A cron scheduler for Que
370
373
  test_files: []