que-scheduler 3.3.0 → 3.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 5707036df23117e995d696388d17bfc0488eef9c7ba30211b1a3700e6f0fb94f
4
- data.tar.gz: 3133e78529f3658c9509e0a1240189424caefc8219b7da035530e68fdbfb38c6
3
+ metadata.gz: 53cadc9fafa321124d4f52caa66503d8590cb1eaa11ee2ac86e0208c5d6a0fef
4
+ data.tar.gz: 2a5731c305089b58c3841f51b67c2dc698718e123a9b68395e92e1dfff37a693
5
5
  SHA512:
6
- metadata.gz: 4609167d5adfc2e7b9c583d3cbece89dd34e8624512b0c8c257804c05934964a343336b192944ce2790fde4f4b6029173df2f76d28fe7daee72e0a1ae473bad2
7
- data.tar.gz: d55658a7923c50010a8a38713dc25c0f08d182004cb5156a336921f55b35e91631edb042f5e76f194d58ed04245ea8f08c78dd64386c2479ee4bdb8ba9a4512a
6
+ metadata.gz: '087a32c4e9245c3bbaa26fb0e891f2b232c2617bad351d354eb502c0edb853a32ec786601a71e796fe5baa1b13f3691f1dc1c17f55d1b1973648101cd60602c7'
7
+ data.tar.gz: 1ba223766fa1d52b79b7ebcdcd534da366636cc306f538e6bc067830a2999b57913c1ade0f5a381287883222aedadc4aa380f9fb479223838468f7df3dfa5d73
data/README.md CHANGED
@@ -22,12 +22,13 @@ needs to be run, enqueueing those jobs, then enqueueing itself to check again la
22
22
  look for it is `config/que_schedule.yml`. They are essentially the same as resque-scheduler
23
23
  files, but with additional features.
24
24
 
25
- 1. Add a migration to start the job scheduler and prepare the audit table. Note that this migration will fail if Que is set to execute jobs synchronously, i.e. `Que::Job.run_synchronously = true`.
25
+ 1. Add a migration to start the job scheduler and prepare the audit table. Note that this migration
26
+ will fail if Que is set to execute jobs synchronously, i.e. `Que::Job.run_synchronously = true`.
26
27
 
27
28
  ```ruby
28
29
  class CreateQueSchedulerSchema < ActiveRecord::Migration
29
30
  def change
30
- Que::Scheduler::Migrations.migrate!(version: 4)
31
+ Que::Scheduler::Migrations.migrate!(version: 5)
31
32
  end
32
33
  end
33
34
  ```
@@ -65,10 +66,21 @@ BatchOrders:
65
66
  cron: "0 0 * * *"
66
67
  priority: 25
67
68
 
68
- # Specify job arguments
69
+ # Specify array job arguments
69
70
  SendOrders:
70
71
  cron: "0 0 * * *"
71
72
  args: ['open']
73
+
74
+ # Specify hash job arguments. Note, this appears as a single hash to `run`, not as kwargs.
75
+ SendPreorders:
76
+ cron: "0 0 * * *"
77
+ args:
78
+ order_type: special
79
+
80
+ # Specify a single nil argument
81
+ SendPostorders:
82
+ cron: "0 0 * * *"
83
+ args: ~ # See https://stackoverflow.com/a/51990876/1267203
72
84
 
73
85
  # Use simpler cron syntax
74
86
  SendBilling:
@@ -136,8 +148,54 @@ end
136
148
 
137
149
  ## Scheduler Audit
138
150
 
139
- An audit table _que_scheduler_audit_ is written to by the scheduler to keep a history of what jobs
140
- were enqueued when. It is created by the included migration tasks.
151
+ An audit table `que_scheduler_audit` is written to by the scheduler to keep a history of when the
152
+ scheduler ran to calculate what was necessary to run (if anything). It is created by the included
153
+ migration tasks.
154
+
155
+ Additionally, there is the audit table `que_scheduler_audit_enqueued`. This logs every job that
156
+ the scheduler enqueues.
157
+
158
+ When there is a major version (breaking) change, a migration should be run in. The version of the
159
+ migration proceeds at a faster rate than the version of the gem. To run in all the migrations required
160
+ up to a number, just migrate to that number with one line, and it will perform all the intermediary steps.
161
+
162
+ ie, This will perform all migrations necessary up to the latest version, skipping any already
163
+ performed.
164
+
165
+ ```ruby
166
+ class CreateQueSchedulerSchema < ActiveRecord::Migration
167
+ def change
168
+ Que::Scheduler::Migrations.migrate!(version: 5)
169
+ end
170
+ end
171
+ ```
172
+
173
+ The changes in past migrations were:
174
+
175
+ | Version | Changes |
176
+ |:-------:|---------------------------------------------------------------------------------|
177
+ | 1 | Enqueued the main Que::Scheduler. This is the job that performs the scheduling. |
178
+ | 2 | Added the audit table `que_scheduler_audit`. |
179
+ | 3 | Added the audit table `que_scheduler_audit_enqueued`. |
180
+ | 4 | Updated the the audit tables to use bigints |
181
+ | 5 | Dropped an unnecessary index |
182
+
183
+ ## Built in optional job for audit clear down
184
+
185
+ que-scheduler comes with the `QueSchedulerAuditClearDownJob` job built in that you can optionally
186
+ schedule to clear down audit rows if you don't need to retain them indefinitely. You should add this
187
+ to your own scheduler config yaml.
188
+
189
+ For example:
190
+
191
+ ```yaml
192
+ # This will clear down the oldest que-scheduler audit rows. Since que-scheduler
193
+ # runs approximately every minute, 129600 is 90 days.
194
+ Que::Scheduler::Jobs::QueSchedulerAuditClearDownJob:
195
+ cron: "0 0 * * *"
196
+ args:
197
+ retain_row_count: 129600
198
+ ```
141
199
 
142
200
  ## HA Redundancy and DB restores
143
201
 
@@ -172,23 +230,6 @@ then reschedules itself. The flow is as follows:
172
230
  1. After a deploy that changes the schedule, the job notices any new jobs to schedule, and knows which
173
231
  ones to forget. It does not need to be re-enqueued or restarted.
174
232
 
175
- ## DB Migrations
176
-
177
- When there is a major version (breaking) change, a migration should be run in. The version of the
178
- migration proceeds at a faster rate than the version of the gem. To run in all the migrations required
179
- up to a number, just migrate to that number with one line, and it will perform all the intermediary steps.
180
-
181
- ie, `Que::Scheduler::Migrations.migrate!(version: 4)` will perform all migrations necessary to
182
- reach migration version `4`.
183
-
184
- As of migration `4`, two elements are added to the DB for que-scheduler to run.
185
-
186
- 1. The first is the scheduler job itself, which runs forever, re-enqueuing itself to performs its
187
- duties.
188
- 1. The second part comprises the audit table `que_scheduler_audit` and the "enqueued" table
189
- `que_scheduler_audit_enqueued`. The first tracks when the scheduler calculated what was necessary to run
190
- (if anything). The second then logs every job that the scheduler enqueues.
191
-
192
233
  ## Testing Configuration
193
234
 
194
235
  You can add tests to validate your configuration during the spec phase. This will perform a variety
@@ -236,3 +277,4 @@ This gem was inspired by the makers of the excellent [Que](https://github.com/ch
236
277
  * @joehorsnell
237
278
  * @bnauta
238
279
  * @papodaca
280
+ * @krzyzak
@@ -5,3 +5,4 @@ require 'que/scheduler/scheduler_job'
5
5
  require 'que/scheduler/db'
6
6
  require 'que/scheduler/audit'
7
7
  require 'que/scheduler/migrations'
8
+ require 'que/scheduler/jobs/que_scheduler_audit_clear_down_job'
@@ -1,6 +1,5 @@
1
1
  require 'hashie'
2
2
  require 'fugit'
3
- require 'backports/2.4.0/hash/compact'
4
3
 
5
4
  # This is the definition of one scheduleable job in the que-scheduler config yml file.
6
5
  module Que
@@ -18,7 +17,7 @@ module Que
18
17
  property :cron, transform_with: ->(v) { Fugit::Cron.parse(v) }
19
18
  property :queue
20
19
  property :priority
21
- property :args
20
+ property :args_array
22
21
  property :schedule_type, default: DEFINED_JOB_TYPE_DEFAULT
23
22
 
24
23
  class << self
@@ -116,7 +115,6 @@ module Que
116
115
  return [] if missed_times.empty?
117
116
 
118
117
  options = to_h.slice(:args, :queue, :priority, :job_class).compact
119
- args_array = args.is_a?(Array) ? args : Array(args)
120
118
 
121
119
  if schedule_type == DefinedJob::DEFINED_JOB_TYPE_EVERY_EVENT
122
120
  missed_times.map do |time_missed|
@@ -0,0 +1,45 @@
1
+ require 'que'
2
+
3
+ # This job can optionally be scheduled to clear down the que-scheduler audit log if it
4
+ # isn't required in the long term.
5
+ module Que
6
+ module Scheduler
7
+ module Jobs
8
+ class QueSchedulerAuditClearDownJob < Que::Job
9
+ class << self
10
+ def build_sql(table_name)
11
+ <<-SQL
12
+ WITH deleted AS (
13
+ DELETE FROM #{table_name}
14
+ WHERE scheduler_job_id <= (
15
+ SELECT scheduler_job_id FROM que_scheduler_audit
16
+ ORDER BY scheduler_job_id DESC
17
+ LIMIT 1 OFFSET $1
18
+ ) RETURNING *
19
+ ) SELECT count(*) FROM deleted;
20
+ SQL
21
+ end
22
+ end
23
+
24
+ DELETE_AUDIT_ENQUEUED_SQL = build_sql('que_scheduler_audit_enqueued').freeze
25
+ DELETE_AUDIT_SQL = build_sql('que_scheduler_audit').freeze
26
+
27
+ # Very low priority
28
+ Que::Scheduler::VersionSupport.set_priority(self, 100)
29
+
30
+ def run(options)
31
+ retain_row_count = options.fetch(:retain_row_count)
32
+ Que::Scheduler::Db.transaction do
33
+ # This may delete zero or more than `retain_row_count` depending on if anything was
34
+ # scheduled in each of the past schedule runs
35
+ Que::Scheduler::VersionSupport.execute(DELETE_AUDIT_ENQUEUED_SQL, [retain_row_count])
36
+ # This will delete all but `retain_row_count` oldest rows
37
+ count = Que::Scheduler::VersionSupport.execute(DELETE_AUDIT_SQL, [retain_row_count])
38
+ log = "#{self.class} cleared down #{count.first.fetch(:count)} rows"
39
+ ::Que.log(event: 'que-scheduler'.to_sym, message: log)
40
+ end
41
+ end
42
+ end
43
+ end
44
+ end
45
+ end
@@ -0,0 +1 @@
1
+ CREATE UNIQUE INDEX index_que_scheduler_audit_on_scheduler_job_id ON que_scheduler_audit USING btree (scheduler_job_id);
@@ -0,0 +1 @@
1
+ DROP INDEX index_que_scheduler_audit_on_scheduler_job_id;
@@ -24,14 +24,32 @@ module Que
24
24
  end.to_h
25
25
  end
26
26
 
27
- private
28
-
29
27
  def hash_item_to_defined_job(name, defined_job_hash)
28
+ # Que stores arguments as a json array. If the args we have to provide are already an
29
+ # array we can can simply pass them through. If it is a single non-nil value, then we make
30
+ # an array with one item which is that value (this includes if it is a hash). It could
31
+ # also be a single nil value.
32
+ args_array =
33
+ if !defined_job_hash.key?('args')
34
+ # No args were requested
35
+ []
36
+ else
37
+ args = defined_job_hash['args']
38
+ if args.is_a?(Array)
39
+ # An array of args was requested
40
+ args
41
+ else
42
+ # A single value, a nil, or a hash was requested. que expects this to
43
+ # be enqueued as an array of 1 item
44
+ [args]
45
+ end
46
+ end
47
+
30
48
  Que::Scheduler::DefinedJob.create(
31
49
  name: name,
32
50
  job_class: defined_job_hash['class'] || name,
33
51
  queue: defined_job_hash['queue'],
34
- args: defined_job_hash['args'],
52
+ args_array: args_array,
35
53
  priority: defined_job_hash['priority'],
36
54
  cron: defined_job_hash['cron'],
37
55
  schedule_type: defined_job_hash['schedule_type']&.to_sym
@@ -34,8 +34,8 @@ module Que
34
34
  end
35
35
  end
36
36
 
37
- def enqueue_required_jobs(result, logs)
38
- result.missed_jobs.map do |to_enqueue|
37
+ def enqueue_required_jobs(calculator_result, logs)
38
+ calculator_result.missed_jobs.map do |to_enqueue|
39
39
  to_enqueue.enqueue.tap do |enqueued_job|
40
40
  check_enqueued_job(to_enqueue, enqueued_job, logs)
41
41
  end
@@ -32,6 +32,15 @@ module Que
32
32
  active_job_version && active_job_version > Gem::Version.create('5')
33
33
  end
34
34
 
35
+ def active_job_version_supports_queues?
36
+ # Supporting queue name in ActiveJob was removed in Rails 4.2.3
37
+ # https://github.com/rails/rails/pull/19498
38
+ # and readded in Rails 6.0.3
39
+ # https://github.com/rails/rails/pull/38635
40
+ ToEnqueue.active_job_version && ToEnqueue.active_job_version >=
41
+ Gem::Version.create('6.0.3')
42
+ end
43
+
35
44
  private
36
45
 
37
46
  def type_from_job_class(job_class)
@@ -87,8 +96,6 @@ module Que
87
96
  EnqueuedJobType.new(enqueued_values)
88
97
  end
89
98
 
90
- private
91
-
92
99
  def calculate_enqueued_values(job)
93
100
  # Now read the just inserted job back out of the DB to get the actual values that will
94
101
  # be used when the job is worked.
@@ -101,9 +108,8 @@ module Que
101
108
  scheduled_at_float ? Time.zone.at(scheduled_at_float) : nil
102
109
  end
103
110
 
104
- # Rails doesn't support queues for ActiveJob
105
- # https://github.com/rails/rails/pull/19498
106
- used_queue = nil
111
+ # Rails didn't support queues for ActiveJob for a while
112
+ used_queue = data[:queue_name] if ToEnqueue.active_job_version_supports_queues?
107
113
 
108
114
  # We can't get the priority out of the DB, as the returned `job` doesn't give us access
109
115
  # to the underlying ActiveJob that was scheduled. We have no option but to assume
@@ -125,6 +131,7 @@ module Que
125
131
  job_settings = {
126
132
  priority: priority,
127
133
  wait_until: run_at,
134
+ queue: queue || Que::Scheduler::VersionSupport.default_scheduler_queue,
128
135
  }.compact
129
136
 
130
137
  job_class_set = job_class.set(**job_settings)
@@ -1,5 +1,5 @@
1
1
  module Que
2
2
  module Scheduler
3
- VERSION = '3.3.0'.freeze
3
+ VERSION = '3.4.0'.freeze
4
4
  end
5
5
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: que-scheduler
3
3
  version: !ruby/object:Gem::Version
4
- version: 3.3.0
4
+ version: 3.4.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Harry Lascelles
8
- autorequire:
8
+ autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2020-04-04 00:00:00.000000000 Z
11
+ date: 2020-06-07 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: activesupport
@@ -24,20 +24,6 @@ dependencies:
24
24
  - - ">="
25
25
  - !ruby/object:Gem::Version
26
26
  version: '4.0'
27
- - !ruby/object:Gem::Dependency
28
- name: backports
29
- requirement: !ruby/object:Gem::Requirement
30
- requirements:
31
- - - "~>"
32
- - !ruby/object:Gem::Version
33
- version: '3.10'
34
- type: :runtime
35
- prerelease: false
36
- version_requirements: !ruby/object:Gem::Requirement
37
- requirements:
38
- - - "~>"
39
- - !ruby/object:Gem::Version
40
- version: '3.10'
41
27
  - !ruby/object:Gem::Dependency
42
28
  name: fugit
43
29
  requirement: !ruby/object:Gem::Requirement
@@ -258,14 +244,28 @@ dependencies:
258
244
  requirements:
259
245
  - - '='
260
246
  - !ruby/object:Gem::Version
261
- version: 0.80.0
247
+ version: 0.84.0
262
248
  type: :development
263
249
  prerelease: false
264
250
  version_requirements: !ruby/object:Gem::Requirement
265
251
  requirements:
266
252
  - - '='
267
253
  - !ruby/object:Gem::Version
268
- version: 0.80.0
254
+ version: 0.84.0
255
+ - !ruby/object:Gem::Dependency
256
+ name: rubocop-rspec
257
+ requirement: !ruby/object:Gem::Requirement
258
+ requirements:
259
+ - - ">="
260
+ - !ruby/object:Gem::Version
261
+ version: '0'
262
+ type: :development
263
+ prerelease: false
264
+ version_requirements: !ruby/object:Gem::Requirement
265
+ requirements:
266
+ - - ">="
267
+ - !ruby/object:Gem::Version
268
+ version: '0'
269
269
  - !ruby/object:Gem::Dependency
270
270
  name: sqlite3
271
271
  requirement: !ruby/object:Gem::Requirement
@@ -323,6 +323,7 @@ files:
323
323
  - lib/que/scheduler/db.rb
324
324
  - lib/que/scheduler/defined_job.rb
325
325
  - lib/que/scheduler/enqueueing_calculator.rb
326
+ - lib/que/scheduler/jobs/que_scheduler_audit_clear_down_job.rb
326
327
  - lib/que/scheduler/migrations.rb
327
328
  - lib/que/scheduler/migrations/1/down.sql
328
329
  - lib/que/scheduler/migrations/1/up.sql
@@ -332,6 +333,8 @@ files:
332
333
  - lib/que/scheduler/migrations/3/up.sql
333
334
  - lib/que/scheduler/migrations/4/down.sql
334
335
  - lib/que/scheduler/migrations/4/up.sql
336
+ - lib/que/scheduler/migrations/5/down.sql
337
+ - lib/que/scheduler/migrations/5/up.sql
335
338
  - lib/que/scheduler/schedule.rb
336
339
  - lib/que/scheduler/scheduler_job.rb
337
340
  - lib/que/scheduler/scheduler_job_args.rb
@@ -348,7 +351,7 @@ metadata:
348
351
  changelog_uri: https://github.com/hlascelles/que-scheduler/blob/master/CHANGELOG.md
349
352
  source_code_uri: https://github.com/hlascelles/que-scheduler/
350
353
  bug_tracker_uri: https://github.com/hlascelles/que-scheduler/issues
351
- post_install_message:
354
+ post_install_message:
352
355
  rdoc_options: []
353
356
  require_paths:
354
357
  - lib
@@ -364,7 +367,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
364
367
  version: '0'
365
368
  requirements: []
366
369
  rubygems_version: 3.0.3
367
- signing_key:
370
+ signing_key:
368
371
  specification_version: 4
369
372
  summary: A cron scheduler for Que
370
373
  test_files: []