que-scheduler 3.0.0 → 3.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +64 -11
- data/lib/que-scheduler.rb +3 -0
- data/lib/que/scheduler.rb +1 -0
- data/lib/que/scheduler/config.rb +24 -0
- data/lib/que/scheduler/db.rb +4 -0
- data/lib/que/scheduler/defined_job.rb +4 -6
- data/lib/que/scheduler/migrations.rb +1 -1
- data/lib/que/scheduler/scheduler_job.rb +14 -4
- data/lib/que/scheduler/version.rb +1 -1
- metadata +18 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: b10ffa2b0fda7c713a51d8780d9f6477e7579fcd0ab47f9fa65084a9f31b5780
|
4
|
+
data.tar.gz: 32ed325e8c054851c61be53f1bb18b79f22c5e41f9f4fa2baa20070fbe078110
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: d26040a8470cee8985e62436c55d013700412c6f8928a1952c37c90e13e91bbf7820b62f55118c238108b0020cc627f27cd209051076048003146839e52960d4
|
7
|
+
data.tar.gz: a58efd112aeb2f304394ca7578e10019d0a1c28b7f1796e18a68a6d49ac23f40e50db49b8e31f3258d8a1034cf547ff4a69b5a60a980c67574776eef54bb2419
|
data/README.md
CHANGED
@@ -18,11 +18,11 @@ needs to be run, enqueueing those jobs, then enqueueing itself to check again la
|
|
18
18
|
```ruby
|
19
19
|
gem 'que-scheduler'
|
20
20
|
```
|
21
|
-
1. Specify a schedule
|
22
|
-
look for it is `config/que_schedule.yml`. They are essentially the same as resque-scheduler
|
21
|
+
1. Specify a schedule in a yml file (see below). The default location that que-scheduler will
|
22
|
+
look for it is `config/que_schedule.yml`. They are essentially the same as resque-scheduler
|
23
23
|
files, but with additional features.
|
24
24
|
|
25
|
-
1. Add a migration to start the job scheduler.
|
25
|
+
1. Add a migration to start the job scheduler and prepare the audit table.
|
26
26
|
|
27
27
|
```ruby
|
28
28
|
class CreateQueSchedulerSchema < ActiveRecord::Migration
|
@@ -35,7 +35,7 @@ files, but with additional features.
|
|
35
35
|
## Schedule configuration
|
36
36
|
|
37
37
|
The schedule file is a list of que job classes with arguments and a schedule frequency (in crontab
|
38
|
-
syntax). The format is similar to the resque-scheduler
|
38
|
+
syntax). The format is similar to the resque-scheduler format, though priorities must be supplied as
|
39
39
|
integers, and job classes must be migrated from Resque to Que. Cron syntax can be anything
|
40
40
|
understood by [fugit](https://github.com/floraison/fugit#fugitcron).
|
41
41
|
|
@@ -111,11 +111,22 @@ A job can have a `schedule_type` assigned to it. Valid values are:
|
|
111
111
|
This feature ensures that jobs which *must run* for a certain cron match will always eventually
|
112
112
|
execute, even after a total system crash, or even a DB backup restore.
|
113
113
|
|
114
|
-
##
|
114
|
+
## Gem configuration
|
115
115
|
|
116
|
-
You can configure some aspects of the gem with
|
116
|
+
You can configure some aspects of the gem with an initializer. The default is given below.
|
117
117
|
|
118
|
-
|
118
|
+
```ruby
|
119
|
+
Que::Scheduler.configure do |config|
|
120
|
+
# The location of the schedule yaml file.
|
121
|
+
config.schedule_location = ENV.fetch('QUE_SCHEDULER_CONFIG_LOCATION', 'config/que_schedule.yml')
|
122
|
+
|
123
|
+
# Specify a transaction block adapter. By default, que-scheduler uses the one supplied by que.
|
124
|
+
# However, if, for example you rely on listeners to ActiveRecord's exact `transaction` method, or
|
125
|
+
# Sequel's DB.after_commit helper, then you can supply it here.
|
126
|
+
config.transaction_adapter = ::Que.method(:transaction)
|
127
|
+
end
|
128
|
+
|
129
|
+
```
|
119
130
|
|
120
131
|
## Scheduler Audit
|
121
132
|
|
@@ -134,7 +145,7 @@ in a coherent state with the rest of your database.
|
|
134
145
|
|
135
146
|
## Concurrent scheduler detection
|
136
147
|
|
137
|
-
No matter how many tasks you have defined in your
|
148
|
+
No matter how many tasks you have defined in your schedule, you will only ever need one que-scheduler
|
138
149
|
job enqueued. que-scheduler knows this, and it will check before performing any operations that
|
139
150
|
there is only one of itself present.
|
140
151
|
|
@@ -144,17 +155,59 @@ it will rollback correctly and try again. It won't schedule jobs twice for a cro
|
|
144
155
|
|
145
156
|
## How it works
|
146
157
|
|
147
|
-
que-scheduler is a job that reads a
|
158
|
+
que-scheduler is a job that reads a schedule file, enqueues any jobs it determines that need to be run,
|
148
159
|
then reschedules itself. The flow is as follows:
|
149
160
|
|
150
161
|
1. The que-scheduler job runs for the very first time.
|
151
|
-
1. que-scheduler loads the schedule
|
162
|
+
1. que-scheduler loads the schedule file. It will not schedule any other jobs, except itself,
|
152
163
|
as it has never run before.
|
153
164
|
1. Some time later it runs again. It knows what jobs it should be monitoring, and notices that some
|
154
165
|
have are due. It enqueues those jobs and then itself. Repeat.
|
155
|
-
1. After a deploy that changes the
|
166
|
+
1. After a deploy that changes the schedule, the job notices any new jobs to schedule, and knows which
|
156
167
|
ones to forget. It does not need to be re-enqueued or restarted.
|
157
168
|
|
169
|
+
## DB Migrations
|
170
|
+
|
171
|
+
When there is a major version (breaking) change, a migration should be run in. The version of the
|
172
|
+
migration proceeds at a faster rate than the version of the gem. To run in all the migrations required
|
173
|
+
up to a number, just migrate to that number with one line, and it will perform all the intermediary steps.
|
174
|
+
|
175
|
+
ie, `Que::Scheduler::Migrations.migrate!(version: 4)` will perform all migrations necessary to
|
176
|
+
reach migration version `4`.
|
177
|
+
|
178
|
+
As of migration `4`, two elements are added to the DB for que-scheduler to run.
|
179
|
+
|
180
|
+
1. The first is the scheduler job itself, which runs forever, re-enqueuing itself to performs its
|
181
|
+
duties.
|
182
|
+
1. The second part comprises the audit table `que_scheduler_audit` and the "enqueued" table
|
183
|
+
`que_scheduler_audit`. The first tracks when the scheduler calculated what was necessary to run
|
184
|
+
(if anything). The second then logs every job that the scheduler enqueues.
|
185
|
+
|
186
|
+
## Upgrading
|
187
|
+
|
188
|
+
que-scheduler uses [semantic versioning](https://semver.org/), so major version changes will usually
|
189
|
+
require additional actions to be taken upgrading from one major version to another.
|
190
|
+
|
191
|
+
Major feature changes are listed below. The full
|
192
|
+
[CHANGELOG](https://github.com/hlascelles/que-scheduler/blob/master/CHANGELOG.md) can be found in
|
193
|
+
the root of the project.
|
194
|
+
|
195
|
+
#### Versions 3.x
|
196
|
+
- Addition of a config initializer.
|
197
|
+
- Addition of numerous extra columns to the audit table.
|
198
|
+
- Required cumulative migration: `Que::Scheduler::Migrations.migrate!(version: 4)`
|
199
|
+
#### Versions 2.x
|
200
|
+
- Introduction of the audit table.
|
201
|
+
- Support for older versions of postgres
|
202
|
+
- Required cumulative migration: `Que::Scheduler::Migrations.migrate!(version: 3)`
|
203
|
+
#### Versions 1.x
|
204
|
+
- Sequel support
|
205
|
+
- Config specified Timezone support
|
206
|
+
- Required migration adding the initial job: `Que::Scheduler::SchedulerJob.enqueue`
|
207
|
+
#### Versions 0.x
|
208
|
+
- The first public release.
|
209
|
+
- Required migration adding the initial job: `Que::Scheduler::SchedulerJob.enqueue`
|
210
|
+
|
158
211
|
## System requirements
|
159
212
|
|
160
213
|
Your [postgres](https://www.postgresql.org/) database must be at least version 9.4.0.
|
data/lib/que/scheduler.rb
CHANGED
@@ -0,0 +1,24 @@
|
|
1
|
+
require 'que'
|
2
|
+
|
3
|
+
module Que
|
4
|
+
module Scheduler
|
5
|
+
class << self
|
6
|
+
attr_accessor :configuration
|
7
|
+
end
|
8
|
+
|
9
|
+
def self.configure
|
10
|
+
self.configuration ||= Configuration.new
|
11
|
+
yield(configuration)
|
12
|
+
end
|
13
|
+
|
14
|
+
class Configuration
|
15
|
+
attr_accessor :schedule_location
|
16
|
+
attr_accessor :transaction_adapter
|
17
|
+
end
|
18
|
+
end
|
19
|
+
end
|
20
|
+
|
21
|
+
Que::Scheduler.configure do |config|
|
22
|
+
config.schedule_location = ENV.fetch('QUE_SCHEDULER_CONFIG_LOCATION', 'config/que_schedule.yml')
|
23
|
+
config.transaction_adapter = ::Que.method(:transaction)
|
24
|
+
end
|
data/lib/que/scheduler/db.rb
CHANGED
@@ -6,10 +6,6 @@ require 'backports/2.4.0/hash/compact'
|
|
6
6
|
module Que
|
7
7
|
module Scheduler
|
8
8
|
class DefinedJob < Hashie::Dash
|
9
|
-
QUE_SCHEDULER_CONFIG_LOCATION =
|
10
|
-
ENV.fetch('QUE_SCHEDULER_CONFIG_LOCATION', 'config/que_schedule.yml')
|
11
|
-
ERROR_MESSAGE_SUFFIX = "in que-scheduler config #{QUE_SCHEDULER_CONFIG_LOCATION}".freeze
|
12
|
-
|
13
9
|
include Hashie::Extensions::Dash::PropertyTranslation
|
14
10
|
|
15
11
|
SCHEDULE_TYPES = [
|
@@ -55,7 +51,8 @@ module Que
|
|
55
51
|
|
56
52
|
class << self
|
57
53
|
def defined_jobs
|
58
|
-
|
54
|
+
schedule_string = IO.read(Que::Scheduler.configuration.schedule_location)
|
55
|
+
@defined_jobs ||= YAML.safe_load(schedule_string).map do |k, v|
|
59
56
|
Que::Scheduler::DefinedJob.new(
|
60
57
|
{
|
61
58
|
name: k,
|
@@ -73,7 +70,8 @@ module Que
|
|
73
70
|
private
|
74
71
|
|
75
72
|
def err_field(field, value)
|
76
|
-
|
73
|
+
schedule = Que::Scheduler.configuration.schedule_location
|
74
|
+
raise "Invalid #{field} '#{value}' in que-scheduler schedule #{schedule}"
|
77
75
|
end
|
78
76
|
end
|
79
77
|
|
@@ -15,7 +15,7 @@ module Que
|
|
15
15
|
@priority = 0
|
16
16
|
|
17
17
|
def run(options = nil)
|
18
|
-
::
|
18
|
+
Que::Scheduler::Db.transaction do
|
19
19
|
assert_one_scheduler_job
|
20
20
|
scheduler_job_args = SchedulerJobArgs.build(options)
|
21
21
|
logs = ["que-scheduler last ran at #{scheduler_job_args.last_run_time}."]
|
@@ -43,14 +43,24 @@ module Que
|
|
43
43
|
else
|
44
44
|
job_class.enqueue(*args, remaining_hash)
|
45
45
|
end
|
46
|
+
check_enqueued_job(enqueued_job, job_class, args, logs)
|
47
|
+
end.compact
|
48
|
+
end
|
49
|
+
|
50
|
+
private
|
51
|
+
|
52
|
+
def check_enqueued_job(enqueued_job, job_class, args, logs)
|
53
|
+
if enqueued_job.is_a?(Que::Job)
|
46
54
|
job_id = enqueued_job.attrs.fetch('job_id')
|
47
|
-
logs << "que-scheduler enqueueing #{job_class} #{job_id} with: #{
|
55
|
+
logs << "que-scheduler enqueueing #{job_class} #{job_id} with args: #{args}"
|
48
56
|
enqueued_job
|
57
|
+
else
|
58
|
+
# This can happen if a middleware nixes the enqueue call
|
59
|
+
logs << "que-scheduler called enqueue on #{job_class} but did not receive a #{Que::Job}"
|
60
|
+
nil
|
49
61
|
end
|
50
62
|
end
|
51
63
|
|
52
|
-
private
|
53
|
-
|
54
64
|
def assert_one_scheduler_job
|
55
65
|
schedulers = Que::Scheduler::Db.count_schedulers
|
56
66
|
return if schedulers == 1
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: que-scheduler
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 3.
|
4
|
+
version: 3.1.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Harry Lascelles
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2018-
|
11
|
+
date: 2018-06-01 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: activesupport
|
@@ -136,6 +136,20 @@ dependencies:
|
|
136
136
|
- - ">="
|
137
137
|
- !ruby/object:Gem::Version
|
138
138
|
version: '0'
|
139
|
+
- !ruby/object:Gem::Dependency
|
140
|
+
name: database_cleaner
|
141
|
+
requirement: !ruby/object:Gem::Requirement
|
142
|
+
requirements:
|
143
|
+
- - ">="
|
144
|
+
- !ruby/object:Gem::Version
|
145
|
+
version: '0'
|
146
|
+
type: :development
|
147
|
+
prerelease: false
|
148
|
+
version_requirements: !ruby/object:Gem::Requirement
|
149
|
+
requirements:
|
150
|
+
- - ">="
|
151
|
+
- !ruby/object:Gem::Version
|
152
|
+
version: '0'
|
139
153
|
- !ruby/object:Gem::Dependency
|
140
154
|
name: fasterer
|
141
155
|
requirement: !ruby/object:Gem::Requirement
|
@@ -298,8 +312,10 @@ extensions: []
|
|
298
312
|
extra_rdoc_files: []
|
299
313
|
files:
|
300
314
|
- README.md
|
315
|
+
- lib/que-scheduler.rb
|
301
316
|
- lib/que/scheduler.rb
|
302
317
|
- lib/que/scheduler/audit.rb
|
318
|
+
- lib/que/scheduler/config.rb
|
303
319
|
- lib/que/scheduler/db.rb
|
304
320
|
- lib/que/scheduler/defined_job.rb
|
305
321
|
- lib/que/scheduler/engine.rb
|