schked 1.4.0 → 2.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +116 -3
- data/lib/schked/adapters/active_record.rb +123 -0
- data/lib/schked/adapters/sequel.rb +108 -0
- data/lib/schked/callbacks.rb +111 -0
- data/lib/schked/cli.rb +22 -1
- data/lib/schked/config.rb +65 -1
- data/lib/schked/database_connection.rb +75 -0
- data/lib/schked/job_run_store.rb +30 -0
- data/lib/schked/liveness_probe.rb +190 -0
- data/lib/schked/migration_generator.rb +42 -0
- data/lib/schked/railtie.rb +10 -0
- data/lib/schked/redis_job_run_store.rb +68 -0
- data/lib/schked/schedule_dsl.rb +83 -0
- data/lib/schked/version.rb +1 -1
- data/lib/schked/worker.rb +84 -38
- data/lib/schked.rb +9 -0
- metadata +27 -4
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: de7c669767af942f29f4c3414334cc7e996d8d41c3e9375ef2a249cc7ccc3b22
|
|
4
|
+
data.tar.gz: a55cb9f96f1eb18d43452ad0e61ec1d722c51d316f4e6b6604d7627b82aaaa2c
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 5691b00d0cd7b9575e82a743b45c7e0210593f34648a2bf1d98f7b387538805a9acea000f63724f6a31b81ce6b21a7825919b975023de04e46285eb009788ea8
|
|
7
|
+
data.tar.gz: 9184dd5c9190dff9f43c3534487fb4d38b6355fdd93b66d61b859f30dfc9c95d4d469d1298bb2f7f314a824ae577943835cfc530d5ff4dd3646e239a6f75b1ff
|
data/README.md
CHANGED
|
@@ -30,9 +30,9 @@ gem install schked
|
|
|
30
30
|
|
|
31
31
|
## Supported Ruby and Rails versions
|
|
32
32
|
|
|
33
|
-
Schked requires **Ruby
|
|
33
|
+
Schked requires **Ruby 3.0+**.
|
|
34
34
|
|
|
35
|
-
The test matrix covers Ruby **
|
|
35
|
+
The test matrix covers Ruby **3.0, 3.1, 3.2, 3.3, 3.4, and 4.0**. Rails integration tests run on every Ruby; Rails 8 is only included on Ruby **3.2+**.
|
|
36
36
|
|
|
37
37
|
## Usage
|
|
38
38
|
|
|
@@ -80,7 +80,11 @@ bundle exec schked show
|
|
|
80
80
|
|
|
81
81
|
### Duplicate scheduling
|
|
82
82
|
|
|
83
|
-
|
|
83
|
+
Schked ships two coordination strategies for multi-instance deployments. Choose one via `Schked.config.job_run_store`:
|
|
84
|
+
|
|
85
|
+
#### Single-active-instance (default)
|
|
86
|
+
|
|
87
|
+
When you deploy your schedule to production, you want to start new instance before you shut down the current. And you don't want simultaneous working of both. To achieve a seamless transition, Schked uses Redis for a global lock.
|
|
84
88
|
|
|
85
89
|
You can configure Redis client as the following:
|
|
86
90
|
|
|
@@ -88,6 +92,67 @@ You can configure Redis client as the following:
|
|
|
88
92
|
Schked.config.redis = {url: ENV.fetch("REDIS_URL") }
|
|
89
93
|
```
|
|
90
94
|
|
|
95
|
+
This is the default — one instance runs all jobs; standby instances hold the global Redis lock and stay idle. This strategy will continue to be supported because it is the simplest and most predictable for many setups.
|
|
96
|
+
|
|
97
|
+
#### Per-job deduplication
|
|
98
|
+
|
|
99
|
+
When you want every scheduler instance to do useful work (and not require a global leader), opt into the per-job deduplication mode. Each recurring job claims its schedule interval atomically; only one instance wins each interval, so each job still runs exactly once across the cluster.
|
|
100
|
+
|
|
101
|
+
Pick a coordination store:
|
|
102
|
+
|
|
103
|
+
```ruby
|
|
104
|
+
# Redis-backed (default Redis client from Schked.config.redis):
|
|
105
|
+
Schked.config.job_run_store = :redis
|
|
106
|
+
|
|
107
|
+
# Database-backed (no Redis required). The ActiveRecord or Sequel
|
|
108
|
+
# connection pool is auto-detected; override via:
|
|
109
|
+
Schked.config.job_run_store = :database
|
|
110
|
+
Schked.config.database_connection = conn # optional: Sequel::Database, ActiveRecord pool, or AR connection (PostgreSQL, Mysql2, or Trilogy)
|
|
111
|
+
|
|
112
|
+
# Custom store responding to #claim(job_name, window_start) and #cleanup(older_than):
|
|
113
|
+
Schked.config.job_run_store = my_store
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
The database backend runs entirely through the ActiveRecord/Sequel connection pool: claims and the internal cleanup sweep check connections out per operation, so they are thread-safe and survive database restarts and failovers. MySQL 8.0+ is required for the MySQL DDL.
|
|
117
|
+
|
|
118
|
+
Additional tuning:
|
|
119
|
+
|
|
120
|
+
```ruby
|
|
121
|
+
Schked.config.max_skew = 60 # max expected clock skew between instances (seconds)
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
Schedule behavior in deduplication mode:
|
|
125
|
+
|
|
126
|
+
- `every` jobs are aligned to an absolute time grid so all instances share the same phase. The first firing is the next grid point relative to now — not relative to process start.
|
|
127
|
+
- `cron` jobs already align to absolute time natively and need no change.
|
|
128
|
+
- `at` / `in` (one-time) jobs are deduplicated too — the claim is kept for the store's retention period (the Redis TTL / the database sweep window).
|
|
129
|
+
- `interval` jobs **are not supported** and raise `Schked::ScheduleDSL::IntervalNotSupportedError` when scheduled in this mode. Their phase depends on job duration and cannot be grid-aligned. Use `every` or `cron` instead.
|
|
130
|
+
|
|
131
|
+
##### Durability of the coordination store
|
|
132
|
+
|
|
133
|
+
Choosing between `:redis` and `:database` is also choosing how strong the exactly-once guarantee is:
|
|
134
|
+
|
|
135
|
+
- **`:database`** — claims are durable rows guarded by a UNIQUE constraint. They survive database restarts and failovers, and nothing removes them before the internal cleanup sweep (which keeps rows for 24 hours). Prefer this backend when a duplicate run is unacceptable.
|
|
136
|
+
- **`:redis`** — claims are keys with a TTL of `max(10 × max_skew, 1 hour)`, so the guarantee is only as strong as the Redis instance's durability:
|
|
137
|
+
- **Eviction policy** must be `noeviction` (or the instance must never reach `maxmemory`). Claim keys have a TTL, so both `allkeys-*` and `volatile-*` policies can evict them under memory pressure — the evicted window may then be claimed and executed by another instance.
|
|
138
|
+
- **Persistence**: a Redis restart without AOF/RDB loses in-flight claims. If it happens inside the contention window (the `max_skew`-wide interval during which instances race to claim the same slot), the job may run twice. Enable AOF, or use `:database` if you cannot accept that risk.
|
|
139
|
+
|
|
140
|
+
Failure semantics (both backends):
|
|
141
|
+
|
|
142
|
+
- The claim is taken **before** the job runs, so the semantics are *at-most-once per window*: if the winning instance crashes mid-run, that window's execution is lost and Schked does not retry it.
|
|
143
|
+
- The coordination store is a hard dependency: when it is unreachable, the claim fails and the firing is skipped (fail-closed) rather than risking a duplicate.
|
|
144
|
+
|
|
145
|
+
##### Database store migration
|
|
146
|
+
|
|
147
|
+
Run the migration generator to print the `schked_job_runs` DDL:
|
|
148
|
+
|
|
149
|
+
```sh
|
|
150
|
+
bundle exec schked generate-migration # Postgres
|
|
151
|
+
bundle exec schked generate-migration --flavor=mysql
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
Copy the SQL into your application's migration and run it. The gem does not write migration files or run DDL on its own — it stays framework-agnostic.
|
|
155
|
+
|
|
91
156
|
### Callbacks
|
|
92
157
|
|
|
93
158
|
Also, you can define callbacks for errors handling:
|
|
@@ -124,6 +189,54 @@ config/initializers/schked.rb
|
|
|
124
189
|
Schked.config.logger = Logger.new(Rails.root.join("log", "schked.log"))
|
|
125
190
|
```
|
|
126
191
|
|
|
192
|
+
### Liveness probe
|
|
193
|
+
|
|
194
|
+
Schked can expose a small HTTP endpoint for Kubernetes liveness probes. It is **disabled by default** to keep the existing behavior unchanged.
|
|
195
|
+
|
|
196
|
+
Configure it in Ruby:
|
|
197
|
+
|
|
198
|
+
```ruby
|
|
199
|
+
Schked.config.liveness_probe = {
|
|
200
|
+
enabled: true,
|
|
201
|
+
bind: "0.0.0.0",
|
|
202
|
+
port: 8080,
|
|
203
|
+
path: "/healthz",
|
|
204
|
+
heartbeat_interval: 5,
|
|
205
|
+
heartbeat_threshold: 15
|
|
206
|
+
}
|
|
207
|
+
```
|
|
208
|
+
|
|
209
|
+
Or via CLI flags:
|
|
210
|
+
|
|
211
|
+
```sh
|
|
212
|
+
bundle exec schked start --liveness-probe --liveness-bind 0.0.0.0 --liveness-port 8080 --liveness-path /healthz
|
|
213
|
+
```
|
|
214
|
+
|
|
215
|
+
In Rails, set it through the application config:
|
|
216
|
+
|
|
217
|
+
```ruby
|
|
218
|
+
# config/application.rb or config/environments/*.rb
|
|
219
|
+
config.schked.liveness_probe = {
|
|
220
|
+
enabled: true,
|
|
221
|
+
bind: "0.0.0.0",
|
|
222
|
+
port: 8080,
|
|
223
|
+
path: "/healthz",
|
|
224
|
+
heartbeat_interval: 5,
|
|
225
|
+
heartbeat_threshold: 15
|
|
226
|
+
}
|
|
227
|
+
```
|
|
228
|
+
|
|
229
|
+
The endpoint returns `200 OK` while the scheduler is responsive and `503 Service Unavailable` when the heartbeat is stale or during shutdown. The scheduler updates the heartbeat every `heartbeat_interval` seconds (default `5`); if it is not updated within `heartbeat_threshold` seconds (default `15`), the endpoint reports unhealthy. Use it in Kubernetes like this:
|
|
230
|
+
|
|
231
|
+
```yaml
|
|
232
|
+
livenessProbe:
|
|
233
|
+
httpGet:
|
|
234
|
+
path: /healthz
|
|
235
|
+
port: 8080
|
|
236
|
+
initialDelaySeconds: 10
|
|
237
|
+
periodSeconds: 10
|
|
238
|
+
```
|
|
239
|
+
|
|
127
240
|
### Monitoring
|
|
128
241
|
|
|
129
242
|
[Yabeda::Schked](https://github.com/yabeda-rb/yabeda-schked) - built-in metrics for monitoring Schked recurring jobs out of the box! Part of the [yabeda](https://github.com/yabeda-rb/yabeda) suite.
|
|
@@ -0,0 +1,123 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "securerandom"
|
|
4
|
+
|
|
5
|
+
module Schked
|
|
6
|
+
# Adapter for ActiveRecord connection pools (PostgreSQL, Mysql2,
|
|
7
|
+
# Trilogy). Implements the +Schked::JobRunStore+ contract.
|
|
8
|
+
#
|
|
9
|
+
# The adapter wraps a connection *pool*, not a single connection: every
|
|
10
|
+
# operation checks a connection out via +with_connection+ and returns it
|
|
11
|
+
# afterwards. The lease guarantees exclusive use of the checked-out
|
|
12
|
+
# connection, so claims (scheduler thread) and the cleanup sweep (work
|
|
13
|
+
# threads) can safely share the pool, and the pool transparently
|
|
14
|
+
# replaces dead connections after a database restart or failover.
|
|
15
|
+
class Adapters
|
|
16
|
+
class ActiveRecord
|
|
17
|
+
include JobRunStore
|
|
18
|
+
|
|
19
|
+
SUPPORTED_ADAPTERS = %w[PostgreSQL Mysql2 Trilogy].freeze
|
|
20
|
+
|
|
21
|
+
attr_reader :adapter_name
|
|
22
|
+
attr_reader :logger
|
|
23
|
+
|
|
24
|
+
def initialize(pool, logger: Logger.new($stdout))
|
|
25
|
+
# Accept either a ConnectionPool or a concrete adapter connection
|
|
26
|
+
# (+ActiveRecord::Base.connection+) and normalize to the pool.
|
|
27
|
+
@pool = pool.respond_to?(:with_connection) ? pool : pool.pool
|
|
28
|
+
@logger = logger
|
|
29
|
+
validate_adapter!
|
|
30
|
+
end
|
|
31
|
+
|
|
32
|
+
def claim(job_name, window_start)
|
|
33
|
+
validate!(job_name, window_start)
|
|
34
|
+
|
|
35
|
+
ts = window_start.is_a?(Time) ? window_start.to_i : Integer(window_start)
|
|
36
|
+
run_at = Time.now.to_f
|
|
37
|
+
claimer = SecureRandom.uuid
|
|
38
|
+
|
|
39
|
+
@pool.with_connection do |connection|
|
|
40
|
+
if postgres?
|
|
41
|
+
postgres_claim(connection, job_name, ts, run_at, claimer)
|
|
42
|
+
else
|
|
43
|
+
mysql_claim(connection, job_name, ts, run_at, claimer)
|
|
44
|
+
end
|
|
45
|
+
end
|
|
46
|
+
rescue ArgumentError
|
|
47
|
+
raise
|
|
48
|
+
rescue => e
|
|
49
|
+
logger.error("Failed to claim AR job run with error: #{e.message}")
|
|
50
|
+
raise
|
|
51
|
+
end
|
|
52
|
+
|
|
53
|
+
def cleanup(older_than)
|
|
54
|
+
cutoff = older_than.is_a?(Time) ? older_than.to_i : Integer(older_than)
|
|
55
|
+
@pool.with_connection do |connection|
|
|
56
|
+
connection.execute("DELETE FROM #{TABLE} WHERE window_start < #{connection.quote(cutoff)}")
|
|
57
|
+
end
|
|
58
|
+
nil
|
|
59
|
+
rescue => e
|
|
60
|
+
logger.error("Failed to clean up AR job runs with error: #{e.message}")
|
|
61
|
+
raise
|
|
62
|
+
end
|
|
63
|
+
|
|
64
|
+
private
|
|
65
|
+
|
|
66
|
+
TABLE = "schked_job_runs"
|
|
67
|
+
|
|
68
|
+
def validate_adapter!
|
|
69
|
+
name = @pool.with_connection do |connection|
|
|
70
|
+
connection.respond_to?(:adapter_name) ? connection.adapter_name.to_s : ""
|
|
71
|
+
end
|
|
72
|
+
unless SUPPORTED_ADAPTERS.any? { |supported| supported.casecmp?(name) }
|
|
73
|
+
raise ArgumentError,
|
|
74
|
+
"Schked::Adapters::ActiveRecord supports #{SUPPORTED_ADAPTERS.join(", ")} " \
|
|
75
|
+
"connections only, got: #{name.inspect}"
|
|
76
|
+
end
|
|
77
|
+
|
|
78
|
+
# Cached: claim/cleanup dispatch on the dialect without extra
|
|
79
|
+
# adapter_name round-trips on every firing.
|
|
80
|
+
@adapter_name = name
|
|
81
|
+
end
|
|
82
|
+
|
|
83
|
+
def postgres?
|
|
84
|
+
adapter_name.include?("Postgre") || adapter_name.include?("Postgres")
|
|
85
|
+
end
|
|
86
|
+
|
|
87
|
+
# Postgres can decide atomically in a single statement: INSERT ...
|
|
88
|
+
# ON CONFLICT DO NOTHING RETURNING yields a row only for the winner.
|
|
89
|
+
def postgres_claim(connection, job_name, ts, run_at, claimer)
|
|
90
|
+
sql = "INSERT INTO #{TABLE} (job_name, window_start, run_at, claimer) " \
|
|
91
|
+
"VALUES (#{connection.quote(job_name)}, #{connection.quote(ts)}, " \
|
|
92
|
+
"#{connection.quote(run_at)}, #{connection.quote(claimer)}) " \
|
|
93
|
+
"ON CONFLICT (job_name, window_start) DO NOTHING RETURNING id"
|
|
94
|
+
Integer(connection.exec_query(sql, "Schked CLAIM").length).positive?
|
|
95
|
+
end
|
|
96
|
+
|
|
97
|
+
# Rails' mysql2 and trilogy adapters both connect with the
|
|
98
|
+
# CLIENT_FOUND_ROWS capability set unconditionally (see
|
|
99
|
+
# mysql2_adapter.rb / trilogy_adapter.rb), so affected_rows cannot
|
|
100
|
+
# distinguish a fresh insert from a matched duplicate. Instead every
|
|
101
|
+
# claimer writes a unique token and reads it back: the
|
|
102
|
+
# UNIQUE (job_name, window_start) constraint guarantees exactly one
|
|
103
|
+
# row survives, so only the claimer whose token is stored in that
|
|
104
|
+
# row won the window.
|
|
105
|
+
def mysql_claim(connection, job_name, ts, run_at, claimer)
|
|
106
|
+
insert = "INSERT IGNORE INTO #{TABLE} (job_name, window_start, run_at, claimer) " \
|
|
107
|
+
"VALUES (#{connection.quote(job_name)}, #{connection.quote(ts)}, " \
|
|
108
|
+
"#{connection.quote(run_at)}, #{connection.quote(claimer)})"
|
|
109
|
+
connection.execute(insert)
|
|
110
|
+
|
|
111
|
+
select = "SELECT claimer FROM #{TABLE} " \
|
|
112
|
+
"WHERE job_name = #{connection.quote(job_name)} AND window_start = #{connection.quote(ts)}"
|
|
113
|
+
row = connection.exec_query(select, "Schked CLAIM").first
|
|
114
|
+
!row.nil? && row["claimer"] == claimer
|
|
115
|
+
end
|
|
116
|
+
|
|
117
|
+
def validate!(job_name, window_start)
|
|
118
|
+
raise ArgumentError, "job_name must be a non-empty String" if job_name.to_s.empty?
|
|
119
|
+
raise ArgumentError, "window_start must not be nil" if window_start.nil?
|
|
120
|
+
end
|
|
121
|
+
end
|
|
122
|
+
end
|
|
123
|
+
end
|
|
@@ -0,0 +1,108 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "securerandom"
|
|
4
|
+
|
|
5
|
+
module Schked
|
|
6
|
+
# Adapter for Sequel (Postgres or MySQL via +Sequel::Database+).
|
|
7
|
+
# Implements the +Schked::JobRunStore+ contract. Both operations go
|
|
8
|
+
# through Sequel's thread-safe connection pool, so claims (scheduler
|
|
9
|
+
# thread) and the cleanup sweep (work threads) can overlap freely.
|
|
10
|
+
class Adapters
|
|
11
|
+
class Sequel
|
|
12
|
+
include JobRunStore
|
|
13
|
+
|
|
14
|
+
SUPPORTED_DATABASE_TYPES = %w[postgres mysql].freeze
|
|
15
|
+
|
|
16
|
+
attr_reader :logger
|
|
17
|
+
|
|
18
|
+
def initialize(database, logger: Logger.new($stdout))
|
|
19
|
+
@database = database
|
|
20
|
+
@logger = logger
|
|
21
|
+
type = database.database_type.to_s
|
|
22
|
+
unless SUPPORTED_DATABASE_TYPES.include?(type)
|
|
23
|
+
raise ArgumentError,
|
|
24
|
+
"Schked::Adapters::Sequel supports Postgres and MySQL databases only, got: #{type.inspect}"
|
|
25
|
+
end
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
def claim(job_name, window_start)
|
|
29
|
+
validate!(job_name, window_start)
|
|
30
|
+
|
|
31
|
+
ts = window_start.is_a?(Time) ? window_start.to_i : Integer(window_start)
|
|
32
|
+
run_at = Time.now.to_f
|
|
33
|
+
claimer = SecureRandom.uuid
|
|
34
|
+
|
|
35
|
+
if postgres?
|
|
36
|
+
# Sequel's +insert_conflict+ is a PostgreSQL-only dataset method;
|
|
37
|
+
# it returns the new PK on success and +nil+ on conflict, so the
|
|
38
|
+
# winner is decided atomically in a single statement.
|
|
39
|
+
id = dataset
|
|
40
|
+
.insert_conflict(target: UNIQUE_COLUMNS)
|
|
41
|
+
.insert(job_name: job_name, window_start: ts, run_at: run_at, claimer: claimer)
|
|
42
|
+
!id.nil?
|
|
43
|
+
else
|
|
44
|
+
mysql_claim(job_name, ts, run_at, claimer)
|
|
45
|
+
end
|
|
46
|
+
rescue ArgumentError
|
|
47
|
+
raise
|
|
48
|
+
rescue => e
|
|
49
|
+
logger.error("Failed to claim sequel job run with error: #{e.message}")
|
|
50
|
+
raise
|
|
51
|
+
end
|
|
52
|
+
|
|
53
|
+
def cleanup(older_than)
|
|
54
|
+
cutoff = older_than.is_a?(Time) ? older_than.to_i : Integer(older_than)
|
|
55
|
+
dataset.where { window_start < cutoff }.delete
|
|
56
|
+
nil
|
|
57
|
+
rescue => e
|
|
58
|
+
logger.error("Failed to clean up sequel job runs with error: #{e.message}")
|
|
59
|
+
raise
|
|
60
|
+
end
|
|
61
|
+
|
|
62
|
+
def adapter_name
|
|
63
|
+
@database.adapter_scheme.to_s
|
|
64
|
+
end
|
|
65
|
+
|
|
66
|
+
private
|
|
67
|
+
|
|
68
|
+
TABLE = :schked_job_runs
|
|
69
|
+
UNIQUE_COLUMNS = %i[job_name window_start].freeze
|
|
70
|
+
|
|
71
|
+
def postgres?
|
|
72
|
+
@database.database_type == :postgres
|
|
73
|
+
end
|
|
74
|
+
|
|
75
|
+
# MySQL has no +RETURNING+, and affected-rows semantics depend on the
|
|
76
|
+
# CLIENT_FOUND_ROWS connection flag, so the winner cannot be derived
|
|
77
|
+
# from the insert alone. Instead every claimer writes a unique token
|
|
78
|
+
# and reads it back: the UNIQUE (job_name, window_start) constraint
|
|
79
|
+
# guarantees exactly one row survives, so only the claimer whose
|
|
80
|
+
# token is stored in that row won the window. +insert_ignore+ is a
|
|
81
|
+
# MySQL dataset method (+INSERT IGNORE+).
|
|
82
|
+
def mysql_claim(job_name, ts, run_at, claimer)
|
|
83
|
+
sql = dataset
|
|
84
|
+
.insert_ignore
|
|
85
|
+
.insert_sql(job_name: job_name, window_start: ts, run_at: run_at, claimer: claimer)
|
|
86
|
+
|
|
87
|
+
@database.synchronize do |conn|
|
|
88
|
+
conn.query(sql)
|
|
89
|
+
end
|
|
90
|
+
|
|
91
|
+
row = @database[TABLE]
|
|
92
|
+
.where(job_name: job_name, window_start: ts)
|
|
93
|
+
.select(:claimer)
|
|
94
|
+
.first
|
|
95
|
+
!row.nil? && row[:claimer] == claimer
|
|
96
|
+
end
|
|
97
|
+
|
|
98
|
+
def dataset
|
|
99
|
+
@database[TABLE]
|
|
100
|
+
end
|
|
101
|
+
|
|
102
|
+
def validate!(job_name, window_start)
|
|
103
|
+
raise ArgumentError, "job_name must be a non-empty String" if job_name.to_s.empty?
|
|
104
|
+
raise ArgumentError, "window_start must not be nil" if window_start.nil?
|
|
105
|
+
end
|
|
106
|
+
end
|
|
107
|
+
end
|
|
108
|
+
end
|
|
@@ -0,0 +1,111 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Schked
|
|
4
|
+
# Installs rufus-scheduler callbacks that translate rufus events into
|
|
5
|
+
# schked semantics: per-job deduplication claims via the configured
|
|
6
|
+
# +JobRunStore+, logging, and user-registered +Schked.config+ callbacks
|
|
7
|
+
# (+:before_start+, +:after_finish+, +:on_error+, +:around_job+).
|
|
8
|
+
#
|
|
9
|
+
# Extracted from +Schked::Worker+ so the worker stays focused on
|
|
10
|
+
# lifecycle and the callback wiring remains independently testable.
|
|
11
|
+
class Callbacks
|
|
12
|
+
# Marker prefix for schked's own internal jobs (cleanup, liveness
|
|
13
|
+
# heartbeat). These are exempt from dedup claims.
|
|
14
|
+
INTERNAL_JOB_PREFIX = "Schked::Worker#"
|
|
15
|
+
|
|
16
|
+
def initialize(config:, job_run_store: nil)
|
|
17
|
+
@config = config
|
|
18
|
+
@job_run_store = job_run_store
|
|
19
|
+
end
|
|
20
|
+
|
|
21
|
+
def install(scheduler)
|
|
22
|
+
cfg = @config
|
|
23
|
+
store = @job_run_store
|
|
24
|
+
internal = method(:internal_job?)
|
|
25
|
+
# Unlabeled jobs are skipped on every firing; logging the guidance at
|
|
26
|
+
# error level for each firing would spam operators (an `every 10s`
|
|
27
|
+
# job yields 8_640 messages per day). Remember which labels were
|
|
28
|
+
# already reported and downgrade repeat notices to debug.
|
|
29
|
+
missing_as_reported = {}
|
|
30
|
+
|
|
31
|
+
scheduler.define_singleton_method(:extract_job_name) do |job|
|
|
32
|
+
if job
|
|
33
|
+
job.opts[:as] || job.job_id
|
|
34
|
+
else
|
|
35
|
+
"unknown"
|
|
36
|
+
end
|
|
37
|
+
end
|
|
38
|
+
|
|
39
|
+
scheduler.define_singleton_method(:on_error) do |job, error|
|
|
40
|
+
cfg.logger.fatal("Task #{extract_job_name(job)} failed with error: #{error.message}")
|
|
41
|
+
cfg.logger.error(error.backtrace.join("\n")) if error.backtrace
|
|
42
|
+
|
|
43
|
+
cfg.fire_callback(:on_error, job, error)
|
|
44
|
+
end
|
|
45
|
+
|
|
46
|
+
scheduler.define_singleton_method(:on_pre_trigger) do |job, time|
|
|
47
|
+
job_name = extract_job_name(job).to_s
|
|
48
|
+
|
|
49
|
+
if store && internal.call(job_name)
|
|
50
|
+
# Internal schked jobs (cleanup sweep, liveness heartbeat) skip
|
|
51
|
+
# the dedup claim entirely so they always run on every instance.
|
|
52
|
+
elsif store
|
|
53
|
+
unless job.opts[:as]
|
|
54
|
+
# Without an explicit +as:+, the dedup key falls back to
|
|
55
|
+
# +job.job_id+, which encodes the Ruby +object_id+ of the
|
|
56
|
+
# +Rufus::Scheduler::Job+ and is unique per process. Claiming
|
|
57
|
+
# with that key would silently duplicate every run across the
|
|
58
|
+
# cluster. Refuse to claim; the operator must add +as:+ to
|
|
59
|
+
# their schedule for dedup to be correct.
|
|
60
|
+
if missing_as_reported.key?(job_name)
|
|
61
|
+
cfg.logger.debug("Skipping task #{job_name}: still no `as:` label (already reported).")
|
|
62
|
+
else
|
|
63
|
+
cfg.logger.error(
|
|
64
|
+
"Task #{job_name} has no `as:` label and cannot be deduplicated " \
|
|
65
|
+
"(each process generates a unique job_id). Add `as: \"my_job\"` to " \
|
|
66
|
+
"the schedule entry. Skipping this firing."
|
|
67
|
+
)
|
|
68
|
+
missing_as_reported[job_name] = true
|
|
69
|
+
end
|
|
70
|
+
next false
|
|
71
|
+
end
|
|
72
|
+
|
|
73
|
+
window_start = job.previous_time || job.scheduled_at
|
|
74
|
+
begin
|
|
75
|
+
claimed = store.claim(job_name, window_start)
|
|
76
|
+
rescue => e
|
|
77
|
+
# Fail closed, but say what actually happened: the task did not
|
|
78
|
+
# fail — the coordination store is unavailable, and firing is
|
|
79
|
+
# skipped to avoid a duplicate run.
|
|
80
|
+
cfg.logger.fatal("Skipped task #{job_name}: job run store unavailable: #{e.class} #{e.message}")
|
|
81
|
+
cfg.fire_callback(:on_error, job, e)
|
|
82
|
+
next false
|
|
83
|
+
end
|
|
84
|
+
unless claimed
|
|
85
|
+
cfg.logger.info("Skipped task: #{job_name} (already claimed for window_start=#{window_start.to_i})")
|
|
86
|
+
next false
|
|
87
|
+
end
|
|
88
|
+
end
|
|
89
|
+
|
|
90
|
+
cfg.logger.info("Started task: #{extract_job_name(job)}")
|
|
91
|
+
cfg.fire_callback(:before_start, job, time)
|
|
92
|
+
end
|
|
93
|
+
|
|
94
|
+
scheduler.define_singleton_method(:around_trigger) do |job, &block|
|
|
95
|
+
cfg.fire_around_callback(:around_job, job, &block)
|
|
96
|
+
end
|
|
97
|
+
|
|
98
|
+
scheduler.define_singleton_method(:on_post_trigger) do |job, time|
|
|
99
|
+
cfg.logger.info("Finished task: #{extract_job_name(job)}")
|
|
100
|
+
|
|
101
|
+
cfg.fire_callback(:after_finish, job, time)
|
|
102
|
+
end
|
|
103
|
+
|
|
104
|
+
scheduler
|
|
105
|
+
end
|
|
106
|
+
|
|
107
|
+
def internal_job?(job_name)
|
|
108
|
+
job_name.start_with?(INTERNAL_JOB_PREFIX)
|
|
109
|
+
end
|
|
110
|
+
end
|
|
111
|
+
end
|
data/lib/schked/cli.rb
CHANGED
|
@@ -15,7 +15,7 @@ module Schked
|
|
|
15
15
|
.shellsplit
|
|
16
16
|
end
|
|
17
17
|
|
|
18
|
-
super
|
|
18
|
+
super
|
|
19
19
|
end
|
|
20
20
|
|
|
21
21
|
def self.exit_on_failure?
|
|
@@ -26,8 +26,13 @@ module Schked
|
|
|
26
26
|
|
|
27
27
|
desc "start", "Start scheduler"
|
|
28
28
|
option :require, type: :array
|
|
29
|
+
option :liveness_probe, type: :boolean, default: nil, desc: "Enable or disable the liveness probe"
|
|
30
|
+
option :liveness_bind, type: :string, desc: "Address the liveness probe binds to"
|
|
31
|
+
option :liveness_port, type: :numeric, desc: "Port the liveness probe listens on"
|
|
32
|
+
option :liveness_path, type: :string, desc: "HTTP path for the liveness probe"
|
|
29
33
|
def start
|
|
30
34
|
load_requires
|
|
35
|
+
apply_liveness_probe_options
|
|
31
36
|
|
|
32
37
|
Schked.worker.wait
|
|
33
38
|
end
|
|
@@ -42,6 +47,12 @@ module Schked
|
|
|
42
47
|
puts "====="
|
|
43
48
|
end
|
|
44
49
|
|
|
50
|
+
desc "generate-migration", "Print DDL SQL for the schked_job_runs table to stdout"
|
|
51
|
+
option :flavor, type: :string, default: "postgres", desc: "DDL flavor (postgres or mysql)"
|
|
52
|
+
def generate_migration(flavor = nil)
|
|
53
|
+
puts MigrationGenerator.sql(flavor || options[:flavor])
|
|
54
|
+
end
|
|
55
|
+
|
|
45
56
|
private
|
|
46
57
|
|
|
47
58
|
def load_requires
|
|
@@ -50,5 +61,15 @@ module Schked
|
|
|
50
61
|
# We have to load Schked at here, because of Rails and our railtie.
|
|
51
62
|
require "schked"
|
|
52
63
|
end
|
|
64
|
+
|
|
65
|
+
def apply_liveness_probe_options
|
|
66
|
+
overrides = {}
|
|
67
|
+
overrides[:enabled] = options[:liveness_probe] unless options[:liveness_probe].nil?
|
|
68
|
+
overrides[:bind] = options[:liveness_bind] if options[:liveness_bind]
|
|
69
|
+
overrides[:port] = options[:liveness_port] if options[:liveness_port]
|
|
70
|
+
overrides[:path] = options[:liveness_path] if options[:liveness_path]
|
|
71
|
+
|
|
72
|
+
Schked.config.liveness_probe = Schked.config.liveness_probe.to_h.merge(overrides) if overrides.any?
|
|
73
|
+
end
|
|
53
74
|
end
|
|
54
75
|
end
|
data/lib/schked/config.rb
CHANGED
|
@@ -4,10 +4,23 @@ require "logger"
|
|
|
4
4
|
|
|
5
5
|
module Schked
|
|
6
6
|
class Config
|
|
7
|
+
VALID_JOB_RUN_STORES = %i[redis database].freeze
|
|
8
|
+
|
|
7
9
|
attr_writer :logger,
|
|
8
10
|
:do_not_load_root_schedule,
|
|
9
11
|
:redis,
|
|
10
|
-
:standalone
|
|
12
|
+
:standalone,
|
|
13
|
+
:job_run_store,
|
|
14
|
+
:max_skew,
|
|
15
|
+
:database_connection
|
|
16
|
+
|
|
17
|
+
def liveness_probe
|
|
18
|
+
@liveness_probe ||= LivenessProbeConfig.new
|
|
19
|
+
end
|
|
20
|
+
|
|
21
|
+
def liveness_probe=(value)
|
|
22
|
+
@liveness_probe = value.is_a?(LivenessProbeConfig) ? value : LivenessProbeConfig.new(value)
|
|
23
|
+
end
|
|
11
24
|
|
|
12
25
|
def paths
|
|
13
26
|
@paths ||= []
|
|
@@ -73,8 +86,59 @@ module Schked
|
|
|
73
86
|
!!@standalone
|
|
74
87
|
end
|
|
75
88
|
|
|
89
|
+
attr_reader :job_run_store
|
|
90
|
+
|
|
91
|
+
def max_skew
|
|
92
|
+
@max_skew ||= 60
|
|
93
|
+
end
|
|
94
|
+
|
|
95
|
+
attr_reader :database_connection
|
|
96
|
+
|
|
97
|
+
def dedup_enabled?
|
|
98
|
+
!@job_run_store.nil?
|
|
99
|
+
end
|
|
100
|
+
|
|
101
|
+
# Validates all configuration options. Called by the worker during
|
|
102
|
+
# initialization so the worker can remain agnostic about which options
|
|
103
|
+
# exist and which combinations are legal.
|
|
104
|
+
def validate!
|
|
105
|
+
validate_job_run_store!
|
|
106
|
+
validate_max_skew!
|
|
107
|
+
end
|
|
108
|
+
|
|
76
109
|
private
|
|
77
110
|
|
|
111
|
+
def validate_max_skew!
|
|
112
|
+
return if @max_skew.nil?
|
|
113
|
+
|
|
114
|
+
begin
|
|
115
|
+
skew = Integer(@max_skew)
|
|
116
|
+
rescue ArgumentError, TypeError
|
|
117
|
+
raise ArgumentError,
|
|
118
|
+
"Schked `max_skew` must be a positive number of seconds, got: #{@max_skew.inspect}"
|
|
119
|
+
end
|
|
120
|
+
|
|
121
|
+
return if skew.positive?
|
|
122
|
+
|
|
123
|
+
raise ArgumentError,
|
|
124
|
+
"Schked `max_skew` must be a positive number of seconds, got: #{@max_skew.inspect}"
|
|
125
|
+
end
|
|
126
|
+
|
|
127
|
+
def validate_job_run_store!
|
|
128
|
+
return if @job_run_store.nil?
|
|
129
|
+
|
|
130
|
+
message = "Schked `job_run_store` must be one of #{VALID_JOB_RUN_STORES.inspect}, " \
|
|
131
|
+
"a Symbol, or an object responding to #claim and #cleanup; got: #{@job_run_store.inspect}"
|
|
132
|
+
|
|
133
|
+
valid = if @job_run_store.is_a?(Symbol)
|
|
134
|
+
VALID_JOB_RUN_STORES.include?(@job_run_store)
|
|
135
|
+
else
|
|
136
|
+
@job_run_store.respond_to?(:claim) && @job_run_store.respond_to?(:cleanup)
|
|
137
|
+
end
|
|
138
|
+
|
|
139
|
+
raise ArgumentError, message unless valid
|
|
140
|
+
end
|
|
141
|
+
|
|
78
142
|
def callbacks
|
|
79
143
|
@callbacks ||= Hash.new { |hsh, key| hsh[key] = [] }
|
|
80
144
|
end
|