stablemates-workhorse 0.9.0 → 0.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +372 -0
- data/README.md +1 -1
- data/lib/stablemates/workhorse/admin.rb +49 -7
- data/lib/stablemates/workhorse/context.rb +5 -2
- data/lib/stablemates/workhorse/contracts.rb +34 -9
- data/lib/stablemates/workhorse/dashboard/backend.rb +30 -26
- data/lib/stablemates/workhorse/dashboard/bundle.json +1 -1
- data/lib/stablemates/workhorse/dashboard/read-surface-1.tar.gz +0 -0
- data/lib/stablemates/workhorse/dashboard/v1_generated.rb +3 -1
- data/lib/stablemates/workhorse/dashboard.rb +24 -5
- data/lib/stablemates/workhorse/errors.rb +5 -0
- data/lib/stablemates/workhorse/executor.rb +6 -2
- data/lib/stablemates/workhorse/queue.rb +13 -7
- data/lib/stablemates/workhorse/sql_catalogue_generated.rb +163 -48
- data/lib/stablemates/workhorse/telemetry.rb +101 -2
- data/lib/stablemates/workhorse/version.rb +1 -1
- data/lib/stablemates/workhorse/worker/batch.rb +50 -3
- data/lib/stablemates/workhorse/worker/heartbeat.rb +605 -104
- data/lib/stablemates/workhorse/worker/listener.rb +9 -5
- data/lib/stablemates/workhorse/worker/statement_watch.rb +86 -0
- data/lib/stablemates/workhorse/worker.rb +541 -97
- data/lib/stablemates/workhorse.rb +1 -0
- metadata +2 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 35dc8eaadac2ae86c30ad3cf2a688ffe571b7c7a8cd564ee00844685fbeb4ea6
|
|
4
|
+
data.tar.gz: 34dfeb5010846d31ea3f7b5751e14bf7ec4ed4783a9223d4e2f483534b419669
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 99884c5d12e4501d537c66272a668be206f796ef0f8ac7663d863898caa3f0e4a530af7a79902d81983e671b76f146bc3e54f529a7cf7d6170d44ecaadc96d5f
|
|
7
|
+
data.tar.gz: dee85878fb8e6862120d5863d8b4cd34cdd18d6e4281e1e5976f9215af12dc0c2b58aef3090e3b42e439868fe367d63f515e66a8968a73d33903c626cb5247c4
|
data/CHANGELOG.md
CHANGED
|
@@ -3,6 +3,378 @@
|
|
|
3
3
|
`stablemates-workhorse` gem versions and release notes live here. The gem carries the version the
|
|
4
4
|
other SDKs carry, because every tag names one release of all of them.
|
|
5
5
|
|
|
6
|
+
## 0.10.0 — 2026-10-26
|
|
7
|
+
|
|
8
|
+
The npm packages, Python distribution, Go module, Rust crate, Ruby gem, JVM artifacts, and .NET
|
|
9
|
+
packages release from one source commit.
|
|
10
|
+
|
|
11
|
+
Requires **schema v76**, Ruby **3.3** or newer, and PostgreSQL **15** or newer.
|
|
12
|
+
|
|
13
|
+
### Upgrade steps
|
|
14
|
+
|
|
15
|
+
**Migrate the schema before starting updated processes.** Run `workhorse schema migrate` before any
|
|
16
|
+
process of this release starts, producers too. Migrations 0075 through 0087 move the schema to
|
|
17
|
+
version **86**. The SDK compatibility floor rises from 73 to **76**, because every SDK statement
|
|
18
|
+
catalogue now names `dashboard_retention_preview_v1` and `deregister_worker_v2`. An updated worker
|
|
19
|
+
refuses to start on a schema below version 76. A 0.9 process keeps working on version 86, because a
|
|
20
|
+
migration inside a major line only adds.
|
|
21
|
+
|
|
22
|
+
### A failed reporter callback gives a catalogued log record
|
|
23
|
+
|
|
24
|
+
A raise from `on_registration_error` or `on_notification_error` gave a log line with the error
|
|
25
|
+
message and no event name (SM-1726). Now the line is the `workhorse.worker.reporter_failed` event
|
|
26
|
+
with `workhorse.worker.id`, `workhorse.worker.reporter` (`registration_error` or
|
|
27
|
+
`notification_error`), and `error.type`. It no longer carries the error message. The logger level,
|
|
28
|
+
the standard error fallback, and the bound do not change.
|
|
29
|
+
|
|
30
|
+
### The worker refuses a `poll_interval` above 2,147,483.647 seconds
|
|
31
|
+
|
|
32
|
+
Before this change, `poll_interval:` had no upper limit (SM-1778). Now `Worker.new` raises
|
|
33
|
+
`ArgumentError` for a value above 2,147,483.647 seconds, the limit of the registry column. Every
|
|
34
|
+
SDK accepts an explicit poll interval from 1 through 2,147,483,647 ms (ADR 0112). Zero stays
|
|
35
|
+
refused, and `nil` keeps the default.
|
|
36
|
+
|
|
37
|
+
### The Rack dashboard refuses a cross-origin read
|
|
38
|
+
|
|
39
|
+
A Rack CORS middleware can add CORS headers to `Stablemates::Workhorse::Dashboard` responses. If
|
|
40
|
+
that policy allowed credentials, a page on another origin could read the dashboard, task payloads
|
|
41
|
+
included, with the operator's cookies (SM-1675). Now the application answers 403 with
|
|
42
|
+
`{"error":"A same-origin request is required"}` for a page, an asset, or a read procedure whose
|
|
43
|
+
`Origin` header names another origin.
|
|
44
|
+
|
|
45
|
+
The same-origin check compared the `Host` text exactly, so `Host: example.test:443` with
|
|
46
|
+
`Origin: https://example.test` failed. Now letter case and the default port do not matter, as in
|
|
47
|
+
the allowed hosts check. An `Origin` with a path, a query, a fragment, or a user fails.
|
|
48
|
+
|
|
49
|
+
A request without `Origin`, such as a top-level navigation, and a same-origin request still
|
|
50
|
+
work. A mutation keeps its own refusal. The change needs no migration.
|
|
51
|
+
|
|
52
|
+
Upgrade step: if a proxy in front of the host ends TLS, the host must see the scheme that the
|
|
53
|
+
browser used. Otherwise the dashboard page's own module scripts carry an `Origin` that does not
|
|
54
|
+
match, and the host refuses them.
|
|
55
|
+
|
|
56
|
+
### The dashboard Schedules page shows the last skip of a schedule
|
|
57
|
+
|
|
58
|
+
Migration 0087 (`0087-show-the-last-schedule-skip-on-the-dashboard.sql`) moves the schema to version 86. The `dashboard/v1` cron document gains `lastSkippedAt`, `lastSkippedCount`, and
|
|
59
|
+
`lastSkippedThrough` on each application schedule, an additive change. The bundled Schedules page
|
|
60
|
+
shows a `Skipped` badge with the count and the time of the last skip of a `skip` schedule (SM-1708).
|
|
61
|
+
The migration does not raise the SDK compatibility floor.
|
|
62
|
+
|
|
63
|
+
### The worker counts each failed heartbeat round
|
|
64
|
+
|
|
65
|
+
A heartbeat round can fail before PostgreSQL answers, for example when the connection breaks or the
|
|
66
|
+
round reaches its deadline. Before this change, such a round left no metric (SM-1704).
|
|
67
|
+
|
|
68
|
+
Now the worker counts each failed round once in the new counter
|
|
69
|
+
`workhorse.worker.heartbeat.round_failure`, with the unit `{round}`. The bounded attribute
|
|
70
|
+
`workhorse.heartbeat.failure_reason` gives the reason: `timeout`, `connection`, `database`, or
|
|
71
|
+
`other`. This applies to the reserved heartbeat connection and to `shared_heartbeats: true`. A
|
|
72
|
+
failed round still gives no ownership verdict, so `workhorse.worker.heartbeat.failure` does not
|
|
73
|
+
change. The worker adds no log record. The new instrument is an addition.
|
|
74
|
+
|
|
75
|
+
### A canceled task leaves its batch while it waits
|
|
76
|
+
|
|
77
|
+
A batch handler waits up to its linger for a partial batch. Before this change, a waiting task
|
|
78
|
+
stayed in the batch after a cancel request or a lease loss cancelled its cancellation token. The
|
|
79
|
+
batch handler then got a task that the worker no longer owned, and the task settled only after the
|
|
80
|
+
linger (SM-1709).
|
|
81
|
+
|
|
82
|
+
Now the worker removes such a task from the batch at once. The task raises the `CancelledError` of
|
|
83
|
+
the token reason, and the batch handler never gets it. The other waiting tasks keep their order and
|
|
84
|
+
dispatch. The TypeScript, Go, and Rust workers already remove such a task.
|
|
85
|
+
|
|
86
|
+
This includes the `:shutdown` cancellation at the end of the shutdown grace. Before, a task that
|
|
87
|
+
still waited then dispatched, and the batch handler ran its payload after the deadline. Now the task
|
|
88
|
+
goes back to its queue without a charged attempt. Before the deadline, the drain still dispatches a
|
|
89
|
+
lingering batch when all its tasks have arrived.
|
|
90
|
+
|
|
91
|
+
### The worker waits longer after each not_due expiry answer
|
|
92
|
+
|
|
93
|
+
When the local timer of a deadline or an execution timeout fires, the worker asks PostgreSQL to
|
|
94
|
+
expire the attempt. If the clock of PostgreSQL has not reached its expiry, PostgreSQL answers
|
|
95
|
+
`not_due`. Before this change, the worker asked again every 5 ms. Each request locks the task row,
|
|
96
|
+
so the heartbeat of the same attempt answered `busy`. If the worker clock ran ahead by more than one
|
|
97
|
+
lease, the attempt could lose its lease before the database expiry (SM-1701).
|
|
98
|
+
|
|
99
|
+
Now the first retry waits 5 ms, and each later retry waits twice as long, to a maximum of 1 s. The
|
|
100
|
+
TypeScript, Go, and Rust workers already use this schedule.
|
|
101
|
+
|
|
102
|
+
### The Ruby client measures signal and human wait values as PostgreSQL does
|
|
103
|
+
|
|
104
|
+
PostgreSQL limits a signal payload, a human wait context, and a human wait result to 65,536 bytes of
|
|
105
|
+
`octet_length(value::jsonb::text)`. That text has a space after each `:` and `,`, so it can be
|
|
106
|
+
longer than compact JSON. Before this change, the Ruby client counted the compact JSON. A value near
|
|
107
|
+
the limit passed that check, and then PostgreSQL refused the statement with a database error
|
|
108
|
+
(SM-1639).
|
|
109
|
+
|
|
110
|
+
Now `Queue#send_signal`, `Queue#complete_human_wait`, and `HandlerContext#wait_for_human` measure
|
|
111
|
+
the canonical JSONB text before any statement. An oversized value raises `ArgumentError`.
|
|
112
|
+
`HandlerContext#wait_for_human` also checks the context before its tier lookup, which can read
|
|
113
|
+
PostgreSQL. The limit does not change.
|
|
114
|
+
|
|
115
|
+
### The worker warns when a maintenance pass or a claim waits longer than the stop budget
|
|
116
|
+
|
|
117
|
+
A maintenance pass or a claim can wait for a lock that another session holds. Before this change,
|
|
118
|
+
the worker logged nothing while it waited. A test run blocked for about one hour, and no log named
|
|
119
|
+
the statement (SM-1785).
|
|
120
|
+
|
|
121
|
+
Now `Worker` logs `workhorse.worker.statement_slow` once, at warn level, if a maintenance pass or a
|
|
122
|
+
claim waits longer than the stop budget. The stop budget is `shutdown_grace`, 25 s by default. A
|
|
123
|
+
grace of 0 turns the warning off. The record carries `workhorse.worker.id`,
|
|
124
|
+
`workhorse.worker.statement` (`maintenance` or `claim`), and `workhorse.worker.elapsed_ms`.
|
|
125
|
+
`workhorse.worker.elapsed_ms` is the wait at the moment of the warning. The statement continues to
|
|
126
|
+
wait, and no other behavior changes.
|
|
127
|
+
|
|
128
|
+
### execution_timeout refuses a nonzero value that rounds to 0 ms
|
|
129
|
+
|
|
130
|
+
The queue rounds `execution_timeout:` to the nearest millisecond. Before this change, a positive
|
|
131
|
+
value below 0.0005 seconds rounded to 0, and the task had no timeout. A small negative value, such
|
|
132
|
+
as -0.0004, also set no timeout (SM-1714).
|
|
133
|
+
|
|
134
|
+
Now the queue raises `ArgumentError` for a nonzero value that rounds to 0 ms or is negative. `nil`
|
|
135
|
+
and an exact 0 still set no timeout. ADR 0114 records the rule for all seven SDKs.
|
|
136
|
+
|
|
137
|
+
### The worker intervals refuse a value above the SQL integer limit
|
|
138
|
+
|
|
139
|
+
`register_worker_v1` takes `maintenance_interval`, `maintenance_routine_interval`, and
|
|
140
|
+
`registry_interval` as PostgreSQL `integer` milliseconds. Before this change, `Worker.new` accepted
|
|
141
|
+
a larger value, and then each registration failed (SM-1668).
|
|
142
|
+
|
|
143
|
+
Now `Worker.new` raises `ArgumentError` if one of these intervals is not from 0.1 through
|
|
144
|
+
2,147,483.647 seconds. `disable_registry: true` still turns off registration. This is a behavior
|
|
145
|
+
change for a caller who passes a larger value. ADR 0112 records the range for all seven SDKs.
|
|
146
|
+
|
|
147
|
+
### An anonymous error class records the name of its superclass
|
|
148
|
+
|
|
149
|
+
Ruby gives an anonymous class a nil name. Before this change, a handler that raised an instance of
|
|
150
|
+
`Class.new(StandardError)` recorded JSON null as the failure `name` (SM-1668).
|
|
151
|
+
|
|
152
|
+
Now the failure envelope records the name of the nearest named superclass, `StandardError` in that
|
|
153
|
+
example. Logs and telemetry use the same name for `error.type`.
|
|
154
|
+
|
|
155
|
+
### retry_delay refuses a small negative value and rounds exact values exactly
|
|
156
|
+
|
|
157
|
+
`fail_v1` takes the retry delay as a PostgreSQL `integer`. Before this change, the worker rounded a
|
|
158
|
+
`retry_delay` to milliseconds first. Thus a negative value such as `-0.0004` became 0 and passed.
|
|
159
|
+
The worker also converted the value to a Float first, so `Rational(1001, 2000)`, which is exactly
|
|
160
|
+
500.5 ms, became 500 ms (SM-1668).
|
|
161
|
+
|
|
162
|
+
Now the worker raises `ArgumentError` for every negative `retry_delay`, at construction for a fixed
|
|
163
|
+
value and before `fail_v1` for a callable result. It converts Integer, Rational, and BigDecimal
|
|
164
|
+
seconds exactly and rounds half up, so that value becomes 501 ms. ADR 0111 records the rule for all
|
|
165
|
+
seven SDKs.
|
|
166
|
+
|
|
167
|
+
### A raising reporter callback no longer stops the worker
|
|
168
|
+
|
|
169
|
+
An application can give `on_registration_error` to the worker to send an alert. Before this change,
|
|
170
|
+
a registration refresh could fail, and then the callback could raise. The error left the registry
|
|
171
|
+
path and could stop the run. The worker also discarded a raise from `on_notification_error` without
|
|
172
|
+
a record (SM-1696).
|
|
173
|
+
|
|
174
|
+
Now the worker rescues a `StandardError` from either callback and continues. It writes the failure
|
|
175
|
+
to `logger` at the error level. Without a logger, it writes the failure to standard error. For each
|
|
176
|
+
callback, it writes the first failure. Then it writes at most one failure in each 60 s, with the
|
|
177
|
+
count of the failures that it did not write. It never calls the failed callback for its own failure.
|
|
178
|
+
|
|
179
|
+
### Workers use the database clock for maintenance
|
|
180
|
+
|
|
181
|
+
Maintenance writes shared state: the due time of each routine and the retention cutoffs. Before
|
|
182
|
+
this change, the Ruby worker sent its host clock as the maintenance time. If that clock was two days
|
|
183
|
+
fast, other workers skipped a routine until that future time. Retention also deleted a 13-day-old
|
|
184
|
+
schedule occurrence under a 14-day policy (SM-1645).
|
|
185
|
+
|
|
186
|
+
Now `Worker#run` and `Worker#run_once` bind the text `now`, and PostgreSQL resolves it to its own
|
|
187
|
+
time at the start of the call.
|
|
188
|
+
|
|
189
|
+
The schema does not change.
|
|
190
|
+
|
|
191
|
+
### The dashboard names an action that the host does not provide
|
|
192
|
+
|
|
193
|
+
A dashboard host answers the `enqueueTest` and `setSchedulePaused` procedures only when the
|
|
194
|
+
application gives a handler for them. Before this change, a writable host without that handler
|
|
195
|
+
answered `FORBIDDEN` with the message "This procedure is not available". Now the message is "This
|
|
196
|
+
dashboard host does not provide the <action> action", as in the TypeScript dashboard server. A
|
|
197
|
+
read-only host keeps the message "This dashboard is read-only". The code `FORBIDDEN` and the HTTP
|
|
198
|
+
status 403 do not change (SM-1705).
|
|
199
|
+
|
|
200
|
+
### A lost notification subscription restarts the poll backoff
|
|
201
|
+
|
|
202
|
+
A worker without a live subscription waits 250 ms after an empty claim, and doubles that wait
|
|
203
|
+
toward 5,000 ms. Before this change, the empty claims of a live period counted toward that backoff.
|
|
204
|
+
After a loss, a worker that was idle for some time therefore waited the 5,000 ms ceiling at once,
|
|
205
|
+
and a new task waited up to 5 s (SM-1648).
|
|
206
|
+
|
|
207
|
+
Now the backoff restarts at 250 ms when the subscription is lost. The `subscription-recovery` case
|
|
208
|
+
in `protocol/v1/runtime.json` checks this rule in every SDK.
|
|
209
|
+
|
|
210
|
+
### A polling-only worker keeps its listener connection for claims
|
|
211
|
+
|
|
212
|
+
A worker without a `cohorts` option caps its default cohort count by the connections that its pool
|
|
213
|
+
can lend. Before this change, a worker with `polling_only: true` also subtracted a connection for a
|
|
214
|
+
listener that it never opens. With a pool of 4 and concurrency 64, it used 2 cohorts instead of 3
|
|
215
|
+
(SM-1668).
|
|
216
|
+
|
|
217
|
+
Now a polling-only worker reserves no listener connection. A worker that listens still reserves one,
|
|
218
|
+
and a worker with dedicated heartbeats still reserves one more.
|
|
219
|
+
|
|
220
|
+
### A handler error with text that jsonb cannot store settles by its retry policy
|
|
221
|
+
|
|
222
|
+
PostgreSQL `jsonb` cannot store a NUL character or an unpaired surrogate. Before this change, a
|
|
223
|
+
handler error with such text made `fail_v1` refuse the failure envelope. The worker stopped, and the
|
|
224
|
+
task stayed active until lease recovery (SM-1650).
|
|
225
|
+
|
|
226
|
+
Now `error_envelope` replaces each NUL character and each byte that is not UTF-8 with U+FFFD. It
|
|
227
|
+
does this in the name, the message, and the stack. The JVM and .NET workers use the same rule. The
|
|
228
|
+
retry policy then settles the task. All other text does not change.
|
|
229
|
+
|
|
230
|
+
The `protocol/v1/failures.json` fixtures pin the rule for every SDK.
|
|
231
|
+
|
|
232
|
+
### A refused completion settles a due deadline or attempt timeout
|
|
233
|
+
|
|
234
|
+
A handler can return just before its deadline or attempt timeout, and the completion write can then
|
|
235
|
+
arrive after that boundary. PostgreSQL refuses the completion. Before this change, the worker only
|
|
236
|
+
tried to acknowledge a cancellation, then recorded `lease_expired` and left the task to lease
|
|
237
|
+
recovery. The task then settled up to one lease later with the wrong outcome. Now
|
|
238
|
+
`reconcile_rejected_completion` calls `expire_owned_telemetry_v1` under the attempt's fence before
|
|
239
|
+
the completion path returns. The attempt then ends with the outcome of the due boundary. A
|
|
240
|
+
cancellation request and a lost lease keep their own outcomes (SM-1649).
|
|
241
|
+
|
|
242
|
+
### Workers on one pool renew their leases in one transaction
|
|
243
|
+
|
|
244
|
+
On the dedicated heartbeat connection, a round sent one transaction for each worker and each 100
|
|
245
|
+
leases. Each transaction needs at least 3 round trips to PostgreSQL. With 8 workers on one pool and
|
|
246
|
+
80 ms for each round trip, workers lost leases while PostgreSQL still answered (SM-1631). Now the heartbeat thread of the pool sends one transaction for each round, with one `heartbeat_pool_v1` statement for every worker and every 100 leases. The
|
|
247
|
+
transaction locks the task rows of every worker in it until its `COMMIT`. A round renews all of its
|
|
248
|
+
leases or none. The settings statement now sets `statement_timeout` and
|
|
249
|
+
`idle_in_transaction_session_timeout` each to one quarter of the time that remains, not to one half
|
|
250
|
+
and one eighth. The statement needs no schema change. The `shared_heartbeats` opt-out does not change (SM-1633).
|
|
251
|
+
|
|
252
|
+
### The dashboard sets the tier and the history of a queue
|
|
253
|
+
|
|
254
|
+
The Rack dashboard answers two new mutations, `setQueueTier` and `setQueueHistory`. `setQueueTier`
|
|
255
|
+
calls `Admin#set_queue_tier` with the authenticated actor and answers `{status: "set", tier}`. If
|
|
256
|
+
Workhorse refuses the switch, the dashboard answers the guard as the status: `live_tasks`,
|
|
257
|
+
`concurrency_policy`, or `rate_limit_policy`. Then `tier` is `null`, and the queue keeps its tier.
|
|
258
|
+
`setQueueHistory` calls `Admin#set_queue_history`. An absent setting keeps its current value. A
|
|
259
|
+
read-only dashboard refuses both mutations (SM-966).
|
|
260
|
+
|
|
261
|
+
### `Admin` moves a queue between tiers and sets its history
|
|
262
|
+
|
|
263
|
+
`Admin#set_queue_tier(queue, tier, audit:)` moves a queue to the `:fast` or `:full` tier and returns
|
|
264
|
+
the tier after the change. If the tier differs and the queue holds a live task, the method raises
|
|
265
|
+
`FastTierUnsupportedError`.
|
|
266
|
+
`Admin#set_queue_history(queue, record_attempts: nil, record_claims: nil)` chooses the history a
|
|
267
|
+
fast-tier queue records. A `nil` setting keeps its current value. The method returns a
|
|
268
|
+
`QueueHistory`. The other SDKs already have these controls (SM-1417).
|
|
269
|
+
|
|
270
|
+
### A worker on a pool of 1 polls under the opt-out
|
|
271
|
+
|
|
272
|
+
Before this change, a worker with `shared_heartbeats: true` started the shared listener on any pool.
|
|
273
|
+
On a pool of 1, the listener held the only connection. Registration, claims, and handler statements
|
|
274
|
+
then waited for the pool timeout, and no task ran (SM-1585).
|
|
275
|
+
|
|
276
|
+
Now, with `shared_heartbeats: true`, the worker starts the listener only when the pool `size` is an
|
|
277
|
+
Integer of at least 2. On a pool of 1, or on a pool without a `size`, the worker polls. Each run
|
|
278
|
+
then logs `workhorse.worker.polling_only` once at warn level. The message names
|
|
279
|
+
`workhorse.worker.queues` and `workhorse.worker.pool_max_size`. A worker with `polling_only: true`
|
|
280
|
+
logs no warning. Without the opt-out, nothing changes.
|
|
281
|
+
|
|
282
|
+
### A contract compares a stored schema number with all its digits
|
|
283
|
+
|
|
284
|
+
Before this change, the Ruby SDK read a stored contract schema with `JSON.parse`, which reads a
|
|
285
|
+
fraction as a Float. Another SDK can store `"minimum": 0.30000000000000000001`, which a Float cannot
|
|
286
|
+
hold. The Ruby SDK then accepted `0.3`. Now a fraction that a Float cannot hold stays a Rational,
|
|
287
|
+
and the contract keywords compare it exactly (SM-1541, ADR 0039). `ContractSchema.new` still
|
|
288
|
+
accepts only JSON values.
|
|
289
|
+
|
|
290
|
+
### A worker reserves its heartbeat connection at startup, and refuses to start without it
|
|
291
|
+
|
|
292
|
+
Before this change, a worker took the heartbeat connection of its pool only when a task arrived,
|
|
293
|
+
and returned it when no task remained. If the pool had no free connection then, the worker still
|
|
294
|
+
claimed tasks. The heartbeat waited for a connection, and the leases of those tasks could expire
|
|
295
|
+
(SM-1577). These changes apply to a worker without `shared_heartbeats: true`:
|
|
296
|
+
|
|
297
|
+
- **A startup reservation failure now ends the run.** This changes behavior. `run` and `run_once`
|
|
298
|
+
first reserve the heartbeat connection, then borrow one other connection for the compatibility
|
|
299
|
+
query and return it at once. Both steps must end within the larger of `heartbeat` and 5 s. This
|
|
300
|
+
bound includes a connect and the query itself, for every kind of pool. If a step does not end in
|
|
301
|
+
time, the run raises the new `Stablemates::Workhorse::WorkerStartupError` before the worker
|
|
302
|
+
registers. Its message names the pool, the budget, and the `shared_heartbeats: true` opt-out. A
|
|
303
|
+
compatibility query that fails keeps its own error type. Before this change, the same worker started and claimed tasks. A
|
|
304
|
+
supervisor that restarts a failed process, such as `run_worker_processes`, starts the worker
|
|
305
|
+
again.
|
|
306
|
+
- **The reservation lasts while a worker runs.** The heartbeat keeps its connection while any worker
|
|
307
|
+
on the pool runs, also when no task runs. The last worker that stops returns it. Thus a pool keeps
|
|
308
|
+
one heartbeat connection for its running workers, apart from the short replacement after a lost
|
|
309
|
+
connection.
|
|
310
|
+
- **No claim without a heartbeat connection.** While the pool has no usable heartbeat connection,
|
|
311
|
+
the worker claims no task. A claim checks this after it has its connection, immediately before
|
|
312
|
+
each attempt, and only for its own run. A fast-tier completion then settles its task with a claim
|
|
313
|
+
limit of 0. A rejected or busy lease does not stop claims. The heartbeat checks its connection
|
|
314
|
+
between rounds, also when no task runs. If the connection is lost, or a round fails, the heartbeat
|
|
315
|
+
asks for a new one at once. It waits for a checkout at most one round bound, and at most one
|
|
316
|
+
heartbeat interval between failed attempts. When the new connection is ready, the worker claims at
|
|
317
|
+
once.
|
|
318
|
+
- **One shutdown deadline.** `shutdown_grace` now starts when dispatch ends. Every wait of the stop
|
|
319
|
+
path ends by that deadline or one unwind window later. This includes the registration statements
|
|
320
|
+
and the release of the listener and the heartbeat connection. Thus a stop takes at most
|
|
321
|
+
`shutdown_grace` plus the unwind window. A claim that returns after the deadline starts no task,
|
|
322
|
+
and its leases expire. The heartbeat connection stays reserved until that claim statement
|
|
323
|
+
settles. A stop during the startup reservation ends the run without an error.
|
|
324
|
+
- **A new run waits for the stop statements of the previous run.** The next `run` or `run_once` of
|
|
325
|
+
the same worker waits for them within its startup budget. Then a late deregistration cannot
|
|
326
|
+
delete the registry row of the new run. If a statement still runs when the budget ends, the run
|
|
327
|
+
raises `WorkerStartupError`. A stop during this wait ends the run without an error.
|
|
328
|
+
|
|
329
|
+
### Throttle works on a fast-tier queue
|
|
330
|
+
|
|
331
|
+
Before this change, a throttled enqueue into a fast-tier queue raised the fast-tier error with the feature
|
|
332
|
+
`throttle`. Throttle only decides if Workhorse accepts an enqueue request, so it needs no state of
|
|
333
|
+
the full tier (SM-928, ADR 0077). Migration 0075
|
|
334
|
+
(`0075-allow-throttle-on-fast-tier-queues.sql`) moves the schema to version 74. After it,
|
|
335
|
+
`enqueue_throttle_v1` and `enqueue_many_v1` accept throttle on a fast-tier queue. The outcomes are
|
|
336
|
+
`accepted` and `coalesced`, as on the full tier.
|
|
337
|
+
|
|
338
|
+
A coalesced request on a fast-tier task writes no `throttled` event, because a fast-tier task
|
|
339
|
+
records only the history that its queue enables. The SDK still emits the throttle log and the
|
|
340
|
+
enqueue outcome metric. In a batch with a debounce or a throttle member, a fast-tier rejection of
|
|
341
|
+
one member now reports the position of that member in the batch. Debounce on a fast-tier queue is
|
|
342
|
+
still rejected.
|
|
343
|
+
|
|
344
|
+
The migration does not raise the SDK compatibility floor. No SDK statement changes. The migration changes no stored data.
|
|
345
|
+
|
|
346
|
+
### The retention preview reads through a dashboard function
|
|
347
|
+
|
|
348
|
+
Before this change, each dashboard host read the retention policy, sent a shared statement that
|
|
349
|
+
counted rows directly in the tables, and capped the counts in its own code. ADR 0040 asks for one
|
|
350
|
+
SQL implementation for each dashboard procedure (SM-1478). Migration 0076
|
|
351
|
+
(`0076-add-the-dashboard-retention-preview-function.sql`) moves the schema to version 75 and adds
|
|
352
|
+
`dashboard_retention_preview_v1`. The Ruby dashboard host now calls it. The `previewRetentionPolicy` response does
|
|
353
|
+
not change.
|
|
354
|
+
|
|
355
|
+
Every SDK statement catalogue names the new function, so the SDK compatibility floor rises from
|
|
356
|
+
schema 73 to 75. Run `workhorse schema migrate` before the first process of this release starts.
|
|
357
|
+
A 0.9 process keeps working on schema 75. The migration changes no stored data.
|
|
358
|
+
|
|
359
|
+
### A late worker deregistration cannot delete the row of a later run
|
|
360
|
+
|
|
361
|
+
Before this change, a worker that stopped sent `deregister_worker_v1`, which deletes the registry
|
|
362
|
+
row by worker ID only. If the stop path was slow and the same worker ID started again, the late
|
|
363
|
+
deregistration could delete the row of the new run and its operator pause state. Each SDK guarded
|
|
364
|
+
against this in its own code (SM-1589). Migration
|
|
365
|
+
0077 (`0077-fence-the-worker-deregistration-by-instance.sql`) moves the schema to version 76 and
|
|
366
|
+
adds `deregister_worker_v2`. It deletes the row only while the row names the instance of the run
|
|
367
|
+
that stops. `deregister_worker_v1` does not change.
|
|
368
|
+
|
|
369
|
+
`Stablemates::Workhorse::Worker` now calls `deregister_worker_v2` with the instance of the run that
|
|
370
|
+
stops.
|
|
371
|
+
|
|
372
|
+
Each SDK keeps its guard for this release: a new run still waits for the stop statements of the
|
|
373
|
+
previous run, or it refuses to start. Every SDK statement catalogue names the new function, so the
|
|
374
|
+
SDK compatibility floor rises from schema 75 to 76. Run `workhorse schema migrate` before the first
|
|
375
|
+
process of this release starts. A 0.9 process keeps working on schema 76. The migration changes no
|
|
376
|
+
stored data.
|
|
377
|
+
|
|
6
378
|
## 0.9.0 — 2026-10-20
|
|
7
379
|
|
|
8
380
|
The npm packages, Python distribution, Go module, Rust crate, Ruby gem, and JVM artifacts release from one source commit.
|
data/README.md
CHANGED
|
@@ -30,7 +30,7 @@ bundle add connection_pool
|
|
|
30
30
|
Install the schema once, as a deployment step. The application never installs or migrates it.
|
|
31
31
|
|
|
32
32
|
```bash
|
|
33
|
-
npx --package @stablemates/workhorse@0.
|
|
33
|
+
npx --package @stablemates/workhorse@0.10.0 workhorse schema install
|
|
34
34
|
```
|
|
35
35
|
|
|
36
36
|
The machine that runs that deployment step needs Node.js 22 or newer. The application itself needs
|
|
@@ -92,6 +92,9 @@ module Stablemates
|
|
|
92
92
|
|
|
93
93
|
ExternalWaitPage = Data.define(:items, :next_cursor)
|
|
94
94
|
|
|
95
|
+
# The history a queue records: attempt rows and claim events.
|
|
96
|
+
QueueHistory = Data.define(:record_attempts, :record_claims)
|
|
97
|
+
|
|
95
98
|
WorkerPauseResult = Data.define(:worker_id, :paused, :paused_by, :reason, :paused_at, :last_heartbeat_at)
|
|
96
99
|
|
|
97
100
|
WorkerRegistryEntry = Data.define(
|
|
@@ -102,16 +105,18 @@ module Stablemates
|
|
|
102
105
|
# Inspects and controls tasks, dead letters, workers, and queues through the executor it holds.
|
|
103
106
|
#
|
|
104
107
|
# Like Queue, every call runs on the connection the executor yields, so a control joins the
|
|
105
|
-
# caller's transaction. Every control takes an AdminAudit that
|
|
108
|
+
# caller's transaction. Every control except +set_queue_history+ takes an AdminAudit that
|
|
109
|
+
# PostgreSQL records.
|
|
106
110
|
class Admin
|
|
107
111
|
MAX_PAGE_SIZE = 1000
|
|
108
112
|
MAX_REDRIVE_BATCH_SIZE = 1000
|
|
109
113
|
MAX_PAYLOAD_BYTES = 1_048_576
|
|
110
114
|
MAX_REDACT_KEYS = 50
|
|
111
115
|
TASK_STATES = %i[blocked scheduled ready active succeeded failed canceled].freeze
|
|
116
|
+
QUEUE_TIERS = %i[fast full].freeze
|
|
112
117
|
REDRIVE_STATUSES = %i[redriven replayed eligible not_found not_failed].freeze
|
|
113
118
|
private_constant :MAX_PAGE_SIZE, :MAX_REDRIVE_BATCH_SIZE, :MAX_PAYLOAD_BYTES, :MAX_REDACT_KEYS,
|
|
114
|
-
:TASK_STATES, :REDRIVE_STATUSES
|
|
119
|
+
:TASK_STATES, :QUEUE_TIERS, :REDRIVE_STATUSES
|
|
115
120
|
|
|
116
121
|
def initialize(executor)
|
|
117
122
|
@executor = Executor.for(executor)
|
|
@@ -294,12 +299,14 @@ module Stablemates
|
|
|
294
299
|
|
|
295
300
|
def resume_queue(queue, audit:) = set_queue_paused(queue, false, audit)
|
|
296
301
|
|
|
297
|
-
# Deletes
|
|
298
|
-
#
|
|
302
|
+
# Deletes the blocked, ready, and scheduled tasks of the queue, and returns how many tasks it
|
|
303
|
+
# deleted or canceled. If a task outside the purge waits for a purged task, the purge cancels
|
|
304
|
+
# that task instead of a delete. The rule also applies to a wait through other purged tasks.
|
|
305
|
+
# The count includes the canceled tasks.
|
|
299
306
|
#
|
|
300
|
-
# Do not call it in a transaction that
|
|
301
|
-
#
|
|
302
|
-
#
|
|
307
|
+
# Do not call it in a transaction that holds task row locks, for example after a cancel or an
|
|
308
|
+
# enqueue. Two purges run one after the other, so such a transaction can deadlock with another
|
|
309
|
+
# purge.
|
|
303
310
|
def purge_queue(queue, audit:)
|
|
304
311
|
name(queue, "queue")
|
|
305
312
|
audit(audit)
|
|
@@ -310,6 +317,37 @@ module Stablemates
|
|
|
310
317
|
Values.parse_integer(rows.first.fetch("deleted_count"))
|
|
311
318
|
end
|
|
312
319
|
|
|
320
|
+
# Moves a queue between the +:fast+ and +:full+ tiers and returns the tier after the change.
|
|
321
|
+
# If the tier differs, Workhorse refuses the change with FastTierUnsupportedError while the
|
|
322
|
+
# queue holds a live task, so a task never changes tier. The audit's +request_id+ is not recorded.
|
|
323
|
+
def set_queue_tier(queue, tier, audit:)
|
|
324
|
+
name(queue, "queue")
|
|
325
|
+
raise ArgumentError, "tier must be :fast or :full" unless QUEUE_TIERS.include?(tier)
|
|
326
|
+
|
|
327
|
+
audit(audit)
|
|
328
|
+
assert_compatible
|
|
329
|
+
rows = @executor.rows(SqlCatalogue::SET_QUEUE_TIER, [queue, tier.to_s, audit.actor, audit.reason])
|
|
330
|
+
raise ArgumentError, "workhorse.set_queue_tier_v1 returned #{rows.length} rows; expected one" unless rows.one?
|
|
331
|
+
|
|
332
|
+
rows.first.fetch("tier").to_sym
|
|
333
|
+
end
|
|
334
|
+
|
|
335
|
+
# Chooses the history a fast-tier queue records. A nil setting keeps its current value, and
|
|
336
|
+
# the QueueHistory reports both settings after the change.
|
|
337
|
+
def set_queue_history(queue, record_attempts: nil, record_claims: nil)
|
|
338
|
+
name(queue, "queue")
|
|
339
|
+
optional_boolean(record_attempts, "record_attempts")
|
|
340
|
+
optional_boolean(record_claims, "record_claims")
|
|
341
|
+
assert_compatible
|
|
342
|
+
rows = @executor.rows(SqlCatalogue::SET_QUEUE_HISTORY,
|
|
343
|
+
[queue, record_attempts&.to_s, record_claims&.to_s])
|
|
344
|
+
raise ArgumentError, "workhorse.set_queue_history_v1 returned #{rows.length} rows; expected one" unless rows.one?
|
|
345
|
+
|
|
346
|
+
row = rows.first
|
|
347
|
+
QueueHistory.new(record_attempts: Values.parse_boolean(row.fetch("record_attempts")),
|
|
348
|
+
record_claims: Values.parse_boolean(row.fetch("record_claims")))
|
|
349
|
+
end
|
|
350
|
+
|
|
313
351
|
# The TaskGroup, or nil when no such group exists.
|
|
314
352
|
def get_task_group(group_id)
|
|
315
353
|
group_id = TaskGroups.group_id(group_id)
|
|
@@ -412,6 +450,10 @@ module Stablemates
|
|
|
412
450
|
raise ArgumentError, "#{label} must be a String or nil" unless value.nil? || value.is_a?(String)
|
|
413
451
|
end
|
|
414
452
|
|
|
453
|
+
def optional_boolean(value, label)
|
|
454
|
+
raise ArgumentError, "#{label} must be true, false, or nil" unless [true, false, nil].include?(value)
|
|
455
|
+
end
|
|
456
|
+
|
|
415
457
|
def optional_time(value, label) = value.nil? ? nil : Values.timestamp(value, label)
|
|
416
458
|
|
|
417
459
|
def task_filter(queue, type, states, created_after, created_before)
|
|
@@ -222,12 +222,15 @@ module Stablemates
|
|
|
222
222
|
# arrives. +context+ is JSON shown to that person; +timeout+ is a count of seconds.
|
|
223
223
|
def wait_for_human(name, context, timeout: nil)
|
|
224
224
|
external_name(name, "human wait")
|
|
225
|
-
durable!("human waits")
|
|
226
225
|
timeout_ms = external_timeout(timeout, "human wait")
|
|
227
226
|
Values.check_json(context, "human wait context")
|
|
228
227
|
encoded = canonical_json(context)
|
|
228
|
+
# PostgreSQL bounds the context with octet_length(context::jsonb::text), which can be
|
|
229
|
+
# longer than the compact JSON, so the worker measures the same text. The check runs before
|
|
230
|
+
# the tier lookup, which can read PostgreSQL.
|
|
229
231
|
raise ArgumentError, "human wait context must be at most #{MAX_EXTERNAL_VALUE_BYTES} bytes of JSON" if
|
|
230
|
-
encoded
|
|
232
|
+
Values.jsonb_text_bytes(encoded) > MAX_EXTERNAL_VALUE_BYTES
|
|
233
|
+
durable!("human waits")
|
|
231
234
|
|
|
232
235
|
once(:wait_for_human, name, encoded) do
|
|
233
236
|
live!(:wait_for_human)
|
|
@@ -122,7 +122,7 @@ module Stablemates
|
|
|
122
122
|
end
|
|
123
123
|
private_class_method :names?
|
|
124
124
|
|
|
125
|
-
def self.number?(value) = value.is_a?(Integer) || (value.is_a?(Float) && value.finite?)
|
|
125
|
+
def self.number?(value) = value.is_a?(Integer) || value.is_a?(Rational) || (value.is_a?(Float) && value.finite?)
|
|
126
126
|
private_class_method :number?
|
|
127
127
|
|
|
128
128
|
# A non-negative integer, which JSON may write as 3.0.
|
|
@@ -133,6 +133,18 @@ module Stablemates
|
|
|
133
133
|
|
|
134
134
|
def initialize(schema)
|
|
135
135
|
Values.check_json(schema, "contract schema")
|
|
136
|
+
compile(schema)
|
|
137
|
+
end
|
|
138
|
+
|
|
139
|
+
# A schema that +Contracts.parse_schema+ read from PostgreSQL. A Rational in it holds a
|
|
140
|
+
# fraction that a Float cannot hold. Internal to the SDK.
|
|
141
|
+
def self.stored(schema) = allocate.tap { |contract| contract.send(:compile, schema) } # :nodoc:
|
|
142
|
+
|
|
143
|
+
def valid?(instance) = valid_at?(@schema, instance, 0)
|
|
144
|
+
|
|
145
|
+
private
|
|
146
|
+
|
|
147
|
+
def compile(schema)
|
|
136
148
|
violation = ContractSchema.profile_violation(schema)
|
|
137
149
|
raise ArgumentError, violation if violation
|
|
138
150
|
|
|
@@ -140,10 +152,6 @@ module Stablemates
|
|
|
140
152
|
check_references(schema, "$")
|
|
141
153
|
end
|
|
142
154
|
|
|
143
|
-
def valid?(instance) = valid_at?(@schema, instance, 0)
|
|
144
|
-
|
|
145
|
-
private
|
|
146
|
-
|
|
147
155
|
# A reference names the root schema or one root definition. The validators behind the six SDKs
|
|
148
156
|
# resolve those two forms alike, and both name a schema position the profile check visited. The
|
|
149
157
|
# check runs before +valid?+, so +valid?+ cannot fail on the schema.
|
|
@@ -289,7 +297,8 @@ module Stablemates
|
|
|
289
297
|
|
|
290
298
|
# The decimal that JSON writes to PostgreSQL for a number. Float#to_s gives the shortest
|
|
291
299
|
# decimal that reads back as the same Float, so 2**60 as a Float compares as
|
|
292
|
-
# 1152921504606847000, not as its binary value. A non-finite Float stays a Float.
|
|
300
|
+
# 1152921504606847000, not as its binary value. A non-finite Float stays a Float. A Rational
|
|
301
|
+
# from a stored schema is already exact.
|
|
293
302
|
def exact(number) = (number.is_a?(Float) && number.finite?) ? Rational(number.to_s) : number
|
|
294
303
|
|
|
295
304
|
def finite?(number) = !number.is_a?(Float) || number.finite?
|
|
@@ -302,7 +311,7 @@ module Stablemates
|
|
|
302
311
|
end
|
|
303
312
|
when Array then right.is_a?(Array) && left.size == right.size && left.zip(right).all? { |l, r| equal?(l, r) }
|
|
304
313
|
when true, false, nil then left.equal?(right)
|
|
305
|
-
when Integer, Float then number?(right) && exact(left) == exact(right)
|
|
314
|
+
when Integer, Float, Rational then number?(right) && exact(left) == exact(right)
|
|
306
315
|
else left == right
|
|
307
316
|
end
|
|
308
317
|
end
|
|
@@ -333,12 +342,12 @@ module Stablemates
|
|
|
333
342
|
end
|
|
334
343
|
|
|
335
344
|
def decode(row)
|
|
336
|
-
schema =
|
|
345
|
+
schema = parse_schema(row.fetch("schema"))
|
|
337
346
|
raise invalid_definition unless schema.is_a?(Hash) && schema.key?("payload")
|
|
338
347
|
|
|
339
348
|
PayloadContract.new(
|
|
340
349
|
version: row.fetch("version"),
|
|
341
|
-
schema: ContractSchema.
|
|
350
|
+
schema: ContractSchema.stored(schema["payload"]),
|
|
342
351
|
payload_max_bytes: Values.parse_integer(row.fetch("payload_max_bytes")),
|
|
343
352
|
result_max_bytes: Values.parse_integer(row.fetch("result_max_bytes")),
|
|
344
353
|
payload_redact_keys: Values.parse_text_array(row.fetch("payload_redact_keys")),
|
|
@@ -350,6 +359,22 @@ module Stablemates
|
|
|
350
359
|
|
|
351
360
|
def invalid_definition = Error.new("invalid contract definition returned by PostgreSQL")
|
|
352
361
|
|
|
362
|
+
# The jsonb text of a stored schema document. jsonb keeps every digit of a number that another
|
|
363
|
+
# SDK synchronized. A fraction that a Float cannot hold stays a Rational, so the contract
|
|
364
|
+
# compares it exactly.
|
|
365
|
+
def parse_schema(text) = text.nil? ? nil : exact_numbers(JSON.parse(text, decimal_class: Rational))
|
|
366
|
+
|
|
367
|
+
def exact_numbers(value)
|
|
368
|
+
case value
|
|
369
|
+
when Rational
|
|
370
|
+
float = value.to_f
|
|
371
|
+
(float.finite? && Rational(float.to_s) == value) ? float : value
|
|
372
|
+
when Array then value.map { |item| exact_numbers(item) }
|
|
373
|
+
when Hash then value.transform_values { |item| exact_numbers(item) }
|
|
374
|
+
else value
|
|
375
|
+
end
|
|
376
|
+
end
|
|
377
|
+
|
|
353
378
|
# Compiles every schema, then renders the +sync_contract_definitions_v1+ document.
|
|
354
379
|
def document(contracts)
|
|
355
380
|
raise ArgumentError, "contracts must be a Hash of task type to TaskTypeContracts" unless contracts.is_a?(Hash)
|