stablemates-workhorse 0.9.0 → 0.11.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +579 -0
- data/README.md +3 -2
- data/lib/stablemates/workhorse/admin.rb +49 -7
- data/lib/stablemates/workhorse/context.rb +11 -2
- data/lib/stablemates/workhorse/contracts.rb +34 -9
- data/lib/stablemates/workhorse/dashboard/backend.rb +30 -26
- data/lib/stablemates/workhorse/dashboard/bundle.json +1 -1
- data/lib/stablemates/workhorse/dashboard/read-surface-1.tar.gz +0 -0
- data/lib/stablemates/workhorse/dashboard/v1_generated.rb +3 -1
- data/lib/stablemates/workhorse/dashboard.rb +25 -6
- data/lib/stablemates/workhorse/errors.rb +5 -0
- data/lib/stablemates/workhorse/executor.rb +6 -2
- data/lib/stablemates/workhorse/queue.rb +76 -36
- data/lib/stablemates/workhorse/sql_catalogue_generated.rb +163 -48
- data/lib/stablemates/workhorse/telemetry.rb +101 -2
- data/lib/stablemates/workhorse/version.rb +1 -1
- data/lib/stablemates/workhorse/worker/batch.rb +50 -3
- data/lib/stablemates/workhorse/worker/heartbeat.rb +572 -135
- data/lib/stablemates/workhorse/worker/listener.rb +9 -5
- data/lib/stablemates/workhorse/worker/statement_watch.rb +86 -0
- data/lib/stablemates/workhorse/worker.rb +660 -132
- data/lib/stablemates/workhorse.rb +1 -0
- metadata +2 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 1686623ff5e225f372245830a169d9b248db6bd195ed3f446679e01098bc9610
|
|
4
|
+
data.tar.gz: bde5f7e6517d2a7a3621b437237db776673156693b22fc1a46f42aa28305b563
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: eef44a6c8715cc1d6c0f7983fb42b3662332ad1302fd6d705e60eca9f2d4ed099d32fd53d1303a2ee1a0cdfa9ee08c3624c57516c967c22cb62f0366180bdf4b
|
|
7
|
+
data.tar.gz: e9137fa4c8b8f6a5f20f800cb4abc482e6c750721758fb5432f407c229816b4a4186700e882eeb8859a8d21f8bcfa2f2be6be1ae734ef574a003227a8db82a94
|
data/CHANGELOG.md
CHANGED
|
@@ -3,6 +3,585 @@
|
|
|
3
3
|
`stablemates-workhorse` gem versions and release notes live here. The gem carries the version the
|
|
4
4
|
other SDKs carry, because every tag names one release of all of them.
|
|
5
5
|
|
|
6
|
+
## 0.11.0 — 2026-10-28
|
|
7
|
+
|
|
8
|
+
The npm packages, Python distribution, Go module, Rust crate, Ruby gem, JVM artifacts, and .NET
|
|
9
|
+
packages release from one source commit.
|
|
10
|
+
|
|
11
|
+
Requires **schema v76**, Ruby **3.3** or newer, and PostgreSQL **15** or newer.
|
|
12
|
+
|
|
13
|
+
### Upgrade steps
|
|
14
|
+
|
|
15
|
+
**Migrate the schema before starting updated processes.** Run `workhorse schema migrate` before any
|
|
16
|
+
process of this release starts, producers too. Migrations 0088, 0089, and 0090 move the schema
|
|
17
|
+
from version 86 to version **89**. They replace function bodies and change no stored rows. The SDK
|
|
18
|
+
compatibility floor stays at **76**. A 0.10 process keeps working on version 89, because a
|
|
19
|
+
migration inside a major line only adds.
|
|
20
|
+
|
|
21
|
+
**Give each worker a pool with a known size of at least 3.** `Worker.new` no longer accepts
|
|
22
|
+
`shared_heartbeats:`. Remove the option. If a worker pool reports a size below 3, or reports no
|
|
23
|
+
size, set its size to 3 or more. Each worker process needs 3 connections: the listener, the
|
|
24
|
+
heartbeat connection, and one statement connection. A `polling_only: true` worker opens no
|
|
25
|
+
listener, but it still needs a pool of 3 (SM-1901).
|
|
26
|
+
|
|
27
|
+
### Breaking changes
|
|
28
|
+
|
|
29
|
+
- **`shared_heartbeats:` is removed.** Every worker now reserves a dedicated heartbeat connection
|
|
30
|
+
on its pool (ADR 0071). Before, `shared_heartbeats: true` let heartbeat rounds borrow ordinary
|
|
31
|
+
pool connections and let a pool below 3 run. Then a heartbeat could wait behind claims and
|
|
32
|
+
settlements, so a lease could expire and a task could run twice. Now `Worker.new` raises
|
|
33
|
+
`ArgumentError` when it gets `shared_heartbeats:`, and the message names the removal. A worker
|
|
34
|
+
on a pool with a size below 3, or with no reported size, raises `ArgumentError` at construction.
|
|
35
|
+
The worker no longer logs `workhorse.worker.polling_only`, because its pool always has a
|
|
36
|
+
connection for the listener (SM-1901).
|
|
37
|
+
|
|
38
|
+
### Behavior changes
|
|
39
|
+
|
|
40
|
+
These changes can make code that worked on 0.10 act differently:
|
|
41
|
+
|
|
42
|
+
1. An idempotency TTL of exactly 0 still selects the default. The queue now raises
|
|
43
|
+
`ArgumentError` for a TTL that is not 0 and rounds to 0 ms (SM-1857). See "An idempotency
|
|
44
|
+
TTL that rounds to 0 ms is refused".
|
|
45
|
+
2. The TypeScript `Queue`, `Admin`, and `Worker.runOnce()` now check the schema before their first
|
|
46
|
+
statement (SM-1864). This SDK already did, so it does not change.
|
|
47
|
+
3. After a lease lapses while a handler runs, a stale failure or release now settles a due
|
|
48
|
+
deadline or attempt timeout at once. The worker reports `deadline_exceeded` or `timeout`, not
|
|
49
|
+
`lease_lost` (SM-1884). See "A stale failure or release after a lapsed lease settles its
|
|
50
|
+
cause".
|
|
51
|
+
4. Every SDK removes the heartbeat opt-out (SM-1899, SM-1901, SM-1902). In this SDK,
|
|
52
|
+
`Worker.new` raises `ArgumentError` for `shared_heartbeats:`, and for a pool with a size below
|
|
53
|
+
3 or with no reported size. See "Breaking changes".
|
|
54
|
+
|
|
55
|
+
### A step of the wall clock no longer moves the expiry request
|
|
56
|
+
|
|
57
|
+
A worker asks PostgreSQL to end an attempt when its deadline or its attempt timeout passes. Before
|
|
58
|
+
this change, the worker read the wall clock again after each wait on the lease window (SM-1869). If
|
|
59
|
+
the host clock stepped back 10 minutes, the first expiry request came 10 minutes late. If the clock
|
|
60
|
+
stepped forward, the request came early and got `not_due`. Each retry then locked the task row until
|
|
61
|
+
the real due time.
|
|
62
|
+
|
|
63
|
+
Now the worker converts the expiry to a monotonic due time once, when its expiry
|
|
64
|
+
watch starts. After that, it reads only the monotonic clock.
|
|
65
|
+
|
|
66
|
+
### Row retention lag skips a member of an unfinished task group
|
|
67
|
+
|
|
68
|
+
Terminal cleanup keeps each member of a task group that has not finished, because the group needs
|
|
69
|
+
its member outcomes when it finishes. Before this change, queue health counted such a member as row
|
|
70
|
+
retention lag after it passed its windows (SM-1851). An open or sealed group can stay unfinished for
|
|
71
|
+
any length of time. Thus health reported "Retention cleanup is late for task records, finished
|
|
72
|
+
results", and the lag grew without end. The public demo keeps one open and one sealed group on
|
|
73
|
+
purpose, so its Health page showed the report every day.
|
|
74
|
+
|
|
75
|
+
Migration 0088 (`0088-count-only-prunable-rows-as-row-retention-lag.sql`) moves the schema to
|
|
76
|
+
version 87. Now `queue_health_v1` skips a member of an unfinished group when it measures task
|
|
77
|
+
identity and terminal outcome lag. For a group member, lag counts from the latest of window
|
|
78
|
+
expiry, history release, and group finish. The health document keeps its keys, and the schema floor
|
|
79
|
+
does not change.
|
|
80
|
+
Every process of every SDK version gets the new result when the migration commits.
|
|
81
|
+
|
|
82
|
+
### A stale failure or release after a lapsed lease settles its cause
|
|
83
|
+
|
|
84
|
+
A lease can lapse while a handler runs, with no change to the fence. Then `fail_v1` and
|
|
85
|
+
`release_owned_v1` answer `stale`, but `expire_owned_v1` can still settle a due deadline or attempt
|
|
86
|
+
timeout under that fence. Before this change, the worker settled nothing, and lease recovery
|
|
87
|
+
settled the task later (SM-1884).
|
|
88
|
+
|
|
89
|
+
Now the worker calls `expire_owned_v1` at once and settles the due boundary, as the Go and Rust
|
|
90
|
+
workers do:
|
|
91
|
+
|
|
92
|
+
- After a lapsed lease with a due deadline, the handler outcome was `lease_lost`. Now it is
|
|
93
|
+
`deadline_exceeded`, and the task fails at once.
|
|
94
|
+
- After a lapsed lease with a due attempt timeout, the handler outcome was `lease_lost`. Now it is
|
|
95
|
+
`timeout`, and PostgreSQL applies the retry policy at once.
|
|
96
|
+
- After a lapsed lease with no due boundary, the outcome stays `lease_lost`.
|
|
97
|
+
|
|
98
|
+
The handler span records `deadline_exceeded` or `timeout_exceeded`. Before this change, it
|
|
99
|
+
recorded `stale`.
|
|
100
|
+
|
|
101
|
+
If `fail_v1` or `release_owned_v1` answers `cancel_requested` and `acknowledge_cancel_v1` refuses,
|
|
102
|
+
the worker settles nothing. After a failed cancellation acknowledgement, the handler outcome was
|
|
103
|
+
`unknown`. Now it is `lease_lost`. The handler span records `stale`. Before this change, it recorded
|
|
104
|
+
`unknown`. A late heartbeat round can also deliver the cancellation request, after a failure, a
|
|
105
|
+
release, or a refused completion. If the acknowledgement then refuses, the handler outcome was
|
|
106
|
+
`canceled`. Now it is `lease_lost`.
|
|
107
|
+
|
|
108
|
+
### The dashboard bundle uses Mantine 9.7.4
|
|
109
|
+
|
|
110
|
+
The Rack dashboard backend serves the shared dashboard browser bundle. After 0.10.0, a dependency
|
|
111
|
+
update moved the dashboard to Mantine 9.7.4 and rebuilt that bundle (SM-1839). The gem now carries
|
|
112
|
+
the rebuilt `read-surface-1.tar.gz`. Its `bundle.json` records the sha256
|
|
113
|
+
`a7d61ed1d0433e38a8d227cbcc8f98a0bf668f2d3bb1d47391b1882fd69bf795`.
|
|
114
|
+
|
|
115
|
+
The `dashboard/v1` procedures and the read surface version do not change (SM-1858).
|
|
116
|
+
|
|
117
|
+
### A worker no longer keeps the tier entries of a task that the executor refuses
|
|
118
|
+
|
|
119
|
+
The worker records each claimed task in its dispatch state. A fast-tier claim records the task ID
|
|
120
|
+
in `@fast_task_ids`. A full-tier claim records the task ID and a deferred tier read in
|
|
121
|
+
`@tier_reads`. Only the end of a handler run removed these entries.
|
|
122
|
+
|
|
123
|
+
Before this change, the handler executor could refuse a task, for example after its shutdown. The
|
|
124
|
+
task then never ran, and the worker kept its tier entry for the life of the `Worker` object
|
|
125
|
+
(SM-1853). Each refusal kept the entry of at most one task.
|
|
126
|
+
|
|
127
|
+
Now the worker removes the task from both collections when the executor refuses the task.
|
|
128
|
+
|
|
129
|
+
### The dashboard page tells each cache not to store it
|
|
130
|
+
|
|
131
|
+
The dashboard page holds `window.workhorseDashboard`, which contains the `auditActor` of the
|
|
132
|
+
operator. Before this change, the Rack dashboard backend sent the page with only `Content-Type`. A
|
|
133
|
+
shared cache in front of the application could store the page by heuristic. A later visitor could
|
|
134
|
+
then get the stored page and see the actor name of an earlier operator. A browser could also show
|
|
135
|
+
the page from its cache after logout (SM-1870).
|
|
136
|
+
|
|
137
|
+
Now the Rack dashboard backend sends the page with `Cache-Control: no-store`, as the TypeScript host
|
|
138
|
+
does. Each asset keeps `Cache-Control: public, max-age=31536000, immutable`.
|
|
139
|
+
|
|
140
|
+
### An idempotency TTL that rounds to 0 ms is refused
|
|
141
|
+
|
|
142
|
+
The queue rounds `Idempotency#ttl` to the nearest millisecond. Before this change, a positive TTL
|
|
143
|
+
below 0.0005 s, or a small negative TTL such as -0.0004 s, rounded to 0. The queue then kept the key
|
|
144
|
+
for the default 24 hours without a warning (SM-1857).
|
|
145
|
+
|
|
146
|
+
Now the queue raises `ArgumentError` for a TTL that is not 0 and rounds to 0 ms. An exact 0 still
|
|
147
|
+
selects the default. ADR 0115 records the rule for all seven SDKs.
|
|
148
|
+
|
|
149
|
+
### A stalled registration refresh no longer delays a stop
|
|
150
|
+
|
|
151
|
+
A worker refreshes its `worker_registry` row at each registry interval. Before this change, the
|
|
152
|
+
dispatch loop sent that refresh itself, with no time limit. If the statement stalled, for example on
|
|
153
|
+
a lock, `stop` could not start the stop path until the statement returned. Also, the draining
|
|
154
|
+
registration and the deregistration ran on two threads in no order. If the registration committed
|
|
155
|
+
last, the row of the stopped worker came back until the registry prune removed it (SM-1868).
|
|
156
|
+
|
|
157
|
+
Now each registry write runs on its own thread, after the write before it. The dispatch loop starts
|
|
158
|
+
a periodic refresh only when no registry write still runs, and it does not wait for the refresh.
|
|
159
|
+
Thus `run` returns within `shutdown_grace` plus the unwind window, and the deregistration runs
|
|
160
|
+
after every registration write of the run.
|
|
161
|
+
|
|
162
|
+
### Recovery no longer locks a timed-out attempt that waits for its cancellation
|
|
163
|
+
|
|
164
|
+
An operator cancels a running task after its execution timeout passed, and its lease is still live.
|
|
165
|
+
Before migration 0089, each maintenance tick locked that attempt in its timeout scan, but settled
|
|
166
|
+
nothing, because the cancellation wins over the timeout. While the tick ran, the heartbeat of the
|
|
167
|
+
owning worker reported the lease busy. The locked rows also used the recovery limit, so a real
|
|
168
|
+
timeout behind them waited for their leases to expire (SM-1866).
|
|
169
|
+
|
|
170
|
+
Now the full-tier timeout scan skips such an attempt while its lease is live and its deadline is
|
|
171
|
+
ahead, as the fast tier does. If the deadline passed after the timeout, recovery closes the task
|
|
172
|
+
`canceled` at once.
|
|
173
|
+
|
|
174
|
+
### Fast-tier recovery closes a canceled attempt when its deadline passes
|
|
175
|
+
|
|
176
|
+
A fast-tier task has a pending cancellation and a live lease. Its execution timeout passes, and
|
|
177
|
+
later its deadline passes. Before migration 0090, each maintenance tick locked that attempt and
|
|
178
|
+
settled nothing until the lease expired. The cancellation blocks the timeout, and recovery took the
|
|
179
|
+
deadline only when the deadline was not later than the timeout (SM-1881).
|
|
180
|
+
|
|
181
|
+
Now fast-tier recovery closes the task `canceled` when its deadline passes, as the full tier does.
|
|
182
|
+
|
|
183
|
+
### The dashboard Schedules page reads a bounded number of occurrences
|
|
184
|
+
|
|
185
|
+
Before migration 0089, `dashboard_cron_v1` counted every retained occurrence of each listed
|
|
186
|
+
schedule. A schedule that fires each second keeps about 1.2 million occurrences in the default
|
|
187
|
+
retention of 14 days. Each poll of the Schedules page read all of them (SM-1866).
|
|
188
|
+
|
|
189
|
+
Now the function reads at most 1,001 occurrences for each listed schedule. `occurrenceCount` stops
|
|
190
|
+
at 1,000, and the new optional key `occurrenceCountCapped` is `true` when more occurrences are
|
|
191
|
+
retained. The bundled Schedules page then shows `1,000+`. `lastFiredAt` is the fire time of the
|
|
192
|
+
newest occurrence. The `dashboard/v1` change is additive. A database below schema 88 returns an
|
|
193
|
+
exact count and no `occurrenceCountCapped`. The page change rebuilds the shared dashboard bundle
|
|
194
|
+
again. The sha256 in "The dashboard bundle uses Mantine 9.7.4" is the sha256 of that bundle.
|
|
195
|
+
|
|
196
|
+
### The worker registry prune finds expired kept pauses through their index
|
|
197
|
+
|
|
198
|
+
Before migration 0089, `prune_worker_registry_v1` deleted expired kept pauses with a condition that
|
|
199
|
+
the planner could not match to `worker_pause_tombstone_expires_idx`. Each maintenance pass read the
|
|
200
|
+
whole table. Now the delete reads only the expired rows through the index (SM-1866).
|
|
201
|
+
|
|
202
|
+
### A child set and a task group check every request before their first statement
|
|
203
|
+
|
|
204
|
+
Before this change, `HandlerContext#run_children` and `#run_children_all` checked the options of
|
|
205
|
+
each child while they serialized it. Thus the first child could read its payload contract before
|
|
206
|
+
the SDK refused the second child. `Queue#create_task_group` and `Queue#task_group` checked the
|
|
207
|
+
callbacks after the compatibility query and the member contract reads. A member add through
|
|
208
|
+
`HandlerContext#task_group` read the membership of its task first (SM-1883).
|
|
209
|
+
|
|
210
|
+
Now these calls check the options of every request before their first statement, also before the
|
|
211
|
+
deferred tier read of a child set. The errors do not change. Only their time changes.
|
|
212
|
+
|
|
213
|
+
## 0.10.0 — 2026-10-26
|
|
214
|
+
|
|
215
|
+
The npm packages, Python distribution, Go module, Rust crate, Ruby gem, JVM artifacts, and .NET
|
|
216
|
+
packages release from one source commit.
|
|
217
|
+
|
|
218
|
+
Requires **schema v76**, Ruby **3.3** or newer, and PostgreSQL **15** or newer.
|
|
219
|
+
|
|
220
|
+
### Upgrade steps
|
|
221
|
+
|
|
222
|
+
**Migrate the schema before starting updated processes.** Run `workhorse schema migrate` before any
|
|
223
|
+
process of this release starts, producers too. Migrations 0075 through 0087 move the schema to
|
|
224
|
+
version **86**. The SDK compatibility floor rises from 73 to **76**, because every SDK statement
|
|
225
|
+
catalogue now names `dashboard_retention_preview_v1` and `deregister_worker_v2`. An updated worker
|
|
226
|
+
refuses to start on a schema below version 76. A 0.9 process keeps working on version 86, because a
|
|
227
|
+
migration inside a major line only adds.
|
|
228
|
+
|
|
229
|
+
### A failed reporter callback gives a catalogued log record
|
|
230
|
+
|
|
231
|
+
A raise from `on_registration_error` or `on_notification_error` gave a log line with the error
|
|
232
|
+
message and no event name (SM-1726). Now the line is the `workhorse.worker.reporter_failed` event
|
|
233
|
+
with `workhorse.worker.id`, `workhorse.worker.reporter` (`registration_error` or
|
|
234
|
+
`notification_error`), and `error.type`. It no longer carries the error message. The logger level,
|
|
235
|
+
the standard error fallback, and the bound do not change.
|
|
236
|
+
|
|
237
|
+
### The worker refuses a `poll_interval` above 2,147,483.647 seconds
|
|
238
|
+
|
|
239
|
+
Before this change, `poll_interval:` had no upper limit (SM-1778). Now `Worker.new` raises
|
|
240
|
+
`ArgumentError` for a value above 2,147,483.647 seconds, the limit of the registry column. Every
|
|
241
|
+
SDK accepts an explicit poll interval from 1 through 2,147,483,647 ms (ADR 0112). Zero stays
|
|
242
|
+
refused, and `nil` keeps the default.
|
|
243
|
+
|
|
244
|
+
### The Rack dashboard refuses a cross-origin read
|
|
245
|
+
|
|
246
|
+
A Rack CORS middleware can add CORS headers to `Stablemates::Workhorse::Dashboard` responses. If
|
|
247
|
+
that policy allowed credentials, a page on another origin could read the dashboard, task payloads
|
|
248
|
+
included, with the operator's cookies (SM-1675). Now the application answers 403 with
|
|
249
|
+
`{"error":"A same-origin request is required"}` for a page, an asset, or a read procedure whose
|
|
250
|
+
`Origin` header names another origin.
|
|
251
|
+
|
|
252
|
+
The same-origin check compared the `Host` text exactly, so `Host: example.test:443` with
|
|
253
|
+
`Origin: https://example.test` failed. Now letter case and the default port do not matter, as in
|
|
254
|
+
the allowed hosts check. An `Origin` with a path, a query, a fragment, or a user fails.
|
|
255
|
+
|
|
256
|
+
A request without `Origin`, such as a top-level navigation, and a same-origin request still
|
|
257
|
+
work. A mutation keeps its own refusal. The change needs no migration.
|
|
258
|
+
|
|
259
|
+
Upgrade step: if a proxy in front of the host ends TLS, the host must see the scheme that the
|
|
260
|
+
browser used. Otherwise the dashboard page's own module scripts carry an `Origin` that does not
|
|
261
|
+
match, and the host refuses them.
|
|
262
|
+
|
|
263
|
+
### The dashboard Schedules page shows the last skip of a schedule
|
|
264
|
+
|
|
265
|
+
Migration 0087 (`0087-show-the-last-schedule-skip-on-the-dashboard.sql`) moves the schema to version 86. The `dashboard/v1` cron document gains `lastSkippedAt`, `lastSkippedCount`, and
|
|
266
|
+
`lastSkippedThrough` on each application schedule, an additive change. The bundled Schedules page
|
|
267
|
+
shows a `Skipped` badge with the count and the time of the last skip of a `skip` schedule (SM-1708).
|
|
268
|
+
The migration does not raise the SDK compatibility floor.
|
|
269
|
+
|
|
270
|
+
### The worker counts each failed heartbeat round
|
|
271
|
+
|
|
272
|
+
A heartbeat round can fail before PostgreSQL answers, for example when the connection breaks or the
|
|
273
|
+
round reaches its deadline. Before this change, such a round left no metric (SM-1704).
|
|
274
|
+
|
|
275
|
+
Now the worker counts each failed round once in the new counter
|
|
276
|
+
`workhorse.worker.heartbeat.round_failure`, with the unit `{round}`. The bounded attribute
|
|
277
|
+
`workhorse.heartbeat.failure_reason` gives the reason: `timeout`, `connection`, `database`, or
|
|
278
|
+
`other`. This applies to the reserved heartbeat connection and to `shared_heartbeats: true`. A
|
|
279
|
+
failed round still gives no ownership verdict, so `workhorse.worker.heartbeat.failure` does not
|
|
280
|
+
change. The worker adds no log record. The new instrument is an addition.
|
|
281
|
+
|
|
282
|
+
### A canceled task leaves its batch while it waits
|
|
283
|
+
|
|
284
|
+
A batch handler waits up to its linger for a partial batch. Before this change, a waiting task
|
|
285
|
+
stayed in the batch after a cancel request or a lease loss cancelled its cancellation token. The
|
|
286
|
+
batch handler then got a task that the worker no longer owned, and the task settled only after the
|
|
287
|
+
linger (SM-1709).
|
|
288
|
+
|
|
289
|
+
Now the worker removes such a task from the batch at once. The task raises the `CancelledError` of
|
|
290
|
+
the token reason, and the batch handler never gets it. The other waiting tasks keep their order and
|
|
291
|
+
dispatch. The TypeScript, Go, and Rust workers already remove such a task.
|
|
292
|
+
|
|
293
|
+
This includes the `:shutdown` cancellation at the end of the shutdown grace. Before, a task that
|
|
294
|
+
still waited then dispatched, and the batch handler ran its payload after the deadline. Now the task
|
|
295
|
+
goes back to its queue without a charged attempt. Before the deadline, the drain still dispatches a
|
|
296
|
+
lingering batch when all its tasks have arrived.
|
|
297
|
+
|
|
298
|
+
### The worker waits longer after each not_due expiry answer
|
|
299
|
+
|
|
300
|
+
When the local timer of a deadline or an execution timeout fires, the worker asks PostgreSQL to
|
|
301
|
+
expire the attempt. If the clock of PostgreSQL has not reached its expiry, PostgreSQL answers
|
|
302
|
+
`not_due`. Before this change, the worker asked again every 5 ms. Each request locks the task row,
|
|
303
|
+
so the heartbeat of the same attempt answered `busy`. If the worker clock ran ahead by more than one
|
|
304
|
+
lease, the attempt could lose its lease before the database expiry (SM-1701).
|
|
305
|
+
|
|
306
|
+
Now the first retry waits 5 ms, and each later retry waits twice as long, to a maximum of 1 s. The
|
|
307
|
+
TypeScript, Go, and Rust workers already use this schedule.
|
|
308
|
+
|
|
309
|
+
### The Ruby client measures signal and human wait values as PostgreSQL does
|
|
310
|
+
|
|
311
|
+
PostgreSQL limits a signal payload, a human wait context, and a human wait result to 65,536 bytes of
|
|
312
|
+
`octet_length(value::jsonb::text)`. That text has a space after each `:` and `,`, so it can be
|
|
313
|
+
longer than compact JSON. Before this change, the Ruby client counted the compact JSON. A value near
|
|
314
|
+
the limit passed that check, and then PostgreSQL refused the statement with a database error
|
|
315
|
+
(SM-1639).
|
|
316
|
+
|
|
317
|
+
Now `Queue#send_signal`, `Queue#complete_human_wait`, and `HandlerContext#wait_for_human` measure
|
|
318
|
+
the canonical JSONB text before any statement. An oversized value raises `ArgumentError`.
|
|
319
|
+
`HandlerContext#wait_for_human` also checks the context before its tier lookup, which can read
|
|
320
|
+
PostgreSQL. The limit does not change.
|
|
321
|
+
|
|
322
|
+
### The worker warns when a maintenance pass or a claim waits longer than the stop budget
|
|
323
|
+
|
|
324
|
+
A maintenance pass or a claim can wait for a lock that another session holds. Before this change,
|
|
325
|
+
the worker logged nothing while it waited. A test run blocked for about one hour, and no log named
|
|
326
|
+
the statement (SM-1785).
|
|
327
|
+
|
|
328
|
+
Now `Worker` logs `workhorse.worker.statement_slow` once, at warn level, if a maintenance pass or a
|
|
329
|
+
claim waits longer than the stop budget. The stop budget is `shutdown_grace`, 25 s by default. A
|
|
330
|
+
grace of 0 turns the warning off. The record carries `workhorse.worker.id`,
|
|
331
|
+
`workhorse.worker.statement` (`maintenance` or `claim`), and `workhorse.worker.elapsed_ms`.
|
|
332
|
+
`workhorse.worker.elapsed_ms` is the wait at the moment of the warning. The statement continues to
|
|
333
|
+
wait, and no other behavior changes.
|
|
334
|
+
|
|
335
|
+
### execution_timeout refuses a nonzero value that rounds to 0 ms
|
|
336
|
+
|
|
337
|
+
The queue rounds `execution_timeout:` to the nearest millisecond. Before this change, a positive
|
|
338
|
+
value below 0.0005 seconds rounded to 0, and the task had no timeout. A small negative value, such
|
|
339
|
+
as -0.0004, also set no timeout (SM-1714).
|
|
340
|
+
|
|
341
|
+
Now the queue raises `ArgumentError` for a nonzero value that rounds to 0 ms or is negative. `nil`
|
|
342
|
+
and an exact 0 still set no timeout. ADR 0114 records the rule for all seven SDKs.
|
|
343
|
+
|
|
344
|
+
### The worker intervals refuse a value above the SQL integer limit
|
|
345
|
+
|
|
346
|
+
`register_worker_v1` takes `maintenance_interval`, `maintenance_routine_interval`, and
|
|
347
|
+
`registry_interval` as PostgreSQL `integer` milliseconds. Before this change, `Worker.new` accepted
|
|
348
|
+
a larger value, and then each registration failed (SM-1668).
|
|
349
|
+
|
|
350
|
+
Now `Worker.new` raises `ArgumentError` if one of these intervals is not from 0.1 through
|
|
351
|
+
2,147,483.647 seconds. `disable_registry: true` still turns off registration. This is a behavior
|
|
352
|
+
change for a caller who passes a larger value. ADR 0112 records the range for all seven SDKs.
|
|
353
|
+
|
|
354
|
+
### An anonymous error class records the name of its superclass
|
|
355
|
+
|
|
356
|
+
Ruby gives an anonymous class a nil name. Before this change, a handler that raised an instance of
|
|
357
|
+
`Class.new(StandardError)` recorded JSON null as the failure `name` (SM-1668).
|
|
358
|
+
|
|
359
|
+
Now the failure envelope records the name of the nearest named superclass, `StandardError` in that
|
|
360
|
+
example. Logs and telemetry use the same name for `error.type`.
|
|
361
|
+
|
|
362
|
+
### retry_delay refuses a small negative value and rounds exact values exactly
|
|
363
|
+
|
|
364
|
+
`fail_v1` takes the retry delay as a PostgreSQL `integer`. Before this change, the worker rounded a
|
|
365
|
+
`retry_delay` to milliseconds first. Thus a negative value such as `-0.0004` became 0 and passed.
|
|
366
|
+
The worker also converted the value to a Float first, so `Rational(1001, 2000)`, which is exactly
|
|
367
|
+
500.5 ms, became 500 ms (SM-1668).
|
|
368
|
+
|
|
369
|
+
Now the worker raises `ArgumentError` for every negative `retry_delay`, at construction for a fixed
|
|
370
|
+
value and before `fail_v1` for a callable result. It converts Integer, Rational, and BigDecimal
|
|
371
|
+
seconds exactly and rounds half up, so that value becomes 501 ms. ADR 0111 records the rule for all
|
|
372
|
+
seven SDKs.
|
|
373
|
+
|
|
374
|
+
### A raising reporter callback no longer stops the worker
|
|
375
|
+
|
|
376
|
+
An application can give `on_registration_error` to the worker to send an alert. Before this change,
|
|
377
|
+
a registration refresh could fail, and then the callback could raise. The error left the registry
|
|
378
|
+
path and could stop the run. The worker also discarded a raise from `on_notification_error` without
|
|
379
|
+
a record (SM-1696).
|
|
380
|
+
|
|
381
|
+
Now the worker rescues a `StandardError` from either callback and continues. It writes the failure
|
|
382
|
+
to `logger` at the error level. Without a logger, it writes the failure to standard error. For each
|
|
383
|
+
callback, it writes the first failure. Then it writes at most one failure in each 60 s, with the
|
|
384
|
+
count of the failures that it did not write. It never calls the failed callback for its own failure.
|
|
385
|
+
|
|
386
|
+
### Workers use the database clock for maintenance
|
|
387
|
+
|
|
388
|
+
Maintenance writes shared state: the due time of each routine and the retention cutoffs. Before
|
|
389
|
+
this change, the Ruby worker sent its host clock as the maintenance time. If that clock was two days
|
|
390
|
+
fast, other workers skipped a routine until that future time. Retention also deleted a 13-day-old
|
|
391
|
+
schedule occurrence under a 14-day policy (SM-1645).
|
|
392
|
+
|
|
393
|
+
Now `Worker#run` and `Worker#run_once` bind the text `now`, and PostgreSQL resolves it to its own
|
|
394
|
+
time at the start of the call.
|
|
395
|
+
|
|
396
|
+
The schema does not change.
|
|
397
|
+
|
|
398
|
+
### The dashboard names an action that the host does not provide
|
|
399
|
+
|
|
400
|
+
A dashboard host answers the `enqueueTest` and `setSchedulePaused` procedures only when the
|
|
401
|
+
application gives a handler for them. Before this change, a writable host without that handler
|
|
402
|
+
answered `FORBIDDEN` with the message "This procedure is not available". Now the message is "This
|
|
403
|
+
dashboard host does not provide the <action> action", as in the TypeScript dashboard server. A
|
|
404
|
+
read-only host keeps the message "This dashboard is read-only". The code `FORBIDDEN` and the HTTP
|
|
405
|
+
status 403 do not change (SM-1705).
|
|
406
|
+
|
|
407
|
+
### A lost notification subscription restarts the poll backoff
|
|
408
|
+
|
|
409
|
+
A worker without a live subscription waits 250 ms after an empty claim, and doubles that wait
|
|
410
|
+
toward 5,000 ms. Before this change, the empty claims of a live period counted toward that backoff.
|
|
411
|
+
After a loss, a worker that was idle for some time therefore waited the 5,000 ms ceiling at once,
|
|
412
|
+
and a new task waited up to 5 s (SM-1648).
|
|
413
|
+
|
|
414
|
+
Now the backoff restarts at 250 ms when the subscription is lost. The `subscription-recovery` case
|
|
415
|
+
in `protocol/v1/runtime.json` checks this rule in every SDK.
|
|
416
|
+
|
|
417
|
+
### A polling-only worker keeps its listener connection for claims
|
|
418
|
+
|
|
419
|
+
A worker without a `cohorts` option caps its default cohort count by the connections that its pool
|
|
420
|
+
can lend. Before this change, a worker with `polling_only: true` also subtracted a connection for a
|
|
421
|
+
listener that it never opens. With a pool of 4 and concurrency 64, it used 2 cohorts instead of 3
|
|
422
|
+
(SM-1668).
|
|
423
|
+
|
|
424
|
+
Now a polling-only worker reserves no listener connection. A worker that listens still reserves one,
|
|
425
|
+
and a worker with dedicated heartbeats still reserves one more.
|
|
426
|
+
|
|
427
|
+
### A handler error with text that jsonb cannot store settles by its retry policy
|
|
428
|
+
|
|
429
|
+
PostgreSQL `jsonb` cannot store a NUL character or an unpaired surrogate. Before this change, a
|
|
430
|
+
handler error with such text made `fail_v1` refuse the failure envelope. The worker stopped, and the
|
|
431
|
+
task stayed active until lease recovery (SM-1650).
|
|
432
|
+
|
|
433
|
+
Now `error_envelope` replaces each NUL character and each byte that is not UTF-8 with U+FFFD. It
|
|
434
|
+
does this in the name, the message, and the stack. The JVM and .NET workers use the same rule. The
|
|
435
|
+
retry policy then settles the task. All other text does not change.
|
|
436
|
+
|
|
437
|
+
The `protocol/v1/failures.json` fixtures pin the rule for every SDK.
|
|
438
|
+
|
|
439
|
+
### A refused completion settles a due deadline or attempt timeout
|
|
440
|
+
|
|
441
|
+
A handler can return just before its deadline or attempt timeout, and the completion write can then
|
|
442
|
+
arrive after that boundary. PostgreSQL refuses the completion. Before this change, the worker only
|
|
443
|
+
tried to acknowledge a cancellation, then recorded `lease_expired` and left the task to lease
|
|
444
|
+
recovery. The task then settled up to one lease later with the wrong outcome. Now
|
|
445
|
+
`reconcile_rejected_completion` calls `expire_owned_telemetry_v1` under the attempt's fence before
|
|
446
|
+
the completion path returns. The attempt then ends with the outcome of the due boundary. A
|
|
447
|
+
cancellation request and a lost lease keep their own outcomes (SM-1649).
|
|
448
|
+
|
|
449
|
+
### Workers on one pool renew their leases in one transaction
|
|
450
|
+
|
|
451
|
+
On the dedicated heartbeat connection, a round sent one transaction for each worker and each 100
|
|
452
|
+
leases. Each transaction needs at least 3 round trips to PostgreSQL. With 8 workers on one pool and
|
|
453
|
+
80 ms for each round trip, workers lost leases while PostgreSQL still answered (SM-1631). Now the heartbeat thread of the pool sends one transaction for each round, with one `heartbeat_pool_v1` statement for every worker and every 100 leases. The
|
|
454
|
+
transaction locks the task rows of every worker in it until its `COMMIT`. A round renews all of its
|
|
455
|
+
leases or none. The settings statement now sets `statement_timeout` and
|
|
456
|
+
`idle_in_transaction_session_timeout` each to one quarter of the time that remains, not to one half
|
|
457
|
+
and one eighth. The statement needs no schema change. The `shared_heartbeats` opt-out does not change (SM-1633).
|
|
458
|
+
|
|
459
|
+
### The dashboard sets the tier and the history of a queue
|
|
460
|
+
|
|
461
|
+
The Rack dashboard answers two new mutations, `setQueueTier` and `setQueueHistory`. `setQueueTier`
|
|
462
|
+
calls `Admin#set_queue_tier` with the authenticated actor and answers `{status: "set", tier}`. If
|
|
463
|
+
Workhorse refuses the switch, the dashboard answers the guard as the status: `live_tasks`,
|
|
464
|
+
`concurrency_policy`, or `rate_limit_policy`. Then `tier` is `null`, and the queue keeps its tier.
|
|
465
|
+
`setQueueHistory` calls `Admin#set_queue_history`. An absent setting keeps its current value. A
|
|
466
|
+
read-only dashboard refuses both mutations (SM-966).
|
|
467
|
+
|
|
468
|
+
### `Admin` moves a queue between tiers and sets its history
|
|
469
|
+
|
|
470
|
+
`Admin#set_queue_tier(queue, tier, audit:)` moves a queue to the `:fast` or `:full` tier and returns
|
|
471
|
+
the tier after the change. If the tier differs and the queue holds a live task, the method raises
|
|
472
|
+
`FastTierUnsupportedError`.
|
|
473
|
+
`Admin#set_queue_history(queue, record_attempts: nil, record_claims: nil)` chooses the history a
|
|
474
|
+
fast-tier queue records. A `nil` setting keeps its current value. The method returns a
|
|
475
|
+
`QueueHistory`. The other SDKs already have these controls (SM-1417).
|
|
476
|
+
|
|
477
|
+
### A worker on a pool of 1 polls under the opt-out
|
|
478
|
+
|
|
479
|
+
Before this change, a worker with `shared_heartbeats: true` started the shared listener on any pool.
|
|
480
|
+
On a pool of 1, the listener held the only connection. Registration, claims, and handler statements
|
|
481
|
+
then waited for the pool timeout, and no task ran (SM-1585).
|
|
482
|
+
|
|
483
|
+
Now, with `shared_heartbeats: true`, the worker starts the listener only when the pool `size` is an
|
|
484
|
+
Integer of at least 2. On a pool of 1, or on a pool without a `size`, the worker polls. Each run
|
|
485
|
+
then logs `workhorse.worker.polling_only` once at warn level. The message names
|
|
486
|
+
`workhorse.worker.queues` and `workhorse.worker.pool_max_size`. A worker with `polling_only: true`
|
|
487
|
+
logs no warning. Without the opt-out, nothing changes.
|
|
488
|
+
|
|
489
|
+
### A contract compares a stored schema number with all its digits
|
|
490
|
+
|
|
491
|
+
Before this change, the Ruby SDK read a stored contract schema with `JSON.parse`, which reads a
|
|
492
|
+
fraction as a Float. Another SDK can store `"minimum": 0.30000000000000000001`, which a Float cannot
|
|
493
|
+
hold. The Ruby SDK then accepted `0.3`. Now a fraction that a Float cannot hold stays a Rational,
|
|
494
|
+
and the contract keywords compare it exactly (SM-1541, ADR 0039). `ContractSchema.new` still
|
|
495
|
+
accepts only JSON values.
|
|
496
|
+
|
|
497
|
+
### A worker reserves its heartbeat connection at startup, and refuses to start without it
|
|
498
|
+
|
|
499
|
+
Before this change, a worker took the heartbeat connection of its pool only when a task arrived,
|
|
500
|
+
and returned it when no task remained. If the pool had no free connection then, the worker still
|
|
501
|
+
claimed tasks. The heartbeat waited for a connection, and the leases of those tasks could expire
|
|
502
|
+
(SM-1577). These changes apply to a worker without `shared_heartbeats: true`:
|
|
503
|
+
|
|
504
|
+
- **A startup reservation failure now ends the run.** This changes behavior. `run` and `run_once`
|
|
505
|
+
first reserve the heartbeat connection, then borrow one other connection for the compatibility
|
|
506
|
+
query and return it at once. Both steps must end within the larger of `heartbeat` and 5 s. This
|
|
507
|
+
bound includes a connect and the query itself, for every kind of pool. If a step does not end in
|
|
508
|
+
time, the run raises the new `Stablemates::Workhorse::WorkerStartupError` before the worker
|
|
509
|
+
registers. Its message names the pool, the budget, and the `shared_heartbeats: true` opt-out. A
|
|
510
|
+
compatibility query that fails keeps its own error type. Before this change, the same worker started and claimed tasks. A
|
|
511
|
+
supervisor that restarts a failed process, such as `run_worker_processes`, starts the worker
|
|
512
|
+
again.
|
|
513
|
+
- **The reservation lasts while a worker runs.** The heartbeat keeps its connection while any worker
|
|
514
|
+
on the pool runs, also when no task runs. The last worker that stops returns it. Thus a pool keeps
|
|
515
|
+
one heartbeat connection for its running workers, apart from the short replacement after a lost
|
|
516
|
+
connection.
|
|
517
|
+
- **No claim without a heartbeat connection.** While the pool has no usable heartbeat connection,
|
|
518
|
+
the worker claims no task. A claim checks this after it has its connection, immediately before
|
|
519
|
+
each attempt, and only for its own run. A fast-tier completion then settles its task with a claim
|
|
520
|
+
limit of 0. A rejected or busy lease does not stop claims. The heartbeat checks its connection
|
|
521
|
+
between rounds, also when no task runs. If the connection is lost, or a round fails, the heartbeat
|
|
522
|
+
asks for a new one at once. It waits for a checkout at most one round bound, and at most one
|
|
523
|
+
heartbeat interval between failed attempts. When the new connection is ready, the worker claims at
|
|
524
|
+
once.
|
|
525
|
+
- **One shutdown deadline.** `shutdown_grace` now starts when dispatch ends. Every wait of the stop
|
|
526
|
+
path ends by that deadline or one unwind window later. This includes the registration statements
|
|
527
|
+
and the release of the listener and the heartbeat connection. Thus a stop takes at most
|
|
528
|
+
`shutdown_grace` plus the unwind window. A claim that returns after the deadline starts no task,
|
|
529
|
+
and its leases expire. The heartbeat connection stays reserved until that claim statement
|
|
530
|
+
settles. A stop during the startup reservation ends the run without an error.
|
|
531
|
+
- **A new run waits for the stop statements of the previous run.** The next `run` or `run_once` of
|
|
532
|
+
the same worker waits for them within its startup budget. Then a late deregistration cannot
|
|
533
|
+
delete the registry row of the new run. If a statement still runs when the budget ends, the run
|
|
534
|
+
raises `WorkerStartupError`. A stop during this wait ends the run without an error.
|
|
535
|
+
|
|
536
|
+
### Throttle works on a fast-tier queue
|
|
537
|
+
|
|
538
|
+
Before this change, a throttled enqueue into a fast-tier queue raised the fast-tier error with the feature
|
|
539
|
+
`throttle`. Throttle only decides if Workhorse accepts an enqueue request, so it needs no state of
|
|
540
|
+
the full tier (SM-928, ADR 0077). Migration 0075
|
|
541
|
+
(`0075-allow-throttle-on-fast-tier-queues.sql`) moves the schema to version 74. After it,
|
|
542
|
+
`enqueue_throttle_v1` and `enqueue_many_v1` accept throttle on a fast-tier queue. The outcomes are
|
|
543
|
+
`accepted` and `coalesced`, as on the full tier.
|
|
544
|
+
|
|
545
|
+
A coalesced request on a fast-tier task writes no `throttled` event, because a fast-tier task
|
|
546
|
+
records only the history that its queue enables. The SDK still emits the throttle log and the
|
|
547
|
+
enqueue outcome metric. In a batch with a debounce or a throttle member, a fast-tier rejection of
|
|
548
|
+
one member now reports the position of that member in the batch. Debounce on a fast-tier queue is
|
|
549
|
+
still rejected.
|
|
550
|
+
|
|
551
|
+
The migration does not raise the SDK compatibility floor. No SDK statement changes. The migration changes no stored data.
|
|
552
|
+
|
|
553
|
+
### The retention preview reads through a dashboard function
|
|
554
|
+
|
|
555
|
+
Before this change, each dashboard host read the retention policy, sent a shared statement that
|
|
556
|
+
counted rows directly in the tables, and capped the counts in its own code. ADR 0040 asks for one
|
|
557
|
+
SQL implementation for each dashboard procedure (SM-1478). Migration 0076
|
|
558
|
+
(`0076-add-the-dashboard-retention-preview-function.sql`) moves the schema to version 75 and adds
|
|
559
|
+
`dashboard_retention_preview_v1`. The Ruby dashboard host now calls it. The `previewRetentionPolicy` response does
|
|
560
|
+
not change.
|
|
561
|
+
|
|
562
|
+
Every SDK statement catalogue names the new function, so the SDK compatibility floor rises from
|
|
563
|
+
schema 73 to 75. Run `workhorse schema migrate` before the first process of this release starts.
|
|
564
|
+
A 0.9 process keeps working on schema 75. The migration changes no stored data.
|
|
565
|
+
|
|
566
|
+
### A late worker deregistration cannot delete the row of a later run
|
|
567
|
+
|
|
568
|
+
Before this change, a worker that stopped sent `deregister_worker_v1`, which deletes the registry
|
|
569
|
+
row by worker ID only. If the stop path was slow and the same worker ID started again, the late
|
|
570
|
+
deregistration could delete the row of the new run and its operator pause state. Each SDK guarded
|
|
571
|
+
against this in its own code (SM-1589). Migration
|
|
572
|
+
0077 (`0077-fence-the-worker-deregistration-by-instance.sql`) moves the schema to version 76 and
|
|
573
|
+
adds `deregister_worker_v2`. It deletes the row only while the row names the instance of the run
|
|
574
|
+
that stops. `deregister_worker_v1` does not change.
|
|
575
|
+
|
|
576
|
+
`Stablemates::Workhorse::Worker` now calls `deregister_worker_v2` with the instance of the run that
|
|
577
|
+
stops.
|
|
578
|
+
|
|
579
|
+
Each SDK keeps its guard for this release: a new run still waits for the stop statements of the
|
|
580
|
+
previous run, or it refuses to start. Every SDK statement catalogue names the new function, so the
|
|
581
|
+
SDK compatibility floor rises from schema 75 to 76. Run `workhorse schema migrate` before the first
|
|
582
|
+
process of this release starts. A 0.9 process keeps working on schema 76. The migration changes no
|
|
583
|
+
stored data.
|
|
584
|
+
|
|
6
585
|
## 0.9.0 — 2026-10-20
|
|
7
586
|
|
|
8
587
|
The npm packages, Python distribution, Go module, Rust crate, Ruby gem, and JVM artifacts release from one source commit.
|
data/README.md
CHANGED
|
@@ -30,7 +30,7 @@ bundle add connection_pool
|
|
|
30
30
|
Install the schema once, as a deployment step. The application never installs or migrates it.
|
|
31
31
|
|
|
32
32
|
```bash
|
|
33
|
-
npx --package @stablemates/workhorse@0.
|
|
33
|
+
npx --package @stablemates/workhorse@0.11.0 workhorse schema install
|
|
34
34
|
```
|
|
35
35
|
|
|
36
36
|
The machine that runs that deployment step needs Node.js 22 or newer. The application itself needs
|
|
@@ -108,7 +108,8 @@ The gem is `stablemates-workhorse`, and its namespace is `Stablemates::Workhorse
|
|
|
108
108
|
that was `:non_replaceable` also carries a `reason`: `:incompatible_key_mode`, `:not_pending`, or
|
|
109
109
|
`:window_elapsed_pending`.
|
|
110
110
|
- `Worker` takes a pool, registers handlers by task type, and runs them under a lease with a shared
|
|
111
|
-
heartbeat connection.
|
|
111
|
+
heartbeat connection. The pool must report a maximum size of at least 3 connections, or the
|
|
112
|
+
constructor raises `ArgumentError`. `run_worker_process` drains the worker on `TERM` or `INT`.
|
|
112
113
|
- `HandlerContext` offers checkpoints, durable sleeps, signal and human waits, child tasks, and
|
|
113
114
|
progress. PostgreSQL owns every durable decision, and an unresolved wait suspends the task.
|
|
114
115
|
- `Admin` lists, inspects, and repairs tasks, dead letters, waits, workers, and queues.
|