stablemates-workhorse 0.10.0 → 0.12.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +245 -0
- data/README.md +3 -2
- data/lib/stablemates/workhorse/context.rb +6 -0
- data/lib/stablemates/workhorse/dashboard/bundle.json +1 -1
- data/lib/stablemates/workhorse/dashboard/read-surface-1.tar.gz +0 -0
- data/lib/stablemates/workhorse/dashboard.rb +1 -1
- data/lib/stablemates/workhorse/errors.rb +3 -3
- data/lib/stablemates/workhorse/queue.rb +63 -29
- data/lib/stablemates/workhorse/sql_catalogue_generated.rb +4 -1
- data/lib/stablemates/workhorse/telemetry.rb +5 -5
- data/lib/stablemates/workhorse/version.rb +1 -1
- data/lib/stablemates/workhorse/worker/heartbeat.rb +14 -78
- data/lib/stablemates/workhorse/worker.rb +216 -107
- metadata +1 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: f684dcd8a1e590688046e5b175ba97db4c4b62320d78c12353c5bf2fdf294ca4
|
|
4
|
+
data.tar.gz: 748bf2d0d2cd17eb8717c1d5066f7add52c5c60895ffb5975038dac3803041ec
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 7c298a44ca80299142012bf23d118876cb036c207974625dd02bd49b262cad60eefaef0ea455d22153c4a052c69dbdbcd8788d3c896bac0c8c9cdd237751239b
|
|
7
|
+
data.tar.gz: fffb3cbaa9f41f1f0efb59a6eff7f7fc4da4a840cf35b53def9a044ad3321cacbc380f0ddae14bf8209a789b61e8e16cfcddaccc106e723f2cd02171fa4b13d3
|
data/CHANGELOG.md
CHANGED
|
@@ -3,6 +3,251 @@
|
|
|
3
3
|
`stablemates-workhorse` gem versions and release notes live here. The gem carries the version the
|
|
4
4
|
other SDKs carry, because every tag names one release of all of them.
|
|
5
5
|
|
|
6
|
+
## 0.12.0 — 2026-11-07
|
|
7
|
+
|
|
8
|
+
The npm packages, Python distribution, Go module, Rust crate, Ruby gem, JVM artifacts, and .NET
|
|
9
|
+
packages release from one source commit.
|
|
10
|
+
|
|
11
|
+
Requires **schema v76**, Ruby **3.3** or newer, and PostgreSQL **15** or newer.
|
|
12
|
+
|
|
13
|
+
### Upgrade steps
|
|
14
|
+
|
|
15
|
+
**Migrate the schema before starting updated processes.** Run `workhorse schema migrate` before any
|
|
16
|
+
process of this release starts, producers too. Migrations 0091 and 0092 move the schema from
|
|
17
|
+
version 89 to version **91**. They replace function bodies and change no stored rows. The SDK
|
|
18
|
+
compatibility floor stays at **76**. A 0.11 process keeps working on version 91, because a
|
|
19
|
+
migration inside a major line only adds.
|
|
20
|
+
|
|
21
|
+
### A full-tier queue no longer makes PostgreSQL log an error at each tier probe
|
|
22
|
+
|
|
23
|
+
Before a tier probe, the worker now reads the tier of the queue from `queue_control` with one
|
|
24
|
+
`SELECT`. It sends the fast claim only to a fast-tier queue. Before, PostgreSQL rejected the probe
|
|
25
|
+
of each full-tier queue every 30 seconds and wrote that error to its log. A tier change still
|
|
26
|
+
reaches a running worker within 30 seconds. The change needs no migration and does not change the
|
|
27
|
+
SDK compatibility floor.
|
|
28
|
+
|
|
29
|
+
### The dashboard task detail shows the task group of a member task
|
|
30
|
+
|
|
31
|
+
Migration 0091 (`0091-show-the-task-group-in-the-task-detail.sql`) moves the schema to version 90.
|
|
32
|
+
The `dashboard/v1` task detail document gains `group`, with the group ID, the member position, and
|
|
33
|
+
the group state of a member task, an additive change. The migration does not raise the SDK
|
|
34
|
+
compatibility floor.
|
|
35
|
+
|
|
36
|
+
### The dashboard task estimate follows the current size of the task table
|
|
37
|
+
|
|
38
|
+
Migration 0092 (`0092-scale-the-dashboard-task-estimate-by-the-table-size.sql`) moves the schema to
|
|
39
|
+
version 91. `dashboard_task_estimate_v1` now scales the stored rows per page of the task table by
|
|
40
|
+
its current number of pages. After a bulk enqueue, the dashboard sidebar counts and the Queues page
|
|
41
|
+
use their estimates before autoanalyze runs. The migration does not raise the SDK compatibility
|
|
42
|
+
floor.
|
|
43
|
+
|
|
44
|
+
## 0.11.0 — 2026-10-28
|
|
45
|
+
|
|
46
|
+
The npm packages, Python distribution, Go module, Rust crate, Ruby gem, JVM artifacts, and .NET
|
|
47
|
+
packages release from one source commit.
|
|
48
|
+
|
|
49
|
+
Requires **schema v76**, Ruby **3.3** or newer, and PostgreSQL **15** or newer.
|
|
50
|
+
|
|
51
|
+
### Upgrade steps
|
|
52
|
+
|
|
53
|
+
**Migrate the schema before starting updated processes.** Run `workhorse schema migrate` before any
|
|
54
|
+
process of this release starts, producers too. Migrations 0088, 0089, and 0090 move the schema
|
|
55
|
+
from version 86 to version **89**. They replace function bodies and change no stored rows. The SDK
|
|
56
|
+
compatibility floor stays at **76**. A 0.10 process keeps working on version 89, because a
|
|
57
|
+
migration inside a major line only adds.
|
|
58
|
+
|
|
59
|
+
**Give each worker a pool with a known size of at least 3.** `Worker.new` no longer accepts
|
|
60
|
+
`shared_heartbeats:`. Remove the option. If a worker pool reports a size below 3, or reports no
|
|
61
|
+
size, set its size to 3 or more. Each worker process needs 3 connections: the listener, the
|
|
62
|
+
heartbeat connection, and one statement connection. A `polling_only: true` worker opens no
|
|
63
|
+
listener, but it still needs a pool of 3 (SM-1901).
|
|
64
|
+
|
|
65
|
+
### Breaking changes
|
|
66
|
+
|
|
67
|
+
- **`shared_heartbeats:` is removed.** Every worker now reserves a dedicated heartbeat connection
|
|
68
|
+
on its pool (ADR 0071). Before, `shared_heartbeats: true` let heartbeat rounds borrow ordinary
|
|
69
|
+
pool connections and let a pool below 3 run. Then a heartbeat could wait behind claims and
|
|
70
|
+
settlements, so a lease could expire and a task could run twice. Now `Worker.new` raises
|
|
71
|
+
`ArgumentError` when it gets `shared_heartbeats:`, and the message names the removal. A worker
|
|
72
|
+
on a pool with a size below 3, or with no reported size, raises `ArgumentError` at construction.
|
|
73
|
+
The worker no longer logs `workhorse.worker.polling_only`, because its pool always has a
|
|
74
|
+
connection for the listener (SM-1901).
|
|
75
|
+
|
|
76
|
+
### Behavior changes
|
|
77
|
+
|
|
78
|
+
These changes can make code that worked on 0.10 act differently:
|
|
79
|
+
|
|
80
|
+
1. An idempotency TTL of exactly 0 still selects the default. The queue now raises
|
|
81
|
+
`ArgumentError` for a TTL that is not 0 and rounds to 0 ms (SM-1857). See "An idempotency
|
|
82
|
+
TTL that rounds to 0 ms is refused".
|
|
83
|
+
2. The TypeScript `Queue`, `Admin`, and `Worker.runOnce()` now check the schema before their first
|
|
84
|
+
statement (SM-1864). This SDK already did, so it does not change.
|
|
85
|
+
3. After a lease lapses while a handler runs, a stale failure or release now settles a due
|
|
86
|
+
deadline or attempt timeout at once. The worker reports `deadline_exceeded` or `timeout`, not
|
|
87
|
+
`lease_lost` (SM-1884). See "A stale failure or release after a lapsed lease settles its
|
|
88
|
+
cause".
|
|
89
|
+
4. Every SDK removes the heartbeat opt-out (SM-1899, SM-1901, SM-1902). In this SDK,
|
|
90
|
+
`Worker.new` raises `ArgumentError` for `shared_heartbeats:`, and for a pool with a size below
|
|
91
|
+
3 or with no reported size. See "Breaking changes".
|
|
92
|
+
|
|
93
|
+
### A step of the wall clock no longer moves the expiry request
|
|
94
|
+
|
|
95
|
+
A worker asks PostgreSQL to end an attempt when its deadline or its attempt timeout passes. Before
|
|
96
|
+
this change, the worker read the wall clock again after each wait on the lease window (SM-1869). If
|
|
97
|
+
the host clock stepped back 10 minutes, the first expiry request came 10 minutes late. If the clock
|
|
98
|
+
stepped forward, the request came early and got `not_due`. Each retry then locked the task row until
|
|
99
|
+
the real due time.
|
|
100
|
+
|
|
101
|
+
Now the worker converts the expiry to a monotonic due time once, when its expiry
|
|
102
|
+
watch starts. After that, it reads only the monotonic clock.
|
|
103
|
+
|
|
104
|
+
### Row retention lag skips a member of an unfinished task group
|
|
105
|
+
|
|
106
|
+
Terminal cleanup keeps each member of a task group that has not finished, because the group needs
|
|
107
|
+
its member outcomes when it finishes. Before this change, queue health counted such a member as row
|
|
108
|
+
retention lag after it passed its windows (SM-1851). An open or sealed group can stay unfinished for
|
|
109
|
+
any length of time. Thus health reported "Retention cleanup is late for task records, finished
|
|
110
|
+
results", and the lag grew without end. The public demo keeps one open and one sealed group on
|
|
111
|
+
purpose, so its Health page showed the report every day.
|
|
112
|
+
|
|
113
|
+
Migration 0088 (`0088-count-only-prunable-rows-as-row-retention-lag.sql`) moves the schema to
|
|
114
|
+
version 87. Now `queue_health_v1` skips a member of an unfinished group when it measures task
|
|
115
|
+
identity and terminal outcome lag. For a group member, lag counts from the latest of window
|
|
116
|
+
expiry, history release, and group finish. The health document keeps its keys, and the schema floor
|
|
117
|
+
does not change.
|
|
118
|
+
Every process of every SDK version gets the new result when the migration commits.
|
|
119
|
+
|
|
120
|
+
### A stale failure or release after a lapsed lease settles its cause
|
|
121
|
+
|
|
122
|
+
A lease can lapse while a handler runs, with no change to the fence. Then `fail_v1` and
|
|
123
|
+
`release_owned_v1` answer `stale`, but `expire_owned_v1` can still settle a due deadline or attempt
|
|
124
|
+
timeout under that fence. Before this change, the worker settled nothing, and lease recovery
|
|
125
|
+
settled the task later (SM-1884).
|
|
126
|
+
|
|
127
|
+
Now the worker calls `expire_owned_v1` at once and settles the due boundary, as the Go and Rust
|
|
128
|
+
workers do:
|
|
129
|
+
|
|
130
|
+
- After a lapsed lease with a due deadline, the handler outcome was `lease_lost`. Now it is
|
|
131
|
+
`deadline_exceeded`, and the task fails at once.
|
|
132
|
+
- After a lapsed lease with a due attempt timeout, the handler outcome was `lease_lost`. Now it is
|
|
133
|
+
`timeout`, and PostgreSQL applies the retry policy at once.
|
|
134
|
+
- After a lapsed lease with no due boundary, the outcome stays `lease_lost`.
|
|
135
|
+
|
|
136
|
+
The handler span records `deadline_exceeded` or `timeout_exceeded`. Before this change, it
|
|
137
|
+
recorded `stale`.
|
|
138
|
+
|
|
139
|
+
If `fail_v1` or `release_owned_v1` answers `cancel_requested` and `acknowledge_cancel_v1` refuses,
|
|
140
|
+
the worker settles nothing. After a failed cancellation acknowledgement, the handler outcome was
|
|
141
|
+
`unknown`. Now it is `lease_lost`. The handler span records `stale`. Before this change, it recorded
|
|
142
|
+
`unknown`. A late heartbeat round can also deliver the cancellation request, after a failure, a
|
|
143
|
+
release, or a refused completion. If the acknowledgement then refuses, the handler outcome was
|
|
144
|
+
`canceled`. Now it is `lease_lost`.
|
|
145
|
+
|
|
146
|
+
### The dashboard bundle uses Mantine 9.7.4
|
|
147
|
+
|
|
148
|
+
The Rack dashboard backend serves the shared dashboard browser bundle. After 0.10.0, a dependency
|
|
149
|
+
update moved the dashboard to Mantine 9.7.4 and rebuilt that bundle (SM-1839). The gem now carries
|
|
150
|
+
the rebuilt `read-surface-1.tar.gz`. Its `bundle.json` records the sha256
|
|
151
|
+
`a7d61ed1d0433e38a8d227cbcc8f98a0bf668f2d3bb1d47391b1882fd69bf795`.
|
|
152
|
+
|
|
153
|
+
The `dashboard/v1` procedures and the read surface version do not change (SM-1858).
|
|
154
|
+
|
|
155
|
+
### A worker no longer keeps the tier entries of a task that the executor refuses
|
|
156
|
+
|
|
157
|
+
The worker records each claimed task in its dispatch state. A fast-tier claim records the task ID
|
|
158
|
+
in `@fast_task_ids`. A full-tier claim records the task ID and a deferred tier read in
|
|
159
|
+
`@tier_reads`. Only the end of a handler run removed these entries.
|
|
160
|
+
|
|
161
|
+
Before this change, the handler executor could refuse a task, for example after its shutdown. The
|
|
162
|
+
task then never ran, and the worker kept its tier entry for the life of the `Worker` object
|
|
163
|
+
(SM-1853). Each refusal kept the entry of at most one task.
|
|
164
|
+
|
|
165
|
+
Now the worker removes the task from both collections when the executor refuses the task.
|
|
166
|
+
|
|
167
|
+
### The dashboard page tells each cache not to store it
|
|
168
|
+
|
|
169
|
+
The dashboard page holds `window.workhorseDashboard`, which contains the `auditActor` of the
|
|
170
|
+
operator. Before this change, the Rack dashboard backend sent the page with only `Content-Type`. A
|
|
171
|
+
shared cache in front of the application could store the page by heuristic. A later visitor could
|
|
172
|
+
then get the stored page and see the actor name of an earlier operator. A browser could also show
|
|
173
|
+
the page from its cache after logout (SM-1870).
|
|
174
|
+
|
|
175
|
+
Now the Rack dashboard backend sends the page with `Cache-Control: no-store`, as the TypeScript host
|
|
176
|
+
does. Each asset keeps `Cache-Control: public, max-age=31536000, immutable`.
|
|
177
|
+
|
|
178
|
+
### An idempotency TTL that rounds to 0 ms is refused
|
|
179
|
+
|
|
180
|
+
The queue rounds `Idempotency#ttl` to the nearest millisecond. Before this change, a positive TTL
|
|
181
|
+
below 0.0005 s, or a small negative TTL such as -0.0004 s, rounded to 0. The queue then kept the key
|
|
182
|
+
for the default 24 hours without a warning (SM-1857).
|
|
183
|
+
|
|
184
|
+
Now the queue raises `ArgumentError` for a TTL that is not 0 and rounds to 0 ms. An exact 0 still
|
|
185
|
+
selects the default. ADR 0115 records the rule for all seven SDKs.
|
|
186
|
+
|
|
187
|
+
### A stalled registration refresh no longer delays a stop
|
|
188
|
+
|
|
189
|
+
A worker refreshes its `worker_registry` row at each registry interval. Before this change, the
|
|
190
|
+
dispatch loop sent that refresh itself, with no time limit. If the statement stalled, for example on
|
|
191
|
+
a lock, `stop` could not start the stop path until the statement returned. Also, the draining
|
|
192
|
+
registration and the deregistration ran on two threads in no order. If the registration committed
|
|
193
|
+
last, the row of the stopped worker came back until the registry prune removed it (SM-1868).
|
|
194
|
+
|
|
195
|
+
Now each registry write runs on its own thread, after the write before it. The dispatch loop starts
|
|
196
|
+
a periodic refresh only when no registry write still runs, and it does not wait for the refresh.
|
|
197
|
+
Thus `run` returns within `shutdown_grace` plus the unwind window, and the deregistration runs
|
|
198
|
+
after every registration write of the run.
|
|
199
|
+
|
|
200
|
+
### Recovery no longer locks a timed-out attempt that waits for its cancellation
|
|
201
|
+
|
|
202
|
+
An operator cancels a running task after its execution timeout passed, and its lease is still live.
|
|
203
|
+
Before migration 0089, each maintenance tick locked that attempt in its timeout scan, but settled
|
|
204
|
+
nothing, because the cancellation wins over the timeout. While the tick ran, the heartbeat of the
|
|
205
|
+
owning worker reported the lease busy. The locked rows also used the recovery limit, so a real
|
|
206
|
+
timeout behind them waited for their leases to expire (SM-1866).
|
|
207
|
+
|
|
208
|
+
Now the full-tier timeout scan skips such an attempt while its lease is live and its deadline is
|
|
209
|
+
ahead, as the fast tier does. If the deadline passed after the timeout, recovery closes the task
|
|
210
|
+
`canceled` at once.
|
|
211
|
+
|
|
212
|
+
### Fast-tier recovery closes a canceled attempt when its deadline passes
|
|
213
|
+
|
|
214
|
+
A fast-tier task has a pending cancellation and a live lease. Its execution timeout passes, and
|
|
215
|
+
later its deadline passes. Before migration 0090, each maintenance tick locked that attempt and
|
|
216
|
+
settled nothing until the lease expired. The cancellation blocks the timeout, and recovery took the
|
|
217
|
+
deadline only when the deadline was not later than the timeout (SM-1881).
|
|
218
|
+
|
|
219
|
+
Now fast-tier recovery closes the task `canceled` when its deadline passes, as the full tier does.
|
|
220
|
+
|
|
221
|
+
### The dashboard Schedules page reads a bounded number of occurrences
|
|
222
|
+
|
|
223
|
+
Before migration 0089, `dashboard_cron_v1` counted every retained occurrence of each listed
|
|
224
|
+
schedule. A schedule that fires each second keeps about 1.2 million occurrences in the default
|
|
225
|
+
retention of 14 days. Each poll of the Schedules page read all of them (SM-1866).
|
|
226
|
+
|
|
227
|
+
Now the function reads at most 1,001 occurrences for each listed schedule. `occurrenceCount` stops
|
|
228
|
+
at 1,000, and the new optional key `occurrenceCountCapped` is `true` when more occurrences are
|
|
229
|
+
retained. The bundled Schedules page then shows `1,000+`. `lastFiredAt` is the fire time of the
|
|
230
|
+
newest occurrence. The `dashboard/v1` change is additive. A database below schema 88 returns an
|
|
231
|
+
exact count and no `occurrenceCountCapped`. The page change rebuilds the shared dashboard bundle
|
|
232
|
+
again. The sha256 in "The dashboard bundle uses Mantine 9.7.4" is the sha256 of that bundle.
|
|
233
|
+
|
|
234
|
+
### The worker registry prune finds expired kept pauses through their index
|
|
235
|
+
|
|
236
|
+
Before migration 0089, `prune_worker_registry_v1` deleted expired kept pauses with a condition that
|
|
237
|
+
the planner could not match to `worker_pause_tombstone_expires_idx`. Each maintenance pass read the
|
|
238
|
+
whole table. Now the delete reads only the expired rows through the index (SM-1866).
|
|
239
|
+
|
|
240
|
+
### A child set and a task group check every request before their first statement
|
|
241
|
+
|
|
242
|
+
Before this change, `HandlerContext#run_children` and `#run_children_all` checked the options of
|
|
243
|
+
each child while they serialized it. Thus the first child could read its payload contract before
|
|
244
|
+
the SDK refused the second child. `Queue#create_task_group` and `Queue#task_group` checked the
|
|
245
|
+
callbacks after the compatibility query and the member contract reads. A member add through
|
|
246
|
+
`HandlerContext#task_group` read the membership of its task first (SM-1883).
|
|
247
|
+
|
|
248
|
+
Now these calls check the options of every request before their first statement, also before the
|
|
249
|
+
deferred tier read of a child set. The errors do not change. Only their time changes.
|
|
250
|
+
|
|
6
251
|
## 0.10.0 — 2026-10-26
|
|
7
252
|
|
|
8
253
|
The npm packages, Python distribution, Go module, Rust crate, Ruby gem, JVM artifacts, and .NET
|
data/README.md
CHANGED
|
@@ -30,7 +30,7 @@ bundle add connection_pool
|
|
|
30
30
|
Install the schema once, as a deployment step. The application never installs or migrates it.
|
|
31
31
|
|
|
32
32
|
```bash
|
|
33
|
-
npx --package @stablemates/workhorse@0.
|
|
33
|
+
npx --package @stablemates/workhorse@0.12.0 workhorse schema install
|
|
34
34
|
```
|
|
35
35
|
|
|
36
36
|
The machine that runs that deployment step needs Node.js 22 or newer. The application itself needs
|
|
@@ -108,7 +108,8 @@ The gem is `stablemates-workhorse`, and its namespace is `Stablemates::Workhorse
|
|
|
108
108
|
that was `:non_replaceable` also carries a `reason`: `:incompatible_key_mode`, `:not_pending`, or
|
|
109
109
|
`:window_elapsed_pending`.
|
|
110
110
|
- `Worker` takes a pool, registers handlers by task type, and runs them under a lease with a shared
|
|
111
|
-
heartbeat connection.
|
|
111
|
+
heartbeat connection. The pool must report a maximum size of at least 3 connections, or the
|
|
112
|
+
constructor raises `ArgumentError`. `run_worker_process` drains the worker on `TERM` or `INT`.
|
|
112
113
|
- `HandlerContext` offers checkpoints, durable sleeps, signal and human waits, child tasks, and
|
|
113
114
|
progress. PostgreSQL owns every durable decision, and an unresolved wait suspends the task.
|
|
114
115
|
- `Admin` lists, inspects, and repairs tasks, dead letters, waits, workers, and queues.
|
|
@@ -63,6 +63,10 @@ module Stablemates
|
|
|
63
63
|
|
|
64
64
|
def submit(outcome) = @outcome.compare_and_set(nil, outcome)
|
|
65
65
|
|
|
66
|
+
# Replaces a cancellation that PostgreSQL refused to acknowledge with a lost lease. A
|
|
67
|
+
# heartbeat round can submit the cancellation request first, but the worker settled nothing.
|
|
68
|
+
def refuse_cancellation = @outcome.compare_and_set(:cancelled, :lease_expired)
|
|
69
|
+
|
|
66
70
|
# Records that a fenced durable write released the task for the suspension +outcome+. The
|
|
67
71
|
# attempt then suspends whichever outcome won, because the heartbeat reports that release as a
|
|
68
72
|
# lost lease.
|
|
@@ -353,6 +357,8 @@ module Stablemates
|
|
|
353
357
|
children.each { |child| child_name(child.name) }
|
|
354
358
|
raise ArgumentError, "child names must be unique" unless children.map(&:name).uniq.length == children.length
|
|
355
359
|
|
|
360
|
+
# Every child is checked before the deferred tier read and the first contract read.
|
|
361
|
+
children.each { |child| @queue.check_child_request(@task, child.task_type, child.payload, child.options) }
|
|
356
362
|
durable!("child tasks")
|
|
357
363
|
|
|
358
364
|
build = lambda do |versions = {}|
|
|
Binary file
|
|
@@ -322,7 +322,7 @@ module Stablemates
|
|
|
322
322
|
"window.workhorseDashboard = #{JSON.generate(config).gsub(/[<>&
]/, SCRIPT_ESCAPES)}"
|
|
323
323
|
}
|
|
324
324
|
.sub("<!--__WORKHORSE_BROWSER_MODULES__-->") { modules }
|
|
325
|
-
[200, {"content-type" => "text/html; charset=utf-8"}, [html]]
|
|
325
|
+
[200, {"content-type" => "text/html; charset=utf-8", "cache-control" => "no-store"}, [html]]
|
|
326
326
|
end
|
|
327
327
|
|
|
328
328
|
def asset(name)
|
|
@@ -298,9 +298,9 @@ module Stablemates
|
|
|
298
298
|
end
|
|
299
299
|
end
|
|
300
300
|
|
|
301
|
-
# A worker
|
|
302
|
-
#
|
|
303
|
-
#
|
|
301
|
+
# A worker could not reserve its dedicated heartbeat connection, or could not borrow a
|
|
302
|
+
# connection for its readiness check, within its startup budget. The message names the pool and
|
|
303
|
+
# the budget. A pool error is the +cause+.
|
|
304
304
|
class WorkerStartupError < Error; end
|
|
305
305
|
|
|
306
306
|
# Any other PG::Error. The original is the +cause+, and +sqlstate+ is its SQLSTATE or nil.
|
|
@@ -120,6 +120,8 @@ module Stablemates
|
|
|
120
120
|
# SDK.
|
|
121
121
|
def add_to_task_group_as_member(task, worker_id, requests) # :nodoc:
|
|
122
122
|
TaskGroups.validate_add(requests)
|
|
123
|
+
# Every request is checked before the membership read.
|
|
124
|
+
request_inputs(requests)
|
|
123
125
|
link = find_task_group(task.id)
|
|
124
126
|
raise ArgumentError, "Task #{task.id} is not a member of a task group" unless link&.role == :member
|
|
125
127
|
|
|
@@ -338,6 +340,32 @@ module Stablemates
|
|
|
338
340
|
# queue of +parent+. Internal to the SDK.
|
|
339
341
|
def serialize_child_request(parent, task_type, payload, options, contracts = {}, version: :current, # :nodoc:
|
|
340
342
|
stored_queue: nil)
|
|
343
|
+
request, input = child_input(parent, task_type, payload, options, stored_queue)
|
|
344
|
+
key = (version == :current) ? request.task_type : [request.task_type, version]
|
|
345
|
+
contract = contracts.fetch(key) do
|
|
346
|
+
contracts[key] = if version != :current
|
|
347
|
+
version && Contracts.load(@executor, request.task_type, version)
|
|
348
|
+
else
|
|
349
|
+
Contracts.load(@executor, request.task_type).tap do |loaded|
|
|
350
|
+
@lock.synchronize { @contracts[request.task_type] = loaded }
|
|
351
|
+
end
|
|
352
|
+
end
|
|
353
|
+
end
|
|
354
|
+
stamp_contract(input, request, contract)
|
|
355
|
+
input
|
|
356
|
+
end
|
|
357
|
+
|
|
358
|
+
# Checks a child request with no round trip. A child set checks every child with this before
|
|
359
|
+
# its first contract read. Internal to the SDK.
|
|
360
|
+
def check_child_request(parent, task_type, payload, options) # :nodoc:
|
|
361
|
+
child_input(parent, task_type, payload, options, nil)
|
|
362
|
+
nil
|
|
363
|
+
end
|
|
364
|
+
|
|
365
|
+
private
|
|
366
|
+
|
|
367
|
+
# The request and the input of a child, without its contract stamp.
|
|
368
|
+
def child_input(parent, task_type, payload, options, stored_queue)
|
|
341
369
|
request = EnqueueRequest.new(task_type: task_type, payload: payload, **options)
|
|
342
370
|
if request.idempotency || request.debounce || request.throttle || request.dependencies
|
|
343
371
|
raise ArgumentError, "Child tasks cannot use coalescing or dependency enqueue options"
|
|
@@ -354,22 +382,9 @@ module Stablemates
|
|
|
354
382
|
input["tags"] = tags(request.tags)
|
|
355
383
|
input["traceContext"] = parent.trace_context unless parent.trace_context.nil?
|
|
356
384
|
input["runAt"] = Values.timestamp(request.run_at, "run at") if request.run_at
|
|
357
|
-
|
|
358
|
-
contract = contracts.fetch(key) do
|
|
359
|
-
contracts[key] = if version != :current
|
|
360
|
-
version && Contracts.load(@executor, request.task_type, version)
|
|
361
|
-
else
|
|
362
|
-
Contracts.load(@executor, request.task_type).tap do |loaded|
|
|
363
|
-
@lock.synchronize { @contracts[request.task_type] = loaded }
|
|
364
|
-
end
|
|
365
|
-
end
|
|
366
|
-
end
|
|
367
|
-
stamp_contract(input, request, contract)
|
|
368
|
-
input
|
|
385
|
+
[request, input]
|
|
369
386
|
end
|
|
370
387
|
|
|
371
|
-
private
|
|
372
|
-
|
|
373
388
|
# Serializes +requests+, passes their JSON document to the block, and returns the rows the block
|
|
374
389
|
# returns. On a contract mismatch, PostgreSQL enqueued nothing: the client reloads the named
|
|
375
390
|
# contracts and calls the block one more time.
|
|
@@ -388,20 +403,25 @@ module Stablemates
|
|
|
388
403
|
end
|
|
389
404
|
|
|
390
405
|
def serialize_requests(requests)
|
|
391
|
-
now = Time.now
|
|
392
406
|
trace_context = Telemetry.inject_trace_context
|
|
393
|
-
inputs = requests
|
|
394
|
-
|
|
395
|
-
input["traceContext"] = trace_context if trace_context
|
|
396
|
-
input
|
|
397
|
-
rescue ArgumentError => e
|
|
398
|
-
raise ArgumentError, "enqueue request #{index + 1}: #{e.message}"
|
|
399
|
-
end
|
|
407
|
+
inputs = request_inputs(requests)
|
|
408
|
+
inputs.each { |input| input["traceContext"] = trace_context } if trace_context
|
|
400
409
|
assert_compatible
|
|
401
410
|
inputs.zip(requests) { |input, request| apply_contract(input, request) }
|
|
402
411
|
JSON.generate(inputs)
|
|
403
412
|
end
|
|
404
413
|
|
|
414
|
+
# The inputs of +requests+, without a round trip, so a batch refuses an invalid option of any
|
|
415
|
+
# request before its first statement.
|
|
416
|
+
def request_inputs(requests)
|
|
417
|
+
now = Time.now
|
|
418
|
+
requests.each_with_index.map do |request, index|
|
|
419
|
+
serialize_request(request, now)
|
|
420
|
+
rescue ArgumentError => e
|
|
421
|
+
raise ArgumentError, "enqueue request #{index + 1}: #{e.message}"
|
|
422
|
+
end
|
|
423
|
+
end
|
|
424
|
+
|
|
405
425
|
# Creates a group with +requests+ as its member tasks, and seals it when +seal+ is true. A
|
|
406
426
|
# contract mismatch of a member or a callback creates nothing, and the retry serializes the
|
|
407
427
|
# callbacks again under the contracts the mismatch reloaded.
|
|
@@ -413,6 +433,11 @@ module Stablemates
|
|
|
413
433
|
raise ArgumentError, "task group callbacks cannot use enqueue idempotency, debounce, or throttle"
|
|
414
434
|
end
|
|
415
435
|
end
|
|
436
|
+
# The callbacks are serialized after the member contracts are read, so their options are
|
|
437
|
+
# checked here, before the first statement.
|
|
438
|
+
now = Time.now
|
|
439
|
+
callbacks.each { |kind, request| callback_input(kind, request, now) unless request.nil? }
|
|
440
|
+
request_inputs(requests)
|
|
416
441
|
group_id = nil
|
|
417
442
|
rows = submit_requests(requests) do |document|
|
|
418
443
|
created = @executor.rows(SqlCatalogue::CREATE_TASK_GROUP_V1,
|
|
@@ -435,11 +460,7 @@ module Stablemates
|
|
|
435
460
|
document = callbacks.each_with_object({}) do |(kind, request), entries|
|
|
436
461
|
next if request.nil?
|
|
437
462
|
|
|
438
|
-
input =
|
|
439
|
-
serialize_request(request, now)
|
|
440
|
-
rescue ArgumentError => e
|
|
441
|
-
raise ArgumentError, "#{kind} callback: #{e.message}"
|
|
442
|
-
end
|
|
463
|
+
input = callback_input(kind, request, now)
|
|
443
464
|
input.delete("runAt") if request.run_at.nil?
|
|
444
465
|
input["traceContext"] = trace_context if trace_context
|
|
445
466
|
apply_contract(input, request)
|
|
@@ -448,6 +469,12 @@ module Stablemates
|
|
|
448
469
|
JSON.generate(document)
|
|
449
470
|
end
|
|
450
471
|
|
|
472
|
+
def callback_input(kind, request, now)
|
|
473
|
+
serialize_request(request, now)
|
|
474
|
+
rescue ArgumentError => e
|
|
475
|
+
raise ArgumentError, "#{kind} callback: #{e.message}"
|
|
476
|
+
end
|
|
477
|
+
|
|
451
478
|
# +member+ is the task ID, worker ID, and fence token of a running member task, or three nils.
|
|
452
479
|
def add_task_group_members(group_id, requests, member)
|
|
453
480
|
TaskGroups.validate_add(requests)
|
|
@@ -545,14 +572,21 @@ module Stablemates
|
|
|
545
572
|
def idempotency_document(idempotency)
|
|
546
573
|
raise ArgumentError, "idempotency must be an Idempotency" unless idempotency.is_a?(Idempotency)
|
|
547
574
|
|
|
548
|
-
ttl = Values.milliseconds(idempotency.ttl, "idempotency TTL", 0..MAX_DURATION_MS)
|
|
549
575
|
{
|
|
550
576
|
"key" => required_bytes(idempotency.key, MAX_KEY_BYTES, "idempotency key"),
|
|
551
577
|
"scope" => scope(idempotency.scope),
|
|
552
|
-
"ttlMs" =>
|
|
578
|
+
"ttlMs" => idempotency_ttl(idempotency.ttl)
|
|
553
579
|
}
|
|
554
580
|
end
|
|
555
581
|
|
|
582
|
+
# The protocol value of an idempotency TTL. An exact zero takes the default. A nonzero value
|
|
583
|
+
# that rounds to 0 ms is refused, so a unit mistake cannot select the default.
|
|
584
|
+
def idempotency_ttl(value)
|
|
585
|
+
return DEFAULT_IDEMPOTENCY_TTL_MS if value.is_a?(Numeric) && value.zero?
|
|
586
|
+
|
|
587
|
+
Values.milliseconds(value, "idempotency TTL", 1..MAX_DURATION_MS)
|
|
588
|
+
end
|
|
589
|
+
|
|
556
590
|
def debounce_document(debounce)
|
|
557
591
|
raise ArgumentError, "debounce must be a Debounce" unless debounce.is_a?(Debounce)
|
|
558
592
|
|
|
@@ -11,7 +11,7 @@ module Stablemates
|
|
|
11
11
|
MINIMUM_PROTOCOL_VERSION = 5
|
|
12
12
|
MAXIMUM_PROTOCOL_VERSION = 5
|
|
13
13
|
MINIMUM_SCHEMA_VERSION = 76
|
|
14
|
-
MAXIMUM_SCHEMA_VERSION =
|
|
14
|
+
MAXIMUM_SCHEMA_VERSION = 91
|
|
15
15
|
# PostgreSQL's atomic enqueue batch limit.
|
|
16
16
|
MAX_ENQUEUE_BATCH_SIZE = 1000
|
|
17
17
|
DEFAULT_TASK_VALUE_MAX_BYTES = 1048576
|
|
@@ -347,6 +347,9 @@ module Stablemates
|
|
|
347
347
|
# `queue_control` (internal)
|
|
348
348
|
QUEUE_CONTROL = 'SELECT queue_name, paused, tier, record_attempts, record_claims FROM workhorse.queue_control'
|
|
349
349
|
|
|
350
|
+
# `queue_tier` (internal)
|
|
351
|
+
QUEUE_TIER = 'SELECT COALESCE((SELECT tier FROM workhorse.queue_control WHERE queue_name = $1::text), \'full\') AS tier'
|
|
352
|
+
|
|
350
353
|
# `schedule_definition` (internal)
|
|
351
354
|
SCHEDULE_DEFINITION = 'SELECT DISTINCT namespace FROM workhorse.schedule_definition ORDER BY namespace'
|
|
352
355
|
|
|
@@ -163,11 +163,11 @@ module Stablemates
|
|
|
163
163
|
# errors of a socket), or SocketError (DNS); or a SQLSTATE that starts with 08, or 57P01,
|
|
164
164
|
# 57P02, or 57P03 (a server shutdown). IO::TimeoutError is an IOError, so a socket timeout
|
|
165
165
|
# is a connection error too.
|
|
166
|
-
# - "connection" also covers a pool error that CONNECTION_ERROR_NAMES names
|
|
167
|
-
#
|
|
168
|
-
#
|
|
169
|
-
#
|
|
170
|
-
#
|
|
166
|
+
# - "connection" also covers a pool error that CONNECTION_ERROR_NAMES names: a wait that ends at
|
|
167
|
+
# the pool's own limit, ConnectionPool::TimeoutError, Sequel::PoolTimeout, or
|
|
168
|
+
# ActiveRecord::ConnectionTimeoutError (an ActiveRecord::ConnectionNotEstablished). A round
|
|
169
|
+
# runs on the dedicated heartbeat connection and borrows none from the pool. The classifier
|
|
170
|
+
# keeps these names, so a pool error in a round still counts as "connection".
|
|
171
171
|
# - "database": any other SQLSTATE.
|
|
172
172
|
# - "other": everything else. A Timeout::Error that is not one of these, for example from
|
|
173
173
|
# Timeout.timeout, is "other": only the round deadline is "timeout".
|