stablemates-workhorse 0.6.1 → 0.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +364 -1
- data/README.md +25 -9
- data/lib/stablemates/workhorse/active_job.rb +141 -13
- data/lib/stablemates/workhorse/admin.rb +78 -3
- data/lib/stablemates/workhorse/batch_context.rb +6 -2
- data/lib/stablemates/workhorse/checkpoint_fingerprint.rb +118 -0
- data/lib/stablemates/workhorse/context.rb +80 -15
- data/lib/stablemates/workhorse/dashboard/backend.rb +20 -1
- data/lib/stablemates/workhorse/dashboard/bundle.json +1 -1
- data/lib/stablemates/workhorse/dashboard/read-surface-1.tar.gz +0 -0
- data/lib/stablemates/workhorse/dashboard/v1_generated.rb +149 -1
- data/lib/stablemates/workhorse/dashboard.rb +45 -7
- data/lib/stablemates/workhorse/errors.rb +79 -1
- data/lib/stablemates/workhorse/executor.rb +3 -0
- data/lib/stablemates/workhorse/queue.rb +169 -16
- data/lib/stablemates/workhorse/sql_catalogue_generated.rb +70 -6
- data/lib/stablemates/workhorse/task_group.rb +240 -0
- data/lib/stablemates/workhorse/telemetry.rb +18 -2
- data/lib/stablemates/workhorse/version.rb +1 -1
- data/lib/stablemates/workhorse/worker.rb +43 -6
- data/lib/stablemates/workhorse.rb +3 -0
- metadata +23 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 807d43f9b16d79430d292a3e6c70acdc55bb8cc3f2010b7e03f0356fb0731c90
|
|
4
|
+
data.tar.gz: f84ab68d2d6d03450983e74dc337880546815472ece79d8cdf1d650679d3912a
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 706d9cfd755c1bc971116245009433dba6d4880dc7b7dec081f486d95f5395395a69b55527527b33252f4cf4bbc5e1a1b94c74ad91b267fe6893884c3bdb09ef
|
|
7
|
+
data.tar.gz: dd7e13f971e790cc10c4f07843a4b7ec9f086f32365c665f681206bf1376d1d7d03bf11e6dfa78aacc75ea8cb2ce292da3246b9471431d7fa6b1630c58c46a17
|
data/CHANGELOG.md
CHANGED
|
@@ -3,6 +3,369 @@
|
|
|
3
3
|
`stablemates-workhorse` gem versions and release notes live here. The gem carries the version the
|
|
4
4
|
other SDKs carry, because every tag names one release of all of them.
|
|
5
5
|
|
|
6
|
+
## 0.8.0 — 2026-10-13
|
|
7
|
+
|
|
8
|
+
The npm packages, Python distribution, Go module, Rust crate, and Ruby gem release from one source commit.
|
|
9
|
+
|
|
10
|
+
Requires **schema v69**, Ruby **3.3** or newer, and PostgreSQL **15** or newer.
|
|
11
|
+
|
|
12
|
+
### Upgrade steps
|
|
13
|
+
|
|
14
|
+
**Migrate the schema before starting updated processes.** The upgrade is a rolling deployment:
|
|
15
|
+
|
|
16
|
+
1. Run `workhorse schema migrate`. The final schema version is **71**.
|
|
17
|
+
2. Roll out the updated processes.
|
|
18
|
+
|
|
19
|
+
The SDK compatibility floor is schema version **69**, which migration 0070 reaches. The floor rises
|
|
20
|
+
from 54. Every SDK statement catalogue now names the task group functions that migration 0066 adds,
|
|
21
|
+
and `save_checkpoint_v2`, which migration 0069 adds. The dashboard of each SDK calls
|
|
22
|
+
`dashboard_task_groups_v1`, which migration 0067 adds, and `list_newest_task_group_events_v1`, which
|
|
23
|
+
migration 0068 adds. Migration 0070 fences a member add, so an updated process does not add tasks to
|
|
24
|
+
a group without that check. An updated process refuses a schema below version 69 at startup. A 0.7
|
|
25
|
+
process keeps working on version 71, because a migration inside a major line only adds.
|
|
26
|
+
|
|
27
|
+
**Give the eighth member when you build a `TaskCheckpoint`.** `TaskCheckpoint` is a `Data` class.
|
|
28
|
+
0.8.0 adds `input_fingerprint` as its eighth member (SM-1264). A call to `TaskCheckpoint.new` with
|
|
29
|
+
seven positional arguments raises `ArgumentError`. A call with keywords and no
|
|
30
|
+
`input_fingerprint:` also raises it. The gem has no compatible constructor. If your code builds a
|
|
31
|
+
`TaskCheckpoint`, for example in a test, give the eighth value. For a checkpoint without an input,
|
|
32
|
+
give `input_fingerprint: nil` or a positional `nil`.
|
|
33
|
+
|
|
34
|
+
Migration 0066 (`0066-add-task-groups-with-completion-callbacks.sql`) adds task groups (SM-1231,
|
|
35
|
+
[ADR 0095](https://github.com/stablemates/workhorse/blob/main/docs/decisions/0095-add-task-groups-with-completion-callbacks.md)).
|
|
36
|
+
A task group is a set of full-tier tasks that Workhorse tracks as one unit. When the group is
|
|
37
|
+
sealed and every member task is finished, Workhorse enqueues its `onFinish`, `onSuccess`, or
|
|
38
|
+
`onFailure` callback task. The migration adds the `task_group`, `task_group_member`,
|
|
39
|
+
`task_group_callback`, and `task_group_event` tables, and a trigger on `task_outcome` that counts
|
|
40
|
+
each member outcome in the transaction that writes it. A queue purge counts a purged member task as
|
|
41
|
+
`canceled`. The `terminal_storage` routine keeps member tasks while their group is not finished,
|
|
42
|
+
reports a new `task_groups` phase after `terminal_tasks`, and deletes a finished group after its
|
|
43
|
+
callback tasks are terminal. New SQLSTATEs: `P1008` (the task group does not accept tasks), `P1009`
|
|
44
|
+
(member limit of 100,000), and `P1010` (an enqueue idempotency hit returns a task that is not a
|
|
45
|
+
member of the group).
|
|
46
|
+
|
|
47
|
+
Migration 0067 (`0067-count-task-group-outcomes-in-counter-shards.sql`) counts task group outcomes
|
|
48
|
+
in counter shards (SM-1245,
|
|
49
|
+
[ADR 0096](https://github.com/stablemates/workhorse/blob/main/docs/decisions/0096-count-task-group-outcomes-in-counter-shards.md)).
|
|
50
|
+
Each group gets 32 rows in the new `task_group_shard` table, and a member outcome locks only its
|
|
51
|
+
shard row at commit. The outcome locks the group row only when the pending count of its shard
|
|
52
|
+
becomes 0. The migration moves the counts of existing groups into the shards. It locks `task_group`
|
|
53
|
+
in `EXCLUSIVE` mode while it runs, so task group changes wait for it. The counts on a `task_group`
|
|
54
|
+
row stay 0 until the group finishes, so read the counts through `get_task_group_v1`. The migration
|
|
55
|
+
also adds `dashboard_task_groups_v1`, which lists task groups one page at a time. The dashboard of
|
|
56
|
+
each SDK reads the task group list through it (SM-1237).
|
|
57
|
+
|
|
58
|
+
Migration 0068 (`0068-read-the-newest-task-group-events.sql`) adds
|
|
59
|
+
`list_newest_task_group_events_v1`, which returns the newest events of a task group first
|
|
60
|
+
(SM-1254). The task group detail of the dashboard reads its events through it. A group with more
|
|
61
|
+
than 1,000 events now shows its newest 100 events. `eventsTruncated` is true when the group has
|
|
62
|
+
older events that the detail does not show. `list_task_group_events_v1` does not change. The
|
|
63
|
+
migration changes no stored data.
|
|
64
|
+
|
|
65
|
+
Migration 0069 (`0069-check-the-input-of-a-checkpoint-on-replay.sql`) adds the nullable column
|
|
66
|
+
`task_checkpoint.input_fingerprint` and `save_checkpoint_v2` (SM-1211,
|
|
67
|
+
[ADR 0098](https://github.com/stablemates/workhorse/blob/main/docs/decisions/0098-check-the-input-of-a-checkpoint-on-replay.md)).
|
|
68
|
+
If a checkpoint name exists, `save_checkpoint_v2` returns `input_conflict` when the stored and the
|
|
69
|
+
given input fingerprints are both present and differ. `save_checkpoint_v1` now calls
|
|
70
|
+
`save_checkpoint_v2` with no fingerprint, so its results do not change. The checkpoint reads of each
|
|
71
|
+
statement catalogue return the new column. Existing rows keep a NULL fingerprint. The migration adds
|
|
72
|
+
the column without a table scan.
|
|
73
|
+
|
|
74
|
+
Migration 0070 (`0070-fence-a-task-group-member-add-and-index-the-group-list.sql`) fences a task
|
|
75
|
+
group member add (SM-1273). `add_task_group_members_v1` now locks the runtime row of the member task
|
|
76
|
+
first, before the shard rows and the group row, as a completion does. Then it refuses the add with
|
|
77
|
+
`P1008` and the reason `not_running_member` if the lease of the member task expired, its deadline or
|
|
78
|
+
attempt timeout passed, or a cancel was requested. Before, a worker whose lease expired could add
|
|
79
|
+
tasks after Workhorse gave the member task to a new worker. The signature does not change. The
|
|
80
|
+
migration also adds `task_group_state_created_idx`, so `dashboard_task_groups_v1` reads one page of
|
|
81
|
+
a state without a read of the groups of other states. The migration is a non-transactional step: it
|
|
82
|
+
builds the index with `CREATE INDEX CONCURRENTLY`, so writes to `task_group` continue. If it fails
|
|
83
|
+
part-way, drop the invalid index before the rerun, as `docs/schema-lifecycle.md` describes. It
|
|
84
|
+
changes no stored data.
|
|
85
|
+
|
|
86
|
+
Migration 0071 (`0071-lock-purged-runtime-rows-in-identity-order.sql`) makes a queue purge lock its
|
|
87
|
+
tasks in identity order (SM-1282). Before, `purge_queue_internal_v1` locked the runtime rows and the
|
|
88
|
+
task rows of the queue in scan order. An enqueue that named two of those tasks as prerequisites
|
|
89
|
+
locks them in identity order. Thus the purge and the enqueue could wait for each other, and
|
|
90
|
+
PostgreSQL aborted one of them with `40P01`. The purge now locks every runtime row, then every task
|
|
91
|
+
row, each in identity order. The signature and the deleted count do not change. The migration does
|
|
92
|
+
not raise the SDK compatibility floor, because no SDK statement changes and every process gets the
|
|
93
|
+
new lock order when the migration commits. It changes no stored data.
|
|
94
|
+
|
|
95
|
+
Migration 0072 (`0072-purge-a-queue-that-holds-a-prerequisite-of-a-live-dependent.sql`) lets a
|
|
96
|
+
queue purge succeed when it holds a prerequisite of a waiting dependent (SM-1284,
|
|
97
|
+
[ADR 0101](https://github.com/stablemates/workhorse/blob/main/docs/decisions/0101-cancel-a-purged-prerequisite-whose-dependent-stays.md)).
|
|
98
|
+
Before, the purge deleted that prerequisite, and PostgreSQL refused the delete with `23001`. The
|
|
99
|
+
whole purge then failed, also when the dependent was in the same queue. Now the purge deletes a
|
|
100
|
+
dependent in the same purge with its prerequisite. If the dependent stays, for example in another
|
|
101
|
+
queue, the purge cancels the prerequisite instead of a delete. The dependent's own
|
|
102
|
+
`on_cancellation` policy then decides its outcome, as after a cancel. The canceled task keeps its
|
|
103
|
+
history and its keys. The purge count, `deleted_count` and `deletedCount`, includes the canceled
|
|
104
|
+
tasks. The purge locks the dependents after its own tasks, as a cancel does, so it has no
|
|
105
|
+
deadlock that a cancel of the same tasks does not also have. Two purges run one after the
|
|
106
|
+
other, also for different queues. The migration does not raise the SDK compatibility
|
|
107
|
+
floor, because no SDK statement or result shape changes. It changes no stored data.
|
|
108
|
+
|
|
109
|
+
### Task groups
|
|
110
|
+
|
|
111
|
+
The Ruby SDK adds task groups (SM-1236). Use the block form to create a group, add its member
|
|
112
|
+
tasks, and seal it in one statement, so in one transaction:
|
|
113
|
+
|
|
114
|
+
```ruby
|
|
115
|
+
result = queue.task_group(on_finish: Stablemates::Workhorse::EnqueueRequest.new(
|
|
116
|
+
task_type: "import.report", payload: {"importId" => 42}
|
|
117
|
+
)) do |group|
|
|
118
|
+
rows.each { |row| group.enqueue("import.row", {"row" => row}) }
|
|
119
|
+
end
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
- `Queue#task_group` takes the `on_finish:`, `on_success:`, and `on_failure:` callbacks. Each
|
|
123
|
+
callback is an `EnqueueRequest` without idempotency, debounce, or throttle. The block receives a
|
|
124
|
+
`TaskGroupBuilder` with `add` and `enqueue`. If the block raises, nothing is enqueued.
|
|
125
|
+
- `Queue#create_task_group`, `Queue#add_to_task_group`, and `Queue#seal_task_group` build a group in
|
|
126
|
+
more than one transaction.
|
|
127
|
+
- `Queue#get_task_group`, `Queue#list_task_group_members`, and `Queue#find_task_group` read a
|
|
128
|
+
group. `Admin` has `get_task_group`, `list_task_group_members`, `list_task_group_events`,
|
|
129
|
+
`cancel_task_group`, `list_task_group_drift`, and `repair_task_group_drift`.
|
|
130
|
+
- `HandlerContext#task_group` and `BatchHandlerContext#task_group` return a `TaskGroupContext`.
|
|
131
|
+
Its `get` returns the `TaskGroupLink` of the task, with the group ID and the counts. Its `add`
|
|
132
|
+
adds tasks from a running member task, also after the seal. If a request names no queue, `add`
|
|
133
|
+
puts the task in the queue of the member task, not in the first queue of the worker (SM-1251,
|
|
134
|
+
[ADR 0097](https://github.com/stablemates/workhorse/blob/main/docs/decisions/0097-put-work-that-a-handler-creates-on-the-queue-of-its-task.md)).
|
|
135
|
+
- `Stablemates::Workhorse::ActiveJob.task_group` runs a block in which each job that the
|
|
136
|
+
Workhorse adapter enqueues becomes a member task. Its callbacks can be Active Job instances,
|
|
137
|
+
such as `ReportJob.new(import.id)`. A job reads its group through
|
|
138
|
+
`Stablemates::Workhorse::ActiveJob.current_task_group`.
|
|
139
|
+
- A job class with `enqueue_after_transaction_commit` set to true cannot join a task group. Rails
|
|
140
|
+
would enqueue that job after the transaction commits, so outside the group. If a task group block
|
|
141
|
+
enqueues such a job through the Workhorse adapter, the SDK raises `ArgumentError` before Rails
|
|
142
|
+
defers the job. Set `enqueue_after_transaction_commit` to false on that job class. The group
|
|
143
|
+
statement runs in the caller's transaction, so the member tasks still commit with it.
|
|
144
|
+
- `TaskGroupClosedError` (`P1008`), `TaskGroupLimitExceededError` (`P1009`), and
|
|
145
|
+
`TaskGroupMembershipConflictError` (`P1010`) report the task group refusals.
|
|
146
|
+
- `Queue#list_task_group_members` and `Admin#list_task_group_members` return a nil `next_after`
|
|
147
|
+
on the last page, also when the page is full (SM-1277). Before, a full last page gave a
|
|
148
|
+
`next_after` that read an empty page. A full page now costs one more read of one member.
|
|
149
|
+
`TaskGroup#callbacks` keeps the order `on_finish`, `on_success`, `on_failure`, and a test now
|
|
150
|
+
asserts it. The change adds no SQL function and no migration.
|
|
151
|
+
|
|
152
|
+
### Checkpoint input
|
|
153
|
+
|
|
154
|
+
A checkpoint finds its stored result by name only. If a deployment changes the input of a checkpoint
|
|
155
|
+
operation, a replay returns the result of the old input and gives no error. A Ruby handler can now
|
|
156
|
+
give the input of a checkpoint (SM-1264,
|
|
157
|
+
[ADR 0098](https://github.com/stablemates/workhorse/blob/main/docs/decisions/0098-check-the-input-of-a-checkpoint-on-replay.md)):
|
|
158
|
+
|
|
159
|
+
```ruby
|
|
160
|
+
context.checkpoint("charge-card", input: {"amountCents" => amount}) { gateway.charge(amount) }
|
|
161
|
+
```
|
|
162
|
+
|
|
163
|
+
- `HandlerContext#checkpoint` and `BatchHandlerContext#checkpoint` take the `input:` keyword, a JSON
|
|
164
|
+
value. The checkpoint stores the fingerprint of the input with its value, through
|
|
165
|
+
`save_checkpoint_v2`. A call without `input:` keeps its current behavior. `input: nil` gives the
|
|
166
|
+
JSON input null.
|
|
167
|
+
- If the stored checkpoint, or an overlapping call with the same name, has the fingerprint of a
|
|
168
|
+
different input, the call raises `CheckpointInputConflictError` and the block does not run. If the
|
|
169
|
+
call or the stored checkpoint has no input, the call compares nothing.
|
|
170
|
+
- `CheckpointInputConflictError` is a subclass of `ConflictError`. The worker fails the task at once
|
|
171
|
+
and keeps the current attempt, as for every replay conflict. The task error names the class.
|
|
172
|
+
- `Stablemates::Workhorse.checkpoint_input_fingerprint(input)` returns the fingerprint: the SHA-256
|
|
173
|
+
digest of the RFC 8785 canonical JSON of the input, as 64 lowercase hexadecimal characters. It
|
|
174
|
+
refuses an Integer or a Float larger than 2**53 - 1 in magnitude, such as `1e21`, because RFC 8785
|
|
175
|
+
writes each number as an IEEE 754 double. Every SDK refuses the same inputs (SM-1274).
|
|
176
|
+
- `TaskCheckpoint#input_fingerprint` returns the stored fingerprint, or nil. `Admin#get_checkpoint`,
|
|
177
|
+
`Admin#list_checkpoints`, and `HandlerContext#get_checkpoint` return it.
|
|
178
|
+
|
|
179
|
+
### Behaviour changes
|
|
180
|
+
|
|
181
|
+
- If a child task request names no queue, `run_child`, `run_children`, and `run_children_all` put
|
|
182
|
+
the child in the queue of the parent task (SM-1260, [ADR 0097](https://github.com/stablemates/workhorse/blob/main/docs/decisions/0097-put-work-that-a-handler-creates-on-the-queue-of-its-task.md)).
|
|
183
|
+
Before, the child went to the first queue in the `queues:` list of the worker. To keep the old
|
|
184
|
+
queue, name that queue on the request, for example `queue: "default"`.
|
|
185
|
+
A parent task that created a child before the upgrade can replay that request after the upgrade.
|
|
186
|
+
If the replayed request names no queue, it keeps the queue of the stored child, so the parent
|
|
187
|
+
task joins its child.
|
|
188
|
+
- `Worker#run_once` does not change: a pass that released every task that it claimed returns
|
|
189
|
+
`false`, and the next call claims at once (SM-1259, [ADR 0099](https://github.com/stablemates/workhorse/blob/main/docs/decisions/0099-report-a-released-only-single-pass-as-no-progress.md)). TypeScript
|
|
190
|
+
`runOnce()` now does the same. If you call the single-pass call in a loop, wait after `false`.
|
|
191
|
+
- A continuous worker that released every task of a claim now waits for its next poll before it
|
|
192
|
+
claims again (SM-1247). A worker releases a task when it has no handler for the task type, for
|
|
193
|
+
example during a rolling deployment. Before, the release notification ended the empty wait of the
|
|
194
|
+
same worker, so it claimed the task again after 0 to 50 ms. One listening worker released one task
|
|
195
|
+
about 30 times each second. Now a notification does not end this release wait. Only its deadline,
|
|
196
|
+
a stop, or a pause ends it. The deadline is the poll interval while the listener is active, or the
|
|
197
|
+
poll backoff without a listener. Thus a worker claims a task that it cannot run at most once in
|
|
198
|
+
each poll interval.
|
|
199
|
+
- The Rack dashboard refuses a mutation whose `audit.requestId` is more than 512 UTF-8 bytes
|
|
200
|
+
with `BAD_REQUEST`, before any SQL runs (SM-1253). It measures the value as sent, before trim.
|
|
201
|
+
Before, the SQL audit function refused the value, and the client got an internal server error.
|
|
202
|
+
|
|
203
|
+
## 0.7.1 — 2026-10-09
|
|
204
|
+
|
|
205
|
+
The npm packages, Python distribution, Go module, Rust crate, and Ruby gem release from one source commit.
|
|
206
|
+
|
|
207
|
+
Requires **schema v54**, Ruby **3.3** or newer, and PostgreSQL **15** or newer.
|
|
208
|
+
|
|
209
|
+
The gem has no changes in 0.7.1. It releases at 0.7.1 to keep one version across the
|
|
210
|
+
registries ([ADR 0050](../docs/decisions/0050-release-0-1-0-without-a-prerelease-suffix.md)).
|
|
211
|
+
The release exists because the npm publish of 0.7.0 stopped partway.
|
|
212
|
+
The gem never published at 0.7.0, so 0.7.1 is the first release with the 0.7.0 changes below.
|
|
213
|
+
|
|
214
|
+
**A 0.7.0 database needs no migration.** 0.7.1 adds no migration, so the final schema version is
|
|
215
|
+
**64** and the SDK compatibility floor stays at schema version **54**. An installation on 0.6
|
|
216
|
+
follows the [0.7.0 upgrade steps](https://github.com/stablemates/workhorse/blob/main/CHANGELOG.md#070--2026-10-09).
|
|
217
|
+
|
|
218
|
+
## 0.7.0 — 2026-10-09
|
|
219
|
+
|
|
220
|
+
The npm packages, Python distribution, Go module, Rust crate, and Ruby gem release from one source commit.
|
|
221
|
+
|
|
222
|
+
Requires **schema v54**, Ruby **3.3** or newer, and PostgreSQL **15** or newer.
|
|
223
|
+
|
|
224
|
+
### Upgrade steps
|
|
225
|
+
|
|
226
|
+
**Migrate the schema before starting updated processes.** The upgrade is a rolling deployment: run `workhorse schema migrate`, then roll out the 0.7
|
|
227
|
+
processes. The final schema version is **64**. The SDK compatibility floor is schema version
|
|
228
|
+
**54**, which migration 0055 reaches. A 0.7 process refuses a schema below version 54 at startup,
|
|
229
|
+
and a 0.6 process keeps working on version 64. Each migration commits on its own, and each SQL
|
|
230
|
+
change takes effect when its migration commits.
|
|
231
|
+
|
|
232
|
+
**Migration 0059 clears incomplete rate settings.** A per-key rate limit or a budget rate with one
|
|
233
|
+
or two of its three fields null used to pass its CHECK constraint. The migration clears such a
|
|
234
|
+
setting, and deletes a budget that the clearing leaves without a limit. After the migration,
|
|
235
|
+
PostgreSQL rejects a synchronization that writes an incomplete rate setting. To keep a limit the
|
|
236
|
+
migration cleared, complete it in your synchronized definitions and synchronize after the
|
|
237
|
+
migration.
|
|
238
|
+
|
|
239
|
+
**Accept the new `terminal-cleanup-backlog` health reason.** Queue health can now report it as
|
|
240
|
+
degraded, so code that switches on reason codes should accept it. Its budget,
|
|
241
|
+
`terminal_cleanup_backlog_ms`, starts from your `row_retention_lag_ms` setting, so existing alerts
|
|
242
|
+
evaluate as before. After migration 0065 the two budgets are independent.
|
|
243
|
+
|
|
244
|
+
**Update `dashboard/v1` clients that read `humanWait.context` from a task listing.**
|
|
245
|
+
`dashboard.tasks` and `dashboard.tasksCursor` rows carry `humanWait.quickAction` instead, which is
|
|
246
|
+
`{ label }` or `null`. `dashboard.taskDetail` still returns the full context (SM-1171).
|
|
247
|
+
|
|
248
|
+
**Handler authors: a durable replay conflict now fails its task on the first occurrence.** The
|
|
249
|
+
behaviour changes below describe it. A 0.6 worker still retries a conflict under the task's retry
|
|
250
|
+
policy until the 0.6 processes are gone.
|
|
251
|
+
|
|
252
|
+
### Breaking changes
|
|
253
|
+
|
|
254
|
+
- **`dashboard/v1`:** task listing rows carry `humanWait.quickAction` instead of
|
|
255
|
+
`humanWait.context`, as the upgrade steps describe. The break is taken in place under the ADR 0064
|
|
256
|
+
exception in [`docs/compatibility.md`](../docs/compatibility.md) (SM-1171).
|
|
257
|
+
|
|
258
|
+
### Behaviour changes
|
|
259
|
+
|
|
260
|
+
Durable execution and workers:
|
|
261
|
+
|
|
262
|
+
- Durable checkpoint, timer, child, child-set, and human-decision replay conflicts fail the task on
|
|
263
|
+
their first occurrence. The worker preserves the current attempt and records the conflict class.
|
|
264
|
+
Before, a conflict retried under the task's retry policy, and every retry met the same conflict.
|
|
265
|
+
Redaction still hides error details. Transient failures, lease loss, child-limit errors, and
|
|
266
|
+
already-waiting signal errors keep their existing behaviour. Conflict settlement bypasses the
|
|
267
|
+
worker's retry-delay callback (SM-1107, migration 0055).
|
|
268
|
+
- A renamed individual child on replay raises `ConflictError` with the stored and requested names,
|
|
269
|
+
and `ConflictError.new` takes a `stored_name:` keyword. Before, it raised a child-limit error. A
|
|
270
|
+
second child after joining the retained child in the same handler run still exceeds the child
|
|
271
|
+
limit (SM-1106, migration 0054).
|
|
272
|
+
- A handler that raises the `CancelledError` of its `:shutdown` cancellation no longer charges an
|
|
273
|
+
attempt. The worker hands its task to `release_owned_v1`, as it does for a task without a handler.
|
|
274
|
+
Before, `CancelledError(:shutdown)` reached `fail_v1`, so a task on its last attempt failed
|
|
275
|
+
because its worker stopped (SM-1164).
|
|
276
|
+
- The `workhorse.handler` span descends only from the task's stored trace context. A task without
|
|
277
|
+
one starts a new trace instead of joining a span that a context-propagating executor carries into
|
|
278
|
+
the handler thread (SM-1179).
|
|
279
|
+
|
|
280
|
+
Dashboard host:
|
|
281
|
+
|
|
282
|
+
- `Dashboard.new(audit_actor: ...)` no longer replaces the actor of a `Principal` that `authorize`
|
|
283
|
+
returns. The dashboard records the principal's actor, as the `dashboard/v1` protocol requires.
|
|
284
|
+
`audit_actor` still names the actor when `authorize` returns `true`, and defaults to `dashboard`
|
|
285
|
+
there (SM-1152).
|
|
286
|
+
- `Dashboard` refuses an RPC body over 2 MiB, or a malformed declared length, with `413` before
|
|
287
|
+
parsing. An unexpected schema-compatibility failure, such as a connection error, answers a generic
|
|
288
|
+
`503` and is written to `rack.errors` instead of escaping the Rack app (SM-1168).
|
|
289
|
+
|
|
290
|
+
Maintenance, health, and administration, from the SQL changes:
|
|
291
|
+
|
|
292
|
+
- Terminal cleanup repeats its batch while each one fills, for up to one second per pass. A pass
|
|
293
|
+
that still ends with a full batch makes its follow-up due five seconds later. Full-tier and
|
|
294
|
+
fast-tier tasks share every batch (SM-1160, migration 0058).
|
|
295
|
+
- `tick_v1` runs the expired-lease scan only when no tick ran it within half the shortest
|
|
296
|
+
maintenance interval of the live registered workers. A worker that opts out of the registry with
|
|
297
|
+
`disable_registry: true` does not shorten that spacing. Recovery of an expired lease can then take
|
|
298
|
+
longer (SM-1167, SM-1192, migration 0061).
|
|
299
|
+
- Row retention lag counts from the scheduled history pass that released the row, so health no
|
|
300
|
+
longer reports task records and finished results as late after every daily pass (SM-1134,
|
|
301
|
+
migration 0056).
|
|
302
|
+
- Queue health raises `terminal-cleanup-backlog` once `terminal_cleanup_backlog_since` is older than
|
|
303
|
+
its budget (SM-1178, SM-1197, migrations 0063 and 0065).
|
|
304
|
+
- `run_task_now_v1` reports `not_scheduled` for a blocked task instead of raising an error. A rate
|
|
305
|
+
synchronization refills each changed bucket at the old rate up to one clock reading. An update
|
|
306
|
+
can no longer change a dependency edge's endpoints or outcome policies (SM-1165, migration 0059).
|
|
307
|
+
- A full-tier task that a worker without a handler released can be canceled, and deadline recovery
|
|
308
|
+
terminalizes it. Both close it like never-started work (SM-1158, migration 0057).
|
|
309
|
+
|
|
310
|
+
Dashboard, from the shared browser bundle:
|
|
311
|
+
|
|
312
|
+
- Pausing a schedule asks for confirmation, as resuming one already did (SM-1147).
|
|
313
|
+
- The Events pager stops at the last page the server accepts and offers older events through a
|
|
314
|
+
custom range. The activity chart shows loading, error, and stale states with a Retry button
|
|
315
|
+
(SM-1171).
|
|
316
|
+
- The Schedules page says how many schedules past the first 50 it does not show, and the System
|
|
317
|
+
page lists the terminal cleanup backlog check (SM-1171, SM-1178).
|
|
318
|
+
|
|
319
|
+
### Migrations
|
|
320
|
+
|
|
321
|
+
Migration 0054 is the first since 0.6.1. The
|
|
322
|
+
[root changelog](https://github.com/stablemates/workhorse/blob/main/CHANGELOG.md) describes each in
|
|
323
|
+
full.
|
|
324
|
+
|
|
325
|
+
- **0054** adds versioned child functions and the nullable `task_child.last_seen_fence_token`
|
|
326
|
+
fence marker. Older clients keep their v1 functions (SM-1106).
|
|
327
|
+
- **0055** reserves a retry delay of -1 in `fail_v1` and `fast_fail_v1` as the terminal failure
|
|
328
|
+
override. It is the SDK compatibility floor, schema version 54 (SM-1107).
|
|
329
|
+
- **0056** changes `queue_health_v1` to count row retention lag from the releasing history pass
|
|
330
|
+
(SM-1134).
|
|
331
|
+
- **0057** changes `cancel_v1` and `terminalize_deadline_v1` to close a released task like
|
|
332
|
+
never-started work (SM-1158).
|
|
333
|
+
- **0058** lets terminal cleanup keep pace and share its batches across tiers, and adds two
|
|
334
|
+
`maintenance_state` columns (SM-1160).
|
|
335
|
+
- **0059** closes five SQL integrity gaps. It adds `rate_limit_policy_per_key_complete_check`,
|
|
336
|
+
`budget_rate_complete_check`, and a trigger that rejects a dependency edge update (SM-1165).
|
|
337
|
+
- **0060** judges `fast_complete_many_v1` and `fast_acknowledge_cancel_v1` at the time they act
|
|
338
|
+
(SM-1163).
|
|
339
|
+
- **0061** adds `fast_task_outcome_failed_finished_idx`, bounds `aggregate_stats_v1`, and spaces the
|
|
340
|
+
expired-lease scan in `tick_v1` with the new `maintenance_state.lease_recovery_started_at` column
|
|
341
|
+
(SM-1167).
|
|
342
|
+
- **0062** adds `dashboard_human_wait_quick_action_v1` and bounds four dashboard read functions
|
|
343
|
+
(SM-1171).
|
|
344
|
+
- **0063** raises the `terminal-cleanup-backlog` health reason and adds
|
|
345
|
+
`terminal_cleanup_follow_up_delay_ms_v1` (SM-1178).
|
|
346
|
+
- **0064** disables JIT for `aggregate_stats_v1` (SM-1193).
|
|
347
|
+
- **0065** gives `terminal-cleanup-backlog` its own budget in `queue_health_policy`, adds
|
|
348
|
+
`sync_queue_health_policy_v2`, and keeps `sync_queue_health_policy_v1` (SM-1197).
|
|
349
|
+
|
|
350
|
+
### Added
|
|
351
|
+
|
|
352
|
+
- `Queue#health` reports the backlog budget as `budgets.terminalCleanupBacklogMs` with schema 64,
|
|
353
|
+
and `status.reasons` can carry `terminal-cleanup-backlog` (SM-1178, SM-1197).
|
|
354
|
+
|
|
355
|
+
### Dependencies
|
|
356
|
+
|
|
357
|
+
- The gemspec declares `logger` at least 1.6 and below 2. Ruby 4.0 no longer ships `logger` as a
|
|
358
|
+
default gem. On Ruby 4.0, 0.6.1 raises `LoadError` at `require "stablemates/workhorse"` unless
|
|
359
|
+
the application already installs `logger` (SM-1138).
|
|
360
|
+
- The dashboard requires `cgi/escape` instead of `cgi`, which Ruby 4.0 reduces to a stub that warns
|
|
361
|
+
(SM-1138).
|
|
362
|
+
|
|
363
|
+
### Recipes
|
|
364
|
+
|
|
365
|
+
- A new tested recipe enqueues through Sequel on its native `pg` connection (SM-1114).
|
|
366
|
+
- The quick start and the agent playbook print their results as JSON (SM-1139), and a batch handler
|
|
367
|
+
example shows how batch-handler tasks are enqueued (SM-1135).
|
|
368
|
+
|
|
6
369
|
## 0.6.1 — 2026-10-02
|
|
7
370
|
|
|
8
371
|
The npm packages, Python distribution, Go module, Rust crate, and Ruby gem release from one source commit.
|
|
@@ -150,7 +513,7 @@ Requires **schema v43**, Ruby **3.3** or newer, and PostgreSQL **15** or newer.
|
|
|
150
513
|
renews the lease, `Worker#stop` still wakes the dispatcher, and a second signal still exits the
|
|
151
514
|
process. The first failure of each logger is written to standard error.
|
|
152
515
|
- Add the Active Job adapter, selected with `config.active_job.queue_adapter =
|
|
153
|
-
:stablemates_workhorse`. It requires Active Job 8.0 or later. A default job runs under the
|
|
516
|
+
:stablemates_workhorse`. It requires Active Job 8.0 or later. A default job runs under the
|
|
154
517
|
`active_job` task type. A typed job declares its task type with `workhorse_options` and carries
|
|
155
518
|
one JSON `Hash`, so another SDK can enqueue or run it. `Stablemates::Workhorse::ActiveJob.handle`
|
|
156
519
|
registers both formats on a `Worker`.
|
data/README.md
CHANGED
|
@@ -7,10 +7,7 @@ PostgreSQL.
|
|
|
7
7
|
> release may change behaviour, so read the
|
|
8
8
|
> [changelog](https://github.com/stablemates/workhorse/blob/main/ruby/CHANGELOG.md) before you
|
|
9
9
|
> upgrade. It will not ask you to recreate your database: migrations are ordered, and inside a major
|
|
10
|
-
> line a migration only adds, so a running deployment upgrades in place.
|
|
11
|
-
> migration 0025: a database from before 0.5.0 crosses it offline, with the
|
|
12
|
-
> [0.5.0 upgrade steps](https://github.com/stablemates/workhorse/blob/main/CHANGELOG.md#050--2026-09-28).
|
|
13
|
-
> The upgrade from 0.5 to 0.6 only adds.
|
|
10
|
+
> line a migration only adds, so a running deployment upgrades in place.
|
|
14
11
|
|
|
15
12
|
An AI agent should read [the Workhorse documentation index](https://workhorse.run/llms.txt) first.
|
|
16
13
|
|
|
@@ -26,15 +23,17 @@ bundle add connection_pool
|
|
|
26
23
|
Install the schema once, as a deployment step. The application never installs or migrates it.
|
|
27
24
|
|
|
28
25
|
```bash
|
|
29
|
-
npx --package @stablemates/workhorse@0.
|
|
26
|
+
npx --package @stablemates/workhorse@0.8.0 workhorse schema install
|
|
30
27
|
```
|
|
31
28
|
|
|
32
29
|
The machine that runs that deployment step needs Node.js 22 or newer. The application itself needs
|
|
33
30
|
no Node.js.
|
|
34
31
|
|
|
35
|
-
Pin that version to the gem version the application depends on. The two come from one commit of
|
|
36
|
-
|
|
37
|
-
|
|
32
|
+
Pin that version to the gem version the application depends on. The two come from one commit of this
|
|
33
|
+
repository, so the numbers match. The application refuses a schema below its schema floor, the
|
|
34
|
+
oldest schema version it runs against. A schema at or above the floor is compatible while it still
|
|
35
|
+
serves the application's protocol. A schema tool from an older release can leave a schema below that
|
|
36
|
+
floor.
|
|
38
37
|
|
|
39
38
|
Runtime processes verify compatibility instead of changing the schema. Call
|
|
40
39
|
`Queue#assert_compatible` or `Admin#assert_compatible` at startup. A refusal is
|
|
@@ -46,6 +45,7 @@ Requires Ruby 3.3 or newer and PostgreSQL 15 through 18.
|
|
|
46
45
|
|
|
47
46
|
```ruby
|
|
48
47
|
require "connection_pool"
|
|
48
|
+
require "json"
|
|
49
49
|
require "pg"
|
|
50
50
|
require "stablemates/workhorse"
|
|
51
51
|
|
|
@@ -62,7 +62,7 @@ end
|
|
|
62
62
|
worker.run_once # production uses Stablemates::Workhorse.run_worker_process(worker)
|
|
63
63
|
|
|
64
64
|
task = Stablemates::Workhorse::Admin.new(pool).get_task(enqueued.task_id)
|
|
65
|
-
puts "#{task.state} #{task.result}" if task
|
|
65
|
+
puts "#{task.state} #{JSON.generate(task.result)}" if task
|
|
66
66
|
```
|
|
67
67
|
|
|
68
68
|
Handlers receive at-least-once delivery. Use stable provider idempotency keys around external
|
|
@@ -70,6 +70,22 @@ effects. `ruby/examples/` also holds a transactional enqueue, a dedicated worker
|
|
|
70
70
|
orchestration of child tasks, signals, and human decisions. An integration spec runs every one of
|
|
71
71
|
them against PostgreSQL.
|
|
72
72
|
|
|
73
|
+
## Task groups, checkpoint inputs, and single passes
|
|
74
|
+
|
|
75
|
+
- A [task group](https://workhorse.run/docs/task-groups) tracks many tasks as one unit. After the
|
|
76
|
+
group is sealed and every member task is finished, Workhorse enqueues the callback tasks of the
|
|
77
|
+
group. The `Queue#task_group` block creates a group, adds its tasks, and seals it in one
|
|
78
|
+
transaction.
|
|
79
|
+
- A handler can give the [input of a
|
|
80
|
+
checkpoint](https://workhorse.run/docs/durable-execution#check-the-input-of-a-checkpoint). Give it
|
|
81
|
+
as `context.checkpoint(name, input: input) { ... }`. If a later run gives a different input, the
|
|
82
|
+
call raises `CheckpointInputConflictError`, the task fails, and the operation does not run again.
|
|
83
|
+
If the stored checkpoint or the call has no input, Workhorse compares nothing.
|
|
84
|
+
- `Worker#run_once` returns true only when a handler ran. It returns false after a pass that only
|
|
85
|
+
[released the tasks that it
|
|
86
|
+
claimed](https://workhorse.run/docs/workers#return-a-task-that-the-worker-cannot-run). If you call
|
|
87
|
+
it in a loop, wait after false.
|
|
88
|
+
|
|
73
89
|
## Package boundary
|
|
74
90
|
|
|
75
91
|
The gem is `stablemates-workhorse`, and its namespace is `Stablemates::Workhorse`.
|