stablemates-workhorse 0.12.0 → 0.15.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: f684dcd8a1e590688046e5b175ba97db4c4b62320d78c12353c5bf2fdf294ca4
4
- data.tar.gz: 748bf2d0d2cd17eb8717c1d5066f7add52c5c60895ffb5975038dac3803041ec
3
+ metadata.gz: ada675ecd38ee7f528c98894b840ae3efbb604f66055bbc405bd5e25a972b5c1
4
+ data.tar.gz: 7c800a1b76bdba58e2e9eb566814cd064a483c0dfe689ffd8b06b441850a1317
5
5
  SHA512:
6
- metadata.gz: 7c298a44ca80299142012bf23d118876cb036c207974625dd02bd49b262cad60eefaef0ea455d22153c4a052c69dbdbcd8788d3c896bac0c8c9cdd237751239b
7
- data.tar.gz: fffb3cbaa9f41f1f0efb59a6eff7f7fc4da4a840cf35b53def9a044ad3321cacbc380f0ddae14bf8209a789b61e8e16cfcddaccc106e723f2cd02171fa4b13d3
6
+ metadata.gz: 745a120d3e5ed782c595e512a9597c09a630b317b7004bf71e16677bcedede1f7d571b90dbaeda178adc07817880f4cd4e957fb1183ac3e2dc6ff3d35cbc2b44
7
+ data.tar.gz: 2ff9147d43d755d67dca231d5172f844c3fc810bebb0c70239d5e90e26df0ec3f77711f97b719c8d7d2fdbc641f9e13e48b0253b2fc8eba09be25c640118c646
data/CHANGELOG.md CHANGED
@@ -3,6 +3,487 @@
3
3
  `stablemates-workhorse` gem versions and release notes live here. The gem carries the version the
4
4
  other SDKs carry, because every tag names one release of all of them.
5
5
 
6
+ ## 0.15.0 — 2026-11-21
7
+
8
+ The npm packages, Python distribution, Go module, Rust crate, Ruby gem, JVM artifacts, and .NET
9
+ packages release from one source commit.
10
+
11
+ Requires **schema v99**, Ruby **3.3** or newer, and PostgreSQL **15** or newer.
12
+
13
+ ### Upgrade steps
14
+
15
+ **Migrate the schema, then roll out the new processes.** Run `workhorse schema migrate`. Migration
16
+ 0106 moves the schema from version 104 to version **105**. It replaces two function bodies and changes
17
+ no stored rows. The SDK compatibility floor stays at **99**, so a 0.14 process keeps working on
18
+ version 105, and the upgrade is a rolling deployment.
19
+
20
+ [The upgrade notes](../docs/upgrading.md#upgrade-from-014x-to-0150) give each step.
21
+
22
+ ### Cleanup no longer reads the whole key and dependency tables
23
+
24
+ Under a generic plan, `prune_enqueue_idempotency_v1` and `prune_released_dependencies_v1` read the
25
+ whole `enqueue_idempotency` or `task_dependency` table to delete the rows of one pass (issue 1263).
26
+ They now delete the rows that they locked by their `ctid`. The result does not change, and this SDK
27
+ does not change. Migration 0106 moves the schema to version 105, and the floor stays 99.
28
+
29
+ ## 0.14.0 — 2026-11-13
30
+
31
+ There is no 0.13.0. A task queue does not tempt fate.
32
+
33
+ The npm packages, Python distribution, Go module, Rust crate, Ruby gem, JVM artifacts, and .NET
34
+ packages release from one source commit.
35
+
36
+ Requires **schema v99**, Ruby **3.3** or newer, and PostgreSQL **15** or newer.
37
+
38
+ ### Upgrade steps
39
+
40
+ **Migrate the schema before starting updated processes.** Run `workhorse schema migrate` before any
41
+ process of this release starts, producers too. Migrations 0093 through 0105 move the schema from
42
+ version 91 to version **104**. The SDK compatibility floor rises from 76 to **99**, so a process of
43
+ this release refuses every mutation on a schema below version 99. A 0.12 process keeps working on
44
+ version 104, because a migration inside a major line only adds.
45
+
46
+ If you run cold exporters, stop each one before the migration and let it finish its uploads.
47
+ Migration 0096 fences each export segment that is exporting. It is the only migration that changes
48
+ stored rows.
49
+
50
+ Migration 0102 builds `task_runtime_active_queue_shard_idx` with `CREATE INDEX CONCURRENTLY`,
51
+ outside a transaction. Its time grows with the size of `task_runtime`. If the build fails,
52
+ PostgreSQL keeps an invalid index, and the schema stays at version 100. Drop the index with
53
+ `DROP INDEX CONCURRENTLY IF EXISTS workhorse.task_runtime_active_queue_shard_idx`, and run
54
+ `workhorse schema migrate` again.
55
+
56
+ [The upgrade notes](../docs/upgrading.md#upgrade-from-012x-to-0140) give each step and each change that needs a
57
+ change in application code.
58
+
59
+ ### Breaking changes
60
+
61
+ These changes are not additions. Change the application before you deploy this release:
62
+
63
+ 1. `Queue#enqueue`, `TaskGroupBuilder#enqueue`, `HandlerContext#run_child`, and the `options` of a
64
+ `ChildTaskRequest` raise `ArgumentError` for `task_type:` or `payload:`. See "Breaking: enqueue
65
+ options cannot name the task type or the payload".
66
+ 2. The Active Job adapter requires Active Job 8.1. See "The Active Job adapter requires Active Job
67
+ 8.1".
68
+
69
+ ### Behavior changes
70
+
71
+ These changes can make code that worked on 0.12 act differently:
72
+
73
+ 1. Each read of the embedded dashboard has a statement timeout. See "The embedded dashboard bounds
74
+ each read and drops undeclared input keys".
75
+ 2. The dashboard answers an exception of an application procedure with 500 and a fixed message. See
76
+ "The dashboard keeps the text of an application exception out of its answer".
77
+ 3. An enqueue refuses a queue name above 256 UTF-8 bytes, and a stored schedule with such a name no
78
+ longer fires. See "Queue purge, tier read, queue name, and claim lease use their locks".
79
+ 4. At repeatable read or serializable, an enqueue can fail with SQLSTATE `40001` after a tier change,
80
+ and `set_queue_tier_v1` refuses these levels with `0A000`. See "Queue purge, tier read, queue
81
+ name, and claim lease use their locks".
82
+ 5. Schedule synchronization refuses an enabled schedule with a concurrency key on a fast-tier queue,
83
+ with SQLSTATE `P1007`. See "One failed schedule no longer stops the schedule pass".
84
+ 6. A signal to a terminal task with no wait of that name answers `stale`, not `not_waiting`. See
85
+ "Three replay answers move to new function versions".
86
+ 7. The worker cuts a long failure envelope and adds `"truncated": true`. See "A failure envelope has
87
+ a size limit".
88
+ 8. An oversized checkpoint value and each child limit refusal fail the task at once, with no retry.
89
+ See "An oversized checkpoint value fails the task without a retry" and "Each child limit refusal
90
+ ends the task".
91
+ 9. Queue health reports the new degraded reasons `maintenance-failing`, `schedule-failing`, and
92
+ `terminal-cleanup-stalled`. Code that switches on reason codes should accept them.
93
+
94
+ ### Each failure in the shutdown unwind window releases its task
95
+
96
+ A Ruby worker stops, and at the end of the shutdown grace it cancels a handler with `:shutdown`.
97
+ Before, only a `CancelledError(:shutdown)` returned the task to its queue. Each other error, and
98
+ each result that the task could not store, failed the attempt through `fail_v1`. Thus a task on its
99
+ last attempt failed because its worker stopped. Now the cancellation decides, not the class of the
100
+ error. The worker reads the token when the handler returns, and a batch member reads it when the
101
+ batch handler returns. If the token holds `:shutdown` then, each `StandardError` releases the task
102
+ through `release_owned_v1`. A result that cannot be stored also releases it (SM-1933, ADR 0117). A
103
+ handler that fails before the cancellation keeps its usual settlement.
104
+
105
+ ### The Active Job adapter requires Active Job 8.1
106
+
107
+ Rails 8.0 reached the end of its security support on 2026-11-07, so the adapter now requires Active
108
+ Job 8.1 or later. With Active Job 8.0, selecting or requiring the adapter raises `LoadError`. An
109
+ application that uses only the native API still boots on Rails 8.0. An application that uses the
110
+ adapter must upgrade Rails to 8.1 before it takes this release, or stay on 0.12.
111
+
112
+ The compatibility table published that date. No earlier changelog announced the drop, which the
113
+ compatibility policy requires. ADR 0058 records this one exception: Rails no longer fixes 8.0, so
114
+ Workhorse does not extend its support.
115
+
116
+ ### Debounce and enqueue decide with the clock after their locks
117
+
118
+ Migration 0093 (`0093-decide-debounce-with-one-clock-and-keep-pending-debounce-keys.sql`) moves the
119
+ schema to version 92. A debounce request that waits for a lock past its window no longer writes a
120
+ second task, and it no longer deadlocks with run-now or a queue purge of its task. Maintenance keeps
121
+ the key of a pending debounced task, so a late request on a paused queue gets
122
+ `window_elapsed_pending`. An enqueue batch reads its timestamp after its locks. The migration does
123
+ not raise the SDK compatibility floor.
124
+
125
+ ### Queue purge, tier read, queue name, and claim lease use their locks
126
+
127
+ Migration 0094 (`0094-purge-locked-tasks-lock-stale-tiers-and-start-leases-after-locks.sql`) moves
128
+ the schema to version 93. A queue purge removes only the tasks that it locked, so a task that a
129
+ producer commits during the purge stays and its worker can complete it. An enqueue at repeatable
130
+ read or serializable fails with SQLSTATE `40001` after a newer tier change, and a tier change
131
+ outside read committed fails with SQLSTATE `0A000`. An enqueue to a queue name above 256 UTF-8
132
+ bytes now fails, and schedule synchronization refuses such a name. A stored schedule with such a
133
+ name no longer fires. A claim computes its lease from a clock sample taken after
134
+ its row locks. The migration does not raise the SDK compatibility floor.
135
+
136
+ ### A failed expiry request keeps the handler running
137
+
138
+ A task reached its deadline or its execution timeout, and the Ruby worker asked PostgreSQL to
139
+ expire the attempt. If that request failed, for example during a failover, the worker cancelled the
140
+ handler, wrote nothing for the attempt, and ended its run with the error. Now a failed request does
141
+ not confirm the expiry. The worker logs `workhorse.worker.expiration_request_failed` at warning
142
+ level, keeps the handler running, and asks again with the backoff that it uses for `not_due`. The
143
+ lease watchdog still ends an attempt that nobody renews (SM-1934).
144
+
145
+ ### One failed schedule no longer stops the schedule pass
146
+
147
+ Migration 0095 (`0095-isolate-each-schedule-in-a-pass-and-match-two-wildcard-day-fields.sql`) moves
148
+ the schema to version 94. Before it, one schedule whose fire failed in the enqueue rolled back the
149
+ schedule pass of every offered namespace, and the worker run ended. Now each schedule runs in a subtransaction. A
150
+ failed schedule keeps its position and stores its error in `last_error_at`, `last_error_code`, and
151
+ `last_error_message` of `schedule_definition`. The other schedules fire, and a later pass that fires
152
+ the schedule clears the error. An error of SQLSTATE class `40` or `57` still fails the whole pass.
153
+ Schedule synchronization refuses an enabled schedule with a concurrency key on a fast-tier queue,
154
+ with SQLSTATE `P1007`. The dashboard schedule read returns `lastErrorAt`, `lastErrorCode`, and
155
+ `lastErrorMessage`. A cron expression with `?` in day-of-week and a wildcard day-of-month now
156
+ matches each day. The migration does not raise the SDK compatibility floor (SM-1927, SM-1947,
157
+ ADR 0118).
158
+
159
+ ### Cold export waits for the transactions that can still write into a day
160
+
161
+ A history row gets its time when it is written, but other sessions see it only after its
162
+ transaction commits. Cold export gave out a day when the day had ended and the statistics rollup
163
+ had passed it. If a transaction wrote into the day and committed later, its row was in no export
164
+ object, and history retention later deleted it. `claim_cold_export_segment_v1` now waits for a
165
+ snapshot frontier. At least 60 s after the end of the day, the claim records a new transaction ID
166
+ in the new `cold_export_dataset` columns `frontier_day_end` and `frontier_xid`. It gives out the
167
+ day only after each transaction older than that ID has ended. A long write transaction in any
168
+ database of the server delays the export of a day until it ends. An exporter calls the claim in a transaction of its own. It
169
+ reads all pages of a day in one `REPEATABLE READ` transaction that starts after the claim. A re-lease passes the same frontier. The migration fences each segment that is exporting.
170
+ A completion from before the migration then fails, and the day is given out again. `send_signal_v1` and `complete_human_wait_v1`, when no wait exists, and
171
+ `record_batch_event_v1` take their transaction ID before they write their event (SM-1928,
172
+ ADR 0121).
173
+
174
+ After a stop of the statistics rollup of more than two days, each catch-up pass deleted some
175
+ minute buckets that it had just written. The hour and day tiers lost those counts. Statistics retention now keeps each minute bucket and each hour bucket until no later pass
176
+ rebuilds a coarser tier from it (SM-1978). A history retention pass that the rollup or the
177
+ export held back recorded `succeeded` and waited for the next local date. It now records
178
+ `incomplete` and runs again on the next offer (SM-1979). The statistics guide states that the
179
+ rollup does not count a history row whose transaction commits after the recompute window passed
180
+ its minute (SM-1959).
181
+
182
+ Migration 0096 (`0096-export-a-day-after-its-snapshot-frontier-and-keep-held-retention-due.sql`)
183
+ moves the schema to version 95. It adds the two nullable columns and
184
+ `cold_export_frontier_margin_ms_v1`, and replaces the bodies of `claim_cold_export_segment_v1`,
185
+ `send_signal_v1`, `complete_human_wait_v1`, `record_batch_event_v1`, `rollup_stats_v1`, and
186
+ `retain_history_v1`. It does not raise the SDK compatibility floor and changes only the exporting
187
+ rows of `cold_export_segment`. Run `workhorse schema migrate`.
188
+
189
+ ### The Active Job adapter checks no result contract
190
+
191
+ A typed Active Job task whose result contract refuses `null` ran its job body, and then the worker
192
+ failed the attempt, because the adapter handler returns `nil`. Each retry ran the job body again.
193
+ Now the worker does not check the result of an adapter handler against a result contract, as ADR
194
+ 0075 and `docs/parity.md` say. The task succeeds, and the job body runs one time (SM-1931). This
195
+ also applies to a default job under the `active_job` task type. A native Ruby handler keeps its
196
+ result check.
197
+
198
+ ### A pool that no worker uses can be collected
199
+
200
+ All the workers on one pool share one listener and one heartbeat connection. Their registries held
201
+ each pool, so a process that built and dropped many pools kept each of them. Now each registry is
202
+ an `ObjectSpace::WeakMap`. Workers on one live pool still share one listener and one heartbeat, and
203
+ a pool can be collected after its last worker stops (SM-1985).
204
+
205
+ ### The embedded dashboard bounds each read and drops undeclared input keys
206
+
207
+ An operator opened an events page with a wide range, a search string, and an exact count. The read
208
+ ran with no time bound and held one connection. Now `Stablemates::Workhorse::Dashboard` runs each
209
+ read in its own read-only transaction with a transaction-local `statement_timeout`. The default is
210
+ 15,000 ms, and the new `statement_timeout_ms:` keyword changes it. A read that PostgreSQL cancels
211
+ answers the `TIMEOUT` RPC error with status 408. A read on a connection inside a caller transaction
212
+ has no bound. Each statement and each read takes a lock of its connection, so threads that share
213
+ one connection cannot run a statement inside the read transaction of another thread. The dashboard also drops each input key that the `dashboard/v1` contract does not
214
+ declare, as the TypeScript host does (SM-1966, SM-2000).
215
+
216
+ ### Terminal cleanup and queue health use one retention rule
217
+
218
+ Terminal cleanup locked the oldest finished tasks, up to four times its limit, and only then removed
219
+ each task that a schedule occurrence, an enqueue key, a dependency edge, or a young child keeps.
220
+ When such tasks filled that window, each pass deleted nothing. Cleanup now locks only the tasks that
221
+ it can delete, so a kept task no longer stops the deletion of other tasks (SM-1935). Each batch
222
+ examines one slice of four times its share of the limit, at least 1,000 and at most 100,000 rows,
223
+ after a position that the new table `terminal_prune_position` stores. A second walk takes the batch after each batch that deleted tasks, and it
224
+ reaches the tasks behind that position. Thus many kept tasks do not make a batch
225
+ read more, and cleanup no longer deletes in strict age order. When a batch deletes the
226
+ last target of a redrive source and has room left, it deletes that source in the same batch,
227
+ and it continues down a redrive chain while it has room.
228
+ `retentionPruneStarved` still reports a pass that a dependency edge stops.
229
+
230
+ Queue health counted a task as `retention-lag` although cleanup keeps it on purpose. Such a task is
231
+ a redrive source with a target, a prerequisite of a live dependent, or a parent of a child that is
232
+ not yet eligible. It can also be a task of an unexpired schedule occurrence or enqueue key. Health
233
+ now skips these tasks, and the lag of such a task counts from the end of its hold. Health still
234
+ counts a task that a runtime row, a history row, or an expired occurrence keeps, because cleanup is
235
+ then late (SM-1852). When cleanup deletes an expired key or an edge to a finished dependent, it records the end of
236
+ that hold in the new column `task_outcome.held_until`, so health counts the task from it. If
237
+ another transaction holds the outcome of the task, the key or edge stays for a later pass. An
238
+ enqueue that reuses an expired key, the occurrence cleanup, a cleanup batch that deletes a
239
+ child, a dependent, or a redrive target, a queue purge, and the deletion of a finished task group
240
+ save the end of the hold in the same way. Health adds no reason for an old unfinished task group.
241
+
242
+ Each health poll read every task and every terminal outcome when no row was past its windows,
243
+ because the cutoffs read a volatile clock. A held history watermark and the members of finished task
244
+ groups also made the read grow. Health now reads the clock one time for its retention cutoffs. Each
245
+ retention walk stops at its cutoff, at the watermark, at the first row that cannot be due earlier,
246
+ and after 1,000 rows. Behind more kept rows than that, row lag is null, and
247
+ `terminal-cleanup-backlog` reports a late cleanup (SM-1936, SM-1990).
248
+
249
+ Each retention window of N days is now N times 24 hours. Before, PostgreSQL stepped calendar days in
250
+ the session TimeZone, so across a daylight-saving change cleanup and the lag moved by one hour. In a
251
+ UTC session nothing changes (ADR 0122).
252
+
253
+ **Accept the new `maintenance-failing` and `schedule-failing` health reasons.** Queue health reports
254
+ `maintenance-failing`, with a `routine` field, when three recorded runs of a maintenance routine
255
+ failed one after another after its last completion. It reports `schedule-failing` when an
256
+ enabled schedule stored an error in its last schedule pass. Both are degraded. Code that switches on
257
+ reason codes should accept them. A dashboard package older than this schema does not show the two
258
+ new reasons on its System page (SM-1992).
259
+
260
+ Migration 0097 (`0097-give-cleanup-and-health-one-retention-rule-in-fixed-hours.sql`) moves the
261
+ schema to version 96. It adds the table `terminal_prune_position`, the nullable column
262
+ `task_outcome.held_until`, and thirteen internal functions, and it
263
+ replaces the bodies of `queue_health_v1`, `evaluate_queue_health_v1`,
264
+ `prune_full_terminal_tasks_internal_v1`, `prune_fast_terminal_tasks_internal_v1`,
265
+ `prune_terminal_storage_v1`, `prune_released_dependencies_v1`, `retain_history_v1`,
266
+ `sync_retention_policy_v1`, `override_retention_policy_v1`, `prune_enqueue_idempotency_v1`,
267
+ `rollup_stats_v1`, `dashboard_retention_preview_v1`,
268
+ `enqueue_batch_v1`, `purge_queue_internal_v1`, `prune_schedule_occurrences_v1`, and
269
+ `maintain_task_groups_v1`. It rewrites no row and does not raise the
270
+ SDK compatibility floor. Run `workhorse schema migrate`.
271
+
272
+ ### Three replay answers move to new function versions
273
+
274
+ A Python, Go, or Rust worker parked a task until an absolute time with digits below the
275
+ millisecond. A TypeScript, Ruby, JVM, or .NET worker replayed the same instant at milliseconds, and
276
+ `schedule_wait_v1` answered `conflict`, so the task failed with no retry. The worker now calls
277
+ `schedule_wait_v2`, which compares the stored target and the sent target at millisecond precision
278
+ and stores the target as sent. A different millisecond still answers `conflict`. The Ruby
279
+ SDK sends a `Time` in milliseconds (SM-1945).
280
+
281
+ A signal to a task that failed, completed, or was canceled before it declared the wait answered
282
+ `not_waiting` and recorded a `signal_rejected` event each time. A sender that tried again had no
283
+ end condition, and the events kept the task out of terminal cleanup. The SDK now calls
284
+ `send_signal_v2`. For a terminal task with no wait of that name, it answers `stale` and records no
285
+ event. A live task that has not reached the wait still answers `not_waiting` (SM-1946).
286
+
287
+ A task holds one child set. A handler that asked for a second set got `ConflictError`, which says
288
+ that the handler code changed. The SDK now calls `create_children_v2`, which answers
289
+ `limit_exceeded` for a second set in the activation that created or joined the first set. The SDK
290
+ raises `LimitExceededError`, and the worker still fails the task with no retry. A second, different `run_children` call in one activation raised `ConflictError` before any statement. It now raises the same limit error. A changed set
291
+ from another activation still raises the conflict error. The other child limit refusals now end the task too, as
292
+ a later entry states (SM-1956).
293
+
294
+ A group callback that failed to enqueue at the finish left only a `callback_failed` event. The
295
+ group snapshot now shows the failure on that callback. `TaskGroupCallback` gains `failure`, a `TaskGroupCallbackFailure` with `code`, `message`, and `failed_at`, or `nil`. Retention deletes this
296
+ record with the group (SM-1955, ADR 0095).
297
+
298
+ Migration 0098 (`0098-compare-wait-targets-in-milliseconds-and-report-second-child-sets.sql`)
299
+ moves the schema to version 97. It adds `schedule_wait_v2`, `send_signal_v2`, and
300
+ `create_children_v2`, and replaces the body of `get_task_group_v1`. The `_v1` functions keep their
301
+ answers. With this migration, the SDK floor rises from 76 to 97. Migrations 0099 and 0100
302
+ raise it to 99. Run `workhorse schema migrate` before the first process of this release
303
+ starts.
304
+
305
+ ### The dashboard keeps the audit of each settings change
306
+
307
+ An operator decreased `taskEventRetentionDays` on the Settings page and gave a reason. The next
308
+ retention pass deleted the older history, and no table named the operator or the reason.
309
+ `Stablemates::Workhorse::Dashboard` now calls `override_retention_policy_v2`, `revert_retention_policy_v2`,
310
+ `override_maintenance_policy_v2`, or `revert_maintenance_policy_v2`. Each records the authenticated
311
+ actor, the reason, a preview and a digest of the request ID, and the time for each changed setting
312
+ in the new nullable `operator_audit` column of `retention_policy` or `maintenance_policy`. The
313
+ `settings` read shows it as `lastChanges.<setting>` of each policy, and the Settings page shows who made
314
+ the last change and why. A revert keeps its own entry. `override_*_policy_v1`, `revert_*_policy_v1`,
315
+ and each `sync_*_policy_v1` remove the entries of the settings whose value they change, so a setting
316
+ never names an actor who did not set its value (SM-1996).
317
+
318
+ ### The dashboard can withhold error stacks and opens each derived record
319
+
320
+ `redact_error_stacks:` of `Stablemates::Workhorse::Dashboard` removes the stack of each worker error from task detail, the event feed, and event
321
+ detail, also from `details.error` of a `failed` or `retry_scheduled` event. It is off by default.
322
+ `dashboard_task_detail_v1`, `dashboard_events_v1`, and `dashboard_event_detail_v1` do the
323
+ redaction from the `redactErrorStacks` input key. The host sets the key after it drops the
324
+ undeclared input keys, so a request cannot turn the redaction off, and the event search does not
325
+ match a withheld stack (SM-1968).
326
+
327
+ A record that the dashboard derives for a fast-tier task has an MD5 identity, and event detail
328
+ refused almost each one with an input validation error. `eventDetail` now accepts any UUID in
329
+ canonical form, and `dashboard_event_detail_v1` finds the derived record (SM-1999). The comment
330
+ and the architecture page state that a `tasks` request with no tag, queue, or task type reads each
331
+ task (SM-1961).
332
+
333
+ Migration 0099 (`0099-keep-the-settings-audit-withhold-error-stacks-and-find-derived-records.sql`)
334
+ moves the schema to version 98. It adds the two nullable columns, the four audited functions, and
335
+ three helper functions, and replaces the bodies of the six released policy writers and of five
336
+ dashboard functions. It changes no rows. The dashboard host calls the audited functions, so the
337
+ SDK compatibility floor rises from 97 to 98. Run `workhorse schema migrate` before the new
338
+ processes start.
339
+
340
+ ### An oversized checkpoint value fails the task without a retry
341
+
342
+ A checkpoint value above 1,048,576 bytes of jsonb text made PostgreSQL raise a plain exception, and
343
+ the worker retried the task. The operation of the checkpoint runs before the save. Thus each retry ran
344
+ the operation again, with its effects, and no attempt could store the value. The worker now calls
345
+ `save_checkpoint_v3`, which answers `value_too_large` and stores nothing. The error
346
+ `Stablemates::Workhorse::CheckpointValueSizeLimitError` gives the checkpoint name, the measured bytes, and the limit. The worker fails the
347
+ task at once, also when the handler wraps the error. Correct the handler, and then redrive the task. The
348
+ limit counts the jsonb text, which has a space after each `,` and `:`, so only PostgreSQL measures it
349
+ (SM-1982).
350
+
351
+ ### A failure envelope has a size limit
352
+
353
+ A handler error with a very large message, such as a full response body, was stored as it came, and
354
+ a retry wrote it three times. The worker now cuts the `name`, the `message`, and the `stack` of the
355
+ failure envelope to 1,024, 16,384, and 65,536 UTF-8 bytes. Each cut is at a character boundary.
356
+ When the worker cuts a field, it adds `"truncated": true`. It sends the envelope to the new `fail_v2`, which applies
357
+ the same cut. Thus a client that does not cut cannot store a longer field (SM-1983). Migration 0100
358
+ adds these functions, and the schema floor rises to 99. `fail_v1`, `fast_fail_v1`, and
359
+ `save_checkpoint_v2` keep their answers.
360
+
361
+ ### Each child limit refusal ends the task
362
+
363
+ A handler replays the same child calls on each attempt, so a child limit refusal came again on each
364
+ retry. Each retry also ran again the work that no checkpoint holds, until `maxAttempts`. The worker
365
+ now fails the task at the first attempt with no retry, and keeps the current attempt. This applies
366
+ to a second single child, a single child after a child set, a set of more than 100 children, and a
367
+ child join whose results are larger than the result limit of the task. A second child set already
368
+ ended the task. The error is `LimitExceededError` with the operation `:run_child` or `:run_children`, or `ChildResultLimitExceededError` for a join that is too large, and the failure
369
+ envelope records the name of its type. The worker finds the error also when the handler wraps it. If a
370
+ handler relied on a retry after such a refusal, correct its calls, and then redrive the task
371
+ (SM-2025).
372
+
373
+ ### The expired-lease scan runs at most once in 500 ms
374
+
375
+ `tick_v1` ran the expired-lease scan of the full tier when no tick ran it within half the shortest
376
+ maintenance interval of the live registered workers. The scan reads every active lease. Thus one
377
+ worker with a `maintenance_interval:` of 0.1 s made the fleet scan every 50 ms. Its ticks held the
378
+ tick lock for almost all the time, and the ticks of the other workers skipped (SM-1960). The gate
379
+ interval now has a floor of 1,000 ms, the default interval, so the scan runs at most once in 500 ms
380
+ while a live worker is registered. If you set an interval below 1,000 ms to find expired leases
381
+ sooner, the full tier now finds an expired lease up to 500 ms plus that interval after it expires. A
382
+ short interval still makes that worker tick, promote, and recover deadlines, attempt timeouts, and
383
+ fast-tier leases at its own rate. A fleet at the default interval or above scans as before.
384
+
385
+ ### A capacity release counts one admission shard through an index
386
+
387
+ When an active task left a queue with a concurrency policy, the release trigger counted the active
388
+ leases of its admission shard with a filter that no index served. It thus read almost every active
389
+ lease of the queue (SM-1991). The count now reads only the leases of that shard through the new
390
+ index `task_runtime_active_queue_shard_idx`, and it gives the same answers. Migration 0101 changes
391
+ the two function bodies, and migration 0102 builds the index with `CREATE INDEX CONCURRENTLY`,
392
+ outside a transaction. The build does not block writes, but its time grows with the size of
393
+ `task_runtime`. Each claim on the full tier writes one more index entry, also on a queue with no
394
+ concurrency policy. In return, a release on a queue with a policy reads the leases of one shard, not
395
+ of the whole queue. A heartbeat stays a HOT update. The schema moves to version 101, and the floor
396
+ stays 99. If the build fails, drop the invalid index with `DROP INDEX CONCURRENTLY IF EXISTS
397
+ workhorse.task_runtime_active_queue_shard_idx` and run `workhorse schema migrate` again.
398
+
399
+ ### Breaking: enqueue options cannot name the task type or the payload
400
+
401
+ A wrapper called `queue.enqueue("send-invoice", payload, **options)`, and `options` held
402
+ `task_type: "refund"` from a configuration file. Ruby keeps the last value of a repeated keyword,
403
+ so Workhorse enqueued a `refund` task. Now `Queue#enqueue`, `TaskGroupBuilder#enqueue`,
404
+ `HandlerContext#run_child`, and the `options` of a `ChildTaskRequest` raise `ArgumentError` for
405
+ `task_type:` or `payload:` before any statement. The message names the keyword and says to use the
406
+ positional argument. `HandlerContext#run_child` now checks its whole request before it reads the
407
+ queue tier, as `run_children` does. `EnqueueRequest.new` keeps both keywords. `enqueue_many`,
408
+ `add_to_task_group`, the Active Job adapter, and `workhorse_options(task_type:)` do not change. To
409
+ upgrade, move the task type or the payload from the options into the positional arguments
410
+ (SM-2039).
411
+
412
+ ### The dashboard keeps the text of an application exception out of its answer
413
+
414
+ If an application procedure of `Dashboard`, from `procedures:`, `enqueue_test:`, or
415
+ `set_schedule_paused:`, raises an error other than `RpcError`, the dashboard now answers 500
416
+ `INTERNAL_SERVER_ERROR` with the message `Internal server error`. This includes `ArgumentError` and
417
+ `JSON::ParserError`. An `ArgumentError` that a driver or a library raises in a built-in procedure
418
+ gets the same answer. The dashboard writes each procedure failure to `rack.errors`. Input
419
+ validation errors, JSON parser errors of the request, the refusals that Workhorse writes, and
420
+ `RpcError` keep their answers (SM-2043).
421
+
422
+ ### An enqueue call takes its locks in one order
423
+
424
+ Migration 0103 (`0103-take-enqueue-locks-in-key-tier-row-order.sql`) moves the schema to version 102. A batch that replaces a debounced task and names a dependent of that task as a prerequisite no
425
+ longer deadlocks with a cancel or a rejecting completion of the task. At repeatable read or
426
+ serializable, a group cancel and a keyed group add no longer deadlock with a purge or an enqueue
427
+ that creates the same `queue_control` row. A few cases can still end one transaction with `40P01`,
428
+ as the root changelog states, and that transaction can run again. At repeatable read, a group cancel
429
+ and the completion of the last member of the group can now end one transaction with `40P01`, where
430
+ the cancel failed with `40001` before. A read committed enqueue takes no new lock. The migration
431
+ does not raise the SDK compatibility floor.
432
+
433
+ ### The embedded dashboard sends a signal through `send_signal_v2`
434
+
435
+ An operator sent a signal from the dashboard to a task that ended with no wait of that name. The
436
+ embedded dashboard called `send_signal_v1`, so it answered `not_waiting` and recorded a
437
+ `signal_rejected` event each time. The SDK `Queue` already called `send_signal_v2`. The dashboard
438
+ host now calls `send_signal_v2` too. For such a task, it answers `stale` and records no event. A
439
+ live task that has not reached the wait still answers `not_waiting`. The task drawer says that the
440
+ task or its wait has ended (SM-2026).
441
+
442
+ ### The embedded dashboard shows a group callback that failed to enqueue
443
+
444
+ A finished group that failed to enqueue a callback showed that callback with no task and no
445
+ reason. The `taskGroupDetail` and `cancelTaskGroup` answers now add `failure` to such a callback,
446
+ with `code`, `message`, and `failedAt`. The key is optional and new in `dashboard/v1`, an additive
447
+ change. The group list does not carry it. A schema below version 97 returns no `failure`. The group
448
+ detail page shows the failure as text (SM-2027).
449
+
450
+ ### Queue health keeps the end of each hold, also on the fast tier
451
+
452
+ An enqueue that reuses an expired key, and a queue purge, never wait for the outcome of the old
453
+ task. When another transaction held that outcome, they dropped the end of the hold, and health
454
+ counted the old task from its windows (SM-2022). A fast-tier task had the same result after the key
455
+ phase deleted its expired key, because `fast_task_outcome` has no `held_until` (SM-2024). Each such
456
+ end now goes to the new table `terminal_hold_end`, whose insert takes no row lock, and health counts
457
+ the task from it. The cold export rows of `fast_task_outcome` do not change.
458
+
459
+ ### A health poll does not read the same held rows again, and a stalled cleanup is reported
460
+
461
+ Behind more than 1,000 rows that a hold keeps on purpose, each health poll read those rows again
462
+ (SM-2021). A poll now stores where they end, and the next poll starts there while none of them can
463
+ become eligible. A poll writes only at read committed in a writable transaction, and it waits for no lock. The new degraded
464
+ reason `terminal-cleanup-stalled` reports terminal cleanup that did not complete a pass for longer
465
+ than `row_retention_lag_ms` plus `terminal_cleanup_interval_ms` while finished tasks exist (SM-2019).
466
+ Before a first completion, it waits until a finished task is past both row windows. Code that switches on reason codes should accept it.
467
+
468
+ ### Terminal cleanup tests each array of its hold code once
469
+
470
+ The hold code of terminal cleanup and the queue purge tested batch membership with
471
+ `NOT x = ANY(array)`, which a generic plan evaluates once for each row (SM-2023). Each such test now
472
+ takes one sort or one hash over the batch, and the results do not change. Migration 0104 moves the
473
+ schema to version 103. It adds two tables, rewrites no row, and does not raise the floor. Run
474
+ `workhorse schema migrate`.
475
+
476
+ ### Event detail reads only the history of its task
477
+
478
+ The dashboard derives the history records of a fast-tier task, and no index
479
+ holds their identities. Thus `dashboard_event_detail_v1` read each row of `fast_task_runtime` and
480
+ `fast_task_outcome` to find one record (SM-2029). The page now sends the task of the record as the new optional
481
+ `taskId` key of `eventDetail`, and the embedded dashboard accepts it and passes it to
482
+ `dashboard_event_detail_v1`. The function then reads only the history rows of that task, and a
483
+ record of another task returns `NOT_FOUND`. A request without `taskId` gets the same answer as
484
+ before. The host works on schema version 103 and earlier, which ignore the key. Migration 0105 moves the
485
+ schema to version 104, and the floor stays 99.
486
+
6
487
  ## 0.12.0 — 2026-11-07
7
488
 
8
489
  The npm packages, Python distribution, Go module, Rust crate, Ruby gem, JVM artifacts, and .NET
data/README.md CHANGED
@@ -15,6 +15,8 @@ PostgreSQL.
15
15
  > [changelog](https://github.com/stablemates/workhorse/blob/main/ruby/CHANGELOG.md) before you
16
16
  > upgrade. It will not ask you to recreate your database: migrations are ordered, and inside a major
17
17
  > line a migration only adds, so a running deployment upgrades in place.
18
+ > Migration 0025 is the one exception. If your database is from a release before 0.5.0, upgrade it
19
+ > offline with [these steps](https://workhorse.run/docs/installation#upgrade-a-04x-database-offline).
18
20
 
19
21
  An AI agent should read [the Workhorse documentation index](https://workhorse.run/llms.txt) first.
20
22
 
@@ -30,7 +32,7 @@ bundle add connection_pool
30
32
  Install the schema once, as a deployment step. The application never installs or migrates it.
31
33
 
32
34
  ```bash
33
- npx --package @stablemates/workhorse@0.12.0 workhorse schema install
35
+ npx --package @stablemates/workhorse@0.15.0 workhorse schema install
34
36
  ```
35
37
 
36
38
  The machine that runs that deployment step needs Node.js 22 or newer. The application itself needs
@@ -3,8 +3,8 @@
3
3
  require "active_job"
4
4
  require "stablemates/workhorse" unless defined?(::Stablemates::Workhorse::Worker)
5
5
 
6
- unless ::ActiveJob.gem_version >= Gem::Version.new("8.0")
7
- raise LoadError, "the Workhorse Active Job adapter requires Active Job 8.0 or later " \
6
+ unless ::ActiveJob.gem_version >= Gem::Version.new("8.1")
7
+ raise LoadError, "the Workhorse Active Job adapter requires Active Job 8.1 or later " \
8
8
  "(found #{::ActiveJob.gem_version})"
9
9
  end
10
10
 
@@ -78,11 +78,16 @@ module Stablemates
78
78
  typed[task_type] = job_class
79
79
  end
80
80
  admin = Admin.new(worker.executor)
81
- worker.handle(TASK_TYPE) { |payload, context| running(worker, context) { run_default(payload, context.task) } }
81
+ # The adapter stores no result, so no result contract applies to its task types (ADR 0075).
82
+ default = Worker::UncheckedResultHandler.new do |payload, context|
83
+ running(worker, context) { run_default(payload, context.task) }
84
+ end
85
+ worker.handle(TASK_TYPE, &default)
82
86
  typed.each do |task_type, job_class|
83
- worker.handle(task_type) do |payload, context|
87
+ handler = Worker::UncheckedResultHandler.new do |payload, context|
84
88
  running(worker, context) { run_typed(job_class, payload, context.task, admin) }
85
89
  end
90
+ worker.handle(task_type, &handler)
86
91
  end
87
92
  worker
88
93
  end
@@ -172,8 +172,11 @@ module Stablemates
172
172
 
173
173
  live!(:checkpoint)
174
174
  value = block.call
175
- row = write(SqlCatalogue::SAVE_CHECKPOINT_V2, name, Values.json(value, "checkpoint value"), fingerprint)
175
+ row = write(SqlCatalogue::SAVE_CHECKPOINT_V3, name, Values.json(value, "checkpoint value"), fingerprint)
176
176
  check_input(name, row["input_fingerprint"], fingerprint) if row.fetch("status") == "input_conflict"
177
+ if row.fetch("status") == "value_too_large"
178
+ raise CheckpointValueSizeLimitError.new(name, Integer(row.fetch("value_bytes")), Integer(row.fetch("max_bytes")))
179
+ end
177
180
  status = expect(:checkpoint, row, %w[saved existing], name)
178
181
  log(:debug, "workhorse.task.checkpoint_saved", "Task checkpoint persisted",
179
182
  "workhorse.checkpoint.name" => name, "workhorse.checkpoint.status" => status)
@@ -199,6 +202,8 @@ module Stablemates
199
202
  raise ArgumentError, "wait time must be no more than 365 days in the future" if
200
203
  (time - Time.now) * 1000 > MAX_WAIT_MS
201
204
 
205
+ # The protocol text cuts the time to milliseconds with %L, so a target of .1239996 s sends
206
+ # .123, also before 1970. schedule_wait_v2 compares at milliseconds.
202
207
  schedule_wait(name, nil, Values.timestamp(time, "wait time"))
203
208
  end
204
209
 
@@ -256,6 +261,8 @@ module Stablemates
256
261
  # dependency options.
257
262
  def run_child(name, task_type, payload, **enqueue_options)
258
263
  child_name(name)
264
+ # The request is checked before the deferred tier read, as each child of a child set is.
265
+ @queue.check_child_request(@task, task_type, payload, enqueue_options)
259
266
  durable!("child tasks")
260
267
  build = lambda do |versions = {}|
261
268
  canonical_json(@queue.serialize_child_request(@task, task_type, payload, enqueue_options, {},
@@ -330,7 +337,7 @@ module Stablemates
330
337
  def schedule_wait(name, duration_ms, wake_at)
331
338
  once(:sleep, name, wake_at.nil? ? :relative : "absolute:#{wake_at}") do
332
339
  live!(:sleep)
333
- row = write(SqlCatalogue::SCHEDULE_WAIT_V1, name, duration_ms, wake_at)
340
+ row = write(SqlCatalogue::SCHEDULE_WAIT_V2, name, duration_ms, wake_at)
334
341
  status = row.fetch("status")
335
342
  case status
336
343
  when "stale" then raise LeaseLostError.new(@task.id, :sleep)
@@ -372,12 +379,14 @@ module Stablemates
372
379
  encoded = initial(build)
373
380
  once(:run_children, :set, "#{mode}:#{encoded}", "child set") do
374
381
  live!(:run_children)
375
- row = write(SqlCatalogue::CREATE_CHILDREN_V1, encoded, mode)
376
- row = replayed(row, encoded, build) { |accepted| write(SqlCatalogue::CREATE_CHILDREN_V1, accepted, mode) }
382
+ row = write(SqlCatalogue::CREATE_CHILDREN_V2, encoded, mode)
383
+ row = replayed(row, encoded, build) { |accepted| write(SqlCatalogue::CREATE_CHILDREN_V2, accepted, mode) }
377
384
  status = row.fetch("status")
378
385
  case status
379
386
  when "stale" then raise LeaseLostError.new(@task.id, :run_children)
380
387
  when "conflict" then raise ConflictError.new(:run_children, "child set")
388
+ # The set has at most MAX_CHILDREN children, which the check above proves, so the limit
389
+ # is the second child set of this activation.
381
390
  when "limit_exceeded" then raise LimitExceededError.new(:run_children, "child set")
382
391
  when "result_too_large"
383
392
  raise ChildResultLimitExceededError.new(Values.parse_integer(row["result_bytes"]).to_i,
@@ -472,6 +481,9 @@ module Stablemates
472
481
  if overlap
473
482
  overlap.call(current[0])
474
483
  elsif current[0] != request
484
+ # A task holds one child set, so a second set in the same activation is a limit.
485
+ raise LimitExceededError.new(:run_children, "child set") if kind == :run_children
486
+
475
487
  raise ConflictError.new(kind, conflict_name)
476
488
  end
477
489