stablemates-workhorse 0.7.1 → 0.9.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e72b7d14542014971d81d818e073acaadbaef569c2516ba66427583e607fd0f3
-  data.tar.gz: 0e1d31974f65086473c750137249edd45722d75aa3b630e6c0a4c6c66bf7eb2e
+  metadata.gz: b030c71db8e2fa0f3b2e14e4ac34442270e1643ea1105efe904bc965ab6e4823
+  data.tar.gz: 90256bbd761dfa37265e9e0fce4f5f49534e378361a625bbc6bc27cf24b7634b
 SHA512:
-  metadata.gz: 3de1cc0e31810df55903ec87b96e553edc96bfa2e7a699e6b926b669a995d6231ae03d5bd1ccfa62c8908d1ed6095e3a775998d092ac27b28fd92c8c9288cd20
-  data.tar.gz: 4683b2cebb636655c2c5c608d56059cedb5e6d837c0eb96735443aa78d2abad27371c3231737ae5be3d8cdfa504bf179a468f70cd15a038cf4516f0388e02459
+  metadata.gz: 9d580a9360fe1772b51567673be3d787ea287ba7df8693d93d8ea4b153620f0a4c0b4c5ad492593830127ccd2c78a1063b5ac5f9174bbbea6aae3f1ca2880b70
+  data.tar.gz: d353d8e0ce3d2b03b869212dee916bcead4c41034de9403c83ab1f3e3a74eaaadfa36bf0c0c0c03f22728efdee770ef437c3de369f51d1f36765f5861948679f

data/CHANGELOG.md

@@ -3,6 +3,323 @@
 `stablemates-workhorse` gem versions and release notes live here. The gem carries the version the
 other SDKs carry, because every tag names one release of all of them.
 
+## 0.9.0 — 2026-10-20
+
+The npm packages, Python distribution, Go module, Rust crate, Ruby gem, and JVM artifacts release from one source commit.
+
+Requires **schema v73**, Ruby **3.3** or newer, and PostgreSQL **15** or newer.
+
+### Upgrade steps
+
+**Migrate the schema before starting updated processes.** Run `workhorse schema migrate` before any
+process of this release starts, producers too. Migrations 0073 and 0074 move the schema to version
+**73**. The SDK compatibility floor rises from 69 to **73**, because every SDK statement catalogue
+now names `heartbeat_many_v2` and `heartbeat_v2`. An updated process refuses a schema below version
+73 at startup. A 0.8 process keeps working on version 73, because a migration inside a major line
+only adds.
+
+### Heartbeats do not wait for a locked task row
+
+Before this release, a heartbeat waited for a task row that another transaction locked. If a
+settlement held the row, the heartbeat then kept the row locked after the settlement committed, and
+a claim skipped the ready task (SM-1434). The worker now renews leases through `heartbeat_many_v2`
+(SM-1443, ADR 0105). It skips a locked row and reports that lease as `busy`: no renewal and no
+ownership verdict. The worker keeps a busy lease, retries only the busy leases after a short delay
+with jitter, and does not move the regular heartbeat schedule. A busy answer never extends the local
+lease watchdog, so a lease that stays busy until the watchdog ends is lost, as before. Old and new
+workers can run together: an old worker keeps calling `heartbeat_many_v1`, which does not change.
+
+**One-lease heartbeat without a row lock wait.** Migration 0074 adds `heartbeat_v2` (SM-1461). It
+renews one lease by the rule of `heartbeat_many_v2`, so it reports a locked row as `busy`. This SDK
+names it in its statement catalogue but does not call it. `heartbeat_v1` does not change.
+
+### Heartbeat rounds keep a renewal margin and end on the server
+
+Before this release, the dedicated heartbeat connection bounded each heartbeat statement by one full
+interval. After a failed round, the heartbeat thread started a new schedule on its new connection.
+With the defaults, one timed-out round thus left about 0 s before the lease watchdog expired
+(SM-1444). Also, when the worker stopped waiting for a round, PostgreSQL continued to run the
+heartbeat statement, and that statement kept its row locks (SM-1442). These changes apply to each
+round on the dedicated heartbeat connection:
+
+- **Rounds start on a fixed schedule.** A round falls due one interval after the previous round
+  started, not after it ended. The schedule uses a monotonic clock. If a round ends after its next
+  slot, the worker starts one overdue round at once and sends no round for the other missed slots.
+- **The round bound is shorter.** A round ends by
+  `max(1 ms, interval × 3 / 4, min(50 ms, interval))`. For an interval of 200 ms or more, this is
+  three quarters of the interval. With the defaults (lease 30 s, interval 10 s), the bound is 7.5 s,
+  so one timed-out round leaves a margin of 2.5 s. An attempt keeps its lease after one timed-out
+  round if `2 × interval + round bound < lease`. The worker does not reject an interval that breaks
+  this rule. Before this release, each statement of a round had its own bound of one interval. Now
+  all statements of one round share one deadline, and a statement that would start after it waits
+  for the next round. Each round starts with the next statement in turn, so a statement that is
+  always slow cannot keep a later statement from its turn.
+- **Each round sets server timeouts.** The round sends `BEGIN`, then one `set_config` statement for
+  `statement_timeout` and `idle_in_transaction_session_timeout`, then `heartbeat_many_v2`, then
+  `COMMIT`. Both settings are transaction-local. The heartbeat statement gets one half of the time
+  that remains, and each idle gap one eighth. If less than 8 ms remains at the settings statement,
+  the round fails before the heartbeat. The worker counts a renewal only after a `COMMIT` that ends
+  before the deadline. No setting bounds `COMMIT` on PostgreSQL 15 and later.
+- **A failed round discards its connection only when a statement may still run** (SM-1492). A
+  round that expires before it sends the heartbeat statement rolls back and keeps its connection.
+  Under load, PostgreSQL can log
+  `canceling statement due to statement timeout` for a heartbeat statement. It can also end the
+  session with `terminating connection due to idle-in-transaction timeout`. Each message means that
+  one round failed. Every task continues, and the next round tries again. A deadlock (`40P01`) also
+  fails the round.
+- **A busy retry with less than 20 ms left sends nothing** (SM-1492). A bounded busy retry runs
+  only if its deadline is at least 20 ms after its start: the 8 ms that a round needs at its
+  settings statement, plus a 12 ms margin. Otherwise the next regular round renews the busy leases.
+- **Replacement connections.** The heartbeat thread keeps the start of the latest round when it
+  replaces a failed connection.
+- **100 ms is the smallest supported heartbeat interval** (SM-1471). This applies to `heartbeat:` of
+  `Worker`, in seconds: the smallest supported value is 0.1. At 100 ms, the round bound is 75 ms.
+  The worker still accepts a shorter interval, and it does not warn. On a loaded host, a shorter
+  interval can make each round miss its deadline, and attempts can then lose their leases.
+
+Rounds under `shared_heartbeats: true` have no client deadline and set no server timeouts, as
+before. They follow the new schedule. The [heartbeat round
+deadline](../docs/architecture/lifecycle.md#heartbeat-round-deadline) and the [heartbeat
+schedule](../docs/architecture/lifecycle.md#heartbeat-schedule) state each rule.
+
+### A payload contract compares the decimals of numbers
+
+Before this release, `ContractSchema` compared an Integer with the binary value of a Float in the
+bound keywords, `const`, `enum`, and `uniqueItems` (SM-1539). Thus `1152921504606846980` failed
+`{"maximum": 1.152921504606847e18}`, although PostgreSQL stores the constant as
+`1152921504606847000`. A contract now compares each Float through the decimal of `Float#to_s`
+(ADR 0039). Ruby still reads a stored schema constant that a double cannot hold as a Float
+(SM-1541).
+
+### A bad worker interval raises ArgumentError
+
+Before this release, a bad value for `heartbeat`, `poll_interval`, `maintenance_interval`,
+`maintenance_routine_interval`, or `registry_interval` raised `NoMethodError`, because these options
+have no upper limit (SM-1511). For example, `heartbeat: 0` did this. Now `Worker.new` raises
+`ArgumentError`, and the message names the option and its lower limit.
+
+### The `workhorse.tasks.retried` description matches `api/telemetry.txt`
+
+The Ruby worker now describes `workhorse.tasks.retried` as "Failed or expired tasks returned to live
+work", as the contract does (SM-1514). The counter already counted expired tasks. Only the
+description changes.
+
+### A wrapped durable replay conflict is terminal
+
+Before this release, the Ruby worker checked only the outermost error for a durable replay conflict
+(ADR 0098). If a handler wrapped a conflict in its own error, the worker recorded a retryable
+failure, and the conflict repeated until `max_attempts` (SM-1512). The worker now follows the cause
+chain of the error, as the Go and Rust workers do. It reads at most 16 errors, so a cyclic chain
+cannot stop it. It reads `Exception#cause`, so an error raised in a `rescue` block also counts. A
+wrapped conflict fails the task at once. The failure envelope records the name of the conflict. Its
+fields `message` and `stack` stay those of the outer error.
+
+### A heartbeat round sends at most 100 leases in each statement
+
+`heartbeat_many_v2` accepts at most 100 leases. A fused completion claim registers its tasks before
+the completing attempts unregister, so a round can hold more leases than the concurrency limit. That
+round raised an error, and no lease in it renewed (SM-1495). The worker now splits a round into
+statements of at most 100 leases. Each statement counts on its own. A failed statement renews none
+of its leases, and the statements before it still count, as before for the statements of each
+worker.
+
+## 0.8.0 — 2026-10-13
+
+The npm packages, Python distribution, Go module, Rust crate, and Ruby gem release from one source commit.
+
+Requires **schema v69**, Ruby **3.3** or newer, and PostgreSQL **15** or newer.
+
+### Upgrade steps
+
+**Migrate the schema before starting updated processes.** The upgrade is a rolling deployment:
+
+1. Run `workhorse schema migrate`. The final schema version is **71**.
+2. Roll out the updated processes.
+
+The SDK compatibility floor is schema version **69**, which migration 0070 reaches. The floor rises
+from 54. Every SDK statement catalogue now names the task group functions that migration 0066 adds,
+and `save_checkpoint_v2`, which migration 0069 adds. The dashboard of each SDK calls
+`dashboard_task_groups_v1`, which migration 0067 adds, and `list_newest_task_group_events_v1`, which
+migration 0068 adds. Migration 0070 fences a member add, so an updated process does not add tasks to
+a group without that check. An updated process refuses a schema below version 69 at startup. A 0.7
+process keeps working on version 71, because a migration inside a major line only adds.
+
+**Give the eighth member when you build a `TaskCheckpoint`.** `TaskCheckpoint` is a `Data` class.
+0.8.0 adds `input_fingerprint` as its eighth member (SM-1264). A call to `TaskCheckpoint.new` with
+seven positional arguments raises `ArgumentError`. A call with keywords and no
+`input_fingerprint:` also raises it. The gem has no compatible constructor. If your code builds a
+`TaskCheckpoint`, for example in a test, give the eighth value. For a checkpoint without an input,
+give `input_fingerprint: nil` or a positional `nil`.
+
+Migration 0066 (`0066-add-task-groups-with-completion-callbacks.sql`) adds task groups (SM-1231,
+[ADR 0095](https://github.com/stablemates/workhorse/blob/main/docs/decisions/0095-add-task-groups-with-completion-callbacks.md)).
+A task group is a set of full-tier tasks that Workhorse tracks as one unit. When the group is
+sealed and every member task is finished, Workhorse enqueues its `onFinish`, `onSuccess`, or
+`onFailure` callback task. The migration adds the `task_group`, `task_group_member`,
+`task_group_callback`, and `task_group_event` tables, and a trigger on `task_outcome` that counts
+each member outcome in the transaction that writes it. A queue purge counts a purged member task as
+`canceled`. The `terminal_storage` routine keeps member tasks while their group is not finished,
+reports a new `task_groups` phase after `terminal_tasks`, and deletes a finished group after its
+callback tasks are terminal. New SQLSTATEs: `P1008` (the task group does not accept tasks), `P1009`
+(member limit of 100,000), and `P1010` (an enqueue idempotency hit returns a task that is not a
+member of the group).
+
+Migration 0067 (`0067-count-task-group-outcomes-in-counter-shards.sql`) counts task group outcomes
+in counter shards (SM-1245,
+[ADR 0096](https://github.com/stablemates/workhorse/blob/main/docs/decisions/0096-count-task-group-outcomes-in-counter-shards.md)).
+Each group gets 32 rows in the new `task_group_shard` table, and a member outcome locks only its
+shard row at commit. The outcome locks the group row only when the pending count of its shard
+becomes 0. The migration moves the counts of existing groups into the shards. It locks `task_group`
+in `EXCLUSIVE` mode while it runs, so task group changes wait for it. The counts on a `task_group`
+row stay 0 until the group finishes, so read the counts through `get_task_group_v1`. The migration
+also adds `dashboard_task_groups_v1`, which lists task groups one page at a time. The dashboard of
+each SDK reads the task group list through it (SM-1237).
+
+Migration 0068 (`0068-read-the-newest-task-group-events.sql`) adds
+`list_newest_task_group_events_v1`, which returns the newest events of a task group first
+(SM-1254). The task group detail of the dashboard reads its events through it. A group with more
+than 1,000 events now shows its newest 100 events. `eventsTruncated` is true when the group has
+older events that the detail does not show. `list_task_group_events_v1` does not change. The
+migration changes no stored data.
+
+Migration 0069 (`0069-check-the-input-of-a-checkpoint-on-replay.sql`) adds the nullable column
+`task_checkpoint.input_fingerprint` and `save_checkpoint_v2` (SM-1211,
+[ADR 0098](https://github.com/stablemates/workhorse/blob/main/docs/decisions/0098-check-the-input-of-a-checkpoint-on-replay.md)).
+If a checkpoint name exists, `save_checkpoint_v2` returns `input_conflict` when the stored and the
+given input fingerprints are both present and differ. `save_checkpoint_v1` now calls
+`save_checkpoint_v2` with no fingerprint, so its results do not change. The checkpoint reads of each
+statement catalogue return the new column. Existing rows keep a NULL fingerprint. The migration adds
+the column without a table scan.
+
+Migration 0070 (`0070-fence-a-task-group-member-add-and-index-the-group-list.sql`) fences a task
+group member add (SM-1273). `add_task_group_members_v1` now locks the runtime row of the member task
+first, before the shard rows and the group row, as a completion does. Then it refuses the add with
+`P1008` and the reason `not_running_member` if the lease of the member task expired, its deadline or
+attempt timeout passed, or a cancel was requested. Before, a worker whose lease expired could add
+tasks after Workhorse gave the member task to a new worker. The signature does not change. The
+migration also adds `task_group_state_created_idx`, so `dashboard_task_groups_v1` reads one page of
+a state without a read of the groups of other states. The migration is a non-transactional step: it
+builds the index with `CREATE INDEX CONCURRENTLY`, so writes to `task_group` continue. If it fails
+part-way, drop the invalid index before the rerun, as `docs/schema-lifecycle.md` describes. It
+changes no stored data.
+
+Migration 0071 (`0071-lock-purged-runtime-rows-in-identity-order.sql`) makes a queue purge lock its
+tasks in identity order (SM-1282). Before, `purge_queue_internal_v1` locked the runtime rows and the
+task rows of the queue in scan order. An enqueue that named two of those tasks as prerequisites
+locks them in identity order. Thus the purge and the enqueue could wait for each other, and
+PostgreSQL aborted one of them with `40P01`. The purge now locks every runtime row, then every task
+row, each in identity order. The signature and the deleted count do not change. The migration does
+not raise the SDK compatibility floor, because no SDK statement changes and every process gets the
+new lock order when the migration commits. It changes no stored data.
+
+Migration 0072 (`0072-purge-a-queue-that-holds-a-prerequisite-of-a-live-dependent.sql`) lets a
+queue purge succeed when it holds a prerequisite of a waiting dependent (SM-1284,
+[ADR 0101](https://github.com/stablemates/workhorse/blob/main/docs/decisions/0101-cancel-a-purged-prerequisite-whose-dependent-stays.md)).
+Before, the purge deleted that prerequisite, and PostgreSQL refused the delete with `23001`. The
+whole purge then failed, also when the dependent was in the same queue. Now the purge deletes a
+dependent in the same purge with its prerequisite. If the dependent stays, for example in another
+queue, the purge cancels the prerequisite instead of a delete. The dependent's own
+`on_cancellation` policy then decides its outcome, as after a cancel. The canceled task keeps its
+history and its keys. The purge count, `deleted_count` and `deletedCount`, includes the canceled
+tasks. The purge locks the dependents after its own tasks, as a cancel does, so it has no
+deadlock that a cancel of the same tasks does not also have. Two purges run one after the
+other, also for different queues. The migration does not raise the SDK compatibility
+floor, because no SDK statement or result shape changes. It changes no stored data.
+
+### Task groups
+
+The Ruby SDK adds task groups (SM-1236). Use the block form to create a group, add its member
+tasks, and seal it in one statement, so in one transaction:
+
+```ruby
+result = queue.task_group(on_finish: Stablemates::Workhorse::EnqueueRequest.new(
+  task_type: "import.report", payload: {"importId" => 42}
+)) do |group|
+  rows.each { |row| group.enqueue("import.row", {"row" => row}) }
+end
+```
+
+- `Queue#task_group` takes the `on_finish:`, `on_success:`, and `on_failure:` callbacks. Each
+  callback is an `EnqueueRequest` without idempotency, debounce, or throttle. The block receives a
+  `TaskGroupBuilder` with `add` and `enqueue`. If the block raises, nothing is enqueued.
+- `Queue#create_task_group`, `Queue#add_to_task_group`, and `Queue#seal_task_group` build a group in
+  more than one transaction.
+- `Queue#get_task_group`, `Queue#list_task_group_members`, and `Queue#find_task_group` read a
+  group. `Admin` has `get_task_group`, `list_task_group_members`, `list_task_group_events`,
+  `cancel_task_group`, `list_task_group_drift`, and `repair_task_group_drift`.
+- `HandlerContext#task_group` and `BatchHandlerContext#task_group` return a `TaskGroupContext`.
+  Its `get` returns the `TaskGroupLink` of the task, with the group ID and the counts. Its `add`
+  adds tasks from a running member task, also after the seal. If a request names no queue, `add`
+  puts the task in the queue of the member task, not in the first queue of the worker (SM-1251,
+  [ADR 0097](https://github.com/stablemates/workhorse/blob/main/docs/decisions/0097-put-work-that-a-handler-creates-on-the-queue-of-its-task.md)).
+- `Stablemates::Workhorse::ActiveJob.task_group` runs a block in which each job that the
+  Workhorse adapter enqueues becomes a member task. Its callbacks can be Active Job instances,
+  such as `ReportJob.new(import.id)`. A job reads its group through
+  `Stablemates::Workhorse::ActiveJob.current_task_group`.
+- A job class with `enqueue_after_transaction_commit` set to true cannot join a task group. Rails
+  would enqueue that job after the transaction commits, so outside the group. If a task group block
+  enqueues such a job through the Workhorse adapter, the SDK raises `ArgumentError` before Rails
+  defers the job. Set `enqueue_after_transaction_commit` to false on that job class. The group
+  statement runs in the caller's transaction, so the member tasks still commit with it.
+- `TaskGroupClosedError` (`P1008`), `TaskGroupLimitExceededError` (`P1009`), and
+  `TaskGroupMembershipConflictError` (`P1010`) report the task group refusals.
+- `Queue#list_task_group_members` and `Admin#list_task_group_members` return a nil `next_after`
+  on the last page, also when the page is full (SM-1277). Before, a full last page gave a
+  `next_after` that read an empty page. A full page now costs one more read of one member.
+  `TaskGroup#callbacks` keeps the order `on_finish`, `on_success`, `on_failure`, and a test now
+  asserts it. The change adds no SQL function and no migration.
+
+### Checkpoint input
+
+A checkpoint finds its stored result by name only. If a deployment changes the input of a checkpoint
+operation, a replay returns the result of the old input and gives no error. A Ruby handler can now
+give the input of a checkpoint (SM-1264,
+[ADR 0098](https://github.com/stablemates/workhorse/blob/main/docs/decisions/0098-check-the-input-of-a-checkpoint-on-replay.md)):
+
+```ruby
+context.checkpoint("charge-card", input: {"amountCents" => amount}) { gateway.charge(amount) }
+```
+
+- `HandlerContext#checkpoint` and `BatchHandlerContext#checkpoint` take the `input:` keyword, a JSON
+  value. The checkpoint stores the fingerprint of the input with its value, through
+  `save_checkpoint_v2`. A call without `input:` keeps its current behavior. `input: nil` gives the
+  JSON input null.
+- If the stored checkpoint, or an overlapping call with the same name, has the fingerprint of a
+  different input, the call raises `CheckpointInputConflictError` and the block does not run. If the
+  call or the stored checkpoint has no input, the call compares nothing.
+- `CheckpointInputConflictError` is a subclass of `ConflictError`. The worker fails the task at once
+  and keeps the current attempt, as for every replay conflict. The task error names the class.
+- `Stablemates::Workhorse.checkpoint_input_fingerprint(input)` returns the fingerprint: the SHA-256
+  digest of the RFC 8785 canonical JSON of the input, as 64 lowercase hexadecimal characters. It
+  refuses an Integer or a Float larger than 2**53 - 1 in magnitude, such as `1e21`, because RFC 8785
+  writes each number as an IEEE 754 double. Every SDK refuses the same inputs (SM-1274).
+- `TaskCheckpoint#input_fingerprint` returns the stored fingerprint, or nil. `Admin#get_checkpoint`,
+  `Admin#list_checkpoints`, and `HandlerContext#get_checkpoint` return it.
+
+### Behaviour changes
+
+- If a child task request names no queue, `run_child`, `run_children`, and `run_children_all` put
+  the child in the queue of the parent task (SM-1260, [ADR 0097](https://github.com/stablemates/workhorse/blob/main/docs/decisions/0097-put-work-that-a-handler-creates-on-the-queue-of-its-task.md)).
+  Before, the child went to the first queue in the `queues:` list of the worker. To keep the old
+  queue, name that queue on the request, for example `queue: "default"`.
+  A parent task that created a child before the upgrade can replay that request after the upgrade.
+  If the replayed request names no queue, it keeps the queue of the stored child, so the parent
+  task joins its child.
+- `Worker#run_once` does not change: a pass that released every task that it claimed returns
+  `false`, and the next call claims at once (SM-1259, [ADR 0099](https://github.com/stablemates/workhorse/blob/main/docs/decisions/0099-report-a-released-only-single-pass-as-no-progress.md)). TypeScript
+  `runOnce()` now does the same. If you call the single-pass call in a loop, wait after `false`.
+- A continuous worker that released every task of a claim now waits for its next poll before it
+  claims again (SM-1247). A worker releases a task when it has no handler for the task type, for
+  example during a rolling deployment. Before, the release notification ended the empty wait of the
+  same worker, so it claimed the task again after 0 to 50 ms. One listening worker released one task
+  about 30 times each second. Now a notification does not end this release wait. Only its deadline,
+  a stop, or a pause ends it. The deadline is the poll interval while the listener is active, or the
+  poll backoff without a listener. Thus a worker claims a task that it cannot run at most once in
+  each poll interval.
+- The Rack dashboard refuses a mutation whose `audit.requestId` is more than 512 UTF-8 bytes
+  with `BAD_REQUEST`, before any SQL runs (SM-1253). It measures the value as sent, before trim.
+  Before, the SQL audit function refused the value, and the client got an internal server error.
+
 ## 0.7.1 — 2026-10-09
 
 The npm packages, Python distribution, Go module, Rust crate, and Ruby gem release from one source commit.
@@ -316,7 +633,7 @@ Requires **schema v43**, Ruby **3.3** or newer, and PostgreSQL **15** or newer.
   renews the lease, `Worker#stop` still wakes the dispatcher, and a second signal still exits the
   process. The first failure of each logger is written to standard error.
 - Add the Active Job adapter, selected with `config.active_job.queue_adapter =
-:stablemates_workhorse`. It requires Active Job 8.0 or later. A default job runs under the
+  :stablemates_workhorse`. It requires Active Job 8.0 or later. A default job runs under the
   `active_job` task type. A typed job declares its task type with `workhorse_options` and carries
   one JSON `Hash`, so another SDK can enqueue or run it. `Stablemates::Workhorse::ActiveJob.handle`
   registers both formats on a `Worker`.

data/README.md

@@ -3,6 +3,13 @@
 The Ruby queue client, operator client, and worker runtime for the Workhorse durable task queue for
 PostgreSQL.
 
+<picture>
+  <source media="(max-width: 600px) and (prefers-color-scheme: dark)" srcset="https://workhorse.run/diagrams/readme-overview-narrow-dark.png">
+  <source media="(max-width: 600px)" srcset="https://workhorse.run/diagrams/readme-overview-narrow-light.png">
+  <source media="(prefers-color-scheme: dark)" srcset="https://workhorse.run/diagrams/readme-overview-dark.png">
+  <img src="https://workhorse.run/diagrams/readme-overview-light.png" width="960" alt="One protocol for your workers and your dashboard">
+</picture>
+
 > **Public beta:** Workhorse is usable for evaluation and early production adoption. A 0.x minor
 > release may change behaviour, so read the
 > [changelog](https://github.com/stablemates/workhorse/blob/main/ruby/CHANGELOG.md) before you
@@ -23,7 +30,7 @@ bundle add connection_pool
 Install the schema once, as a deployment step. The application never installs or migrates it.
 
 ```bash
-npx --package @stablemates/workhorse@0.7.1 workhorse schema install
+npx --package @stablemates/workhorse@0.9.0 workhorse schema install
 ```
 
 The machine that runs that deployment step needs Node.js 22 or newer. The application itself needs
@@ -70,6 +77,22 @@ effects. `ruby/examples/` also holds a transactional enqueue, a dedicated worker
 orchestration of child tasks, signals, and human decisions. An integration spec runs every one of
 them against PostgreSQL.
 
+## Task groups, checkpoint inputs, and single passes
+
+- A [task group](https://workhorse.run/docs/task-groups) tracks many tasks as one unit. After the
+  group is sealed and every member task is finished, Workhorse enqueues the callback tasks of the
+  group. The `Queue#task_group` block creates a group, adds its tasks, and seals it in one
+  transaction.
+- A handler can give the [input of a
+  checkpoint](https://workhorse.run/docs/durable-execution#check-the-input-of-a-checkpoint). Give it
+  as `context.checkpoint(name, input: input) { ... }`. If a later run gives a different input, the
+  call raises `CheckpointInputConflictError`, the task fails, and the operation does not run again.
+  If the stored checkpoint or the call has no input, Workhorse compares nothing.
+- `Worker#run_once` returns true only when a handler ran. It returns false after a pass that only
+  [released the tasks that it
+  claimed](https://workhorse.run/docs/workers#return-a-task-that-the-worker-cannot-run). If you call
+  it in a loop, wait after false.
+
 ## Package boundary
 
 The gem is `stablemates-workhorse`, and its namespace is `Stablemates::Workhorse`.

data/lib/stablemates/workhorse/active_job.rb

@@ -25,8 +25,9 @@ module Stablemates
       # The keywords +workhorse_options+ accepts. A job class that passes any other key raises.
       OPTION_KEYS = %i[task_type max_attempts tags concurrency_key].freeze
       WORKER_KEY = :stablemates_workhorse_active_job_worker
+      CONTEXT_KEY = :stablemates_workhorse_active_job_context
       private_constant :JOB_ID_TAG, :CLASS_TAG, :MAX_OPTION_TAGS, :MAX_TAG_LENGTH, :MAX_TASK_TYPE_BYTES,
-        :WORKER_KEY
+        :WORKER_KEY, :CONTEXT_KEY
 
       # Declares Workhorse options on a job class. Include it and call +workhorse_options+.
       module Options
@@ -77,13 +78,50 @@ module Stablemates
             typed[task_type] = job_class
           end
           admin = Admin.new(worker.executor)
-          worker.handle(TASK_TYPE) { |payload, context| running(worker) { run_default(payload, context.task) } }
+          worker.handle(TASK_TYPE) { |payload, context| running(worker, context) { run_default(payload, context.task) } }
           typed.each do |task_type, job_class|
-            worker.handle(task_type) { |payload, context| running(worker) { run_typed(job_class, payload, context.task, admin) } }
+            worker.handle(task_type) do |payload, context|
+              running(worker, context) { run_typed(job_class, payload, context.task, admin) }
+            end
           end
           worker
         end
 
+        # Creates a task group (ADR 0095), runs the block, and seals the group, in one statement. Every
+        # job the block enqueues through the Workhorse adapter, with +perform_later+ or
+        # +perform_all_later+, becomes a member task. The jobs are enqueued when the block returns,
+        # and each job's +provider_job_id+ is set then. If the block raises, nothing is enqueued.
+        #
+        # A job class with +enqueue_after_transaction_commit+ set to true cannot join a group. Rails
+        # would enqueue it after the transaction commits, so after the block ends and outside the
+        # group. If the block enqueues such a job through the Workhorse adapter, the call raises
+        # ArgumentError before Rails defers the job. Set +enqueue_after_transaction_commit+ to false on
+        # that job class. The group statement runs in the caller's transaction, so the member tasks
+        # still commit with it.
+        #
+        # Each callback is an Active Job instance, such as +ReportJob.new(import.id)+, an
+        # EnqueueRequest, or nil. +on_finish+ runs when every member task is finished, +on_success+
+        # when every member task succeeded, and +on_failure+ when one or more did not succeed.
+        # +queue+ is the Queue that creates the group. It defaults to the queue of the Workhorse
+        # adapter that ActiveJob::Base uses. Returns a TaskGroupBlockResult.
+        def task_group(on_finish: nil, on_success: nil, on_failure: nil, queue: nil, &)
+          raise ArgumentError, "task_group needs a block" unless block_given?
+          raise ArgumentError, "queue must be a Queue or nil" unless queue.nil? || queue.is_a?(Queue)
+
+          queue ||= adapter_queue
+          queue.task_group(on_finish: callback(on_finish), on_success: callback(on_success),
+            on_failure: callback(on_failure), &)
+        end
+
+        # The TaskGroupLink of the job that runs now, as a member task or as a callback task, or nil
+        # for a job with no group. Raises ArgumentError outside a job that a Workhorse worker runs.
+        def current_task_group
+          context = Thread.current[CONTEXT_KEY]
+          raise ArgumentError, "current_task_group needs a job that a Workhorse worker runs" if context.nil?
+
+          context.task_group.get
+        end
+
         # The request that enqueues +job+, or ArgumentError before any statement runs. An invalid
         # priority raises ActiveJob::EnqueueError, which Active Job records on the job.
         def request(job, run_at) # :nodoc:
@@ -104,6 +142,22 @@ module Stablemates
         # The worker running the current job, or nil outside a worker.
         def current_worker = Thread.current[WORKER_KEY] # :nodoc:
 
+        # Raises ArgumentError when a task group block is open and one of +jobs+ goes to the Workhorse
+        # adapter with +enqueue_after_transaction_commit+ set. Rails would enqueue that job after the
+        # commit, outside the group.
+        def refuse_deferral(jobs) # :nodoc:
+          return if TaskGroupBuilder.current.nil?
+
+          jobs.each do |job|
+            next unless job.class.queue_adapter.is_a?(::ActiveJob::QueueAdapters::StablematesWorkhorseAdapter)
+            next unless DEFERRING.include?(job.class.enqueue_after_transaction_commit)
+
+            raise ArgumentError, "#{job.class.name} sets enqueue_after_transaction_commit, so Rails would enqueue " \
+              "it after the transaction commits and outside the task group; set " \
+              "enqueue_after_transaction_commit to false on #{job.class.name}"
+          end
+        end
+
         def validate_options(class_name, options) # :nodoc:
           unknown = options.keys - OPTION_KEYS
           raise ArgumentError, "workhorse_options does not accept #{unknown.map(&:inspect).join(", ")}" unless
@@ -165,13 +219,31 @@ module Stablemates
           adapter_tags + (option || [])
         end
 
-        def running(worker)
-          previous = Thread.current[WORKER_KEY]
+        def running(worker, context)
+          previous = [Thread.current[WORKER_KEY], Thread.current[CONTEXT_KEY]]
           Thread.current[WORKER_KEY] = worker
+          Thread.current[CONTEXT_KEY] = context
           yield
           nil
         ensure
-          Thread.current[WORKER_KEY] = previous
+          Thread.current[WORKER_KEY], Thread.current[CONTEXT_KEY] = previous
+        end
+
+        def callback(job)
+          case job
+          when nil, EnqueueRequest then job
+          when ::ActiveJob::Base then request(job, job.scheduled_at && Time.at(job.scheduled_at.to_f))
+          else raise ArgumentError, "a task group callback must be an Active Job, an EnqueueRequest, or nil"
+          end
+        end
+
+        def adapter_queue
+          adapter = ::ActiveJob::Base.queue_adapter
+          unless adapter.is_a?(::ActiveJob::QueueAdapters::StablematesWorkhorseAdapter)
+            raise ArgumentError, "task_group needs a queue: ActiveJob::Base does not use the Workhorse adapter"
+          end
+
+          adapter.workhorse_queue
         end
 
         def run_default(payload, task)
@@ -204,6 +276,42 @@ module Stablemates
         concurrency_key: "must be a non-empty String or respond to call"
       }.freeze
       private_constant :OPTION_RULES
+
+      # The +enqueue_after_transaction_commit+ values that defer an enqueue. Active Job 8.0 also
+      # accepts the deprecated :always.
+      DEFERRING = [true, :always].freeze
+      private_constant :DEFERRING
+
+      # Checks one job before Rails can defer its enqueue. A prepend on ActiveJob::Base runs before
+      # the module that Rails includes for the deferral. Internal to the SDK.
+      module JobDeferralCheck # :nodoc:
+        private
+
+        def raw_enqueue(...)
+          ::Stablemates::Workhorse::ActiveJob.refuse_deferral([self])
+          super
+        end
+      end
+
+      # Checks the jobs of +perform_all_later+ before Rails can defer them. Internal to the SDK.
+      module BatchDeferralCheck # :nodoc:
+        def perform_all_later(*jobs)
+          ::Stablemates::Workhorse::ActiveJob.refuse_deferral(jobs.flatten)
+          super
+        end
+      end
+      private_constant :JobDeferralCheck, :BatchDeferralCheck
+
+      # The checks wait for ActiveJob::Base. Loading it here would run its load hooks, and Rails
+      # would then look up this adapter before this file defines it.
+      ::ActiveSupport.on_load(:active_job) { prepend(JobDeferralCheck) }
+      ::ActiveJob.singleton_class.prepend(BatchDeferralCheck)
+      # Active Job 8.1 splits +perform_all_later+ in a module that it prepends when Rails includes
+      # the deferral, which can be after this file loads. The check therefore also goes into that
+      # module, so it runs before the split.
+      if ::ActiveJob::EnqueueAfterTransactionCommit.const_defined?(:ActiveJobMethods, false)
+        ::ActiveJob::EnqueueAfterTransactionCommit::ActiveJobMethods.prepend(BatchDeferralCheck)
+      end
     end
   end
 end
@@ -226,8 +334,9 @@ module ActiveJob
 
       # Enqueues +jobs+ in atomic chunks of the shared batch limit and returns how many it enqueued.
       # A job with an invalid priority records the error and is skipped. Each job reports only the
-      # error of this call.
+      # error of this call. Inside the block form of a task group, the jobs join the group instead.
       def enqueue_all(jobs)
+        builder = ::Stablemates::Workhorse::TaskGroupBuilder.current
         pending = jobs.filter_map do |job|
           job.enqueue_error = nil
           [job, workhorse.request(job, job.scheduled_at && Time.at(job.scheduled_at.to_f))]
@@ -236,6 +345,8 @@ module ActiveJob
           job.successfully_enqueued = false
           nil
         end
+        return grouped(builder, pending) if builder
+
         pending.each_slice(::Stablemates::Workhorse::SqlCatalogue::MAX_ENQUEUE_BATCH_SIZE) do |chunk|
           results = queue.enqueue_many(chunk.map(&:last))
           chunk.zip(results) do |(job, _request), result|
@@ -252,20 +363,37 @@ module ActiveJob
         !worker.nil? && worker.stopping?
       end
 
+      # The Queue this adapter enqueues through. Internal to the SDK.
+      def workhorse_queue # :nodoc:
+        @queue || @lock.synchronize do
+          @queue ||= ::Stablemates::Workhorse::Queue.new(
+            @executor || ::Stablemates::Workhorse::ActiveRecordExecutor.new(::ActiveRecord::Base)
+          )
+        end
+      end
+
       private
 
       def workhorse = ::Stablemates::Workhorse::ActiveJob
 
+      def queue = workhorse_queue
+
+      # Inside the block form of a task group, a job joins the group. The group enqueues it when the
+      # block returns.
       def enqueue_one(job, run_at)
-        job.provider_job_id = queue.enqueue_many([workhorse.request(job, run_at)]).first.task_id
+        request = workhorse.request(job, run_at)
+        builder = ::Stablemates::Workhorse::TaskGroupBuilder.current
+        return job.provider_job_id = queue.enqueue_many([request]).first.task_id if builder.nil?
+
+        builder.add_request(request) { |result| job.provider_job_id = result.task_id }
       end
 
-      def queue
-        @queue || @lock.synchronize do
-          @queue ||= ::Stablemates::Workhorse::Queue.new(
-            @executor || ::Stablemates::Workhorse::ActiveRecordExecutor.new(::ActiveRecord::Base)
-          )
+      def grouped(builder, pending)
+        pending.each do |job, request|
+          builder.add_request(request) { |result| job.provider_job_id = result.task_id }
+          job.successfully_enqueued = true
         end
+        pending.length
       end
     end
   end