stablemates-workhorse 0.7.1 → 0.8.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e72b7d14542014971d81d818e073acaadbaef569c2516ba66427583e607fd0f3
-  data.tar.gz: 0e1d31974f65086473c750137249edd45722d75aa3b630e6c0a4c6c66bf7eb2e
+  metadata.gz: 807d43f9b16d79430d292a3e6c70acdc55bb8cc3f2010b7e03f0356fb0731c90
+  data.tar.gz: f84ab68d2d6d03450983e74dc337880546815472ece79d8cdf1d650679d3912a
 SHA512:
-  metadata.gz: 3de1cc0e31810df55903ec87b96e553edc96bfa2e7a699e6b926b669a995d6231ae03d5bd1ccfa62c8908d1ed6095e3a775998d092ac27b28fd92c8c9288cd20
-  data.tar.gz: 4683b2cebb636655c2c5c608d56059cedb5e6d837c0eb96735443aa78d2abad27371c3231737ae5be3d8cdfa504bf179a468f70cd15a038cf4516f0388e02459
+  metadata.gz: 706d9cfd755c1bc971116245009433dba6d4880dc7b7dec081f486d95f5395395a69b55527527b33252f4cf4bbc5e1a1b94c74ad91b267fe6893884c3bdb09ef
+  data.tar.gz: dd7e13f971e790cc10c4f07843a4b7ec9f086f32365c665f681206bf1376d1d7d03bf11e6dfa78aacc75ea8cb2ce292da3246b9471431d7fa6b1630c58c46a17

data/CHANGELOG.md

@@ -3,6 +3,203 @@
 `stablemates-workhorse` gem versions and release notes live here. The gem carries the version the
 other SDKs carry, because every tag names one release of all of them.
 
+## 0.8.0 — 2026-10-13
+
+The npm packages, Python distribution, Go module, Rust crate, and Ruby gem release from one source commit.
+
+Requires **schema v69**, Ruby **3.3** or newer, and PostgreSQL **15** or newer.
+
+### Upgrade steps
+
+**Migrate the schema before starting updated processes.** The upgrade is a rolling deployment:
+
+1. Run `workhorse schema migrate`. The final schema version is **71**.
+2. Roll out the updated processes.
+
+The SDK compatibility floor is schema version **69**, which migration 0070 reaches. The floor rises
+from 54. Every SDK statement catalogue now names the task group functions that migration 0066 adds,
+and `save_checkpoint_v2`, which migration 0069 adds. The dashboard of each SDK calls
+`dashboard_task_groups_v1`, which migration 0067 adds, and `list_newest_task_group_events_v1`, which
+migration 0068 adds. Migration 0070 fences a member add, so an updated process does not add tasks to
+a group without that check. An updated process refuses a schema below version 69 at startup. A 0.7
+process keeps working on version 71, because a migration inside a major line only adds.
+
+**Give the eighth member when you build a `TaskCheckpoint`.** `TaskCheckpoint` is a `Data` class.
+0.8.0 adds `input_fingerprint` as its eighth member (SM-1264). A call to `TaskCheckpoint.new` with
+seven positional arguments raises `ArgumentError`. A call with keywords and no
+`input_fingerprint:` also raises it. The gem has no compatible constructor. If your code builds a
+`TaskCheckpoint`, for example in a test, give the eighth value. For a checkpoint without an input,
+give `input_fingerprint: nil` or a positional `nil`.
+
+Migration 0066 (`0066-add-task-groups-with-completion-callbacks.sql`) adds task groups (SM-1231,
+[ADR 0095](https://github.com/stablemates/workhorse/blob/main/docs/decisions/0095-add-task-groups-with-completion-callbacks.md)).
+A task group is a set of full-tier tasks that Workhorse tracks as one unit. When the group is
+sealed and every member task is finished, Workhorse enqueues its `onFinish`, `onSuccess`, or
+`onFailure` callback task. The migration adds the `task_group`, `task_group_member`,
+`task_group_callback`, and `task_group_event` tables, and a trigger on `task_outcome` that counts
+each member outcome in the transaction that writes it. A queue purge counts a purged member task as
+`canceled`. The `terminal_storage` routine keeps member tasks while their group is not finished,
+reports a new `task_groups` phase after `terminal_tasks`, and deletes a finished group after its
+callback tasks are terminal. New SQLSTATEs: `P1008` (the task group does not accept tasks), `P1009`
+(member limit of 100,000), and `P1010` (an enqueue idempotency hit returns a task that is not a
+member of the group).
+
+Migration 0067 (`0067-count-task-group-outcomes-in-counter-shards.sql`) counts task group outcomes
+in counter shards (SM-1245,
+[ADR 0096](https://github.com/stablemates/workhorse/blob/main/docs/decisions/0096-count-task-group-outcomes-in-counter-shards.md)).
+Each group gets 32 rows in the new `task_group_shard` table, and a member outcome locks only its
+shard row at commit. The outcome locks the group row only when the pending count of its shard
+becomes 0. The migration moves the counts of existing groups into the shards. It locks `task_group`
+in `EXCLUSIVE` mode while it runs, so task group changes wait for it. The counts on a `task_group`
+row stay 0 until the group finishes, so read the counts through `get_task_group_v1`. The migration
+also adds `dashboard_task_groups_v1`, which lists task groups one page at a time. The dashboard of
+each SDK reads the task group list through it (SM-1237).
+
+Migration 0068 (`0068-read-the-newest-task-group-events.sql`) adds
+`list_newest_task_group_events_v1`, which returns the newest events of a task group first
+(SM-1254). The task group detail of the dashboard reads its events through it. A group with more
+than 1,000 events now shows its newest 100 events. `eventsTruncated` is true when the group has
+older events that the detail does not show. `list_task_group_events_v1` does not change. The
+migration changes no stored data.
+
+Migration 0069 (`0069-check-the-input-of-a-checkpoint-on-replay.sql`) adds the nullable column
+`task_checkpoint.input_fingerprint` and `save_checkpoint_v2` (SM-1211,
+[ADR 0098](https://github.com/stablemates/workhorse/blob/main/docs/decisions/0098-check-the-input-of-a-checkpoint-on-replay.md)).
+If a checkpoint name exists, `save_checkpoint_v2` returns `input_conflict` when the stored and the
+given input fingerprints are both present and differ. `save_checkpoint_v1` now calls
+`save_checkpoint_v2` with no fingerprint, so its results do not change. The checkpoint reads of each
+statement catalogue return the new column. Existing rows keep a NULL fingerprint. The migration adds
+the column without a table scan.
+
+Migration 0070 (`0070-fence-a-task-group-member-add-and-index-the-group-list.sql`) fences a task
+group member add (SM-1273). `add_task_group_members_v1` now locks the runtime row of the member task
+first, before the shard rows and the group row, as a completion does. Then it refuses the add with
+`P1008` and the reason `not_running_member` if the lease of the member task expired, its deadline or
+attempt timeout passed, or a cancel was requested. Before, a worker whose lease expired could add
+tasks after Workhorse gave the member task to a new worker. The signature does not change. The
+migration also adds `task_group_state_created_idx`, so `dashboard_task_groups_v1` reads one page of
+a state without a read of the groups of other states. The migration is a non-transactional step: it
+builds the index with `CREATE INDEX CONCURRENTLY`, so writes to `task_group` continue. If it fails
+part-way, drop the invalid index before the rerun, as `docs/schema-lifecycle.md` describes. It
+changes no stored data.
+
+Migration 0071 (`0071-lock-purged-runtime-rows-in-identity-order.sql`) makes a queue purge lock its
+tasks in identity order (SM-1282). Before, `purge_queue_internal_v1` locked the runtime rows and the
+task rows of the queue in scan order. An enqueue that named two of those tasks as prerequisites
+locks them in identity order. Thus the purge and the enqueue could wait for each other, and
+PostgreSQL aborted one of them with `40P01`. The purge now locks every runtime row, then every task
+row, each in identity order. The signature and the deleted count do not change. The migration does
+not raise the SDK compatibility floor, because no SDK statement changes and every process gets the
+new lock order when the migration commits. It changes no stored data.
+
+Migration 0072 (`0072-purge-a-queue-that-holds-a-prerequisite-of-a-live-dependent.sql`) lets a
+queue purge succeed when it holds a prerequisite of a waiting dependent (SM-1284,
+[ADR 0101](https://github.com/stablemates/workhorse/blob/main/docs/decisions/0101-cancel-a-purged-prerequisite-whose-dependent-stays.md)).
+Before, the purge deleted that prerequisite, and PostgreSQL refused the delete with `23001`. The
+whole purge then failed, also when the dependent was in the same queue. Now the purge deletes a
+dependent in the same purge with its prerequisite. If the dependent stays, for example in another
+queue, the purge cancels the prerequisite instead of a delete. The dependent's own
+`on_cancellation` policy then decides its outcome, as after a cancel. The canceled task keeps its
+history and its keys. The purge count, `deleted_count` and `deletedCount`, includes the canceled
+tasks. The purge locks the dependents after its own tasks, as a cancel does, so it has no
+deadlock that a cancel of the same tasks does not also have. Two purges run one after the
+other, also for different queues. The migration does not raise the SDK compatibility
+floor, because no SDK statement or result shape changes. It changes no stored data.
+
+### Task groups
+
+The Ruby SDK adds task groups (SM-1236). Use the block form to create a group, add its member
+tasks, and seal it in one statement, so in one transaction:
+
+```ruby
+result = queue.task_group(on_finish: Stablemates::Workhorse::EnqueueRequest.new(
+  task_type: "import.report", payload: {"importId" => 42}
+)) do |group|
+  rows.each { |row| group.enqueue("import.row", {"row" => row}) }
+end
+```
+
+- `Queue#task_group` takes the `on_finish:`, `on_success:`, and `on_failure:` callbacks. Each
+  callback is an `EnqueueRequest` without idempotency, debounce, or throttle. The block receives a
+  `TaskGroupBuilder` with `add` and `enqueue`. If the block raises, nothing is enqueued.
+- `Queue#create_task_group`, `Queue#add_to_task_group`, and `Queue#seal_task_group` build a group in
+  more than one transaction.
+- `Queue#get_task_group`, `Queue#list_task_group_members`, and `Queue#find_task_group` read a
+  group. `Admin` has `get_task_group`, `list_task_group_members`, `list_task_group_events`,
+  `cancel_task_group`, `list_task_group_drift`, and `repair_task_group_drift`.
+- `HandlerContext#task_group` and `BatchHandlerContext#task_group` return a `TaskGroupContext`.
+  Its `get` returns the `TaskGroupLink` of the task, with the group ID and the counts. Its `add`
+  adds tasks from a running member task, also after the seal. If a request names no queue, `add`
+  puts the task in the queue of the member task, not in the first queue of the worker (SM-1251,
+  [ADR 0097](https://github.com/stablemates/workhorse/blob/main/docs/decisions/0097-put-work-that-a-handler-creates-on-the-queue-of-its-task.md)).
+- `Stablemates::Workhorse::ActiveJob.task_group` runs a block in which each job that the
+  Workhorse adapter enqueues becomes a member task. Its callbacks can be Active Job instances,
+  such as `ReportJob.new(import.id)`. A job reads its group through
+  `Stablemates::Workhorse::ActiveJob.current_task_group`.
+- A job class with `enqueue_after_transaction_commit` set to true cannot join a task group. Rails
+  would enqueue that job after the transaction commits, so outside the group. If a task group block
+  enqueues such a job through the Workhorse adapter, the SDK raises `ArgumentError` before Rails
+  defers the job. Set `enqueue_after_transaction_commit` to false on that job class. The group
+  statement runs in the caller's transaction, so the member tasks still commit with it.
+- `TaskGroupClosedError` (`P1008`), `TaskGroupLimitExceededError` (`P1009`), and
+  `TaskGroupMembershipConflictError` (`P1010`) report the task group refusals.
+- `Queue#list_task_group_members` and `Admin#list_task_group_members` return a nil `next_after`
+  on the last page, also when the page is full (SM-1277). Before, a full last page gave a
+  `next_after` that read an empty page. A full page now costs one more read of one member.
+  `TaskGroup#callbacks` keeps the order `on_finish`, `on_success`, `on_failure`, and a test now
+  asserts it. The change adds no SQL function and no migration.
+
+### Checkpoint input
+
+A checkpoint finds its stored result by name only. If a deployment changes the input of a checkpoint
+operation, a replay returns the result of the old input and gives no error. A Ruby handler can now
+give the input of a checkpoint (SM-1264,
+[ADR 0098](https://github.com/stablemates/workhorse/blob/main/docs/decisions/0098-check-the-input-of-a-checkpoint-on-replay.md)):
+
+```ruby
+context.checkpoint("charge-card", input: {"amountCents" => amount}) { gateway.charge(amount) }
+```
+
+- `HandlerContext#checkpoint` and `BatchHandlerContext#checkpoint` take the `input:` keyword, a JSON
+  value. The checkpoint stores the fingerprint of the input with its value, through
+  `save_checkpoint_v2`. A call without `input:` keeps its current behavior. `input: nil` gives the
+  JSON input null.
+- If the stored checkpoint, or an overlapping call with the same name, has the fingerprint of a
+  different input, the call raises `CheckpointInputConflictError` and the block does not run. If the
+  call or the stored checkpoint has no input, the call compares nothing.
+- `CheckpointInputConflictError` is a subclass of `ConflictError`. The worker fails the task at once
+  and keeps the current attempt, as for every replay conflict. The task error names the class.
+- `Stablemates::Workhorse.checkpoint_input_fingerprint(input)` returns the fingerprint: the SHA-256
+  digest of the RFC 8785 canonical JSON of the input, as 64 lowercase hexadecimal characters. It
+  refuses an Integer or a Float larger than 2**53 - 1 in magnitude, such as `1e21`, because RFC 8785
+  writes each number as an IEEE 754 double. Every SDK refuses the same inputs (SM-1274).
+- `TaskCheckpoint#input_fingerprint` returns the stored fingerprint, or nil. `Admin#get_checkpoint`,
+  `Admin#list_checkpoints`, and `HandlerContext#get_checkpoint` return it.
+
+### Behaviour changes
+
+- If a child task request names no queue, `run_child`, `run_children`, and `run_children_all` put
+  the child in the queue of the parent task (SM-1260, [ADR 0097](https://github.com/stablemates/workhorse/blob/main/docs/decisions/0097-put-work-that-a-handler-creates-on-the-queue-of-its-task.md)).
+  Before, the child went to the first queue in the `queues:` list of the worker. To keep the old
+  queue, name that queue on the request, for example `queue: "default"`.
+  A parent task that created a child before the upgrade can replay that request after the upgrade.
+  If the replayed request names no queue, it keeps the queue of the stored child, so the parent
+  task joins its child.
+- `Worker#run_once` does not change: a pass that released every task that it claimed returns
+  `false`, and the next call claims at once (SM-1259, [ADR 0099](https://github.com/stablemates/workhorse/blob/main/docs/decisions/0099-report-a-released-only-single-pass-as-no-progress.md)). TypeScript
+  `runOnce()` now does the same. If you call the single-pass call in a loop, wait after `false`.
+- A continuous worker that released every task of a claim now waits for its next poll before it
+  claims again (SM-1247). A worker releases a task when it has no handler for the task type, for
+  example during a rolling deployment. Before, the release notification ended the empty wait of the
+  same worker, so it claimed the task again after 0 to 50 ms. One listening worker released one task
+  about 30 times each second. Now a notification does not end this release wait. Only its deadline,
+  a stop, or a pause ends it. The deadline is the poll interval while the listener is active, or the
+  poll backoff without a listener. Thus a worker claims a task that it cannot run at most once in
+  each poll interval.
+- The Rack dashboard refuses a mutation whose `audit.requestId` is more than 512 UTF-8 bytes
+  with `BAD_REQUEST`, before any SQL runs (SM-1253). It measures the value as sent, before trim.
+  Before, the SQL audit function refused the value, and the client got an internal server error.
+
 ## 0.7.1 — 2026-10-09
 
 The npm packages, Python distribution, Go module, Rust crate, and Ruby gem release from one source commit.
@@ -316,7 +513,7 @@ Requires **schema v43**, Ruby **3.3** or newer, and PostgreSQL **15** or newer.
   renews the lease, `Worker#stop` still wakes the dispatcher, and a second signal still exits the
   process. The first failure of each logger is written to standard error.
 - Add the Active Job adapter, selected with `config.active_job.queue_adapter =
-:stablemates_workhorse`. It requires Active Job 8.0 or later. A default job runs under the
+  :stablemates_workhorse`. It requires Active Job 8.0 or later. A default job runs under the
   `active_job` task type. A typed job declares its task type with `workhorse_options` and carries
   one JSON `Hash`, so another SDK can enqueue or run it. `Stablemates::Workhorse::ActiveJob.handle`
   registers both formats on a `Worker`.

data/README.md

@@ -23,7 +23,7 @@ bundle add connection_pool
 Install the schema once, as a deployment step. The application never installs or migrates it.
 
 ```bash
-npx --package @stablemates/workhorse@0.7.1 workhorse schema install
+npx --package @stablemates/workhorse@0.8.0 workhorse schema install
 ```
 
 The machine that runs that deployment step needs Node.js 22 or newer. The application itself needs
@@ -70,6 +70,22 @@ effects. `ruby/examples/` also holds a transactional enqueue, a dedicated worker
 orchestration of child tasks, signals, and human decisions. An integration spec runs every one of
 them against PostgreSQL.
 
+## Task groups, checkpoint inputs, and single passes
+
+- A [task group](https://workhorse.run/docs/task-groups) tracks many tasks as one unit. After the
+  group is sealed and every member task is finished, Workhorse enqueues the callback tasks of the
+  group. The `Queue#task_group` block creates a group, adds its tasks, and seals it in one
+  transaction.
+- A handler can give the [input of a
+  checkpoint](https://workhorse.run/docs/durable-execution#check-the-input-of-a-checkpoint). Give it
+  as `context.checkpoint(name, input: input) { ... }`. If a later run gives a different input, the
+  call raises `CheckpointInputConflictError`, the task fails, and the operation does not run again.
+  If the stored checkpoint or the call has no input, Workhorse compares nothing.
+- `Worker#run_once` returns true only when a handler ran. It returns false after a pass that only
+  [released the tasks that it
+  claimed](https://workhorse.run/docs/workers#return-a-task-that-the-worker-cannot-run). If you call
+  it in a loop, wait after false.
+
 ## Package boundary
 
 The gem is `stablemates-workhorse`, and its namespace is `Stablemates::Workhorse`.

data/lib/stablemates/workhorse/active_job.rb

@@ -25,8 +25,9 @@ module Stablemates
       # The keywords +workhorse_options+ accepts. A job class that passes any other key raises.
       OPTION_KEYS = %i[task_type max_attempts tags concurrency_key].freeze
       WORKER_KEY = :stablemates_workhorse_active_job_worker
+      CONTEXT_KEY = :stablemates_workhorse_active_job_context
       private_constant :JOB_ID_TAG, :CLASS_TAG, :MAX_OPTION_TAGS, :MAX_TAG_LENGTH, :MAX_TASK_TYPE_BYTES,
-        :WORKER_KEY
+        :WORKER_KEY, :CONTEXT_KEY
 
       # Declares Workhorse options on a job class. Include it and call +workhorse_options+.
       module Options
@@ -77,13 +78,50 @@ module Stablemates
             typed[task_type] = job_class
           end
           admin = Admin.new(worker.executor)
-          worker.handle(TASK_TYPE) { |payload, context| running(worker) { run_default(payload, context.task) } }
+          worker.handle(TASK_TYPE) { |payload, context| running(worker, context) { run_default(payload, context.task) } }
           typed.each do |task_type, job_class|
-            worker.handle(task_type) { |payload, context| running(worker) { run_typed(job_class, payload, context.task, admin) } }
+            worker.handle(task_type) do |payload, context|
+              running(worker, context) { run_typed(job_class, payload, context.task, admin) }
+            end
           end
           worker
         end
 
+        # Creates a task group (ADR 0095), runs the block, and seals the group, in one statement. Every
+        # job the block enqueues through the Workhorse adapter, with +perform_later+ or
+        # +perform_all_later+, becomes a member task. The jobs are enqueued when the block returns,
+        # and each job's +provider_job_id+ is set then. If the block raises, nothing is enqueued.
+        #
+        # A job class with +enqueue_after_transaction_commit+ set to true cannot join a group. Rails
+        # would enqueue it after the transaction commits, so after the block ends and outside the
+        # group. If the block enqueues such a job through the Workhorse adapter, the call raises
+        # ArgumentError before Rails defers the job. Set +enqueue_after_transaction_commit+ to false on
+        # that job class. The group statement runs in the caller's transaction, so the member tasks
+        # still commit with it.
+        #
+        # Each callback is an Active Job instance, such as +ReportJob.new(import.id)+, an
+        # EnqueueRequest, or nil. +on_finish+ runs when every member task is finished, +on_success+
+        # when every member task succeeded, and +on_failure+ when one or more did not succeed.
+        # +queue+ is the Queue that creates the group. It defaults to the queue of the Workhorse
+        # adapter that ActiveJob::Base uses. Returns a TaskGroupBlockResult.
+        def task_group(on_finish: nil, on_success: nil, on_failure: nil, queue: nil, &)
+          raise ArgumentError, "task_group needs a block" unless block_given?
+          raise ArgumentError, "queue must be a Queue or nil" unless queue.nil? || queue.is_a?(Queue)
+
+          queue ||= adapter_queue
+          queue.task_group(on_finish: callback(on_finish), on_success: callback(on_success),
+            on_failure: callback(on_failure), &)
+        end
+
+        # The TaskGroupLink of the job that runs now, as a member task or as a callback task, or nil
+        # for a job with no group. Raises ArgumentError outside a job that a Workhorse worker runs.
+        def current_task_group
+          context = Thread.current[CONTEXT_KEY]
+          raise ArgumentError, "current_task_group needs a job that a Workhorse worker runs" if context.nil?
+
+          context.task_group.get
+        end
+
         # The request that enqueues +job+, or ArgumentError before any statement runs. An invalid
         # priority raises ActiveJob::EnqueueError, which Active Job records on the job.
         def request(job, run_at) # :nodoc:
@@ -104,6 +142,22 @@ module Stablemates
         # The worker running the current job, or nil outside a worker.
         def current_worker = Thread.current[WORKER_KEY] # :nodoc:
 
+        # Raises ArgumentError when a task group block is open and one of +jobs+ goes to the Workhorse
+        # adapter with +enqueue_after_transaction_commit+ set. Rails would enqueue that job after the
+        # commit, outside the group.
+        def refuse_deferral(jobs) # :nodoc:
+          return if TaskGroupBuilder.current.nil?
+
+          jobs.each do |job|
+            next unless job.class.queue_adapter.is_a?(::ActiveJob::QueueAdapters::StablematesWorkhorseAdapter)
+            next unless DEFERRING.include?(job.class.enqueue_after_transaction_commit)
+
+            raise ArgumentError, "#{job.class.name} sets enqueue_after_transaction_commit, so Rails would enqueue " \
+              "it after the transaction commits and outside the task group; set " \
+              "enqueue_after_transaction_commit to false on #{job.class.name}"
+          end
+        end
+
         def validate_options(class_name, options) # :nodoc:
           unknown = options.keys - OPTION_KEYS
           raise ArgumentError, "workhorse_options does not accept #{unknown.map(&:inspect).join(", ")}" unless
@@ -165,13 +219,31 @@ module Stablemates
           adapter_tags + (option || [])
         end
 
-        def running(worker)
-          previous = Thread.current[WORKER_KEY]
+        def running(worker, context)
+          previous = [Thread.current[WORKER_KEY], Thread.current[CONTEXT_KEY]]
           Thread.current[WORKER_KEY] = worker
+          Thread.current[CONTEXT_KEY] = context
           yield
           nil
         ensure
-          Thread.current[WORKER_KEY] = previous
+          Thread.current[WORKER_KEY], Thread.current[CONTEXT_KEY] = previous
+        end
+
+        def callback(job)
+          case job
+          when nil, EnqueueRequest then job
+          when ::ActiveJob::Base then request(job, job.scheduled_at && Time.at(job.scheduled_at.to_f))
+          else raise ArgumentError, "a task group callback must be an Active Job, an EnqueueRequest, or nil"
+          end
+        end
+
+        def adapter_queue
+          adapter = ::ActiveJob::Base.queue_adapter
+          unless adapter.is_a?(::ActiveJob::QueueAdapters::StablematesWorkhorseAdapter)
+            raise ArgumentError, "task_group needs a queue: ActiveJob::Base does not use the Workhorse adapter"
+          end
+
+          adapter.workhorse_queue
         end
 
         def run_default(payload, task)
@@ -204,6 +276,42 @@ module Stablemates
         concurrency_key: "must be a non-empty String or respond to call"
       }.freeze
       private_constant :OPTION_RULES
+
+      # The +enqueue_after_transaction_commit+ values that defer an enqueue. Active Job 8.0 also
+      # accepts the deprecated :always.
+      DEFERRING = [true, :always].freeze
+      private_constant :DEFERRING
+
+      # Checks one job before Rails can defer its enqueue. A prepend on ActiveJob::Base runs before
+      # the module that Rails includes for the deferral. Internal to the SDK.
+      module JobDeferralCheck # :nodoc:
+        private
+
+        def raw_enqueue(...)
+          ::Stablemates::Workhorse::ActiveJob.refuse_deferral([self])
+          super
+        end
+      end
+
+      # Checks the jobs of +perform_all_later+ before Rails can defer them. Internal to the SDK.
+      module BatchDeferralCheck # :nodoc:
+        def perform_all_later(*jobs)
+          ::Stablemates::Workhorse::ActiveJob.refuse_deferral(jobs.flatten)
+          super
+        end
+      end
+      private_constant :JobDeferralCheck, :BatchDeferralCheck
+
+      # The checks wait for ActiveJob::Base. Loading it here would run its load hooks, and Rails
+      # would then look up this adapter before this file defines it.
+      ::ActiveSupport.on_load(:active_job) { prepend(JobDeferralCheck) }
+      ::ActiveJob.singleton_class.prepend(BatchDeferralCheck)
+      # Active Job 8.1 splits +perform_all_later+ in a module that it prepends when Rails includes
+      # the deferral, which can be after this file loads. The check therefore also goes into that
+      # module, so it runs before the split.
+      if ::ActiveJob::EnqueueAfterTransactionCommit.const_defined?(:ActiveJobMethods, false)
+        ::ActiveJob::EnqueueAfterTransactionCommit::ActiveJobMethods.prepend(BatchDeferralCheck)
+      end
     end
   end
 end
@@ -226,8 +334,9 @@ module ActiveJob
 
       # Enqueues +jobs+ in atomic chunks of the shared batch limit and returns how many it enqueued.
       # A job with an invalid priority records the error and is skipped. Each job reports only the
-      # error of this call.
+      # error of this call. Inside the block form of a task group, the jobs join the group instead.
       def enqueue_all(jobs)
+        builder = ::Stablemates::Workhorse::TaskGroupBuilder.current
         pending = jobs.filter_map do |job|
           job.enqueue_error = nil
           [job, workhorse.request(job, job.scheduled_at && Time.at(job.scheduled_at.to_f))]
@@ -236,6 +345,8 @@ module ActiveJob
           job.successfully_enqueued = false
           nil
         end
+        return grouped(builder, pending) if builder
+
         pending.each_slice(::Stablemates::Workhorse::SqlCatalogue::MAX_ENQUEUE_BATCH_SIZE) do |chunk|
           results = queue.enqueue_many(chunk.map(&:last))
           chunk.zip(results) do |(job, _request), result|
@@ -252,20 +363,37 @@ module ActiveJob
         !worker.nil? && worker.stopping?
       end
 
+      # The Queue this adapter enqueues through. Internal to the SDK.
+      def workhorse_queue # :nodoc:
+        @queue || @lock.synchronize do
+          @queue ||= ::Stablemates::Workhorse::Queue.new(
+            @executor || ::Stablemates::Workhorse::ActiveRecordExecutor.new(::ActiveRecord::Base)
+          )
+        end
+      end
+
       private
 
       def workhorse = ::Stablemates::Workhorse::ActiveJob
 
+      def queue = workhorse_queue
+
+      # Inside the block form of a task group, a job joins the group. The group enqueues it when the
+      # block returns.
       def enqueue_one(job, run_at)
-        job.provider_job_id = queue.enqueue_many([workhorse.request(job, run_at)]).first.task_id
+        request = workhorse.request(job, run_at)
+        builder = ::Stablemates::Workhorse::TaskGroupBuilder.current
+        return job.provider_job_id = queue.enqueue_many([request]).first.task_id if builder.nil?
+
+        builder.add_request(request) { |result| job.provider_job_id = result.task_id }
       end
 
-      def queue
-        @queue || @lock.synchronize do
-          @queue ||= ::Stablemates::Workhorse::Queue.new(
-            @executor || ::Stablemates::Workhorse::ActiveRecordExecutor.new(::ActiveRecord::Base)
-          )
+      def grouped(builder, pending)
+        pending.each do |job, request|
+          builder.add_request(request) { |result| job.provider_job_id = result.task_id }
+          job.successfully_enqueued = true
         end
+        pending.length
       end
     end
   end

data/lib/stablemates/workhorse/admin.rb

@@ -72,7 +72,11 @@ module Stablemates
 
     BulkRedrivePage = Data.define(:results, :next_cursor)
 
-    TaskCheckpoint = Data.define(:task_id, :name, :value, :attempt, :fence_token, :worker_id, :created_at)
+    # +input_fingerprint+ is the fingerprint of the input that produced +value+, or nil when the write
+    # gave no input (ADR 0098).
+    TaskCheckpoint = Data.define(
+      :task_id, :name, :value, :attempt, :fence_token, :worker_id, :created_at, :input_fingerprint
+    )
 
     TaskWait = Data.define(
       :task_id, :name, :mode, :duration_ms, :requested_wake_at, :wake_at, :attempt, :fence_token,
@@ -290,7 +294,12 @@ module Stablemates
 
       def resume_queue(queue, audit:) = set_queue_paused(queue, false, audit)
 
-      # Deletes every pending task in the queue and returns how many PostgreSQL deleted.
+      # Deletes every pending task in the queue and returns how many tasks it deleted or canceled.
+      # A task that a dependent in another queue waits for is canceled, not deleted.
+      #
+      # Do not call it in a transaction that already holds task row locks, for example after a
+      # cancel or an enqueue. Two purges run one after the other, so such a transaction can
+      # deadlock with another purge.
       def purge_queue(queue, audit:)
         name(queue, "queue")
         audit(audit)
@@ -301,6 +310,71 @@ module Stablemates
         Values.parse_integer(rows.first.fetch("deleted_count"))
       end
 
+      # The TaskGroup, or nil when no such group exists.
+      def get_task_group(group_id)
+        group_id = TaskGroups.group_id(group_id)
+        assert_compatible
+        row = @executor.rows(SqlCatalogue::GET_TASK_GROUP_V1, [group_id]).first
+        row && TaskGroups.group(row)
+      end
+
+      # One TaskGroupMemberPage of at most +limit+ member tasks after the position +after+, in the
+      # order they joined.
+      def list_task_group_members(group_id, after: 0, limit: 100)
+        group_id = TaskGroups.group_id(group_id)
+        limit(limit, TaskGroups::MAX_PAGE_SIZE, "list_task_group_members limit")
+        raise ArgumentError, "after must be a non-negative Integer" unless after.is_a?(Integer) && after >= 0
+
+        assert_compatible
+        rows = @executor.rows(SqlCatalogue::LIST_TASK_GROUP_MEMBERS_V1, [group_id, after.to_s, limit.to_s])
+        last = TaskGroups.probe_after(rows, limit)
+        more = !last.nil? && @executor.rows(SqlCatalogue::LIST_TASK_GROUP_MEMBERS_V1, [group_id, last.to_s, "1"]).any?
+        TaskGroups.member_page(rows, more)
+      end
+
+      # The group's TaskGroupEvent history in time order: create, seal, cancel, finish, callbacks,
+      # and repairs.
+      def list_task_group_events(group_id, limit: 1000)
+        group_id = TaskGroups.group_id(group_id)
+        limit(limit, TaskGroups::MAX_PAGE_SIZE, "list_task_group_events limit")
+        assert_compatible
+        @executor.rows(SqlCatalogue::LIST_TASK_GROUP_EVENTS_V1, [group_id, limit.to_s])
+          .map { |row| TaskGroups.event(row) }
+      end
+
+      # Seals an open group, so no task can join it after this call, not even from a running member
+      # task. Then cancels every member task that is not terminal. The canceled member tasks count as
+      # not succeeded, so the group finishes as usual and +on_finish+ and +on_failure+ run. A
+      # finished group stays unchanged. Returns the TaskGroup, or nil for an unknown group.
+      def cancel_task_group(group_id, audit:)
+        group_id = TaskGroups.group_id(group_id)
+        audit(audit)
+        assert_compatible
+        row = @executor.rows(SqlCatalogue::CANCEL_TASK_GROUP_V1,
+          [group_id, audit.actor, audit.reason, audit.request_id]).first
+        row && TaskGroups.group(row)
+      end
+
+      # The groups that are not finished and whose counters disagree with their member outcomes,
+      # and the sealed groups with no pending member, as TaskGroupDrift. It writes nothing, so it is
+      # the dry run of +repair_task_group_drift+.
+      def list_task_group_drift(limit: 1000)
+        limit(limit, TaskGroups::MAX_DRIFT_LIMIT, "list_task_group_drift limit")
+        assert_compatible
+        @executor.rows(SqlCatalogue::LIST_TASK_GROUP_DRIFT_V1, [limit.to_s]).map { |row| TaskGroups.drift(row) }
+      end
+
+      # Recounts each group +list_task_group_drift+ reports from its member outcomes, and finishes
+      # a sealed group with no pending member. Returns one TaskGroupRepair for each group. The
+      # +counter_repaired+ event of each group records the audit.
+      def repair_task_group_drift(audit:, limit: 1000)
+        audit(audit)
+        limit(limit, TaskGroups::MAX_DRIFT_LIMIT, "repair_task_group_drift limit")
+        assert_compatible
+        @executor.rows(SqlCatalogue::REPAIR_TASK_GROUP_DRIFT_V1,
+          [limit.to_s, audit.actor, audit.reason, audit.request_id]).map { |row| TaskGroups.repair(row) }
+      end
+
       private
 
       def set_queue_paused(queue, paused, audit)
@@ -561,7 +635,8 @@ module Stablemates
           attempt: Values.parse_integer(row.fetch("attempt")),
           fence_token: Values.parse_integer(row.fetch("fence_token")),
           worker_id: row.fetch("worker_id"),
-          created_at: Values.parse_time(row.fetch("created_at"))
+          created_at: Values.parse_time(row.fetch("created_at")),
+          input_fingerprint: row.fetch("input_fingerprint")
         )
       end
 

data/lib/stablemates/workhorse/batch_context.rb

@@ -16,6 +16,9 @@ module Stablemates
       # The member's ClaimedTask.
       def task = @context.task
 
+      # The TaskGroupContext of the member's task.
+      def task_group = @context.task_group
+
       # The member's CancellationToken.
       def cancellation = @context.cancellation
 
@@ -23,8 +26,9 @@ module Stablemates
       def get_checkpoint(name) = @context.get_checkpoint(name)
 
       # Returns the value the member's checkpoint named +name+ saved, running the block to produce
-      # and save it only when no such checkpoint exists.
-      def checkpoint(name, &) = @context.checkpoint(name, &)
+      # and save it only when no such checkpoint exists. The +input+ keyword works as in
+      # HandlerContext#checkpoint.
+      def checkpoint(name, input: HandlerContext::NO_INPUT, &) = @context.checkpoint(name, input: input, &)
 
       # The progress the member reported last, as TaskProgress, or nil.
       def get_progress = @context.get_progress