@fabricorg/platform-host 7.6.0 → 8.1.0

package/README.md

@@ -68,7 +68,8 @@ delay is the remaining lease duration, followed by the worker's next claim; ther
 retry or separate backoff. The next claim increments `attemptCount`. At the cap, or for an unleased,
 non-idempotent, or approval-resume invocation, Host records `failed` with the reason retry was refused,
 through the error-text redactor. Approval resume is excluded because its temporary lease does not
-establish that a recovery worker is configured. Ordinary thrown errors retain terminal failure behavior.
+establish that a recovery worker is configured. Ordinary application governance errors retain terminal failure behavior. Host store errors after
+a recorded provider effect follow the recovery rules below.
 
 The retry region includes Host's single-record store writes for mutation-governance and HITL
 checkpoints. Later attempts reuse these persisted decisions through the `mutationFootprint` and
@@ -76,7 +77,9 @@ checkpoints. Later attempts reuse these persisted decisions through the `mutatio
 A deferral happens before the attempt writes any policy evidence: the complete definition batch must
 evaluate before Host starts appending policy records. Policy persistence, the `ComplianceBlocked`
 event, obligation listing/resolution/persistence, state-machine checks, the handler transaction,
-handlers, adapters, and finalization retain their existing failure/reconciliation/recovery semantics.
+handlers and adapters retain their separate failure semantics. Finalization recovery is bounded by
+`recoveryMaxAttempts`. Its effective default is at least `retryableExecutionMaxAttempts` because
+both count worker claims.
 A retryable obligation-resolver error fails terminally with the policy evidence from that attempt.
 Preview evaluation does not enter this execution catch.
 
@@ -114,10 +117,10 @@ Host does not run adapters after this signal or add a new lifecycle event for it
 committed by earlier attempts are not undone.
 
 The signal does not change failures in pre-handler governance or validation, state-transition guard
-exceptions, adapters, or finalization. Governance and adapter throws retain their existing `failed`
-paths. State guards still fail closed. `after_adapters` finalization failures still escape as
-`RecoverableFinalizationError` for recovery; other finalization throws retain their existing failure
-path. An ordinary handler error still records `failed`. This signal requests an operator decision,
+exceptions, adapters, or finalization. Application governance errors and definite adapter failures retain their
+`failed` paths; post-success errors follow the recovery rules below. State guards still fail closed.
+Transient `after_adapters` finalization failures escape as `RecoverableFinalizationError` for
+recovery within `recoveryMaxAttempts`. An ordinary handler error still records `failed`. This signal requests an operator decision,
 not an automatic retry or a claim that the current transaction committed.
 
 New rows bind that key to the action version, actor/authority binding, and a versioned canonical
@@ -127,16 +130,295 @@ parameter comparison and call `onLegacyRecord`; actor, authority-binding, and ac
 still reject in enforce mode. See `MIGRATION-2-IDEMPOTENCY.md`.
 
 Lifecycle records use deterministic checkpoint ids. Recovered idempotent actions do not repeat an
-adapter that already reached `succeeded`, and event, policy, and adapter writes are append-safe. A
-stale action that was not declared idempotent fails terminally for manual reconciliation instead of
-silently rerunning unknown side effects.
-
-Adapters may classify outcomes as `success`, `transient_failure`, `permanent_failure`, `timeout`,
-or `ambiguous`. The retry layer retries only transient idempotent outcomes; permanent failures,
-timeouts, and ambiguous results are never retried. An ambiguous outcome — where the external effect
-may or may not have been applied — is routed to `reconciliation_required` with durable
-`ExternalReconciliation` evidence and an `AdapterInvocationAmbiguous` lifecycle event rather than
-blindly retrying an unsafe effect.
+adapter that already reached `succeeded`. Built-in stores deduplicate event and policy inserts by
+checkpoint ID. Adapter rows are mutable; stale-worker protection requires the optional
+`writeAdapterInvocation` capability described below. A
+reclaimed non-idempotent action inspects its adapter checkpoints: running or succeeded rows route
+to reconciliation; no rows or only failed rows retain terminal failure for manual review.
+
+Returned `ambiguous` or `timeout` outcomes, timeouts thrown from `adapter.execute` (including an
+aborted adapter deadline), contradictory successful results, and reverse contradictions
+(`success: false` with `outcome: "success"`) route to `reconciliation_required`.
+Host detects a thrown timeout from the adapter deadline's aborted signal or an error message matching
+`/timeout|deadline|timed?\s*out/i`. A definite rejection whose message matches therefore also routes to
+reconciliation; the message matcher does not prove that the provider timed out.
+
+An uninterpretable provider reply (a non-object, `null`, or an array), a getter that throws before Host
+can interpret the reply, and an unrecognised outcome string such as `"timed_out"` also take this path.
+Host copies the reply's contract fields and other own enumerable fields into a plain object before
+interpretation, reading each top-level field once. A getter that throws during the copy takes the
+unreadable-reply path. These outcomes are not retried by the retry helper. Object replies retain the
+previous truthiness reading of `success`, including truthy non-boolean values from JavaScript adapters.
+The reconciliation kind remains `adapter_outcome_ambiguous`.
+
+Unknown-outcome settlement uses `transactionWithEvents` when its transaction supplies
+`updateAdapterInvocation`, `appendExternalReconciliation` on a governance-capable store, and
+`updateLeasedActionInvocation` for a leased invocation. A store advertising conditional adapter writes
+also needs transaction-scoped `writeAdapterInvocation` for leased finalization. In that transaction Host writes the
+`ambiguous` checkpoint, `AdapterInvocationAmbiguous` with its stable event ID, governance
+`ExternalReconciliation` evidence where supported, and the fenced `reconciliation_required` verdict,
+in that order. These writes commit together. Host retries once on a non-lease error; a second
+non-lease failure selects the verdict-first fallback. Lease loss raises
+`RecoverableFinalizationError` without retry or fallback.
+
+Without those transaction members, or after two non-lease transaction failures, the verdict-first
+fallback writes the terminal verdict (retrying once except on lease loss), then the checkpoint,
+event, and reconciliation record. If both checkpoint shapes fail, Host still attempts the ambiguity
+event and reconciliation record. After the verdict commits, Host attempts each remaining evidence
+write even if the event append fails, then surfaces the first error as `RecoverableFinalizationError`.
+An evidence failure can leave a terminal verdict with missing evidence; a later direct execution returns that verdict
+and workers exclude it from recovery, without repairing the evidence gap. If both verdict attempts
+fail, Host surfaces `RecoverableFinalizationError` without attempting an unfenced checkpoint.
+
+With the original action definition and adapter-step order available, recovery checks `ambiguous`
+rows before the parked-handoff shortcut, stale-action gates, authorization, and handler execution.
+This applies to actions declared idempotent and actions not declared idempotent. Settlement uses the
+stored reconciliation metadata or the legacy row's error as its message and reason, without running
+the handler or an adapter. For a running invocation whose action resolves, `executeInvocation` reads
+each adapter step's row before the pending-handoff shortcut, stopping on recorded ambiguity or a
+failed read. A failed read surfaces `RecoverableFinalizationError`, where base 939bcad returned the
+parked result. The shortcut's condition is unchanged from base: a running invocation with
+`pendingCompletion` and no lease owner returns early without requiring `pendingCompletion.parkedAt`.
+If the action no longer resolves, the shortcut still returns the handoff before the unknown-action
+failure. Completion callbacks that settle the invocation require parking, including after a deadline sweep.
+
+`AdapterInvocationRecord.reconciliation` is an optional, redacted checkpoint preserving reason,
+external reference, and the original `adapterEventSubject`. The subject stays out of the returned
+`AdapterReconciliationOutcome`. Recovery uses this checkpoint without scanning `listEvents()`.
+Without stored subject metadata it uses `{ subjectType: "AdapterInvocation", subjectId: adapterInvocationId }`;
+the stable event ID deduplicates an event already written. Custom stores should persist the checkpoint
+through adapter create, update, and read methods.
+
+Ordinary adapter writes do not name the migration 8 column and remain compatible with schemas
+lacking it. If a checkpoint write carrying `reconciliation` fails, fallback retries the row once
+without that field, using legacy `status: "ambiguous"` plus `error`. Unknown outcomes therefore still
+settle without migration 8, but without stored checkpoint metadata. Apply PostgreSQL migration 8
+through `ensureSchema()` in controlled startup or migrations to retain that metadata. An error inside
+an atomic PostgreSQL transaction first rolls it back; the legacy write runs outside that transaction
+on the fallback path.
+
+On a store without atomic support, if the verdict cannot be written at all, no durable write after the
+call succeeds: neither verdict nor checkpoint is saved, as after a crash mid-call. A leftover `running`
+step can then re-run on recovery for an idempotent action under the same `adapterInvocationId`
+idempotency key, within the worker recovery budget. A reclaimed non-idempotent action with a
+running row without a matching durable handoff settles that step as unknown. A succeeded row or
+durable handoff instead ends `reconciliation_required` as an applied effect. Adapters should be
+idempotent or carry that stable key. The same exposure applies if two atomic attempts roll back
+and the fallback verdict cannot be written.
+
+Once an invocation has a durable `succeeded` adapter row or accepted handoff, errors in Host
+store reads or writes raise `RecoverableFinalizationError` instead of ending it `failed`.
+This includes later adapter checkpoints, obligation reads, and the completion write in both
+`before_adapters` and `after_adapters`. Recovery also protects Host store operations before
+and around the handler when an earlier provider effect is recorded. Authorization, policy and handler callbacks
+retain their own failure semantics. Idempotent recovery repairs recorded success/handoff evidence
+without calling that step's provider. Non-idempotent interruption ends `reconciliation_required`
+instead of repairing the invocation. Lease loss retains its fencing error.
+
+If neither record is durable, Host retries the failed handoff or marker write once. After a
+second failure it reuses degraded atomic finalization where available, otherwise writes the
+`reconciliation_required` verdict first, then attempts the succeeded marker with safe output,
+`ExternalOperationAccepted` when applicable, and `AdapterInvocationSucceeded`. Each fallback
+evidence write is attempted independently; the first error raises `RecoverableFinalizationError`
+after the remaining writes are attempted. One exception keeps its 7.x behaviour: when
+unsatisfied obligations end a handed-off invocation `reconciliation_required` and the
+reconciliation-log append fails, Host still returns the verdict. A terminal verdict can retain
+missing evidence. If even the
+verdict cannot be saved, Host throws `RecoverableFinalizationError`; the row can remain `running`.
+Recovery may replay that unrecorded outcome for an idempotent action under its stable
+`adapterInvocationId`, within the worker budget. A reclaimed non-idempotent action instead
+ends `reconciliation_required`. A durable handoff takes precedence over its running row and is
+recorded as an applied effect with its external reference. A running row without such a handoff
+is settled as unknown. A succeeded row also ends `reconciliation_required`. With no rows or only failed rows, the existing
+non-idempotent interruption gates still fail the invocation.
+
+For idempotent recovery, the handler still runs before the adapter loop. The loop reads each
+checkpoint before `getInput()` or adapter lookup. Succeeded rows and durable handoffs use stored
+adapter type, operation, vendor, input and output; a falsy new input or removed adapter does not
+skip evidence repair. Stable event IDs deduplicate replayed accepted and success events; newly
+repaired success events include `recovered: true` and stored input/output when available.
+
+Recovery looks up the stable attestation ID. If present, Host reuses it and reapplies its
+`satisfiesObligations` updates without calling the extractor. If absent and stored output exists,
+Host calls `extractExecutionAttestation` with stored input/output and checkpoint `createdAt` and
+`updatedAt` as start/completion times. The extractor may run again after an interrupted append;
+it must return deterministic evidence for the same input. A handoff checkpoint with no output,
+or withheld output, skips extraction. Repair errors remain recoverable within the budget.
+
+Known limits: a later step's definite provider failure or pre-call refusal can still end the
+invocation `failed` after an earlier step succeeded. Authorization denials, ordinary handler
+failures, validation/state-guard rejection, and unsatisfied policy obligations on recovery retain
+their existing terminal outcomes. These are decisions or application failures, not Host store
+errors. Idempotent recovery can call a provider again for an unrecorded running outcome, or for a
+definite failed row whose invocation failure verdict was lost. The handler still re-runs before
+the adapter loop. Saga compensation is unchanged. Withheld-output and refused-handoff
+reconciliation retain the degraded pattern. Adapter-row fencing and overdue-sweep/late-callback
+rules are described below.
+
+Event appends are first-write-wins by event ID. Host re-appends lifecycle events under stable IDs
+on retry and recovery, and each later attempt carries a fresh `sequence` and timestamps and can carry
+a different payload. A custom store's `appendEvent` and `appendEventWithOutbox` must treat an ID it
+already holds as a no-op that keeps the first event: do not throw, replace, or merge it. A store that
+refuses these appends can leave recovery unable to converge, so the invocation stays `running` until
+the recovery budget settles it, even when the adapter effect succeeded. Custom stores that currently
+throw on a conflicting append need to change. The shipped stores enforce ID uniqueness across the
+whole store, not per tenant or space. The rule covers the outbox too: `appendEventWithOutbox` for an
+ID already stored adds no outbox row, including when the first append went through plain
+`appendEvent`, so that event is never published later. Keep `outbox.shouldPublish` deterministic for
+a given event ID. Custom store authors can verify these paths with the
+[event append conformance checks](#event-append-conformance).
+
+### Conditional adapter-row writes (8.0.0, issue #79)
+
+`PlatformHostStore.writeAdapterInvocation(input)` is optional. Memory and PostgreSQL implement it;
+custom stores can keep their existing methods. Without it, Host retains unchecked adapter creates
+and updates, so a reclaimed worker can overwrite a newer worker's adapter checkpoint.
+
+The input carries `scope` (tenant, space, invocation), a create or update `mutation`, a `condition`,
+and optional `expectedStatuses`. A lease condition requires a `running` parent with the supplied
+`leaseOwner` and `leaseToken`. A verdict condition requires the exact parent `status` and `error`
+written by this execution. Host also captures the handoff row ID as
+`completionAdapterInvocationId`, so `externalCompletionDigest` guards that row's callback output:
+an absent digest requires no recorded completion; a supplied digest requires that same completion.
+Other adapter rows retain the status/error and expected-row-status checks without the digest check.
+Omitting the captured row ID retains the conservative digest check on adapter writes. Absent error
+matches no error. Both parent and adapter must match scope.
+A create must also match the record's scope and refuses a duplicate ID. The method returns `false`
+when its conditions fail. Conditional updates reject `succeeded` or `ambiguous` to `running` or
+`failed`, including when callers omit `expectedStatuses`.
+
+Host captures the supplied execution credentials before loading the invocation. It uses them for
+fenced invocation writes and leased adapter creation, retry preparation, success, failure, and
+ambiguity checkpoints. A refused lease or row-status check raises `RecoverableFinalizationError`.
+This prevents stale checkpoint writes; adapters still need stable idempotency keys because a lease
+cannot cancel a provider call already in flight. Inline writes use the verdict condition after
+this execution writes its own verdict; earlier unleased adapter writes retain their existing path.
+
+Verdict-first fallbacks use the settled verdict condition after the terminal write releases the
+lease. Their expected row status is the last state observed or written by that execution. A changed
+parent status/error or row refuses the adapter update; a changed completion also refuses a write
+to the captured handoff row. These refusals emit
+`invocation.adapter_evidence_refused` (`fabric.platform.invocation.adapter_evidence_refused`) through
+the telemetry hook. The adapter row stays untouched, with no unchecked retry. Host still attempts
+`AdapterInvocationAmbiguous` and reconciliation evidence; withheld-output and refused-handoff
+success evidence may already have been appended before the refused marker write. A refusal is not
+an evidence-collector error. Thrown store errors are retained while each independent fallback
+write is attempted, then the first error is raised as `RecoverableFinalizationError`.
+
+An inline writer preserving a sweep verdict does not adopt it as this execution's own verdict.
+Its adapter evidence keeps the unchecked inline path; conditional settled-parent enrichment is
+used only for a verdict this execution actually wrote. The inline guard reads the current row
+only when the execution snapshot or its patch carries `pendingCompletion`.
+
+Budget exhaustion saves its plain verdict before reading evidence. It then enriches the parent
+through optional `updateSettledActionInvocation`, comparing that execution's exact status, error,
+and completion snapshot. Parent enrichment joins the independent evidence collector, alongside
+adapter checkpoints, events and reconciliation records. Refusal emits
+`invocation.adapter_evidence_refused` with `evidence: "parent_enrichment"` and is not an error;
+a missing `updateSettledActionInvocation` skips only parent enrichment. Evidence writes still run.
+A thrown store error is collected and raised as `RecoverableFinalizationError` after the remaining
+writes are attempted. The two optional store methods are expected together. Successful enrichment
+stores the error and `adapterReconciliation` before returning them; skipped or refused enrichment
+can leave replay with only the plain verdict. Stores omitting `writeAdapterInvocation` retain
+unchecked adapter writes. A retained handoff's reference is included in the initial budget verdict.
+
+Subclasses of the built-in stores that override `createAdapterInvocation` or
+`updateAdapterInvocation` for encryption or auditing must also override `writeAdapterInvocation`.
+Checked writes can bypass the unchecked methods. Alternatively, explicitly disable the capability
+and accept the weaker stale-worker guarantee. Parent annotations similarly have their own
+`updateSettledActionInvocation` seam.
+
+Atomic finalizers require transaction-scoped `writeAdapterInvocation` when the store advertises the
+capability and execution holds a lease. A missing member selects verdict-first fallback before
+transaction evidence writes. PostgreSQL locks the matching parent with `FOR UPDATE` before the
+adapter mutation in one SQL statement. Memory rechecks buffered lease and adapter predicates at
+commit and excludes claims and other conditional writes while applying the commit. A commit-time
+conflict rolls back and raises `RecoverableFinalizationError`. No schema migration is added by this
+capability; it uses the existing parent lease and verdict columns.
+
+External-completion adapter writes use the callback's transaction-scoped `updateAdapterInvocation`
+when available, sharing its invocation lock. These are callback writes rather than worker-lease
+writes; legacy transactions without that member retain the outer-store write. The sweep retains
+its transaction-scoped reconciliation append and parked/overdue settlement rules.
+
+### Bounded worker recovery (8.0.0)
+
+`recoveryMaxAttempts` is an optional positive integer. Its effective default is
+`max(5, retryableExecutionMaxAttempts)`; `Infinity` is rejected. Both caps count the same worker
+claims. Construction rejects an explicit `recoveryMaxAttempts` below
+`retryableExecutionMaxAttempts`. The governance cap stops transient pre-policy deferrals at its
+cap; the recovery budget stops resumed execution above its cap. With the defaults, governance
+retry exhaustion therefore happens on claim 5 before recovery exhaustion could happen on claim 6.
+
+Host checks a resumed leased `running` invocation immediately after lease ownership, before
+normal action resolution, ambiguity scans, authorization, the handler, or adapter work. When
+`attemptCount > recoveryMaxAttempts`, it settles `reconciliation_required` with
+`recovery attempts exhausted (n/max)`. It writes that verdict before resolving the action or
+reading adapter evidence, releasing the lease. A running row without a matching durable handoff
+receives unknown-outcome settlement. A recorded ambiguous row enriches the parent error and
+`adapterReconciliation`, as well as adapter/event evidence, using its stored reconciliation evidence
+and a conditional parent update. A durable handoff takes precedence over that step's running row.
+It is recorded as an applied effect with its external reference in the message and, on governance
+stores, the reconciliation record. It is not marked ambiguous solely because its marker was lost.
+If resolution or row reads throw, or no relevant row exists, the plain verdict already written
+stands. Enrichment failure can retain an evidence gap. Transition telemetry is emitted once when
+the status changes. If the verdict is unwritable, Host throws `RecoverableFinalizationError`;
+later claims continue but do not execute mutation code.
+
+The counter counts worker claims, including the first run, rather than provider calls or completed
+recoveries. With the default of 5: the first run plus four recoveries; the sixth claim settles.
+A raised governance cap raises the implicit recovery budget with it. Inline/direct execution does
+not increment the counter. Approval acceptance sets it to at least 1 without incrementing it on
+resume. Approval resume holds a temporary lease and uses the existing count: a count equal to the
+budget is admitted, while a count above the budget settles reconciliation before the handler.
+Approval does not reset an exhausted budget. A worker dying before reaching the check consumes a
+claim without calling a provider at that point. Parked handoffs have no lease, are not claimed by
+workers, and are outside this budget. The existing parked shortcut and ambiguity scan remain in
+place. `retryableExecutionMaxAttempts` still limits pre-policy deferrals. No new store method is
+required.
+
+**Inline and direct callers.** Inline `submitAction()` without a dispatcher and direct
+`executeInvocation()` can now reject with an error whose `name` is
+`"RecoverableFinalizationError"` where 7.x returned `failed`. The class is not exported. Without a
+lease, the invocation stays `running` and workers do not claim it. Before a handoff is durable,
+call `executeInvocation()` again yourself and cap your own retries. Once a handoff is durable,
+re-running `executeInvocation()` on the unleased running row returns without resuming execution
+(it first settles any step already recorded `ambiguous`).
+If execution stopped before parking, callbacks are refused with `still_executing` until a sweep.
+Declare `completionDeadlineMs` for actions that hand off so the sweep surfaces the row, then
+reconcile it per the [external-completion runbook](../../docs/operations/runbooks.md#overdue-handoffs).
+The leased worker budget does not bound unleased calls. A
+previously leased direct call retains its lease until expiry and can also be recovered by a
+worker. Terminal degraded verdicts can retain evidence gaps and are not automatically repaired.
+
+A transaction implementation may rethrow the handler's error wrapped in its own error; Host
+still treats a failure inside the handler callback as a handler outcome.
+
+**Known limit.** If a later step's adapter row is created but its attempt-preparation write keeps
+failing, the row stays `running` although the provider was not called. For an action not declared
+idempotent, the next claim settles that step as an unknown provider outcome.
+
+
+A deadline sweep retains the handoff and sets `pendingCompletion.overdueAt` with its
+`reconciliation_required` verdict. A late callback settles that verdict only when the handoff also
+has `parkedAt`, proving Host reached parking after its remaining steps and finalization. An unparked
+handoff stays in reconciliation even if later-step ambiguity evidence is missing; the callback's
+redacted result and outcome are recorded as evidence for an operator without applying the action's
+result schema. This evidence-only behavior applies to invocations already
+`reconciliation_required`; callbacks for unparked `running` invocations are refused with
+`still_executing`. Any parked row without `overdueAt`, including a 7.x row, still uses the old
+`"did not complete by"` error substring. That compatibility path is weaker: an unrelated error
+containing the substring on a parked row lacking the marker can also authorize settlement.
+
+Circuit-open refusals and deadline expiry before execution retain `failed` semantics because
+the refused attempt made no provider call. A `prepareAttempt` store error retains its prior failure
+behavior only while the invocation has no durable success or handoff. With an earlier recorded
+effect, an exhausted Host-store error raises `RecoverableFinalizationError`. Returned `permanent_failure`
+and exhausted `transient_failure` outcomes, plus exhausted non-timeout adapter throws, also retain
+`failed` semantics and their existing events, unless reading any field of the reply throws: Host
+copies the reply before interpreting it, so such a reply takes the unreadable-reply path to
+reconciliation. Adapters report definite non-effects with
+`permanent_failure` or `transient_failure`; no additional definite-failure flag is needed.
 
 ### Adapter circuit breakers
 
@@ -250,9 +532,10 @@ in action parameters or workflow history.
 Production stores can implement `AtomicMutationPlatformHostStore.transactionWithEvents`. The
 transaction-scoped `db`, event sequence, event append, event listing, and invocation update methods
 must all use the same database transaction.
-The optional transaction-scoped `updateAdapterInvocation` and `appendExternalReconciliation`
-operations extend this unit of work to degraded adapter finalization, as described under redaction
-below.
+The optional transaction-scoped adapter-update and `appendExternalReconciliation` operations
+extend this unit of work to degraded adapter finalization. Leased finalizers use
+`writeAdapterInvocation` when the store advertises it, otherwise `updateAdapterInvocation`,
+as described under redaction below.
 
 `PostgresPlatformHostStore` enables this capability when constructed with a
 `PostgresPlatformHostTransactionProvider`. The application owns that narrow binder because only the
@@ -265,7 +548,8 @@ With `transactionWithEvents`:
 - `after_adapters`: declared domain events, the Host-generated `StateTransitioned` fact, and
   invocation finalization commit together after required adapter steps succeed or accept a
   parked external handoff. Handler writes have already committed separately. A finalization
-  failure leaves the invocation recoverable; succeeded adapter checkpoints are not repeated.
+  failure leaves the invocation recoverable within `recoveryMaxAttempts`; recovery repairs evidence
+  for succeeded adapter checkpoints without repeating those calls.
 
 Stores without `transactionWithEvents` retain the legacy non-atomic boundary for domain writes,
 events, and invocation persistence. A partial append or finalization failure can leave some
@@ -341,7 +625,7 @@ Three optional `PlatformHostOptions` hooks cover other Host-recorded values.
 | --- | --- |
 | `redactAdapterOutput(actionId, adapterType, operation, output)` | Once after a successful adapter result is converted to output, before handoff handling. The adapter row, `AdapterInvocationSucceeded` payload, and `extractExecutionAttestation` input use the redacted output. |
 | `redactErrorText(context, message)` | At the origin of caught handler/execution errors, adapter result errors and ambiguous-outcome reasons, and completion errors. The invocation/adapter errors, reconciliation messages and reasons, completion evidence, failure events, and returned error reuse that text. |
-| `redactExternalCompletionResult(actionId, provider, result)` | After a completion matches its invocation, before merging, result-schema validation, persistence, and digesting. See [External completion](#external-completion). |
+| `redactExternalCompletionResult(actionId, provider, result)` | After a completion matches its invocation, before persistence and digesting; successful settling callbacks also merge and validate against the result schema. See [External completion](#external-completion). |
 
 `ErrorTextRedactionContext` is exported. It contains `actionId` and `source` (`handler`, `adapter`,
 `external_completion`, or `execution`); adapter errors also supply `adapterType`, `operation`, and
@@ -367,40 +651,53 @@ callback can still land as evidence. References come from the original adapter r
 For withheld output and refused handoffs (immediate completion or a second handoff), Host uses
 `transactionWithEvents` when the transaction supplies `updateAdapterInvocation`,
 `updateLeasedActionInvocation` for a leased invocation, and `appendExternalReconciliation` for
-withheld output on a governance-capable store. These optional members must
+withheld output or refused handoffs on a governance-capable store. Leased finalization also needs
+transaction-scoped `writeAdapterInvocation` when the store advertises it. These optional members must
 share the event/verdict transaction; the adapter update must enforce the store method's supplied
 tenant, space, and invocation scope. Both built-in stores provide them when configured with a
 transaction provider. Custom stores must implement them to get this atomic behavior.
 
-When atomic finalization succeeds, evidence events, any withheld-output reconciliation entry, the adapter's `succeeded`
-marker, and the terminal `reconciliation_required` verdict commit together. Host writes evidence and
+When atomic finalization succeeds, evidence events, any withheld-output or refused-handoff
+reconciliation entry, the adapter's `succeeded` marker, and the terminal `reconciliation_required`
+verdict commit together. Host writes evidence and
 the marker before the fenced verdict, which releases the lease, then emits status telemetry after
 commit. It retries the whole transaction once on a non-lease error. If the retry also fails with a
 non-lease error, Host selects the verdict-first fallback described below. Lease loss is not retried
 or converted to fallback; it raises `RecoverableFinalizationError`. Atomic rollback leaves the
 invocation and adapter row `running` until a retry or fallback commits. If lease loss or an unavailable
 fallback verdict prevents finalization, the invocation remains reclaimable and the adapter can run
-again under the same `adapterInvocationId` idempotency key. Adapters should be idempotent or carry
-that stable idempotency key.
+again under the same `adapterInvocationId` idempotency key for idempotent actions within the
+worker recovery budget. Non-idempotent reclaim inspects its checkpoints instead of replaying a call.
 
 Without the required transaction members, or after two non-lease transaction failures, Host keeps
-the verdict-first order: the `reconciliation_required` verdict precedes evidence and the adapter marker. It retries the verdict
-write once, except on lease loss. A later evidence or marker failure raises
-`RecoverableFinalizationError` after the verdict commits and can leave a terminal verdict with missing
-evidence, including when the same deterministic failure affects the transaction and fallback. If both verdict-write attempts fail, the adapter can run again on recovery,
-the same exposure as a crash mid-step; adapters should be idempotent or carry a stable idempotency key.
-Direct execution returns a durable terminal verdict and recovery excludes it through terminal-status
-checks. Refused-handoff completion reconciliation logging stays after finalization. After atomic
-finalization, a failed log write leaves the committed verdict and adapter marker intact if the
-unfenced invocation-error update succeeds. If that error update also fails, the existing adapter-error
-catch can mark the adapter and invocation `failed`.
+the verdict-first order: the `reconciliation_required` verdict precedes evidence and the adapter marker.
+It retries the verdict write once, except on lease loss. After the verdict is durable, Host attempts
+each evidence write independently: a failed event, reconciliation append, or adapter checkpoint does
+not skip the remaining writes. Host raises the first evidence failure as `RecoverableFinalizationError`
+only after those attempts finish. This includes unknown-outcome settlement and post-success degraded
+finalization. Failed writes can still leave a terminal verdict with missing evidence; terminal replay
+does not repair that evidence gap. If both verdict-write attempts fail, the adapter can run again on
+recovery for an idempotent action within the worker recovery budget, the same exposure as a crash mid-step.
+A reclaimed non-idempotent action settles a running checkpoint as unknown.
+
+A refused handoff's provider reference is included in the step's `AdapterInvocationSucceeded` event
+even when explicit adapter output omits it. Capable atomic transactions append its reconciliation
+record with the step's evidence. The fallback independently attempts the reference-bearing event,
+the governance reconciliation append, the outer `recordCompletionReconciliation` append, and the
+adapter marker. Both reconciliation appends use the same stable id to deduplicate repeated attempts.
+A failure of either append is reported after the remaining fallback evidence writes are attempted.
+A non-sweep verdict may receive a log-unavailable error note; a preserved sweep verdict keeps its error.
+After successful atomic finalization, the outer append retains its existing tolerated-failure behavior.
+If its annotation write fails, Host raises `RecoverableFinalizationError` while keeping the committed
+verdict and adapter marker intact; terminal replay retains the verdict.
 
 `extractExecutionAttestation` receives the redacted output, so an extractor that needs a stripped
-field can fail after the external effect, following the existing extractor-failure behavior.
+field can fail after the external effect. Host leaves a durable success checkpoint recoverable
+within `recoveryMaxAttempts`; repair may call the extractor again.
 
 `redactActionParameters` and `redactAdapterInput` behavior is unchanged. Independently of hook
-configuration, a stale worker that loses its lease inside an adapter step rethrows without marking
-the adapter failed, preserving the new owner's row. Refused-handoff ordering and audit events also
+configuration, a refused fenced invocation write inside an adapter step rethrows. With
+`writeAdapterInvocation`, stale adapter updates also refuse before marking the new owner's row failed. Refused-handoff ordering and audit events also
 change without the new hooks.
 
 Applications own the safety of domain events/results, policy and HITL evidence, explicit
@@ -643,8 +940,10 @@ binding's target; otherwise Host's later fact sets the recorded state to the tar
 
 Configure `redactExternalCompletionResult` to withhold sensitive completion fields. Host applies it
 once to a supplied result (including a result attached to a failed callback). The redacted value
-feeds `externalCompletion.result`; for a completed outcome it also feeds the adapter output,
-`ExternalOperationCompleted` payload, merged invocation result, and returned result. A supplied
+feeds `externalCompletion.result`; for a completed outcome it also feeds the adapter output and
+`ExternalOperationCompleted` payload. Only a settling successful callback merges it into the
+invocation result and validates that merge against the action's result schema. A non-settling
+callback returns the stored invocation result and keeps the callback result in its evidence. A supplied
 `completion.error` passes through `redactErrorText` before recording, including when attached to a
 completed outcome. A missing error uses Host's fixed failure message, which bypasses the hook.
 
@@ -691,13 +990,55 @@ refused. A repeated identical completion is absorbed, identity being a digest of
 reference, outcome, result, error, and evidence. A contradictory one, a flipped outcome or the same
 outcome with different evidence, moves the invocation to
 `reconciliation_required`, keeps the first outcome on record, logs the second, and emits
-`ExternalOperationContradicted`. Under an atomic store the read and write happen under one lock. The
-callback's redacted result is merged with the invocation result, stripped of private fields, and
-validated against the action's result schema before writes. The stored merge is what is validated.
+`ExternalOperationContradicted`. With `transactionWithEvents` and transaction-scoped
+`getActionInvocationForUpdate`, the callback read and verdict write share a lock. The
+successful settling callback's redacted result is merged with the invocation result, stripped of
+private fields, and validated against the action's result schema before writes. A non-settling
+callback records its redacted result/outcome as evidence without schema validation or merging it
+into the invocation result.
 An invocation that ended for another reason while a handoff was pending keeps its status and records
 the callback as evidence. Declare `completionDeadlineMs` on `long-running` actions;
-`reconcileOverdueCompletions()` moves overdue handoffs to reconciliation while keeping the handoff so
-a late completion can still land, and the worker cycle runs that sweep before it claims.
+`reconcileOverdueCompletions()` moves running, unleased, due handoffs to reconciliation and retains
+the handoff with `overdueAt` set to the sweep's current time. The worker cycle runs the sweep before
+claiming work. A callback can settle a parked running invocation, even past its deadline before the sweep. From
+reconciliation, settlement requires the parked, swept handoff or the substring fallback described above;
+a swept unparked handoff retains its verdict and records the callback in `externalCompletion`,
+adapter evidence, and a completion/failure event. It clears the pending handoff without claiming
+that the invocation's remaining steps ran. The markerless parked-row fallback is described above.
+
+For each sweep candidate, Host re-reads status, lease ownership, handoff identity
+(`adapterInvocationId` and `externalReference`), due time, and absence of `overdueAt`. With
+`transactionWithEvents` and `getActionInvocationForUpdate`, this check and the verdict/marker write
+share the row lock. Reconciliation evidence joins that transaction when it exposes
+`appendExternalReconciliation`; otherwise evidence is written on the outer store and can survive a
+rolled-back verdict. A custom transaction exposing `getActionInvocationForUpdate` without
+`appendExternalReconciliation` makes the sweep append on the outer store while holding the
+invocation row lock. If the reconciliation table references that row through a foreign key, the
+outer append can block waiting for the held lock. Implement both members in the same transaction.
+Stores without a transaction lock use an immediate re-read, but retain a race
+between that read and the write: a concurrent callback or execution can still be overwritten.
+Custom stores must preserve and reload the optional `overdueAt` Date in the handoff JSON; PostgreSQL
+hydrates it without a new migration.
+
+Inline writes bypass the extra guard read and transaction when neither the loaded invocation nor
+its patch carries `pendingCompletion`. Ordinary inline completion retains its base store cost.
+Writing a handoff into a running invocation from anything other than its executing step is not
+supported; an execution without that handoff in its snapshot does not guard against such imports.
+Inline execution preserves a stored `reconciliation_required` verdict while committing independent
+adapter checkpoints, events, and reconciliation records, including references from refused handoffs.
+If execution reaches parking after the sweep, it records the handler result and `parkedAt` on the
+same handoff, retaining the sweep's status, error, and `overdueAt`. Successful `after_adapters`
+finalization also commits its deferred domain events: the handler and remaining adapters have
+succeeded. Inline callers receive the latest
+record observed by the preserving writer, merged with its evidence update, after these writes
+succeed; there is no post-attempt read. Leased execution retains `RecoverableFinalizationError`
+on fencing failure. The transaction lock serializes the check with the sweep where available; stores
+without it retain a read/write race. This does not cancel an adapter in flight or undo earlier
+domain writes. Verdict-first fallbacks attempt independent evidence writes before raising the first
+failure as `RecoverableFinalizationError`. Reconciliation-log failures leave the sweep error unchanged. Concurrent inline executions of the
+same invocation can still raise `RecoverableFinalizationError` with `Invocation is no longer running`
+when another execution finalized the row first. This error is internal; callers classifying it can
+match its `name` and inspect the stored invocation before deciding how to recover.
 
 The handoff is persisted the moment it is accepted, before the adapter step is marked done, so a
 worker that dies between the two leaves the handoff and not the marker. A custom store has four
@@ -736,10 +1077,93 @@ Host passes `AdapterInvocationScope` from the store-loaded action invocation to
 must match `tenantId`, `spaceId`, and `actionInvocationId` when scope is supplied. A mismatch returns
 `undefined` on read and leaves the row unchanged on update (a not-found error is permitted).
 The trailing argument is optional to preserve existing custom-store and direct-call compatibility;
-calls without it retain ID-only behavior in the shipped stores. The two adapter isolation checks
+calls without it retain ID-only behavior in the shipped stores. When `writeAdapterInvocation` is present, the write probe also checks its scoped create/update
+refusals and an admitted update in our scope. The two adapter isolation checks
 fail for stores that ignore scope. Deployments running `runTenantIsolationChecks` against a custom
 store will see new failures until that store honours the argument.
 
+## Event append conformance
+
+`eventAppendChecks()` and `runEventAppendChecks(subject)` run six checks for first-write-wins
+event IDs: plain appends, outbox appends, outbox-after-plain, transactional appends,
+transactional outbox appends, and transactional outbox-after-plain. Each sends earlier and later
+duplicates with lower and higher sequences, earlier and later occurrence/recording times, and
+different remaining envelope and outbox metadata fields. Transaction checks cover buffered
+duplicates and duplicates against a committed event; they do not require transaction reads to
+expose buffered writes. Each committed-duplicate transaction appends a fresh plain control event
+after the duplicate and verifies its presence after commit, detecting a silently rolled-back unit
+of work. Event-count checks count only the original probe ID; controls use distinct sequences.
+
+The checks detect common first-write-wins violations: throwing, replacing or merging events,
+keeping both events, selecting the earlier-ordered or later-ordered duplicate, adding outbox
+rows, and resetting or rewriting an existing outbox row in the probed states. They are not
+exhaustive. Known limits include rules that compare ordering fields in mixed directions,
+store-wide uniqueness across spaces or tenants, the dead-letter state, and claims that lease
+foreign rows without returning them. They also cannot detect a buffering transaction that merges
+fields the store never returns (such as `includeProvenance`) from a buffered duplicate, or
+duplicate handling backed by an in-process cache that works during one run but fails across
+restarts. Stores requiring gapless sequences can fail falsely: the first event uses sequence 50
+in a fresh space. So can stores with a unique key over fields the control events share with the
+probe event, such as `actionInvocationId` with the subject.
+
+Outbox-creating checks verify the first event and metadata, then compare whole-row snapshots
+while pending, leased by the probe worker, retry-pending, and settled as published. Retry-pending
+accepts a cleared lease, an expired lease with no owner or the probe worker, or the probe owner
+retained as a record without an expiry. A probe lease that still expires more than five seconds
+after the probe's clock fails with `markOutboxFailed did not release the lease`; an expiry inside
+that margin is treated as clock skew and waited out before the row is claimed again. A lease held
+by another worker fails with the request to pause relays. The checks prescribe neither a positive attempt
+count nor error formatting; they snapshot the values the store returns. Retry availability uses
+whole seconds a few seconds in the past, tolerating coarse timestamp precision and small clock
+skew, then the probe reclaims the row before publication. Snapshots include the full envelope,
+lease and publication fields. Comparisons sort object keys, normalize dates and date strings to milliseconds, treat null like undefined, and
+compare numeric strings in `sequence`, `attemptCount` and `leaseToken` as numbers. An absent
+actor disclosure is accepted; included disclosure or actor identity on the first outbox envelope
+fails because the first metadata does not opt in. A store's first returned records supply the
+snapshot references; extra envelope fields and fields the store omits or returns as null are
+accepted, including required fields. This tolerance does not validate envelope completeness. For
+buffered duplicates in the first transaction, exposed fields that differ between the first input
+and a duplicate must retain the first value. Outbox-after-plain checks write a fresh control
+event through the matching outbox seam and require its row to be the only outbox row visible in
+the probe space, so absence verification cannot pass solely because `listOutbox` returns
+nothing.
+
+`EventAppendSubject<TDb>` supplies `store`, `tenantId`, and optional `reset()`. Only
+`appendEvent` and `listEvents(tenantId, spaceId)` are required; the subject does not need a full
+recoverable store. The result has `passed` and per-check `id`, `status`, `evidence`, and
+optional `error`, following the tenant-isolation suite. Absent outbox or transaction append
+capabilities produce `skipped: ` evidence; `passed` means no failures, so inspect skips for
+capability coverage. An available outbox append without `listOutbox`, `claimOutbox`,
+`markOutboxPublished` or `markOutboxFailed` fails because the check cannot verify or clean up.
+Append errors and probe retirement failures also fail.
+
+Run against test or staging stores, preferably with a dedicated probe tenant. **Pause outbox
+relays for the probe tenant.** Event timestamps and written availability values (including
+retirement) use whole seconds a few seconds in the past. Claims and publication markers use
+real current time. Probe rows are expected to be due when availability follows `recordedAt`;
+stores assigning availability by their own clock may need additional setup. Visible foreign-worker
+leases or status changes before the probe acts fail with a request to pause relays. This detects visible interference; it does not prevent a relay race.
+Append-only events remain in fresh `event-append-probe-${uuid}` spaces. Successful
+outbox-creating checks leave their row **published although nothing was sent**; the space name
+identifies probe rows. Published is needed to exercise duplicate appends against that delivery
+state. Absence-control rows and failure rows instead go through final cleanup, which attempts
+to dead-letter visible probe rows still pending with `event append probe; never published`.
+Operator tooling can requeue those dead-lettered rows for real publication: exclude probe
+spaces from real publication and requeue workflows.
+
+Claims use the real current time, a 30-second lease, and a limit equal to the pending probe row
+count. Cleanup filters listed rows locally by tenant and space and writes only to probe rows. If
+a claim returns foreign rows, the failure names their IDs, the checks write nothing to those
+rows, and no further claims occur in that run. Later outbox checks fail with an explanation that
+they were skipped because of the earlier filter violation; a standalone `check.run(subject)`
+stops claims for that check. Cleanup after a filter violation can dead-letter only probe rows
+returned by the violating claim, using the probe worker and that claim's tokens without another
+claim. The failure lists visible probe rows still pending, including unreturned rows or failed
+retirements, and instructs the operator to dead-letter them by hand. A filter-ignoring claim can
+still impose a short lease and make whatever writes the claim itself makes, such as incrementing attempt counts. Those leases lapse
+within 30 seconds of the claim if the store honours the requested duration; the checks do not
+release foreign leases or change foreign statuses, availability or errors.
+
 ## Runtime observability and health
 
 Platform Host exposes vendor-neutral lifecycle telemetry through the optional `telemetry` sink on