@fabricorg/platform-host 7.5.0 → 8.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -68,7 +68,8 @@ delay is the remaining lease duration, followed by the worker's next claim; ther
68
68
  retry or separate backoff. The next claim increments `attemptCount`. At the cap, or for an unleased,
69
69
  non-idempotent, or approval-resume invocation, Host records `failed` with the reason retry was refused,
70
70
  through the error-text redactor. Approval resume is excluded because its temporary lease does not
71
- establish that a recovery worker is configured. Ordinary thrown errors retain terminal failure behavior.
71
+ establish that a recovery worker is configured. Ordinary application governance errors retain terminal failure behavior. Host store errors after
72
+ a recorded provider effect follow the recovery rules below.
72
73
 
73
74
  The retry region includes Host's single-record store writes for mutation-governance and HITL
74
75
  checkpoints. Later attempts reuse these persisted decisions through the `mutationFootprint` and
@@ -76,10 +77,52 @@ checkpoints. Later attempts reuse these persisted decisions through the `mutatio
76
77
  A deferral happens before the attempt writes any policy evidence: the complete definition batch must
77
78
  evaluate before Host starts appending policy records. Policy persistence, the `ComplianceBlocked`
78
79
  event, obligation listing/resolution/persistence, state-machine checks, the handler transaction,
79
- handlers, adapters, and finalization retain their existing failure/reconciliation/recovery semantics.
80
+ handlers and adapters retain their separate failure semantics. Finalization recovery is bounded by
81
+ `recoveryMaxAttempts`. Its effective default is at least `retryableExecutionMaxAttempts` because
82
+ both count worker claims.
80
83
  A retryable obligation-resolver error fails terminally with the policy evidence from that attempt.
81
84
  Preview evaluation does not enter this execution catch.
82
85
 
86
+ **Handler transaction reconciliation.** A handler or store transaction wrapper can throw
87
+ `ReconciliationRequiredError(message, { cause })` from `@fabricorg/platform` (also exported from
88
+ `@fabricorg/platform/policies`) when a domain effect may already have committed and an operator
89
+ must decide how to proceed. For example, a recovery-receipt store can check its own stored result
90
+ against the action's current declared events and result schema before handing that result back.
91
+ If its own check detects an incompatible receipt from an earlier attempt, it can throw this signal.
92
+ Put non-sensitive operator detail in the message: Host stores the redacted message, not `cause`.
93
+ The cross-copy predicate
94
+ `isReconciliationRequiredError` recognises the stable name and `EXECUTION_RECONCILIATION_REQUIRED`
95
+ code as well as local instances.
96
+
97
+ A `transactionWithEvents` wrapper cannot tell which Host transaction it is wrapping from the public
98
+ callback contract: Host also uses this API for `after_adapters` finalization and external completion.
99
+ Translate only a condition the store detects itself while recovering its own receipt, and let other
100
+ errors propagate unchanged. Do not match Host's internal validation-error messages; those errors
101
+ have no public stable code or message contract.
102
+
103
+ Host honours the signal only at the handler-transaction catch, covering `transactionWithEvents`
104
+ and the fallback `transaction`. It redacts the message with `redactErrorText` using `source: "handler"`,
105
+ records terminal `reconciliation_required`, and returns the error with the action's consistency and
106
+ persisted HITL route/risk tier. Status persistence emits the existing reconciliation telemetry.
107
+ Re-execution returns the terminal record without running the handler. Workers retain their existing
108
+ counting: a returned reconciliation result increments `failed`, not `completed` or `deferred`.
109
+
110
+ Atomic `transactionWithEvents` stores roll back the attempt's domain writes, domain events, and
111
+ Host's `StateTransitioned` together when the transaction throws. On the `transaction`-only fallback,
112
+ Host appends events directly. The built-in stores simply run the callback without rollback: domain
113
+ writes and any events Host appended during the attempt stay persisted beside the
114
+ `reconciliation_required` record, as they already do beside `failed`. A custom fallback transaction
115
+ may roll back its domain writes, but Host does not make its event appends atomic with those writes.
116
+ Host does not run adapters after this signal or add a new lifecycle event for it. Effects or evidence
117
+ committed by earlier attempts are not undone.
118
+
119
+ The signal does not change failures in pre-handler governance or validation, state-transition guard
120
+ exceptions, adapters, or finalization. Application governance errors and definite adapter failures retain their
121
+ `failed` paths; post-success errors follow the recovery rules below. State guards still fail closed.
122
+ Transient `after_adapters` finalization failures escape as `RecoverableFinalizationError` for
123
+ recovery within `recoveryMaxAttempts`. An ordinary handler error still records `failed`. This signal requests an operator decision,
124
+ not an automatic retry or a claim that the current transaction committed.
125
+
83
126
  New rows bind that key to the action version, actor/authority binding, and a versioned canonical
84
127
  parameter digest. Host 3 defaults to `enforce`, which throws `IdempotencyConflictError`; configure
85
128
  `idempotency.conflictMode` as `audit-only` for a preflight soak. Legacy null-digest rows skip only the
@@ -87,16 +130,282 @@ parameter comparison and call `onLegacyRecord`; actor, authority-binding, and ac
87
130
  still reject in enforce mode. See `MIGRATION-2-IDEMPOTENCY.md`.
88
131
 
89
132
  Lifecycle records use deterministic checkpoint ids. Recovered idempotent actions do not repeat an
90
- adapter that already reached `succeeded`, and event, policy, and adapter writes are append-safe. A
91
- stale action that was not declared idempotent fails terminally for manual reconciliation instead of
92
- silently rerunning unknown side effects.
93
-
94
- Adapters may classify outcomes as `success`, `transient_failure`, `permanent_failure`, `timeout`,
95
- or `ambiguous`. The retry layer retries only transient idempotent outcomes; permanent failures,
96
- timeouts, and ambiguous results are never retried. An ambiguous outcome — where the external effect
97
- may or may not have been applied — is routed to `reconciliation_required` with durable
98
- `ExternalReconciliation` evidence and an `AdapterInvocationAmbiguous` lifecycle event rather than
99
- blindly retrying an unsafe effect.
133
+ adapter that already reached `succeeded`. Built-in stores deduplicate event and policy inserts by
134
+ checkpoint ID. Adapter rows are mutable; stale-worker protection requires the optional
135
+ `writeAdapterInvocation` capability described below. A
136
+ reclaimed non-idempotent action inspects its adapter checkpoints: running or succeeded rows route
137
+ to reconciliation; no rows or only failed rows retain terminal failure for manual review.
138
+
139
+ Returned `ambiguous` or `timeout` outcomes, timeouts thrown from `adapter.execute` (including an
140
+ aborted adapter deadline), contradictory successful results, and reverse contradictions
141
+ (`success: false` with `outcome: "success"`) route to `reconciliation_required`.
142
+ Host detects a thrown timeout from the adapter deadline's aborted signal or an error message matching
143
+ `/timeout|deadline|timed?\s*out/i`. A definite rejection whose message matches therefore also routes to
144
+ reconciliation; the message matcher does not prove that the provider timed out.
145
+
146
+ An uninterpretable provider reply (a non-object, `null`, or an array), a getter that throws before Host
147
+ can interpret the reply, and an unrecognised outcome string such as `"timed_out"` also take this path.
148
+ Host copies the reply's contract fields and other own enumerable fields into a plain object before
149
+ interpretation, reading each top-level field once. A getter that throws during the copy takes the
150
+ unreadable-reply path. These outcomes are not retried by the retry helper. Object replies retain the
151
+ previous truthiness reading of `success`, including truthy non-boolean values from JavaScript adapters.
152
+ The reconciliation kind remains `adapter_outcome_ambiguous`.
153
+
154
+ Unknown-outcome settlement uses `transactionWithEvents` when its transaction supplies
155
+ `updateAdapterInvocation`, `appendExternalReconciliation` on a governance-capable store, and
156
+ `updateLeasedActionInvocation` for a leased invocation. A store advertising conditional adapter writes
157
+ also needs transaction-scoped `writeAdapterInvocation` for leased finalization. In that transaction Host writes the
158
+ `ambiguous` checkpoint, `AdapterInvocationAmbiguous` with its stable event ID, governance
159
+ `ExternalReconciliation` evidence where supported, and the fenced `reconciliation_required` verdict,
160
+ in that order. These writes commit together. Host retries once on a non-lease error; a second
161
+ non-lease failure selects the verdict-first fallback. Lease loss raises
162
+ `RecoverableFinalizationError` without retry or fallback.
163
+
164
+ Without those transaction members, or after two non-lease transaction failures, the verdict-first
165
+ fallback writes the terminal verdict (retrying once except on lease loss), then the checkpoint,
166
+ event, and reconciliation record. If both checkpoint shapes fail, Host still attempts the ambiguity
167
+ event and reconciliation record. After the verdict commits, Host attempts each remaining evidence
168
+ write even if the event append fails, then surfaces the first error as `RecoverableFinalizationError`.
169
+ An evidence failure can leave a terminal verdict with missing evidence; a later direct execution returns that verdict
170
+ and workers exclude it from recovery, without repairing the evidence gap. If both verdict attempts
171
+ fail, Host surfaces `RecoverableFinalizationError` without attempting an unfenced checkpoint.
172
+
173
+ With the original action definition and adapter-step order available, recovery checks `ambiguous`
174
+ rows before the parked-handoff shortcut, stale-action gates, authorization, and handler execution.
175
+ This applies to actions declared idempotent and actions not declared idempotent. Settlement uses the
176
+ stored reconciliation metadata or the legacy row's error as its message and reason, without running
177
+ the handler or an adapter. For a running invocation whose action resolves, `executeInvocation` reads
178
+ each adapter step's row before the pending-handoff shortcut, stopping on recorded ambiguity or a
179
+ failed read. A failed read surfaces `RecoverableFinalizationError`, where base 939bcad returned the
180
+ parked result. The shortcut's condition is unchanged from base: a running invocation with
181
+ `pendingCompletion` and no lease owner returns early without requiring `pendingCompletion.parkedAt`.
182
+ If the action no longer resolves, the shortcut still returns the handoff before the unknown-action
183
+ failure. Completion callbacks that settle the invocation require parking, including after a deadline sweep.
184
+
185
+ `AdapterInvocationRecord.reconciliation` is an optional, redacted checkpoint preserving reason,
186
+ external reference, and the original `adapterEventSubject`. The subject stays out of the returned
187
+ `AdapterReconciliationOutcome`. Recovery uses this checkpoint without scanning `listEvents()`.
188
+ Without stored subject metadata it uses `{ subjectType: "AdapterInvocation", subjectId: adapterInvocationId }`;
189
+ the stable event ID deduplicates an event already written. Custom stores should persist the checkpoint
190
+ through adapter create, update, and read methods.
191
+
192
+ Ordinary adapter writes do not name the migration 8 column and remain compatible with schemas
193
+ lacking it. If a checkpoint write carrying `reconciliation` fails, fallback retries the row once
194
+ without that field, using legacy `status: "ambiguous"` plus `error`. Unknown outcomes therefore still
195
+ settle without migration 8, but without stored checkpoint metadata. Apply PostgreSQL migration 8
196
+ through `ensureSchema()` in controlled startup or migrations to retain that metadata. An error inside
197
+ an atomic PostgreSQL transaction first rolls it back; the legacy write runs outside that transaction
198
+ on the fallback path.
199
+
200
+ On a store without atomic support, if the verdict cannot be written at all, no durable write after the
201
+ call succeeds: neither verdict nor checkpoint is saved, as after a crash mid-call. A leftover `running`
202
+ step can then re-run on recovery for an idempotent action under the same `adapterInvocationId`
203
+ idempotency key, within the worker recovery budget. A reclaimed non-idempotent action with a
204
+ running row without a matching durable handoff settles that step as unknown. A succeeded row or
205
+ durable handoff instead ends `reconciliation_required` as an applied effect. Adapters should be
206
+ idempotent or carry that stable key. The same exposure applies if two atomic attempts roll back
207
+ and the fallback verdict cannot be written.
208
+
209
+ Once an invocation has a durable `succeeded` adapter row or accepted handoff, errors in Host
210
+ store reads or writes raise `RecoverableFinalizationError` instead of ending it `failed`.
211
+ This includes later adapter checkpoints, obligation reads, and the completion write in both
212
+ `before_adapters` and `after_adapters`. Recovery also protects Host store operations before
213
+ and around the handler when an earlier provider effect is recorded. Authorization, policy and handler callbacks
214
+ retain their own failure semantics. Idempotent recovery repairs recorded success/handoff evidence
215
+ without calling that step's provider. Non-idempotent interruption ends `reconciliation_required`
216
+ instead of repairing the invocation. Lease loss retains its fencing error.
217
+
218
+ If neither record is durable, Host retries the failed handoff or marker write once. After a
219
+ second failure it reuses degraded atomic finalization where available, otherwise writes the
220
+ `reconciliation_required` verdict first, then attempts the succeeded marker with safe output,
221
+ `ExternalOperationAccepted` when applicable, and `AdapterInvocationSucceeded`. Each fallback
222
+ evidence write is attempted independently; the first error raises `RecoverableFinalizationError`
223
+ after the remaining writes are attempted. One exception keeps its 7.x behaviour: when
224
+ unsatisfied obligations end a handed-off invocation `reconciliation_required` and the
225
+ reconciliation-log append fails, Host still returns the verdict. A terminal verdict can retain
226
+ missing evidence. If even the
227
+ verdict cannot be saved, Host throws `RecoverableFinalizationError`; the row can remain `running`.
228
+ Recovery may replay that unrecorded outcome for an idempotent action under its stable
229
+ `adapterInvocationId`, within the worker budget. A reclaimed non-idempotent action instead
230
+ ends `reconciliation_required`. A durable handoff takes precedence over its running row and is
231
+ recorded as an applied effect with its external reference. A running row without such a handoff
232
+ is settled as unknown. A succeeded row also ends `reconciliation_required`. With no rows or only failed rows, the existing
233
+ non-idempotent interruption gates still fail the invocation.
234
+
235
+ For idempotent recovery, the handler still runs before the adapter loop. The loop reads each
236
+ checkpoint before `getInput()` or adapter lookup. Succeeded rows and durable handoffs use stored
237
+ adapter type, operation, vendor, input and output; a falsy new input or removed adapter does not
238
+ skip evidence repair. Stable event IDs deduplicate replayed accepted and success events; newly
239
+ repaired success events include `recovered: true` and stored input/output when available.
240
+
241
+ Recovery looks up the stable attestation ID. If present, Host reuses it and reapplies its
242
+ `satisfiesObligations` updates without calling the extractor. If absent and stored output exists,
243
+ Host calls `extractExecutionAttestation` with stored input/output and checkpoint `createdAt` and
244
+ `updatedAt` as start/completion times. The extractor may run again after an interrupted append;
245
+ it must return deterministic evidence for the same input. A handoff checkpoint with no output,
246
+ or withheld output, skips extraction. Repair errors remain recoverable within the budget.
247
+
248
+ Known limits: a later step's definite provider failure or pre-call refusal can still end the
249
+ invocation `failed` after an earlier step succeeded. Authorization denials, ordinary handler
250
+ failures, validation/state-guard rejection, and unsatisfied policy obligations on recovery retain
251
+ their existing terminal outcomes. These are decisions or application failures, not Host store
252
+ errors. Idempotent recovery can call a provider again for an unrecorded running outcome, or for a
253
+ definite failed row whose invocation failure verdict was lost. The handler still re-runs before
254
+ the adapter loop. Saga compensation is unchanged. Withheld-output and refused-handoff
255
+ reconciliation retain the degraded pattern. Adapter-row fencing and overdue-sweep/late-callback
256
+ rules are described below.
257
+
258
+ ### Conditional adapter-row writes (8.0.0, issue #79)
259
+
260
+ `PlatformHostStore.writeAdapterInvocation(input)` is optional. Memory and PostgreSQL implement it;
261
+ custom stores can keep their existing methods. Without it, Host retains unchecked adapter creates
262
+ and updates, so a reclaimed worker can overwrite a newer worker's adapter checkpoint.
263
+
264
+ The input carries `scope` (tenant, space, invocation), a create or update `mutation`, a `condition`,
265
+ and optional `expectedStatuses`. A lease condition requires a `running` parent with the supplied
266
+ `leaseOwner` and `leaseToken`. A verdict condition requires the exact parent `status` and `error`
267
+ written by this execution. Host also captures the handoff row ID as
268
+ `completionAdapterInvocationId`, so `externalCompletionDigest` guards that row's callback output:
269
+ an absent digest requires no recorded completion; a supplied digest requires that same completion.
270
+ Other adapter rows retain the status/error and expected-row-status checks without the digest check.
271
+ Omitting the captured row ID retains the conservative digest check on adapter writes. Absent error
272
+ matches no error. Both parent and adapter must match scope.
273
+ A create must also match the record's scope and refuses a duplicate ID. The method returns `false`
274
+ when its conditions fail. Conditional updates reject `succeeded` or `ambiguous` to `running` or
275
+ `failed`, including when callers omit `expectedStatuses`.
276
+
277
+ Host captures the supplied execution credentials before loading the invocation. It uses them for
278
+ fenced invocation writes and leased adapter creation, retry preparation, success, failure, and
279
+ ambiguity checkpoints. A refused lease or row-status check raises `RecoverableFinalizationError`.
280
+ This prevents stale checkpoint writes; adapters still need stable idempotency keys because a lease
281
+ cannot cancel a provider call already in flight. Inline writes use the verdict condition after
282
+ this execution writes its own verdict; earlier unleased adapter writes retain their existing path.
283
+
284
+ Verdict-first fallbacks use the settled verdict condition after the terminal write releases the
285
+ lease. Their expected row status is the last state observed or written by that execution. A changed
286
+ parent status/error or row refuses the adapter update; a changed completion also refuses a write
287
+ to the captured handoff row. These refusals emit
288
+ `invocation.adapter_evidence_refused` (`fabric.platform.invocation.adapter_evidence_refused`) through
289
+ the telemetry hook. The adapter row stays untouched, with no unchecked retry. Host still attempts
290
+ `AdapterInvocationAmbiguous` and reconciliation evidence; withheld-output and refused-handoff
291
+ success evidence may already have been appended before the refused marker write. A refusal is not
292
+ an evidence-collector error. Thrown store errors are retained while each independent fallback
293
+ write is attempted, then the first error is raised as `RecoverableFinalizationError`.
294
+
295
+ An inline writer preserving a sweep verdict does not adopt it as this execution's own verdict.
296
+ Its adapter evidence keeps the unchecked inline path; conditional settled-parent enrichment is
297
+ used only for a verdict this execution actually wrote. The inline guard reads the current row
298
+ only when the execution snapshot or its patch carries `pendingCompletion`.
299
+
300
+ Budget exhaustion saves its plain verdict before reading evidence. It then enriches the parent
301
+ through optional `updateSettledActionInvocation`, comparing that execution's exact status, error,
302
+ and completion snapshot. Parent enrichment joins the independent evidence collector, alongside
303
+ adapter checkpoints, events and reconciliation records. Refusal emits
304
+ `invocation.adapter_evidence_refused` with `evidence: "parent_enrichment"` and is not an error;
305
+ a missing `updateSettledActionInvocation` skips only parent enrichment. Evidence writes still run.
306
+ A thrown store error is collected and raised as `RecoverableFinalizationError` after the remaining
307
+ writes are attempted. The two optional store methods are expected together. Successful enrichment
308
+ stores the error and `adapterReconciliation` before returning them; skipped or refused enrichment
309
+ can leave replay with only the plain verdict. Stores omitting `writeAdapterInvocation` retain
310
+ unchecked adapter writes. A retained handoff's reference is included in the initial budget verdict.
311
+
312
+ Subclasses of the built-in stores that override `createAdapterInvocation` or
313
+ `updateAdapterInvocation` for encryption or auditing must also override `writeAdapterInvocation`.
314
+ Checked writes can bypass the unchecked methods. Alternatively, explicitly disable the capability
315
+ and accept the weaker stale-worker guarantee. Parent annotations similarly have their own
316
+ `updateSettledActionInvocation` seam.
317
+
318
+ Atomic finalizers require transaction-scoped `writeAdapterInvocation` when the store advertises the
319
+ capability and execution holds a lease. A missing member selects verdict-first fallback before
320
+ transaction evidence writes. PostgreSQL locks the matching parent with `FOR UPDATE` before the
321
+ adapter mutation in one SQL statement. Memory rechecks buffered lease and adapter predicates at
322
+ commit and excludes claims and other conditional writes while applying the commit. A commit-time
323
+ conflict rolls back and raises `RecoverableFinalizationError`. No schema migration is added by this
324
+ capability; it uses the existing parent lease and verdict columns.
325
+
326
+ External-completion adapter writes use the callback's transaction-scoped `updateAdapterInvocation`
327
+ when available, sharing its invocation lock. These are callback writes rather than worker-lease
328
+ writes; legacy transactions without that member retain the outer-store write. The sweep retains
329
+ its transaction-scoped reconciliation append and parked/overdue settlement rules.
330
+
331
+ ### Bounded worker recovery (8.0.0)
332
+
333
+ `recoveryMaxAttempts` is an optional positive integer. Its effective default is
334
+ `max(5, retryableExecutionMaxAttempts)`; `Infinity` is rejected. Both caps count the same worker
335
+ claims. Construction rejects an explicit `recoveryMaxAttempts` below
336
+ `retryableExecutionMaxAttempts`. The governance cap stops transient pre-policy deferrals at its
337
+ cap; the recovery budget stops resumed execution above its cap. With the defaults, governance
338
+ retry exhaustion therefore happens on claim 5 before recovery exhaustion could happen on claim 6.
339
+
340
+ Host checks a resumed leased `running` invocation immediately after lease ownership, before
341
+ normal action resolution, ambiguity scans, authorization, the handler, or adapter work. When
342
+ `attemptCount > recoveryMaxAttempts`, it settles `reconciliation_required` with
343
+ `recovery attempts exhausted (n/max)`. It writes that verdict before resolving the action or
344
+ reading adapter evidence, releasing the lease. A running row without a matching durable handoff
345
+ receives unknown-outcome settlement. A recorded ambiguous row enriches the parent error and
346
+ `adapterReconciliation`, as well as adapter/event evidence, using its stored reconciliation evidence
347
+ and a conditional parent update. A durable handoff takes precedence over that step's running row.
348
+ It is recorded as an applied effect with its external reference in the message and, on governance
349
+ stores, the reconciliation record. It is not marked ambiguous solely because its marker was lost.
350
+ If resolution or row reads throw, or no relevant row exists, the plain verdict already written
351
+ stands. Enrichment failure can retain an evidence gap. Transition telemetry is emitted once when
352
+ the status changes. If the verdict is unwritable, Host throws `RecoverableFinalizationError`;
353
+ later claims continue but do not execute mutation code.
354
+
355
+ The counter counts worker claims, including the first run, rather than provider calls or completed
356
+ recoveries. With the default of 5: the first run plus four recoveries; the sixth claim settles.
357
+ A raised governance cap raises the implicit recovery budget with it. Inline/direct execution does
358
+ not increment the counter. Approval acceptance sets it to at least 1 without incrementing it on
359
+ resume. Approval resume holds a temporary lease and uses the existing count: a count equal to the
360
+ budget is admitted, while a count above the budget settles reconciliation before the handler.
361
+ Approval does not reset an exhausted budget. A worker dying before reaching the check consumes a
362
+ claim without calling a provider at that point. Parked handoffs have no lease, are not claimed by
363
+ workers, and are outside this budget. The existing parked shortcut and ambiguity scan remain in
364
+ place. `retryableExecutionMaxAttempts` still limits pre-policy deferrals. No new store method is
365
+ required.
366
+
367
+ **Inline and direct callers.** Inline `submitAction()` without a dispatcher and direct
368
+ `executeInvocation()` can now reject with an error whose `name` is
369
+ `"RecoverableFinalizationError"` where 7.x returned `failed`. The class is not exported. Without a
370
+ lease, the invocation stays `running` and workers do not claim it. Before a handoff is durable,
371
+ call `executeInvocation()` again yourself and cap your own retries. Once a handoff is durable,
372
+ re-running `executeInvocation()` on the unleased running row returns without resuming execution
373
+ (it first settles any step already recorded `ambiguous`).
374
+ If execution stopped before parking, callbacks are refused with `still_executing` until a sweep.
375
+ Declare `completionDeadlineMs` for actions that hand off so the sweep surfaces the row, then
376
+ reconcile it per the [external-completion runbook](../../docs/operations/runbooks.md#overdue-handoffs).
377
+ The leased worker budget does not bound unleased calls. A
378
+ previously leased direct call retains its lease until expiry and can also be recovered by a
379
+ worker. Terminal degraded verdicts can retain evidence gaps and are not automatically repaired.
380
+
381
+ A transaction implementation may rethrow the handler's error wrapped in its own error; Host
382
+ still treats a failure inside the handler callback as a handler outcome.
383
+
384
+ **Known limit.** If a later step's adapter row is created but its attempt-preparation write keeps
385
+ failing, the row stays `running` although the provider was not called. For an action not declared
386
+ idempotent, the next claim settles that step as an unknown provider outcome.
387
+
388
+
389
+ A deadline sweep retains the handoff and sets `pendingCompletion.overdueAt` with its
390
+ `reconciliation_required` verdict. A late callback settles that verdict only when the handoff also
391
+ has `parkedAt`, proving Host reached parking after its remaining steps and finalization. An unparked
392
+ handoff stays in reconciliation even if later-step ambiguity evidence is missing; the callback's
393
+ redacted result and outcome are recorded as evidence for an operator without applying the action's
394
+ result schema. This evidence-only behavior applies to invocations already
395
+ `reconciliation_required`; callbacks for unparked `running` invocations are refused with
396
+ `still_executing`. Any parked row without `overdueAt`, including a 7.x row, still uses the old
397
+ `"did not complete by"` error substring. That compatibility path is weaker: an unrelated error
398
+ containing the substring on a parked row lacking the marker can also authorize settlement.
399
+
400
+ Circuit-open refusals and deadline expiry before execution retain `failed` semantics because
401
+ the refused attempt made no provider call. A `prepareAttempt` store error retains its prior failure
402
+ behavior only while the invocation has no durable success or handoff. With an earlier recorded
403
+ effect, an exhausted Host-store error raises `RecoverableFinalizationError`. Returned `permanent_failure`
404
+ and exhausted `transient_failure` outcomes, plus exhausted non-timeout adapter throws, also retain
405
+ `failed` semantics and their existing events, unless reading any field of the reply throws: Host
406
+ copies the reply before interpreting it, so such a reply takes the unreadable-reply path to
407
+ reconciliation. Adapters report definite non-effects with
408
+ `permanent_failure` or `transient_failure`; no additional definite-failure flag is needed.
100
409
 
101
410
  ### Adapter circuit breakers
102
411
 
@@ -210,6 +519,10 @@ in action parameters or workflow history.
210
519
  Production stores can implement `AtomicMutationPlatformHostStore.transactionWithEvents`. The
211
520
  transaction-scoped `db`, event sequence, event append, event listing, and invocation update methods
212
521
  must all use the same database transaction.
522
+ The optional transaction-scoped adapter-update and `appendExternalReconciliation` operations
523
+ extend this unit of work to degraded adapter finalization. Leased finalizers use
524
+ `writeAdapterInvocation` when the store advertises it, otherwise `updateAdapterInvocation`,
525
+ as described under redaction below.
213
526
 
214
527
  `PostgresPlatformHostStore` enables this capability when constructed with a
215
528
  `PostgresPlatformHostTransactionProvider`. The application owns that narrow binder because only the
@@ -222,7 +535,8 @@ With `transactionWithEvents`:
222
535
  - `after_adapters`: declared domain events, the Host-generated `StateTransitioned` fact, and
223
536
  invocation finalization commit together after required adapter steps succeed or accept a
224
537
  parked external handoff. Handler writes have already committed separately. A finalization
225
- failure leaves the invocation recoverable; succeeded adapter checkpoints are not repeated.
538
+ failure leaves the invocation recoverable within `recoveryMaxAttempts`; recovery repairs evidence
539
+ for succeeded adapter checkpoints without repeating those calls.
226
540
 
227
541
  Stores without `transactionWithEvents` retain the legacy non-atomic boundary for domain writes,
228
542
  events, and invocation persistence. A partial append or finalization failure can leave some
@@ -298,7 +612,7 @@ Three optional `PlatformHostOptions` hooks cover other Host-recorded values.
298
612
  | --- | --- |
299
613
  | `redactAdapterOutput(actionId, adapterType, operation, output)` | Once after a successful adapter result is converted to output, before handoff handling. The adapter row, `AdapterInvocationSucceeded` payload, and `extractExecutionAttestation` input use the redacted output. |
300
614
  | `redactErrorText(context, message)` | At the origin of caught handler/execution errors, adapter result errors and ambiguous-outcome reasons, and completion errors. The invocation/adapter errors, reconciliation messages and reasons, completion evidence, failure events, and returned error reuse that text. |
301
- | `redactExternalCompletionResult(actionId, provider, result)` | After a completion matches its invocation, before merging, result-schema validation, persistence, and digesting. See [External completion](#external-completion). |
615
+ | `redactExternalCompletionResult(actionId, provider, result)` | After a completion matches its invocation, before persistence and digesting; successful settling callbacks also merge and validate against the result schema. See [External completion](#external-completion). |
302
616
 
303
617
  `ErrorTextRedactionContext` is exported. It contains `actionId` and `source` (`handler`, `adapter`,
304
618
  `external_completion`, or `execution`); adapter errors also supply `adapterType`, `operation`, and
@@ -314,29 +628,63 @@ failures refuse with non-retryable `redaction_failed` before writes, allowing a
314
628
  callback after the redactor is repaired.
315
629
 
316
630
  If `redactAdapterOutput` throws or returns null, an array, or a non-object value, the external effect
317
- has already happened. Host persists the `reconciliation_required` verdict before the adapter row.
318
- It retries an idempotent verdict write once after a store error; a lost lease is not retried. After
319
- the verdict is durable, Host emits the retained handoff's `ExternalOperationAccepted`, emits
320
- `AdapterInvocationSucceeded` with redacted input and `outputWithheld: true` without output, appends
321
- fixed-text reconciliation evidence when the store supports it, and marks the adapter `succeeded`.
322
- The events use the normal step deduplication keys. A first accepted reference is retained as
323
- `pendingCompletion`, so its callback can still land as evidence. References come from the original
324
- adapter result. Refused handoffs also persist their verdict before evidence and the adapter marker.
325
-
326
- Once the reconciliation verdict is durable, direct execution returns it and recovery excludes it.
327
- A later evidence or adapter-marker write failure does not change it to completed or failed; Host
328
- throws `RecoverableFinalizationError`. The verdict, evidence and marker are separate writes, so an
329
- evidence-write failure can leave a partial audit trail. If both verdict-write attempts fail, the
330
- adapter can run again on recovery, the same exposure as a crash mid-step. Adapters behind these paths
331
- should be idempotent or carry a stable idempotency key. A lost lease can also prevent this branch
332
- from establishing its own durable verdict; these are not atomic guarantees.
631
+ has already happened. Host withholds output and records a `reconciliation_required` verdict. Its
632
+ safe evidence includes the retained handoff's `ExternalOperationAccepted`,
633
+ `AdapterInvocationSucceeded` with redacted input and `outputWithheld: true` without output, and
634
+ fixed-text reconciliation evidence when the store supports governance. The events use the normal
635
+ step deduplication keys. A first accepted reference is retained as `pendingCompletion`, so its
636
+ callback can still land as evidence. References come from the original adapter result.
637
+
638
+ For withheld output and refused handoffs (immediate completion or a second handoff), Host uses
639
+ `transactionWithEvents` when the transaction supplies `updateAdapterInvocation`,
640
+ `updateLeasedActionInvocation` for a leased invocation, and `appendExternalReconciliation` for
641
+ withheld output or refused handoffs on a governance-capable store. Leased finalization also needs
642
+ transaction-scoped `writeAdapterInvocation` when the store advertises it. These optional members must
643
+ share the event/verdict transaction; the adapter update must enforce the store method's supplied
644
+ tenant, space, and invocation scope. Both built-in stores provide them when configured with a
645
+ transaction provider. Custom stores must implement them to get this atomic behavior.
646
+
647
+ When atomic finalization succeeds, evidence events, any withheld-output or refused-handoff
648
+ reconciliation entry, the adapter's `succeeded` marker, and the terminal `reconciliation_required`
649
+ verdict commit together. Host writes evidence and
650
+ the marker before the fenced verdict, which releases the lease, then emits status telemetry after
651
+ commit. It retries the whole transaction once on a non-lease error. If the retry also fails with a
652
+ non-lease error, Host selects the verdict-first fallback described below. Lease loss is not retried
653
+ or converted to fallback; it raises `RecoverableFinalizationError`. Atomic rollback leaves the
654
+ invocation and adapter row `running` until a retry or fallback commits. If lease loss or an unavailable
655
+ fallback verdict prevents finalization, the invocation remains reclaimable and the adapter can run
656
+ again under the same `adapterInvocationId` idempotency key for idempotent actions within the
657
+ worker recovery budget. Non-idempotent reclaim inspects its checkpoints instead of replaying a call.
658
+
659
+ Without the required transaction members, or after two non-lease transaction failures, Host keeps
660
+ the verdict-first order: the `reconciliation_required` verdict precedes evidence and the adapter marker.
661
+ It retries the verdict write once, except on lease loss. After the verdict is durable, Host attempts
662
+ each evidence write independently: a failed event, reconciliation append, or adapter checkpoint does
663
+ not skip the remaining writes. Host raises the first evidence failure as `RecoverableFinalizationError`
664
+ only after those attempts finish. This includes unknown-outcome settlement and post-success degraded
665
+ finalization. Failed writes can still leave a terminal verdict with missing evidence; terminal replay
666
+ does not repair that evidence gap. If both verdict-write attempts fail, the adapter can run again on
667
+ recovery for an idempotent action within the worker recovery budget, the same exposure as a crash mid-step.
668
+ A reclaimed non-idempotent action settles a running checkpoint as unknown.
669
+
670
+ A refused handoff's provider reference is included in the step's `AdapterInvocationSucceeded` event
671
+ even when explicit adapter output omits it. Capable atomic transactions append its reconciliation
672
+ record with the step's evidence. The fallback independently attempts the reference-bearing event,
673
+ the governance reconciliation append, the outer `recordCompletionReconciliation` append, and the
674
+ adapter marker. Both reconciliation appends use the same stable id to deduplicate repeated attempts.
675
+ A failure of either append is reported after the remaining fallback evidence writes are attempted.
676
+ A non-sweep verdict may receive a log-unavailable error note; a preserved sweep verdict keeps its error.
677
+ After successful atomic finalization, the outer append retains its existing tolerated-failure behavior.
678
+ If its annotation write fails, Host raises `RecoverableFinalizationError` while keeping the committed
679
+ verdict and adapter marker intact; terminal replay retains the verdict.
333
680
 
334
681
  `extractExecutionAttestation` receives the redacted output, so an extractor that needs a stripped
335
- field can fail after the external effect, following the existing extractor-failure behavior.
682
+ field can fail after the external effect. Host leaves a durable success checkpoint recoverable
683
+ within `recoveryMaxAttempts`; repair may call the extractor again.
336
684
 
337
685
  `redactActionParameters` and `redactAdapterInput` behavior is unchanged. Independently of hook
338
- configuration, a stale worker that loses its lease inside an adapter step rethrows without marking
339
- the adapter failed, preserving the new owner's row. Refused-handoff ordering and audit events also
686
+ configuration, a refused fenced invocation write inside an adapter step rethrows. With
687
+ `writeAdapterInvocation`, stale adapter updates also refuse before marking the new owner's row failed. Refused-handoff ordering and audit events also
340
688
  change without the new hooks.
341
689
 
342
690
  Applications own the safety of domain events/results, policy and HITL evidence, explicit
@@ -540,13 +888,30 @@ can filter it. It does not need an `emitsEvents` declaration from the action.
540
888
  For `before_adapters`, the transition is appended with the handler's domain events. For
541
889
  `after_adapters`, it is deferred to finalization after required adapter steps succeed or
542
890
  accept a parked external handoff, immediately after domain events. Adapter failure withholds
543
- both domain events and the transition, leaving Host's recorded entity state unchanged by that
544
- invocation even though handler writes may have committed. With `transactionWithEvents`, the
891
+ both domain events and the transition, leaving the built-in stores' ledger-derived entity state
892
+ unchanged by that invocation even though handler writes may have committed. Custom stores that
893
+ read domain rows can already observe the target state at this point. With `transactionWithEvents`, the
545
894
  transition shares the transaction used for domain events in the selected phase; otherwise it
546
895
  follows the same legacy non-atomic boundary as domain events.
547
896
 
548
- Invalid transitions, failed handlers, replay-only acceptance in the target state, and bindings
549
- without a non-empty entity ID or target do not produce a Host transition fact. If the handler
897
+ For idempotent `after_adapters` actions, Host saves `pendingStateTransition` on the invocation.
898
+ It contains `{ entityType, entityId, fromState, toState }` from the validated transition.
899
+ Custom stores must persist and return this optional invocation-record field through
900
+ `updateActionInvocation`, including the transaction-scoped method. With a mutation transaction,
901
+ Host writes the field after the handler succeeds and result/events validate, just before returning
902
+ from the transaction callback. This keeps the invocation row lock out of the handler run while the
903
+ saved transition commits or rolls back with the domain writes. The `transaction`-only fallback saves
904
+ it separately before the handler. Non-idempotent actions do not write this field. It is retained as
905
+ evidence after completion. PostgreSQL `ensureSchema()` adds `pending_state_transition jsonb`.
906
+
907
+ On an admitted same-invocation replay, Host uses the saved `fromState` when the saved entity type,
908
+ entity ID, and target match the current binding and state. The fact follows the usual event phase
909
+ and stable `state:transition` ID, so built-in stores deduplicate an already appended event.
910
+ Invocations started before upgrading, or stores that omit this evidence, retain replay acceptance
911
+ without a re-emitted transition. Mismatched evidence also suppresses replay emission.
912
+
913
+ Invalid transitions, failed handlers, replay acceptance without matching saved evidence, and
914
+ bindings without a non-empty entity ID or target do not produce a Host transition fact. If the handler
550
915
  returns its own `StateTransitioned` with the bound entity's subject type and ID, Host suppresses
551
916
  its generated fact in either event phase. Applications using that workaround should remove it
552
917
  to use Host emission.
@@ -562,8 +927,10 @@ binding's target; otherwise Host's later fact sets the recorded state to the tar
562
927
 
563
928
  Configure `redactExternalCompletionResult` to withhold sensitive completion fields. Host applies it
564
929
  once to a supplied result (including a result attached to a failed callback). The redacted value
565
- feeds `externalCompletion.result`; for a completed outcome it also feeds the adapter output,
566
- `ExternalOperationCompleted` payload, merged invocation result, and returned result. A supplied
930
+ feeds `externalCompletion.result`; for a completed outcome it also feeds the adapter output and
931
+ `ExternalOperationCompleted` payload. Only a settling successful callback merges it into the
932
+ invocation result and validates that merge against the action's result schema. A non-settling
933
+ callback returns the stored invocation result and keeps the callback result in its evidence. A supplied
567
934
  `completion.error` passes through `redactErrorText` before recording, including when attached to a
568
935
  completed outcome. A missing error uses Host's fixed failure message, which bypasses the hook.
569
936
 
@@ -610,13 +977,55 @@ refused. A repeated identical completion is absorbed, identity being a digest of
610
977
  reference, outcome, result, error, and evidence. A contradictory one, a flipped outcome or the same
611
978
  outcome with different evidence, moves the invocation to
612
979
  `reconciliation_required`, keeps the first outcome on record, logs the second, and emits
613
- `ExternalOperationContradicted`. Under an atomic store the read and write happen under one lock. The
614
- callback's redacted result is merged with the invocation result, stripped of private fields, and
615
- validated against the action's result schema before writes. The stored merge is what is validated.
980
+ `ExternalOperationContradicted`. With `transactionWithEvents` and transaction-scoped
981
+ `getActionInvocationForUpdate`, the callback read and verdict write share a lock. The
982
+ successful settling callback's redacted result is merged with the invocation result, stripped of
983
+ private fields, and validated against the action's result schema before writes. A non-settling
984
+ callback records its redacted result/outcome as evidence without schema validation or merging it
985
+ into the invocation result.
616
986
  An invocation that ended for another reason while a handoff was pending keeps its status and records
617
987
  the callback as evidence. Declare `completionDeadlineMs` on `long-running` actions;
618
- `reconcileOverdueCompletions()` moves overdue handoffs to reconciliation while keeping the handoff so
619
- a late completion can still land, and the worker cycle runs that sweep before it claims.
988
+ `reconcileOverdueCompletions()` moves running, unleased, due handoffs to reconciliation and retains
989
+ the handoff with `overdueAt` set to the sweep's current time. The worker cycle runs the sweep before
990
+ claiming work. A callback can settle a parked running invocation, even past its deadline before the sweep. From
991
+ reconciliation, settlement requires the parked, swept handoff or the substring fallback described above;
992
+ a swept unparked handoff retains its verdict and records the callback in `externalCompletion`,
993
+ adapter evidence, and a completion/failure event. It clears the pending handoff without claiming
994
+ that the invocation's remaining steps ran. The markerless parked-row fallback is described above.
995
+
996
+ For each sweep candidate, Host re-reads status, lease ownership, handoff identity
997
+ (`adapterInvocationId` and `externalReference`), due time, and absence of `overdueAt`. With
998
+ `transactionWithEvents` and `getActionInvocationForUpdate`, this check and the verdict/marker write
999
+ share the row lock. Reconciliation evidence joins that transaction when it exposes
1000
+ `appendExternalReconciliation`; otherwise evidence is written on the outer store and can survive a
1001
+ rolled-back verdict. A custom transaction exposing `getActionInvocationForUpdate` without
1002
+ `appendExternalReconciliation` makes the sweep append on the outer store while holding the
1003
+ invocation row lock. If the reconciliation table references that row through a foreign key, the
1004
+ outer append can block waiting for the held lock. Implement both members in the same transaction.
1005
+ Stores without a transaction lock use an immediate re-read, but retain a race
1006
+ between that read and the write: a concurrent callback or execution can still be overwritten.
1007
+ Custom stores must preserve and reload the optional `overdueAt` Date in the handoff JSON; PostgreSQL
1008
+ hydrates it without a new migration.
1009
+
1010
+ Inline writes bypass the extra guard read and transaction when neither the loaded invocation nor
1011
+ its patch carries `pendingCompletion`. Ordinary inline completion retains its base store cost.
1012
+ Writing a handoff into a running invocation from anything other than its executing step is not
1013
+ supported; an execution without that handoff in its snapshot does not guard against such imports.
1014
+ Inline execution preserves a stored `reconciliation_required` verdict while committing independent
1015
+ adapter checkpoints, events, and reconciliation records, including references from refused handoffs.
1016
+ If execution reaches parking after the sweep, it records the handler result and `parkedAt` on the
1017
+ same handoff, retaining the sweep's status, error, and `overdueAt`. Successful `after_adapters`
1018
+ finalization also commits its deferred domain events: the handler and remaining adapters have
1019
+ succeeded. Inline callers receive the latest
1020
+ record observed by the preserving writer, merged with its evidence update, after these writes
1021
+ succeed; there is no post-attempt read. Leased execution retains `RecoverableFinalizationError`
1022
+ on fencing failure. The transaction lock serializes the check with the sweep where available; stores
1023
+ without it retain a read/write race. This does not cancel an adapter in flight or undo earlier
1024
+ domain writes. Verdict-first fallbacks attempt independent evidence writes before raising the first
1025
+ failure as `RecoverableFinalizationError`. Reconciliation-log failures leave the sweep error unchanged. Concurrent inline executions of the
1026
+ same invocation can still raise `RecoverableFinalizationError` with `Invocation is no longer running`
1027
+ when another execution finalized the row first. This error is internal; callers classifying it can
1028
+ match its `name` and inspect the stored invocation before deciding how to recover.
620
1029
 
621
1030
  The handoff is persisted the moment it is accepted, before the adapter step is marked done, so a
622
1031
  worker that dies between the two leaves the handoff and not the marker. A custom store has four
@@ -655,7 +1064,8 @@ Host passes `AdapterInvocationScope` from the store-loaded action invocation to
655
1064
  must match `tenantId`, `spaceId`, and `actionInvocationId` when scope is supplied. A mismatch returns
656
1065
  `undefined` on read and leaves the row unchanged on update (a not-found error is permitted).
657
1066
  The trailing argument is optional to preserve existing custom-store and direct-call compatibility;
658
- calls without it retain ID-only behavior in the shipped stores. The two adapter isolation checks
1067
+ calls without it retain ID-only behavior in the shipped stores. When `writeAdapterInvocation` is present, the write probe also checks its scoped create/update
1068
+ refusals and an admitted update in our scope. The two adapter isolation checks
659
1069
  fail for stores that ignore scope. Deployments running `runTenantIsolationChecks` against a custom
660
1070
  store will see new failures until that store honours the argument.
661
1071