@fabricorg/platform-host 8.8.0 → 8.10.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -41,7 +41,7 @@ injected `now` clock is also passed to ordinary policy evaluation.
41
41
 
42
42
  Production polling workers can use `createStoreBackedActionDispatcher()` plus
43
43
  `runPlatformActionWorker()`. The pending invocation row is the durable queue item; workers claim
44
- bounded batches with an atomic, renewable lease and `FOR UPDATE SKIP LOCKED`. Every claim advances a
44
+ bounded batches with an atomic, renewable lease and `FOR NO KEY UPDATE SKIP LOCKED`. Every claim advances a
45
45
  fencing token. A heartbeat may renew only a lease that has not expired; after expiry, the work must be
46
46
  reclaimed so its token advances. An expired worker cannot finalize after another worker takes ownership. Pass a stable
47
47
  `idempotencyKey` to `submitAction()` so retries resolve
@@ -76,7 +76,7 @@ Non-idempotent governance retry requires a durable `deferredBeforeEffectAttempt`
76
76
  preceding claim count. Each retry must record its own pre-effect deferral to permit another claim.
77
77
  A missing or stale marker retains interruption failure and manual reconciliation. Marker-write
78
78
  failure ends terminally when the verdict can be persisted; lease loss retains the recoverable
79
- fencing error. Handler and adapter errors are outside this deferral window.
79
+ fencing error. Handler and adapter errors are outside this governance deferral window; handler rollback retry is described below.
80
80
 
81
81
  PostgreSQL migration 9 (`pre_effect_governance_deferral`) adds the nullable marker column.
82
82
  Run `ensureSchema()` before deploying Hosts that may defer non-idempotent actions. Invocation
@@ -96,6 +96,124 @@ both count worker claims.
96
96
  A retryable obligation-resolver error fails terminally with the policy evidence from that attempt.
97
97
  Preview evaluation does not enter this execution catch.
98
98
 
99
+ ## Retry a rolled-back handler transaction
100
+
101
+ Throw `RetryableHandlerError` from `@fabricorg/platform` (or its `/policies` export) to assert
102
+ that the handler's effects stayed inside the supplied Host transaction and it can run again.
103
+ Host can retry that transaction within the same execution attempt. The default budget is three
104
+ total tries; `handlerRetry.maxAttempts` accepts integers from 1 to 10, with 1 disabling retry.
105
+ The worker claim, `attemptCount`, policy evaluations, obligations and approval decision are retained.
106
+ This is separate from `RetryableExecutionError`, whose handler throws retain terminal failure behavior.
107
+
108
+ Retry requires the atomic `transactionWithEvents` path, a throw from the handler call, and
109
+ `ConfirmedTransactionRollback` from that transaction call with the identical thrown error as `cause`.
110
+ Bare errors, other wrappers, failed rollback and errors at commit do not qualify. Errors from
111
+ transactional authorization, state validation, result/child validation, event extraction/persistence,
112
+ staging or finalization preparation do not qualify; neither does a returned `{ success: false }`.
113
+ `ReconciliationRequiredError`, `RetryableExecutionError` and Host's recoverable fencing errors retain
114
+ their existing outcomes and bypass the classifier.
115
+
116
+ There must be no adapter row from the resume scan (including failed/running rows), durable provider
117
+ effect, or pending completion/handoff. Before a retried try runs application code, its transaction
118
+ locks and checks invocation ownership. Leased execution checks the captured owner and token and
119
+ uses a transaction-scoped lease fence. Inline execution requires an unleased running row without
120
+ lease expiry; built-in workers do not claim such rows. Missing locking/fencing support prevents retry.
121
+ Lease loss raises the existing recoverable fencing error rather than a stale owner's failed verdict.
122
+
123
+ Host reparses copied durable parameters and restores copied provenance, admission time and
124
+ transactional authorization input. Uncopyable inputs, failed reparse or a changed parsed value stop
125
+ retrying. Preparation, adapter inputs and staged requests are published from the committed try.
126
+ A compatible replay checkpoint continues to bypass the handler. Exhaustion settles the last handler
127
+ error through `redactErrorText` with `source: "handler"` and the existing status handling.
128
+
129
+ ```ts
130
+ import { isTransientPostgresTransactionError } from "@fabricorg/platform-host";
131
+
132
+ const host = createGovernedActionHost({
133
+ store,
134
+ registry,
135
+ authorization,
136
+ handlerRetry: {
137
+ maxAttempts: 3,
138
+ classify: (error, { actionId }) =>
139
+ actionId === "records.update" && isTransientPostgresTransactionError(error),
140
+ delayMs: (retry) => Math.random() * Math.min(1000, 25 * 2 ** (retry - 1)),
141
+ onRetry: ({ actionInvocationId, attempt, maxAttempts }) =>
142
+ metrics.recordHandlerRetry(actionInvocationId, attempt, maxAttempts),
143
+ },
144
+ });
145
+ ```
146
+
147
+ `classify` must return literal `true` to opt a raw error in; a thrown classifier refuses retry.
148
+ It asserts safety for the whole retried unit, not just the SQL statement that failed.
149
+ `isTransientPostgresTransactionError` recognizes candidate SQLSTATEs `40001` (serialization),
150
+ `40P01` (deadlock victim) and `55P03` (lock unavailable), following up to 16 errors through `cause`
151
+ with cycle and throwing-getter protection. It is not proof of rollback or absence of effects.
152
+
153
+ Without `delayMs`, Host uses full jitter on a 25 ms exponential backoff capped at 1000 ms.
154
+ A configured number or callback result is clamped to 0–1000 ms; callback retry numbers start at 1.
155
+ A throwing or non-finite delay stops retries with the original handler error. `onRetry` runs before
156
+ the delay; `attempt` identifies the upcoming try (2 through `maxAttempts`). Synchronous throws and
157
+ asynchronous rejections are ignored without awaiting the observer. Delay happens between finished
158
+ transactions; Host does not race a transaction against a timer. Retrying delays the worker's other claimed rows. A crash mid-retry uses existing recovery.
159
+
160
+ With retry enabled, in-transaction authorization, state guards and the other transaction callbacks
161
+ run again per try. They and the handler must keep effects inside the supplied transaction. Avoid
162
+ independent commits, HTTP calls, signals, nontransactional service writes and unawaited work.
163
+
164
+ ### Provider rollback evidence
165
+
166
+ Rollback evidence is optional. A provider that propagates bare errors gets no handler retry.
167
+ `PostgresPlatformHostStore` does not infer rollback. Built-in stores expose provider rollback
168
+ confirmation to Host's handler transaction through an internal callback request. Ordinary application
169
+ transactions, store-internal approval/erasure transactions and other Host APIs retain their prior
170
+ error contracts, including callback-error identity and erasure's SQLSTATE mappings, when the
171
+ provider below wraps callback failures.
172
+ The request travels on the callback Host passes to `transactionWithEvents`. Code that decorates a
173
+ built-in store's `transactionWithEvents` and substitutes its own callback drops the request, so
174
+ handler retry stays off for that store; a decorator that passes Host's callback through unchanged
175
+ keeps it.
176
+ An application-owned provider can use this pattern with its transaction-bound domain store:
177
+
178
+ ```ts
179
+ import { ConfirmedTransactionRollback, type PostgresPlatformHostTransactionProvider } from "@fabricorg/platform-host";
180
+
181
+ type ApplicationDb = ReturnType<typeof domainStore.withClient>;
182
+ const transactionProvider: PostgresPlatformHostTransactionProvider<ApplicationDb> = {
183
+ async run(run) {
184
+ const client = await pool.connect();
185
+ try {
186
+ await client.query("BEGIN");
187
+ let result;
188
+ try {
189
+ result = await run({ db: domainStore.withClient(client), sql: client });
190
+ } catch (error) {
191
+ await client.query("ROLLBACK");
192
+ throw new ConfirmedTransactionRollback(error);
193
+ }
194
+ // COMMIT errors remain outside the callback rollback signal.
195
+ try {
196
+ await client.query("COMMIT");
197
+ } catch (error) {
198
+ await client.query("ROLLBACK");
199
+ throw error;
200
+ }
201
+ return result;
202
+ } finally {
203
+ client.release();
204
+ }
205
+ },
206
+ };
207
+ ```
208
+
209
+ The provider throws the wrapper after successful `ROLLBACK` of a failed callback. If rollback
210
+ throws, that rejection supplies no confirmation. Memory confirms a failed callback for the Host
211
+ handler request after successful Host-ledger undo and a configured provider's `restore`. Commit-phase
212
+ errors and providerless approval transactions supply no confirmation. Ordinary public Memory
213
+ transaction callbacks retain their original errors. Custom atomic stores that expose the wrapper
214
+ through their public transaction API are supported; Host's non-handler transaction boundary and
215
+ exported conformance rejection checks normalize that transport evidence before classifying errors.
216
+
99
217
  **Handler transaction reconciliation.** A handler or store transaction wrapper can throw
100
218
  `ReconciliationRequiredError(message, { cause })` from `@fabricorg/platform` (also exported from
101
219
  `@fabricorg/platform/policies`) when a domain effect may already have committed and an operator
@@ -133,11 +251,13 @@ The signal does not change failures in pre-handler governance or validation, sta
133
251
  exceptions, adapters, or finalization. Application governance errors and definite adapter failures retain their
134
252
  `failed` paths; post-success errors follow the recovery rules below. State guards still fail closed.
135
253
  Transient `after_adapters` finalization failures escape as `RecoverableFinalizationError` for
136
- recovery within `recoveryMaxAttempts`. An ordinary handler error still records `failed`. This signal requests an operator decision,
254
+ recovery within `recoveryMaxAttempts`. A handler error without eligible confirmed rollback still records `failed`. This signal requests an operator decision,
137
255
  not an automatic retry or a claim that the current transaction committed.
138
256
 
139
257
  New rows bind that key to the action version, actor/authority binding, and a versioned canonical
140
- parameter digest. Host 3 defaults to `enforce`, which throws `IdempotencyConflictError`; configure
258
+ parameter digest. An empty `authorizationBindingId` is treated as absent for submission
259
+ idempotency identity and receipt ownership, including stored empty IDs from older Hosts.
260
+ Host 3 defaults to `enforce`, which throws `IdempotencyConflictError`; configure
141
261
  `idempotency.conflictMode` as `audit-only` for a preflight soak. Legacy null-digest rows skip only the
142
262
  parameter comparison and call `onLegacyRecord`; actor, authority-binding, and action-version mismatches
143
263
  still reject in enforce mode. See `MIGRATION-2-IDEMPOTENCY.md`.
@@ -345,7 +465,7 @@ and accept the weaker stale-worker guarantee. Parent annotations similarly have
345
465
 
346
466
  Atomic finalizers require transaction-scoped `writeAdapterInvocation` when the store advertises the
347
467
  capability and execution holds a lease. A missing member selects verdict-first fallback before
348
- transaction evidence writes. PostgreSQL locks the matching parent with `FOR UPDATE` before the
468
+ transaction evidence writes. PostgreSQL locks the matching parent with `FOR NO KEY UPDATE` before the
349
469
  adapter mutation in one SQL statement. Memory rechecks buffered lease and adapter predicates at
350
470
  commit and excludes claims and other conditional writes while applying the commit. A commit-time
351
471
  conflict rolls back and raises `RecoverableFinalizationError`. No schema migration is added by this
@@ -564,6 +684,35 @@ When `authorizeExecution` is absent, the Host reuses `authorize` at the executio
564
684
  completed idempotent replay returns its existing result without creating or executing a new
565
685
  mutation.
566
686
 
687
+ Applications whose authority is protected by database locks can also configure
688
+ `authorizeExecutionInTransaction`. Host calls it after state-machine validation and directly before
689
+ the handler (or the handlerless action's event/state-transition work), with the same execution input
690
+ plus the `db` supplied to the handler transaction. Only `true` permits the effect; `false` and
691
+ non-boolean results roll the transaction back and use the same authorization-refusal settlement as
692
+ `authorizeExecution`. A throw rolls the transaction back and settles the invocation as `failed`.
693
+ Unlike `authorizeExecution`, a `RetryableExecutionError` thrown here is not deferred for a later
694
+ attempt, because the hook runs after policy evidence is recorded; retry transient authority-store
695
+ errors, such as a lock timeout, inside the hook. The hook runs on initial execution, approval resume,
696
+ recovery, offline replay and staged dispatch when those paths execute the handler transaction. It is
697
+ intentionally skipped for `authorityMoment: "capture"`, whose durable capture evidence governs instead.
698
+
699
+ The hook guards the handler transaction only. A finalizing action's `finalize` callback runs in a
700
+ later transaction, after adapters, without this hook, and a recovery attempt that resumes from a
701
+ finalization checkpoint skips the handler transaction and therefore the hook. Only the
702
+ pre-transaction `authorizeExecution` check runs on that attempt.
703
+
704
+ With `transactionWithEvents`, the hook shares the application transaction and `db` used by the
705
+ handler. A finalizing action has already locked its invocation row, and state-machine validation has
706
+ already performed its application reads before the hook runs; ordinary actions do not acquire an
707
+ invocation-row lock here. Host does not choose lock strengths for application state or authority
708
+ rows. Applications should document a consistent order from those existing locks to any locks acquired
709
+ by the hook. The legacy `store.transaction` path invokes the hook and handler in the same callback on
710
+ the same `db`, but that base store seam does not itself promise a database transaction. A store
711
+ whose `transaction` is not a real database transaction, such as `PostgresPlatformHostStore` built
712
+ without a transaction provider, gives the hook no atomicity with the handler's writes: a lock the
713
+ hook takes is released before the handler runs. Use a store with `transactionWithEvents` when the
714
+ hook must close the revocation gap.
715
+
567
716
  External gateways should persist a compact admission record bound to the tenant, registration, actor,
568
717
  action, canonical parameter hash, resource scope, and expiry, then pass its opaque non-secret identifier
569
718
  as `authorizationBindingId` to `submitAction()`. Credentials and signed principal proofs do not belong
@@ -781,13 +930,24 @@ The evaluator returns `auto-execute`, `needs-approval`, `escalate`, or `rejected
781
930
  from execution claims.
782
931
  - `rejected` terminally fails before policies and mutation code run.
783
932
 
784
- An approval workflow calls `resumeApprovedInvocation()` instead of invoking the handler directly. The
933
+ An approval workflow can record a decision for a worker or call `resumeApprovedInvocation()` to execute inline. For inline resume, the
785
934
  host authorizes the approver (using `authorizeApproval` when provided), then atomically persists the
786
935
  decision and changes `waiting_for_approval` to either leased `running` or terminal `failed`. This keeps
787
936
  the decision in the mutation ledger and prevents an approval/worker race.
788
937
  `authorizeApproval` receives the parked `actionInvocationId` and durable, post-redaction `invocation`
789
938
  record so the application can verify the approver's authority over that specific invocation.
790
939
 
940
+ `authorizeApprovalInTransaction` is the optional final authority boundary for both inline resume and
941
+ `recordApprovalDecision`. The store locks and classifies the invocation first, then calls the hook with
942
+ the transaction-bound `db` and locked invocation immediately before a decision write. Only `true`
943
+ permits the decision. A refusal leaves inline resume waiting and makes record-only approval return
944
+ `unauthorized` with its configured refusal evidence; a throw rolls back. Applied-decision replay,
945
+ expired decisions, and rows that cannot accept a decision do not call the hook. The built-in
946
+ PostgreSQL store holds the invocation row's `FOR NO KEY UPDATE` lock while the hook runs; the memory store
947
+ holds its serialized mutation unit. Applications choose any authority-row lock strength and should
948
+ acquire authority locks after the invocation lock to keep one lock order. Configuring this hook
949
+ requires `transactionalApprovalAuthorization: true`; Host rejects stores that do not advertise it.
950
+
791
951
  ```ts
792
952
  await host.resumeApprovedInvocation(actionInvocationId, tenantId, spaceId, {
793
953
  approved: true,
@@ -802,6 +962,97 @@ Custom stores remain source-compatible when HITL is unused. To enable HITL they
802
962
  additive `ApprovalPlatformHostStore` capability. Both built-in stores implement it; Postgres users must
803
963
  call `ensureSchema()` so the HITL evidence and approval-decision columns are added.
804
964
 
965
+ ### Record a decision for worker execution
966
+
967
+ `createGovernedActionHost()` also returns `ApprovalDecisionRecordingHost`; the base
968
+ `GovernedActionHost` interface retains its handwritten-host shape. An HTTP endpoint can commit a
969
+ decision without running the handler, adapters or `executeInvocation`:
970
+
971
+ ```ts
972
+ const result = await host.recordApprovalDecision(actionInvocationId, tenantId, spaceId, {
973
+ requestId: decisionRequestId,
974
+ approved: true,
975
+ approverId: reviewer.id,
976
+ approverType: "natural_person",
977
+ evidence: { verificationRef: receiptId },
978
+ });
979
+ // result.outcome === "approved": the row is pending for a worker claim.
980
+ ```
981
+
982
+ `requestId` must be a non-blank string of at most 256 characters. Keep it stable across transport
983
+ retries. The result contains `outcome`, `actionInvocationId`, current `status`, `replayed`, and the
984
+ stored `decision` for `approved` or `denied`, including replays. Non-applied outcomes omit
985
+ `decision`, protecting another approver's identity, reason and evidence. Host snapshots and validates
986
+ evidence before authorization.
987
+
988
+ | Outcome | Durable effect |
989
+ | --- | --- |
990
+ | `approved` | Commit the decision and queue `pending`, clearing lease/error without changing `attemptCount`. |
991
+ | `denied` | Commit the decision and settle `failed`, using the inline denial patch. |
992
+ | `unauthorized` | Entitlement or approval authorization returned false; invocation unchanged. |
993
+ | `expired` | A due waiting approval settles `failed` with `approvalExpiry`; a terminal row already carrying approval expiry and no decision remains unchanged. The attempted decision is not applied. |
994
+ | `already_decided` | An existing decision belongs to a different request, or the same request changes verdict or approver id/type; invocation unchanged. |
995
+ | `late_terminal` | Terminal row without an approval decision or approval expiry; unchanged. |
996
+ | `not_waiting` | Non-terminal row without a decision that is not waiting for approval; unchanged. |
997
+ | `unsupported` | Waiting row outside Host HITL atomic approval; unchanged. |
998
+
999
+ The store's locked classifier checks an applied request id, verdict and approver id/type before
1000
+ checking authorization refusal and row state. A matching retry returns its original `approved` or
1001
+ `denied` outcome with `replayed: true` after pending, running, completion or failure, and appends no
1002
+ events. An opposite verdict is `already_decided`. Older decisions without request ids also produce
1003
+ `already_decided` for an authorized request. For a different request id, authorization refusal
1004
+ returns `unauthorized` before existing-decision classification. An authorized late retry returns
1005
+ `expired` from a terminal row's `approvalExpiry`, including when a sweep expired it. It remains
1006
+ `replayed: false`, because this is row-state classification rather than a stored decision receipt.
1007
+ The compatibility authorizers still run before the transaction; thrown callback errors propagate even
1008
+ on a retry. When `authorizeApprovalInTransaction` is configured, its locked result also has to allow
1009
+ the write. An applied-decision replay still runs the compatibility callbacks but skips the
1010
+ in-transaction hook because it writes nothing.
1011
+
1012
+ The store operation atomically classifies, changes state and appends its events/outbox rows.
1013
+ PostgreSQL requires a transaction provider (`transactionWithEvents`); unsupported stores throw a
1014
+ configuration error before writing. Recording does not use `beginApprovalDecision`, whose inline
1015
+ contract remains leased `running`. Missing scoped rows and invalid ids/evidence throw.
1016
+ An unresolved waiting action returns `unsupported` without changing the row, with or without a
1017
+ deadline, including one already due. Approval authorization is skipped when the action cannot
1018
+ resolve; the locked classifier leaves the invocation unchanged in that case. Inline resume keeps
1019
+ its existing unknown-action behavior.
1020
+
1021
+ The stored decision carries `requestId` and `recordedAtAttempt` in the existing decision JSON.
1022
+ The first worker claim after recording has `attemptCount === recordedAtAttempt + 1`. Host computes
1023
+ `approval_resume` for that claim, exempts the non-idempotent interruption guard and compares the
1024
+ recovery budget using `attemptCount - recordedAtAttempt`. The earlier parking claim therefore does
1025
+ not exhaust this first run at `recoveryMaxAttempts: 1`; an inline-origin wait recorded at zero works
1026
+ the same way. Attempt history is retained. Later claims use the existing recovery checks, budget
1027
+ and `recovery` reason, comparing the absolute `attemptCount` against `recoveryMaxAttempts`
1028
+ without subtracting the recorded count. Retryable governance failures on the approved claim retain leased-attempt
1029
+ retry behavior instead of the inline resume override's terminal behavior.
1030
+
1031
+ The pending row is recoverable through `runPlatformActionWorkerCycle` without a separate enqueue.
1032
+ Hosts without a worker must dispatch the pending row themselves. `executeInvocation` exists but is
1033
+ not a concurrent-safe targeted claim. Upgrade workers before using this API: older workers mishandle
1034
+ recorded approvals' attempt and execution-reason semantics. Saga-parent waits are excluded through
1035
+ `ActionDefinition.kind === "saga"`; Host HITL also requires a `needs-approval` or `escalate` checkpoint.
1036
+
1037
+ Set `approvalDecisionAttemptEvents: true` to append `APPROVAL_DECISION_REFUSED_EVENT`
1038
+ (`ApprovalDecisionRefused`) for non-applied outcomes. It defaults off. The payload contains outcome,
1039
+ request id, approver id/type, attempted `approved` verdict, observed `status` and `attemptedAt` ISO
1040
+ time, without evidence, reason text or parameters. Scoped row lookup supplies tenant and space;
1041
+ a wrong scope throws not-found without an event. Refusal event ids combine invocation identity with
1042
+ the SHA-256 digest of the exact request id (`digestParameters`), preserving punctuation through
1043
+ `lifecycleId` sanitization. Store event-id deduplication retains one event for a repeated request.
1044
+ The attempt factory runs after any update with the pre-patch observed row: waiting on first expiry,
1045
+ failed on a later attempt against that expired row. Applied-decision replays append nothing. The
1046
+ classifier appends within its transaction, including on expiry, so append failure rolls back the
1047
+ state patch and events. Host construction requires atomic approval, transaction and outbox support
1048
+ when the option is on. `approvalLifecycleEvents` independently controls applied decision/expiry events.
1049
+
1050
+ After commit, Host emits `invocation.failed` for an applied denial or expiry. Custom recording stores
1051
+ return `applied` alongside outcome, invocation and replay information so Host can distinguish a new
1052
+ transition from a previously expired row. Replays and unchanged rows emit no additional status
1053
+ telemetry. Approval queues pending without a recording-time status event, matching sibling approval
1054
+ paths; the worker emits execution telemetry when it executes the invocation.
1055
+
805
1056
  ### Approval deadlines, evidence, and lifecycle events
806
1057
 
807
1058
  `HostHitlDecision.approvalExpiresAt` accepts an absolute, valid `Date`. Alternatively,
@@ -1040,6 +1291,14 @@ an offline replay and a re-submission under an existing idempotency key:
1040
1291
  `redactActionParameters` changes it (test "the persisted parameters are the copy that was
1041
1292
  checked …").
1042
1293
 
1294
+ When recording refusals is enabled, the recording path first checks request-only metadata:
1295
+ correlation/causation consistency, provenance source and trace attributes, audit attributes when no
1296
+ audit redactor is configured, and parameter-independent authorization-binding fields. A failure
1297
+ throws the validation error and leaves the key free. This validation does not run redaction or trace
1298
+ hooks. Binding digest comparison and capture-time expiry are checked before recording, except
1299
+ that the digest is not compared for actions declaring `admitSubmission`. Audit validation after a
1300
+ configured redactor retains its normal ordering.
1301
+
1043
1302
  Only JSON-data parameters can be admitted. Anything else is refused as `invalid_parameters`
1044
1303
  directly after `authorize`, before the composition callback and the check run. A SharedArrayBuffer or a view over one is refused this way,
1045
1304
  since the caller could still write to its memory (test "a parameter backed by a SharedArrayBuffer …").
@@ -1059,10 +1318,63 @@ Only `{ admitted: true }`, read as an own data property, admits. Each of these m
1059
1318
  - any other answer (`invalid_decision`);
1060
1319
  - parameters that are not JSON data (`invalid_parameters`).
1061
1320
 
1062
- Host writes nothing for a refused submission: no invocation, idempotency key, event, policy record
1321
+ By default, Host writes nothing for a refused submission: no invocation, idempotency key, event, policy record
1063
1322
  or error, so the key stays free. The message names the action, the refusal and the check's code
1064
1323
  only. Host emits `invocation.admission_refused` telemetry with the refusal.
1065
1324
 
1325
+ ### Opt-in recorded submission refusals
1326
+
1327
+ Set `idempotency: { recordRefusals: true }` to reserve a refused submission's idempotency key.
1328
+ The default is `false`. Activation requires a store implementing `recordSubmissionRefusal`
1329
+ and `conflictMode: "enforce"` (the default); `"audit-only"` is rejected at host creation.
1330
+ This applies to keyed, non-staged submissions from any actor type. With the option on, an
1331
+ empty string is a present key; the default path continues to treat it as keyless. Keyless submissions,
1332
+ including staged children, retain the default behavior.
1333
+
1334
+ Host records entitlement and authorization refusals, and admission kinds `refused` and
1335
+ `invalid_parameters`. Admission `failed`, `timed_out`, and `invalid_decision`, malformed-request
1336
+ errors and store errors do not become recorded refusals. A recording failure surfaces the store
1337
+ error; an error's `recorded: true` confirms that the key is occupied by a refusal matching the
1338
+ caller. A durable insert winner decides whether the key holds a refusal or an executable invocation.
1339
+
1340
+ A refusal occupies one `action_invocations` row with terminal status `submission_refused` and a
1341
+ validated `submissionRefusal` descriptor. Parameters and result are fixed empty objects; agent
1342
+ proposal and provenance attributes are absent. The descriptor carries a fixed class and, for
1343
+ admission, the refusal kind and optional fixed `refusalCode`. An `admitSubmission` refusal code
1344
+ is durable evidence when this option is on: choose it from fixed action codes, without deriving
1345
+ it from submitted values (see `ActionAdmissionDecision` in `packages/platform/actions/index.ts`).
1346
+ Create new refusals through `recordSubmissionRefusal`; generic creation and status updates
1347
+ reject `submission_refused` as an input status. Refusal rows appear in invocation listings.
1348
+ Execution, resume and store mutation paths leave
1349
+ these rows inert. The stored-record erasure operator can redact their payload fields;
1350
+ Host does not schedule automatic retention. See **Erasing stored records**.
1351
+
1352
+ Entitlement, authorization and admission checks still run on replay before Host tries an insert;
1353
+ there is no refusal lookup before the checks (use the separate `readSubmissionOutcome` receipt read below). A matching caller receives the stored refusal,
1354
+ including after the checks start passing. Identity compares actor id and type, authorization
1355
+ binding id, action version, and a parameter digest when the stored refusal has one. A refused
1356
+ caller with a different identity receives its own unrecorded refusal. An admitted caller with a
1357
+ different identity receives `IdempotencyConflictError`. A refused caller does not receive the
1358
+ stored result of an executable invocation that won the key.
1359
+
1360
+ For actions without `admitSubmission`, Host digests parameters at the refusal site; a digest
1361
+ failure records nothing and is surfaced. For actions declaring `admitSubmission`, refusal sites
1362
+ do not digest parameters, including entitlement and authorization refusals: the parameters have
1363
+ not passed admission. These refusal rows are not parameter-bound, so changing parameters alone
1364
+ does not free their keys. Use a new key for a new request.
1365
+
1366
+ A caller who fails authorization can occupy a key within the tenant, space and action scope.
1367
+ Applications must select trusted tenant/space context and suitable key namespaces. If an
1368
+ entitlement or authorization callback returns `false` during an outage, that response becomes
1369
+ a permanent refusal when its insert wins. Throw for an unavailable dependency rather than
1370
+ reporting a definitive denial.
1371
+
1372
+ Before enabling the option, apply PostgreSQL migration 15 through `ensureSchema()` and upgrade
1373
+ all hosts sharing the database. Older hosts do not recognize `submission_refused` and could
1374
+ execute a refusal row. Migration 15 adds nullable `submission_refusal`; the existing status
1375
+ column is unconstrained text and parameters accept the fixed empty object. The memory and
1376
+ PostgreSQL stores use the same key scope as ordinary invocation insertion.
1377
+
1066
1378
  The deadline is judged when the check's answer settles, not by which of the answer and a timer
1067
1379
  wins a race. Host reads the monotonic clock (`performance.now()`) when the check is called and
1068
1380
  again in the handler attached to the check's promise. An answer, or a throw, that settles
@@ -1093,6 +1405,25 @@ submitted …"). Before submitting an action that declares `admitSubmission`: (1
1093
1405
  migration 6, and nothing is written; (2) upgrade every worker, since a worker on an older Host would
1094
1406
  execute such an invocation without checking `admittedAt`.
1095
1407
 
1408
+ ### Read a historical submission receipt
1409
+
1410
+ `host.readSubmissionOutcome({ tenantId, spaceId, actionId, idempotencyKey, actorId,
1411
+ actorType, authorizationBindingId })` reads receipt metadata without rerunning mutation
1412
+ permission or admission checks. The embedding application must supply trusted, authenticated
1413
+ tenant, space and actor identity. The original principal can read its receipt after mutation
1414
+ permission is revoked. The key is exact, including the valid empty string.
1415
+
1416
+ A refusal error with `recorded: false` establishes nothing about key ownership: an admitted
1417
+ invocation or another principal's record may hold it. Use the receipt read to observe a
1418
+ visible durable outcome. `no_record` means no record visible to this principal at this
1419
+ snapshot; it does not establish that the key is free. A returned invocation status is an
1420
+ observation and may change. Refused receipts include only their validated descriptor;
1421
+ erased receipts include only identity, version and recorded status. No pending work is dispatched.
1422
+
1423
+ The operation is available independently of refusal recording and conflict mode. Custom
1424
+ stores implement optional `getSubmissionOutcomeByKey`; an unsupported store throws
1425
+ `SubmissionOutcomeUnsupportedError`. Storage errors propagate. See ADR 0022.
1426
+
1096
1427
  ## External mutation governance
1097
1428
 
1098
1429
  Hosts can configure `resolveMutationGovernance` to persist an approved, provider-neutral
@@ -1338,13 +1669,18 @@ cast in an index expression is not immutable and PostgreSQL refuses it.
1338
1669
 
1339
1670
  ## Tenant isolation
1340
1671
 
1341
- `tenantIsolationChecks()` and `runTenantIsolationChecks(subject)` run nine checks against a host
1672
+ `tenantIsolationChecks()` and `runTenantIsolationChecks(subject)` run ten checks against a host
1342
1673
  and its store. They prove scope on the invocation reads and writes, idempotency keys, event streams,
1343
1674
  worker claims, staged-invocation lists and claims, external completion, an optional governed read,
1344
1675
  and adapter checkpoint reads and writes they probe. The subject supplies a host, a recoverable
1345
1676
  store, a registered action, and two scopes in different tenants. Adapter and staged checks create
1346
1677
  scoped probes and retire their parent invocations afterwards. The staged check is skipped for a
1347
- store that does not advertise transactional staged invocations.
1678
+ store that does not advertise transactional staged invocations. The refusal check is skipped when
1679
+ `recordSubmissionRefusal` is absent. When present, it tests refusal insertion/replay and ordinary
1680
+ invocation creation under one key/action in another tenant and another space. It creates terminal
1681
+ refusal rows and completed probe invocations in the other tenant’s and other space’s fresh probe
1682
+ spaces; refusal rows retain their identity and refusal verdict through the stored-record erasure operator. A PostgreSQL store must have migration
1683
+ 15 applied (`ensureSchema()`) before running the kit, or the new check fails.
1348
1684
 
1349
1685
  Host passes `AdapterInvocationScope` from the store-loaded action invocation to
1350
1686
  `getAdapterInvocation(id, scope)` and `updateAdapterInvocation(id, patch, scope)`. Custom stores
@@ -1356,6 +1692,115 @@ refusals and an admitted update in our scope. The two adapter isolation checks
1356
1692
  fail for stores that ignore scope. Deployments running `runTenantIsolationChecks` against a custom
1357
1693
  store will see new failures until that store honours the argument.
1358
1694
 
1695
+ ## Staged invocation conformance
1696
+
1697
+ `stagedInvocationChecks()` and `runStagedInvocationChecks(subject)` expose 11 destructive
1698
+ checks for isolated test stores. `StagedInvocationSubject.makeHarness()` returns a fresh
1699
+ `StagedContractHarness`: a staged/atomic store and optional `raceCreateThenRotate` and
1700
+ `raceRotateThenCreate` hooks. The hooks retain a transaction during the competing operation;
1701
+ without them the checks use ordinary concurrent or ordered store calls.
1702
+
1703
+ The probes cover principal round trips and fencing, lease owner/status/token/expiry fences,
1704
+ token rotation, child identity retention, create/rotation ordering, parent status gating,
1705
+ and batch replay/drift refusal. They do not establish database-clock authority.
1706
+
1707
+ ```ts
1708
+ const result = await runStagedInvocationChecks({
1709
+ makeHarness: async () => ({ store: await createIsolatedStagedStore() }),
1710
+ });
1711
+ ```
1712
+
1713
+ ## Outbox requeue conformance
1714
+
1715
+ `outboxRequeueChecks()` and `runOutboxRequeueChecks(subject)` expose seven destructive
1716
+ checks. `OutboxRequeueSubject.makeStore()` supplies a fresh store implementing the outbox
1717
+ requeue port, including relay operations and readback. Probes cover budget reset and event
1718
+ identity, concurrent requeue, stale authorization generations, detached evidence, relay races,
1719
+ live-record refusal, and tenant/space refusal.
1720
+
1721
+ ```ts
1722
+ const result = await runOutboxRequeueChecks({ makeStore: createIsolatedRequeueStore });
1723
+ ```
1724
+
1725
+ ## Approval conformance
1726
+
1727
+ `approvalChecks()` and `runApprovalChecks(subject)` expose 44 destructive checks, including
1728
+ approval lifecycle parity. `ApprovalSubject.makeStore()` supplies a fresh `ApprovalCheckStore`
1729
+ with approval, recovery, outbox, and transaction support for the host-based probes. These build
1730
+ Host instances internally. Parameterized variants of a named case run within its check.
1731
+
1732
+ Probes cover recording, replay and contradiction, authorization boundaries, expiry, refusal
1733
+ evidence and telemetry, transaction rollback, inline decision parity, and recovery budgets.
1734
+ They exercise the supplied store with the packaged Host; they do not certify application policy.
1735
+
1736
+ ```ts
1737
+ const result = await runApprovalChecks({ makeStore: createIsolatedApprovalStore });
1738
+ ```
1739
+
1740
+ For these sets, inspect each result's `status`, `evidence`, and optional `error`. Missing optional
1741
+ capabilities yield `skipped`; the recorded approval probes also skip absent transactional
1742
+ approval lifecycle/outbox support. Probe errors and missing readback fail. `passed` means no check
1743
+ failed and can include skips. Checks leave invocation, event, and outbox records under fresh
1744
+ identities/spaces. Use isolated test stores with no competing workers or relays; requeue probes
1745
+ can mark rows published without sending to a real consumer. A store factory must provide the
1746
+ transaction support its advertised capabilities require.
1747
+
1748
+
1749
+ ## Outbox relay conformance
1750
+
1751
+ `outboxRelayChecks()` and `runOutboxRelayChecks({ makeStore })` expose eight destructive
1752
+ checks. `OutboxRelaySubject.makeStore()` returns a fresh `OutboxPlatformHostStore<unknown>`
1753
+ for each check. The public types are `OutboxRelaySubject`, `OutboxRelayCheck` and
1754
+ `OutboxRelayConformanceResult`; `OutboxRelayResult` remains the relay cycle's result type.
1755
+
1756
+ Probes cover claim eligibility (including equal availability and expiry boundaries), claim
1757
+ owner/expiry/attempt/token effects, a one-row batch limit, concurrent claim exclusivity,
1758
+ owner/token settlement fences, settled-row refusal and renewal fences. Each check uses a base
1759
+ time one hour ahead of the wall clock, with claim and renewal expiry modeled through explicit
1760
+ `now` inputs and a margin for wall-clock settlement guards. New rows must be available by their
1761
+ first claim time, at least one hour after the check starts; availability need not equal `recordedAt`.
1762
+ The relay checks require numeric lease tokens that increase on each claim, including reclaim,
1763
+ even though the general outbox types keep token fields and parameters optional.
1764
+ The `claim-limit` check pins only the batch limit: PostgreSQL orders claim
1765
+ candidates but does not guarantee the order of its returned update rows, and the relay does
1766
+ not rely on returned order. When `renewOutboxLeaseAndRead` is absent, the renewal check still
1767
+ exercises `renewOutboxLease` and does not skip.
1768
+
1769
+ Out of scope: claim/listOutbox returned ordering and settlement of expired but unreclaimed
1770
+ leases (the probes settle while leases are unexpired by the wall clock), relay attempt budgets
1771
+ and backoff policy (`src/outbox.ts`, `runOutboxRelayCycle`),
1772
+ error text, dead-letter requeue (`outboxRequeueChecks`), and first-write-wins event appends
1773
+ (`eventAppendChecks`). These checks ship with the package version, so upgrading Host and
1774
+ rerunning them shows where a custom store has fallen behind.
1775
+
1776
+ ```ts
1777
+ const result = await runOutboxRelayChecks({ makeStore: createIsolatedOutboxStore });
1778
+ ```
1779
+
1780
+ ## Invocation idempotency conformance
1781
+
1782
+ `invocationIdempotencyChecks()` and `runInvocationIdempotencyChecks({ makeStore })` expose
1783
+ seven destructive checks. `InvocationIdempotencySubject.makeStore()` returns a fresh
1784
+ `PlatformHostStore<unknown>` for each check, with optional `recordSubmissionRefusal` support.
1785
+ The other public types are `InvocationIdempotencyCheck` and `InvocationIdempotencyResult`.
1786
+
1787
+ Probes cover first-row replay and readback with changed submission fields, tenant/space/action
1788
+ key scope, distinct unkeyed inserts, empty-string keys, concurrent first-write arbitration,
1789
+ admission timestamp retention and refusal/admitted winner arbitration. The concurrent probe
1790
+ reads each submitted candidate id through the base store port. Only `refusal-winner` skips
1791
+ when `recordSubmissionRefusal` is absent; thrown probes and failed readbacks fail. Inspect
1792
+ individual statuses: `passed` means no check failed and can include that optional skip.
1793
+
1794
+ Out of scope: duplicate-id creates without a key (the stores differ), conflict classification
1795
+ (Host's responsibility), and cross-tenant refusal scoping (already covered by
1796
+ `fabric.tenancy.submission-refusal-scoped.v1`). These checks ship with the package version,
1797
+ so upgrading Host and rerunning them shows where a custom store has fallen behind.
1798
+ Use isolated stores with no competing workers; both sets leave records under fresh ids/spaces.
1799
+
1800
+ ```ts
1801
+ const result = await runInvocationIdempotencyChecks({ makeStore: createIsolatedInvocationStore });
1802
+ ```
1803
+
1359
1804
  ## Event append conformance
1360
1805
 
1361
1806
  `eventAppendChecks()` and `runEventAppendChecks(subject)` run six checks for first-write-wins
@@ -1526,7 +1971,8 @@ declarations therefore does not query the migration-11 tables by default.
1526
1971
 
1527
1972
  The child has a Host-only `stagedOrigin` of parent invocation ID plus stage key and, when selected,
1528
1973
  the durable principal name. `authorizeExecution` can inspect `invocation.stagedOrigin.principal`
1529
- alongside the child's durable parameters and actor. Public `SubmitActionInput` has no
1974
+ alongside the child's durable parameters and actor. When configured, `authorizeExecutionInTransaction`
1975
+ receives the same staged origin on the child's handler transaction. Public `SubmitActionInput` has no
1530
1976
  `stagedOrigin`; Host adds it through the internal staged-dispatch path.
1531
1977
  The lookup and unique index intentionally remain keyed by tenant, space, parent invocation ID, and
1532
1978
  stage key: one parent stage key identifies one child, while the principal is evidence for that
@@ -1800,3 +2246,329 @@ Apply PostgreSQL migration 13 (`agent_proposal`) with `ensureSchema()` before th
1800
2246
  proposal submission. Inserts without proposals omit the new column, following the
1801
2247
  `admitted_at` rollout pattern. Existing migration SQL is unchanged; readers accept legacy
1802
2248
  rows without a proposal. See `src/postgres-store.ts` (`ensureSchema`, `toActionRecord`).
2249
+
2250
+ ## Erasing stored records
2251
+
2252
+ `createGovernedActionHost()` also returns the separate `StoredRecordErasureHost`
2253
+ operator capability. `GovernedActionHost` keeps its existing shape. Applications
2254
+ can use `createStoredRecordErasureHost(store, authorization)` without constructing an action
2255
+ runtime. Supply trusted `actorId` and `actorType` from the authenticated application session.
2256
+ `validateErasureInput` requires nonblank actor fields, rejects NUL/lone-surrogate
2257
+ characters in `actorId`, and checks `actorType` with Platform's `isActorType`.
2258
+ The factory's required second argument selects `authorizeStoredRecordErasure` from `PlatformHostAuthorization`; the
2259
+ integrated Host passes its own authorization object.
2260
+
2261
+ After `validateErasureInput`, the operator calls `authorization.authorizeStoredRecordErasure`
2262
+ with exactly `{ actorId, actorType, tenantId, spaceId, requestId, targets }`. Targets are
2263
+ validated, detached copies; the hook receives neither `redact` nor stored payloads.
2264
+ The operator awaits the hook through the authorization object, preserving class receivers,
2265
+ before calling the store, with no prior record reads. Its strict `authorized !== true`
2266
+ guard permits erasure only for a `true` result.
2267
+
2268
+ Absent hooks and non-`true` results throw the exported
2269
+ `StoredRecordErasureAuthorizationError` with `refusal: "not_authorized"`.
2270
+ Absence uses `"Stored-record erasure is refused because no authorizeStoredRecordErasure is configured."`
2271
+ Other refusals use `"Actor <actorId> is not authorized to erase stored records in this scope."`
2272
+ Typed refusals contain no target-existence details. A thrown hook error propagates
2273
+ unchanged before store access, distinguishing a failed check from an explicit denial.
2274
+ Successful operations return `EraseStoredRecordsResult`; validation errors and
2275
+ `ErasureBusyError` remain distinct failures.
2276
+
2277
+ ```ts
2278
+ import { createStoredRecordErasureHost } from "@fabricorg/platform-host";
2279
+
2280
+ // Reuse the application's authorization object; no configured erasure hook means refusal.
2281
+ const operator = createStoredRecordErasureHost(store, authorization);
2282
+ const result = await operator.eraseStoredRecords({
2283
+ actorId: "privacy-operator",
2284
+ actorType: "natural_person",
2285
+ requestId: "privacy-request-2026-11-10",
2286
+ lockTimeoutMs: 250, // Optional; target/connection acquisition budget.
2287
+ workTimeoutMs: 5000, // Optional; rewrite work budget.
2288
+ tenantId: "tenant-a",
2289
+ spaceId: "production",
2290
+ targets: [
2291
+ { kind: "invocation", invocationId: "act_123" },
2292
+ { kind: "event", eventId: "evt_456" },
2293
+ ],
2294
+ redact(record) {
2295
+ // Transform the detached payload fields and return the same record context.
2296
+ // This example removes a known application value, retaining object shapes.
2297
+ return JSON.parse(JSON.stringify(record).replaceAll("value-to-remove", "[erased]"));
2298
+ },
2299
+ });
2300
+ ```
2301
+
2302
+ `ErasablePlatformHostStore<TDb>` extends `PlatformHostStore<TDb>` with
2303
+ `eraseStoredRecords(input: EraseStoredRecordsInput): Promise<EraseStoredRecordsResult>`.
2304
+ This store capability is an unauthenticated persistence port, not an authorization boundary.
2305
+ Direct callers must enforce authorization; the Host operator enforces it through the hook.
2306
+ Both built-in stores use `erasureAudit` to retain the request's `actorId` and `actorType`
2307
+ in the `HostRecordsErased` event envelope. These are identifying metadata, outside the
2308
+ redaction view, rather than erased payload values.
2309
+ The base store port has no new required method. Unsupported stores fail with an
2310
+ explicit capability error. The built-in memory and PostgreSQL stores implement
2311
+ this capability; PostgreSQL requires an application transaction provider.
2312
+ `validateErasureInput` rejects missing or blank scope/request identifiers, malformed
2313
+ targets, NUL/lone-surrogate identifiers, duplicate kind/ID pairs, batches outside 1–100 targets, and non-function
2314
+ redactors before invoking the store. IDs shared by different target kinds are distinct.
2315
+ The result retains request target order, reports `erased`, `already_erased`, `active`,
2316
+ or `not_found`, and counts processed records by record kind without returning payloads.
2317
+ Absent and foreign-scope targets both return `not_found`.
2318
+
2319
+ An invocation target covers its invocation payload fields, adapter inputs/outputs/errors
2320
+ and reconciliation, staged parameters/errors linked as parent or child, policy evaluations,
2321
+ policy obligations, execution attestations, external reconciliations, canonical events,
2322
+ outbox envelopes/errors/trace context, and outbox requeue reason/prior-error evidence.
2323
+ An event target covers the canonical event and its outbox/requeue copies. It does not
2324
+ rewrite copies inside the owning invocation's `finalizationCheckpoint.handlerEvents`,
2325
+ `stateTransition`, result or other invocation evidence: checkpoint events have no event ID
2326
+ to join. Use an invocation target to cover those copies of the logical event. The
2327
+ `ERASURE_COLUMNS` source manifest and schema-discovered inventory tests classify
2328
+ text and JSON columns. The event callback runs once per canonical event in a batch;
2329
+ stores copy its redacted payload and disclosed provenance into the outbox envelope.
2330
+ Outbox-specific errors and trace context have their own callback view. Legacy outbox
2331
+ rows lacking a canonical event expose their payload/provenance in that outbox view. Overlapping
2332
+ targets do not invoke the callback twice for the same stored record in one batch.
2333
+
2334
+ `StoredRecordErasureView` discriminates record kinds and exposes identifying context
2335
+ as readonly properties plus payload field names in `fields`. `redactStoredFields` uses
2336
+ `ERASURE_PROTECTED_PATHS` to omit specific Host-owned paths and restore their values.
2337
+ There is no recursive blacklist of key names. For example, policy fact values under
2338
+ `outcome.guidance.factsUsed[].value` remain visible even when they contain `id`, `type`,
2339
+ `status` or `provider`. Application extensions outside the paths below are visible.
2340
+
2341
+ | Record/field | Protected paths relative to that field |
2342
+ | --- | --- |
2343
+ | Invocation/event/outbox `provenance` | `source`, `correlationId`, `causationId` |
2344
+ | Invocation `authorizationBinding` | `id`, `tenantId`, `actorId`, `actionId`, `parameterDigest`, `governingMoment`, `capturedAt`, `expiresAt` |
2345
+ | Invocation `authorizationReconciliation` | `kind`, `governingMoment`, `executionReason`, `authorizationBindingId`, `provenance.source`, `provenance.correlationId`, `provenance.causationId` |
2346
+ | Invocation `adapterReconciliation` | `kind`, `adapterType`, `operation`, `vendor`, `adapterInvocationId`, `externalReference` |
2347
+ | Invocation `approvalDecision` | `approved`, `approverId`, `approverType`, `decidedAt`, `requestId`, `recordedAtAttempt` |
2348
+ | Invocation `pendingCompletion` | `provider`, `adapterType`, `operation`, `adapterInvocationId`, `externalReference`, `acceptedAt`, `dueAt`, `parkedAt`, `overdueAt`; a non-null pending completion prevents erasure |
2349
+ | Invocation `externalCompletion` | `digest`, `externalReference`, `provider`, `outcome`, `observedAt`, `recordedAt` |
2350
+ | Invocation `pendingStateTransition` | `entityType`, `entityId`, `fromState`, `toState` |
2351
+ | Invocation `approvalExpiry` | `expiresAt`, `expiredAt` |
2352
+ | Invocation `submissionRefusal` | `class`, `refusal`, `refusalCode` |
2353
+ | Invocation `finalizationCheckpoint` | `protocol`, `actionVersion`, `steps.length`, `handlerEvents.length`, `stagedInvocations.length`, `steps[].adapterType`, `steps[].operation`, `skipped`; identity fields `eventType`, `eventSchemaVersion`, `subjectType`, `subjectId` and `payload.fromState/toState` on `handlerEvents[]` and `stateTransition`; `stagedInvocations[].key/actionId/expectedActionVersion/principal` |
2354
+ | Adapter `reconciliation` | `kind`, `adapterType`, `operation`, `vendor`, `adapterInvocationId`, `externalReference`, `adapterEventSubject.subjectType`, `adapterEventSubject.subjectId` |
2355
+ | Policy evaluation `outcome` | `policyId`, `policyVersion`, `policyKind`, `result` |
2356
+ | Execution attestation `attestation` | `provider`, `operation`, `externalOperationId`, `satisfiesObligations`, `startedAt`, `completedAt` |
2357
+ | External reconciliation `reconciliation` | `status`, `provider`, `externalOperationId`, `attempt`, `observedAt` |
2358
+ | Event/outbox `payload` | Top-level `fromState`, `toState`; `HostRecordsErased` additionally retains `requestId`, `targets`, `counts` |
2359
+ | Other payload fields | No protected descendants; record headers remain outside the callback view |
2360
+
2361
+ The store rejects changed context or field names, protected-path insertion, invalid JSON,
2362
+ NUL/lone-surrogate strings, incompatible field types and removal of structural containers.
2363
+ Keep array shapes when they contain protected descendants. Null/absent fields remain
2364
+ null/absent. Supply a synchronous, pure JSON transformation; a thrown error aborts the batch.
2365
+ Settled parameters and results need not retain their execution schemas.
2366
+
2367
+ Invocation erasure checks settlement on its locked closure. Event-only owner observations have the concurrency limit below. Pending,
2368
+ running and waiting-for-approval invocations are active. A terminal invocation also
2369
+ remains active with a pending external completion, a live invocation lease, a pending
2370
+ linked stage, or a live linked stage/outbox lease. An event inherits its owning
2371
+ invocation's active refusal and checks its own outbox lease. Completed, failed,
2372
+ blocked-by-policy, validation-failed, submission-refused and reconciliation-required
2373
+ rows without these conditions are settled. Pending stages can make completed parents
2374
+ active: staged dispatch runs after parent completion. A reconciliation-required row
2375
+ without a pending completion can be erased; marker guards fence later evidence writes.
2376
+
2377
+ The PostgreSQL transaction and memory critical section atomically apply accepted
2378
+ rewrites, scope-bound markers and a sequenced `HostRecordsErased` audit event. The audit
2379
+ contains request ID, target kind/ID/outcome and counts, without erased values, value
2380
+ hashes or a free-text reason. Active/missing targets may coexist with erased targets
2381
+ in one successful batch. A callback failure rolls back rewrites, markers and the audit.
2382
+ Repeated requests reapply the callback to stored redacted data and report
2383
+ `already_erased`; `requestId` identifies audit work rather than suppressing retries.
2384
+
2385
+ PostgreSQL erasure requires a transaction provider running READ COMMITTED; it checks
2386
+ `SHOW transaction_isolation` and refuses other levels. Ordinary writers at stricter
2387
+ isolation levels may receive a serialization failure instead of an erased outcome.
2388
+ Apply migration 16, upgrade **all Host instances sharing the database**, then enable
2389
+ erasure. Older Host SQL and direct application SQL bypass these guards and are unsupported
2390
+ once erasure is enabled. Migration 16 adds metadata columns, markers and erasure-lookup
2391
+ indexes without triggers or write-path functions. Like migrations 1–15 it runs in the
2392
+ migration runner's transaction and takes table DDL locks; ordinary index builds block
2393
+ writes for their duration on large tables. Plan an appropriate maintenance window.
2394
+
2395
+ `eraseStoredRecords` processes sorted targets inside individual savepoints. An invocation
2396
+ target locks its invocation and dependent adapter, governance, staged parent/child, event,
2397
+ outbox and requeue rows. It also locks the stage identified by the child's stored
2398
+ `staged_parent_invocation_id` and `staged_stage_key`, even before that stage has recorded
2399
+ its child ID. A pending or leased origin stage keeps the child active; a settled origin
2400
+ stage is redacted with the child. This follows the stage, not the parent invocation's
2401
+ payloads or other stages; target the parent separately for its checkpoint copies. An event target locks its event and delivery/requeue rows, without
2402
+ locking its invocation owner or sibling rows. Canonical delivery copies are found by event
2403
+ ID. Legacy outbox-only rows with null owner columns are found through the indexed envelope
2404
+ `actionInvocationId`; new inserts populate the owner column explicitly. No advisory lock
2405
+ is taken by erasure. The migration runner retains its separate schema advisory lock.
2406
+
2407
+ The first existing target lock may wait within `lockTimeoutMs` (default 250 ms, including
2408
+ provider acquisition). While retaining target locks, subsequent row locks, including the
2409
+ final tenant/space sequence-row lock, use `NOWAIT`; contention rolls back with
2410
+ `ErasureBusyError`. The audit allocation and insert are the final SQL statements before
2411
+ provider commit. Before opening that transaction, a separate provider transaction commits
2412
+ an `event_sequences` row with `next_sequence=0` using `ON CONFLICT DO NOTHING`. It takes no
2413
+ target locks. It attempts the insert only when it sees no committed row for the tenant and
2414
+ space, so it can wait, within the acquisition budget, only for another transaction's
2415
+ uncommitted first insert of that row. The zero seed consumes no event
2416
+ sequence: the first allocation returns 1. The erasure transaction then locks the existing
2417
+ row with `NOWAIT`; it does not try to insert a missing sequence row while holding targets.
2418
+ The seed can remain after a refused or rolled-back erasure without consuming a sequence.
2419
+ Steady event appends in the same tenant and space can starve erasure: it can repeatedly
2420
+ report busy on this final nonwaiting lock. Each retry repeats callbacks and rewrites before
2421
+ attempting that lock; busy rolls those rewrites back.
2422
+
2423
+ Dependent rows are locked before settlement and callbacks. A claim/renewal committed before
2424
+ those locks is observed. Active/missing targets roll back to their savepoints, releasing
2425
+ locks acquired for that target before processing the next. Locks retained from an earlier
2426
+ overlapping accepted target stay held. Invocation settlement is evaluated on locked rows.
2427
+ For an event-only target the owner's status, stages and sibling leases are observed after
2428
+ the event closure is locked, but are not themselves locked. A concurrent change to that
2429
+ owner/sibling activity can therefore race that observation. Target the invocation when
2430
+ settlement of its complete closure must be serialized.
2431
+
2432
+ Ordinary invocation updates predicate on their own `records_erased_at`. Existing dependent
2433
+ updates use their row stamp; insertion paths without a proven transaction-held owner lock
2434
+ use a materialized owner read with `FOR KEY SHARE` inside the insertion statement. Host's
2435
+ ordinary owner locks change from the released version's `FOR UPDATE` to `FOR NO KEY UPDATE`. Transaction-held owner reads are reused
2436
+ for event and reconciliation insertion. Staged creation fences parent and initial child;
2437
+ dispatch/failure attachment fences the newly attached child. An erased child cannot acquire
2438
+ a stage containing old parameters through these methods. Host terminally rejects an
2439
+ unsettled stage if an older/custom store nevertheless returns an erased child by origin,
2440
+ so the attachment refusal cannot keep that stage retrying indefinitely. Replay conflict updates preserve
2441
+ the stored row, and event reappend remains first-write-wins.
2442
+
2443
+ Outbox, stage and invocation claims lock actual candidate rows with `SKIP LOCKED`, without
2444
+ a blocking parent lock. Stage cancellation also selects candidates with `SKIP LOCKED`; transactional approval
2445
+ expiry uses the optional `lockDueApprovals` transaction member for that selection. Delivery of erased outbox records remains usable: claims,
2446
+ renewals, publish/fail acknowledgement and requeue use stored redacted envelopes; failure
2447
+ withholds supplied error text and requeue withholds supplied reason text while retaining
2448
+ stored prior errors. The optional `renewOutboxLeaseAndRead` capability returns the stored
2449
+ envelope atomically with renewal. Relays use that envelope and reject expired responses;
2450
+ custom stores retain the boolean `renewOutboxLease` API. A custom store implementing
2451
+ erasure without optional renew-and-read keeps a window in which the relay can publish an
2452
+ envelope claimed before erasure.
2453
+
2454
+ Callers that do not erase still pay an owner `FOR KEY SHARE` lock for owned inserts outside
2455
+ a supplied mutation transaction, one locking invocation read per transactional Host event
2456
+ append, and one renew-and-read round trip per relayed record. Payload writes that classify
2457
+ a zero-row refusal add a classification `SELECT`, including retried owned duplicate event
2458
+ appends; conditional boolean/lease writes and ownerless duplicate events omit that query.
2459
+ The Host changeset reports the local invocation/adapter insert/update measurements and
2460
+ their scope.
2461
+
2462
+ These row acquisitions keep erasure from waiting on an ordinary row writer while retaining
2463
+ its earlier target locks; it cannot supply the waiting edge of such a row-lock deadlock
2464
+ cycle. They do not promise freedom from indirect waits in arbitrary application transactions,
2465
+ DDL interference, writer-configured timeouts or external transaction-provider behavior.
2466
+ Memory mirrors erased outcomes using its transaction queue and synchronous commit; it does
2467
+ not provide PostgreSQL's row-level lock footprint.
2468
+
2469
+ Indexed selection, callbacks, rewrites and audit work use `workTimeoutMs` (default 5000 ms).
2470
+ Remaining-budget checks and PostgreSQL statement/idle-transaction timeouts bound SQL and idle
2471
+ lock retention. Neither budget preempts synchronous JavaScript or promises a hard real-time
2472
+ network deadline. Timeout options accept integer milliseconds from 1 to 60000. Providers
2473
+ should commit promptly, preserve original rollback errors and discard failed connections.
2474
+ Ordinary query timeouts retain their PostgreSQL identity; database-caused `ErasureBusyError`
2475
+ retains its cause.
2476
+
2477
+ Erasure removes the invocation `parameterDigest` (SQL `NULL`, including
2478
+ `submission_refused` rows). Store-owned rewrites replace
2479
+ `authorizationBinding.parameterDigest`, `externalCompletion.digest` and the
2480
+ `parameterDigest` of rewritten stage rows with exported `ERASED_DIGEST = "erased"`.
2481
+ The sentinel is outside the 64-hex SHA-256 output space and preserves required string
2482
+ fields and the stage column's `NOT NULL` constraint. `ERASURE_PROTECTED_PATHS` keeps
2483
+ these nested fields out of the callback view and rejects their insertion;
2484
+ `redactStoredFields` overwrites them after merging protected structure. The stores
2485
+ remove or replace top-level digests in the row rewrite, alongside its erasure stamp.
2486
+ Repeating erasure leaves these digest fields absent or set to the same sentinel.
2487
+
2488
+ Private proposal/event comparison columns, memory maps and equality ports are removed.
2489
+ Migration 16 is edited in place because it has not been published; migrations 1–15
2490
+ retain their SQL. `packages/platform-host/tests/erasure-digest-cases.ts` checks raw records in both stores,
2491
+ including repeat erasure, and `erasureChecks` checks the four digest outcomes.
2492
+
2493
+ Host keyed resubmission and recorded-refusal paths evaluate retained actor, binding ID
2494
+ and action-version conflicts first. An enforce-mode conflict raises
2495
+ `IdempotencyConflictError`; otherwise an erased row raises `StoredRecordsErasedError`,
2496
+ including in audit-only mode, before the legacy callback or redacted-result return.
2497
+ A caller that is itself refused while the key holds an erased row is handled separately:
2498
+ against an erased recorded refusal, a retained-identity conflict leaves the caller's own
2499
+ refusal in place and a matching identity raises `StoredRecordsErasedError`; an erased row
2500
+ of any other status leaves the caller's own refusal in place.
2501
+ `createDurableSagaParentLifecycle` preserves its event-type conflict check, then refuses
2502
+ an erased prior event before comparing payloads. Erasure-capable custom stores expose
2503
+ `storedRecordsErased` on the mutation transaction's `getEvent` result.
2504
+ Store batch-replay paths refuse erased stage rows before digest comparisons, including
2505
+ an origin stage erased with its child before child-ID attachment.
2506
+ `packages/platform-host/tests/erasure-digest-cases.ts` pins same/different payload refusals and unchanged events.
2507
+
2508
+ Host overwrites only the digests it computes itself. A hash that another package or the
2509
+ application places inside stored evidence is shown to the redactor and is kept unless the
2510
+ redactor removes it. `@fabricorg/policy-opa` stores one by default: `inputHash`, a SHA-256
2511
+ of the policy input, in the policy outcome's engine metadata. Remove it in the redactor
2512
+ when the policy input can contain erased values, or configure the evaluator's `hashInput`.
2513
+
2514
+ The staged `batchDigest` formula now covers parent ID and sorted keys, actions,
2515
+ versions and optional principals, excluding parameters and parameter digests.
2516
+ Migration 16 replaces persisted `batch_digest` values with `LEGACY_BATCH_DIGEST`.
2517
+ Custom `PlatformHostStore` implementations persisting stages should overwrite stored
2518
+ batch digests with that placeholder and skip only the batch-digest comparison for
2519
+ those rows; the remaining request identity and batch-size comparisons still apply.
2520
+ `stagedInvocationRequestMatches` and `LEGACY_BATCH_DIGEST` are exported from
2521
+ `@fabricorg/platform-host` for this comparison.
2522
+ During a rolling upgrade, an old Host can write payload-covering digests after the
2523
+ migration runs. Finish the rollout before relying on this guarantee, and replace
2524
+ any such late-written digests with the placeholder.
2525
+ Erasure replaces external reconciliation IDs with fresh random `rec_` IDs, including
2526
+ on repeated erasure, following the existing reapplication of the redactor.
2527
+ See `packages/platform-host/tests/erasure-digests.test.ts` and
2528
+ `packages/platform-host/tests/staged-invocations.test.ts` for regression coverage.
2529
+
2530
+ Application hashes outside the protected Host paths remain visible to the redactor.
2531
+
2532
+ | Field | Coverage and reader before erasure | Erasure result |
2533
+ | --- | --- | --- |
2534
+ | Invocation `parameterDigest` | Submitted parameters at identity capture, potentially including values removed by write-time redaction; `idempotencyConflict` and `refusalIdentityConflict` | Removed / SQL `NULL` |
2535
+ | `authorizationBinding.parameterDigest` | Parameters bound to captured authority; `validateAuthorizationBinding` validates the incoming binding, rather than comparing the stored binding digest | `ERASED_DIGEST` |
2536
+ | Staged `parameterDigest` | Canonical child parameters; `stagedInvocationRequestIdentity` feeds store exact-batch comparisons; staged dispatch does not read this field | `ERASED_DIGEST` on rewritten stages |
2537
+ | Staged `batchDigest` | Parent ID plus sorted requests; store `stageInvocations` / `applyStagedBatch` drift checks | Payload-free formula; legacy values replaced by migration 16 |
2538
+ | `externalCompletion.digest` | Provider, reference, outcome, result, error and evidence, excluding observation time; completion duplicate/contradiction and conditional-write comparisons | `ERASED_DIGEST` |
2539
+ | Private proposal / saga payload comparisons | Original proposal and saga payload comparisons | Removed, including columns and memory maps |
2540
+ | Contradiction reconciliation ID | `recordCompletionReconciliation` hashes provider/reference/kind, with the kind carrying a completion-digest prefix | Replaced with a fresh random ID |
2541
+
2542
+ An older Host reading the missing invocation digest treats the erased row as a legacy
2543
+ record and can accept a changed request under its existing key. This is an additional
2544
+ reason to upgrade Host instances sharing storage before the first erasure; older writers
2545
+ also bypass the erasure guards.
2546
+
2547
+ Store SQL guards reject invocation/evidence writes after erasure with `StoredRecordsErasedError`
2548
+ (`HOST_RECORDS_ERASED`). Conditional `updateSettledActionInvocation` and
2549
+ `writeAdapterInvocation` return `false` in both built-in stores. Late approval recording
2550
+ returns its normal replay/refusal outcome without appending fresh attempt evidence for an
2551
+ erased record. External completion refuses before reading its supplied payload (`ExternalCompletionError.refusal ===
2552
+ "erased"`, a permanent refusal). Completion processing loads the scoped invocation
2553
+ before validating supplied completion fields. Erasure retains live memory invocation
2554
+ references during its await-free commit so a delayed read observes the marker. First-write-wins event append retains
2555
+ the existing redacted event. Outbox claims hydrate the stored redacted envelope. For an erasable store, the relay
2556
+ renews the lease and uses the returned stored envelope immediately before publisher handoff and withholds stale claims or
2557
+ renewal responses that arrived after the renewed lease expired;
2558
+ requeue retains that envelope and redacted prior-error evidence. Requeue replaces
2559
+ new operator reason text with `[erased]`, and failure settlement discards new publisher
2560
+ error text for erased events. This prevents those late inputs from restoring old text.
2561
+
2562
+ Apply migration 16 with `ensureSchema()` and upgrade **Host instances sharing the
2563
+ database before the first erasure**. Mixed versions can hold previously loaded payloads.
2564
+ A payload handed to a publisher, adapter, caller, cache or downstream consumer before
2565
+ erasure commits cannot be recalled. This operation does not erase external systems,
2566
+ backups, application domain data or consumer projections; arrange those separately.
2567
+ See webhook-delivery's **Event erasure** section for the equivalent delivery limit.
2568
+
2569
+ `erasureChecks()` and `runErasureChecks(subject)` provide eight destructive checks for
2570
+ isolated test stores. Supply `payloadRecords.seed/read` fixture access to verify the
2571
+ record kinds outside the base store's read port. Missing erasure support or fixture
2572
+ access reports the relevant check as `skipped`; callback/check errors report `failed`.
2573
+ The success probes check for exactly one scoped `HostRecordsErased` event with the
2574
+ request's `actorId` and `actorType`. Both built-in stores run the complete kit in their tests.