@fabricorg/platform-host 7.5.0 → 8.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +220 -0
- package/MIGRATION-8.md +247 -0
- package/README.md +455 -45
- package/dist/index.cjs +5407 -4251
- package/dist/index.cjs.map +1 -1
- package/dist/index.d.cts +1513 -1380
- package/dist/index.d.ts +1513 -1380
- package/dist/index.js +5400 -4246
- package/dist/index.js.map +1 -1
- package/package.json +9 -7
package/README.md
CHANGED
|
@@ -68,7 +68,8 @@ delay is the remaining lease duration, followed by the worker's next claim; ther
|
|
|
68
68
|
retry or separate backoff. The next claim increments `attemptCount`. At the cap, or for an unleased,
|
|
69
69
|
non-idempotent, or approval-resume invocation, Host records `failed` with the reason retry was refused,
|
|
70
70
|
through the error-text redactor. Approval resume is excluded because its temporary lease does not
|
|
71
|
-
establish that a recovery worker is configured. Ordinary
|
|
71
|
+
establish that a recovery worker is configured. Ordinary application governance errors retain terminal failure behavior. Host store errors after
|
|
72
|
+
a recorded provider effect follow the recovery rules below.
|
|
72
73
|
|
|
73
74
|
The retry region includes Host's single-record store writes for mutation-governance and HITL
|
|
74
75
|
checkpoints. Later attempts reuse these persisted decisions through the `mutationFootprint` and
|
|
@@ -76,10 +77,52 @@ checkpoints. Later attempts reuse these persisted decisions through the `mutatio
|
|
|
76
77
|
A deferral happens before the attempt writes any policy evidence: the complete definition batch must
|
|
77
78
|
evaluate before Host starts appending policy records. Policy persistence, the `ComplianceBlocked`
|
|
78
79
|
event, obligation listing/resolution/persistence, state-machine checks, the handler transaction,
|
|
79
|
-
handlers
|
|
80
|
+
handlers and adapters retain their separate failure semantics. Finalization recovery is bounded by
|
|
81
|
+
`recoveryMaxAttempts`. Its effective default is at least `retryableExecutionMaxAttempts` because
|
|
82
|
+
both count worker claims.
|
|
80
83
|
A retryable obligation-resolver error fails terminally with the policy evidence from that attempt.
|
|
81
84
|
Preview evaluation does not enter this execution catch.
|
|
82
85
|
|
|
86
|
+
**Handler transaction reconciliation.** A handler or store transaction wrapper can throw
|
|
87
|
+
`ReconciliationRequiredError(message, { cause })` from `@fabricorg/platform` (also exported from
|
|
88
|
+
`@fabricorg/platform/policies`) when a domain effect may already have committed and an operator
|
|
89
|
+
must decide how to proceed. For example, a recovery-receipt store can check its own stored result
|
|
90
|
+
against the action's current declared events and result schema before handing that result back.
|
|
91
|
+
If its own check detects an incompatible receipt from an earlier attempt, it can throw this signal.
|
|
92
|
+
Put non-sensitive operator detail in the message: Host stores the redacted message, not `cause`.
|
|
93
|
+
The cross-copy predicate
|
|
94
|
+
`isReconciliationRequiredError` recognises the stable name and `EXECUTION_RECONCILIATION_REQUIRED`
|
|
95
|
+
code as well as local instances.
|
|
96
|
+
|
|
97
|
+
A `transactionWithEvents` wrapper cannot tell which Host transaction it is wrapping from the public
|
|
98
|
+
callback contract: Host also uses this API for `after_adapters` finalization and external completion.
|
|
99
|
+
Translate only a condition the store detects itself while recovering its own receipt, and let other
|
|
100
|
+
errors propagate unchanged. Do not match Host's internal validation-error messages; those errors
|
|
101
|
+
have no public stable code or message contract.
|
|
102
|
+
|
|
103
|
+
Host honours the signal only at the handler-transaction catch, covering `transactionWithEvents`
|
|
104
|
+
and the fallback `transaction`. It redacts the message with `redactErrorText` using `source: "handler"`,
|
|
105
|
+
records terminal `reconciliation_required`, and returns the error with the action's consistency and
|
|
106
|
+
persisted HITL route/risk tier. Status persistence emits the existing reconciliation telemetry.
|
|
107
|
+
Re-execution returns the terminal record without running the handler. Workers retain their existing
|
|
108
|
+
counting: a returned reconciliation result increments `failed`, not `completed` or `deferred`.
|
|
109
|
+
|
|
110
|
+
Atomic `transactionWithEvents` stores roll back the attempt's domain writes, domain events, and
|
|
111
|
+
Host's `StateTransitioned` together when the transaction throws. On the `transaction`-only fallback,
|
|
112
|
+
Host appends events directly. The built-in stores simply run the callback without rollback: domain
|
|
113
|
+
writes and any events Host appended during the attempt stay persisted beside the
|
|
114
|
+
`reconciliation_required` record, as they already do beside `failed`. A custom fallback transaction
|
|
115
|
+
may roll back its domain writes, but Host does not make its event appends atomic with those writes.
|
|
116
|
+
Host does not run adapters after this signal or add a new lifecycle event for it. Effects or evidence
|
|
117
|
+
committed by earlier attempts are not undone.
|
|
118
|
+
|
|
119
|
+
The signal does not change failures in pre-handler governance or validation, state-transition guard
|
|
120
|
+
exceptions, adapters, or finalization. Application governance errors and definite adapter failures retain their
|
|
121
|
+
`failed` paths; post-success errors follow the recovery rules below. State guards still fail closed.
|
|
122
|
+
Transient `after_adapters` finalization failures escape as `RecoverableFinalizationError` for
|
|
123
|
+
recovery within `recoveryMaxAttempts`. An ordinary handler error still records `failed`. This signal requests an operator decision,
|
|
124
|
+
not an automatic retry or a claim that the current transaction committed.
|
|
125
|
+
|
|
83
126
|
New rows bind that key to the action version, actor/authority binding, and a versioned canonical
|
|
84
127
|
parameter digest. Host 3 defaults to `enforce`, which throws `IdempotencyConflictError`; configure
|
|
85
128
|
`idempotency.conflictMode` as `audit-only` for a preflight soak. Legacy null-digest rows skip only the
|
|
@@ -87,16 +130,282 @@ parameter comparison and call `onLegacyRecord`; actor, authority-binding, and ac
|
|
|
87
130
|
still reject in enforce mode. See `MIGRATION-2-IDEMPOTENCY.md`.
|
|
88
131
|
|
|
89
132
|
Lifecycle records use deterministic checkpoint ids. Recovered idempotent actions do not repeat an
|
|
90
|
-
adapter that already reached `succeeded
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
`
|
|
99
|
-
|
|
133
|
+
adapter that already reached `succeeded`. Built-in stores deduplicate event and policy inserts by
|
|
134
|
+
checkpoint ID. Adapter rows are mutable; stale-worker protection requires the optional
|
|
135
|
+
`writeAdapterInvocation` capability described below. A
|
|
136
|
+
reclaimed non-idempotent action inspects its adapter checkpoints: running or succeeded rows route
|
|
137
|
+
to reconciliation; no rows or only failed rows retain terminal failure for manual review.
|
|
138
|
+
|
|
139
|
+
Returned `ambiguous` or `timeout` outcomes, timeouts thrown from `adapter.execute` (including an
|
|
140
|
+
aborted adapter deadline), contradictory successful results, and reverse contradictions
|
|
141
|
+
(`success: false` with `outcome: "success"`) route to `reconciliation_required`.
|
|
142
|
+
Host detects a thrown timeout from the adapter deadline's aborted signal or an error message matching
|
|
143
|
+
`/timeout|deadline|timed?\s*out/i`. A definite rejection whose message matches therefore also routes to
|
|
144
|
+
reconciliation; the message matcher does not prove that the provider timed out.
|
|
145
|
+
|
|
146
|
+
An uninterpretable provider reply (a non-object, `null`, or an array), a getter that throws before Host
|
|
147
|
+
can interpret the reply, and an unrecognised outcome string such as `"timed_out"` also take this path.
|
|
148
|
+
Host copies the reply's contract fields and other own enumerable fields into a plain object before
|
|
149
|
+
interpretation, reading each top-level field once. A getter that throws during the copy takes the
|
|
150
|
+
unreadable-reply path. These outcomes are not retried by the retry helper. Object replies retain the
|
|
151
|
+
previous truthiness reading of `success`, including truthy non-boolean values from JavaScript adapters.
|
|
152
|
+
The reconciliation kind remains `adapter_outcome_ambiguous`.
|
|
153
|
+
|
|
154
|
+
Unknown-outcome settlement uses `transactionWithEvents` when its transaction supplies
|
|
155
|
+
`updateAdapterInvocation`, `appendExternalReconciliation` on a governance-capable store, and
|
|
156
|
+
`updateLeasedActionInvocation` for a leased invocation. A store advertising conditional adapter writes
|
|
157
|
+
also needs transaction-scoped `writeAdapterInvocation` for leased finalization. In that transaction Host writes the
|
|
158
|
+
`ambiguous` checkpoint, `AdapterInvocationAmbiguous` with its stable event ID, governance
|
|
159
|
+
`ExternalReconciliation` evidence where supported, and the fenced `reconciliation_required` verdict,
|
|
160
|
+
in that order. These writes commit together. Host retries once on a non-lease error; a second
|
|
161
|
+
non-lease failure selects the verdict-first fallback. Lease loss raises
|
|
162
|
+
`RecoverableFinalizationError` without retry or fallback.
|
|
163
|
+
|
|
164
|
+
Without those transaction members, or after two non-lease transaction failures, the verdict-first
|
|
165
|
+
fallback writes the terminal verdict (retrying once except on lease loss), then the checkpoint,
|
|
166
|
+
event, and reconciliation record. If both checkpoint shapes fail, Host still attempts the ambiguity
|
|
167
|
+
event and reconciliation record. After the verdict commits, Host attempts each remaining evidence
|
|
168
|
+
write even if the event append fails, then surfaces the first error as `RecoverableFinalizationError`.
|
|
169
|
+
An evidence failure can leave a terminal verdict with missing evidence; a later direct execution returns that verdict
|
|
170
|
+
and workers exclude it from recovery, without repairing the evidence gap. If both verdict attempts
|
|
171
|
+
fail, Host surfaces `RecoverableFinalizationError` without attempting an unfenced checkpoint.
|
|
172
|
+
|
|
173
|
+
With the original action definition and adapter-step order available, recovery checks `ambiguous`
|
|
174
|
+
rows before the parked-handoff shortcut, stale-action gates, authorization, and handler execution.
|
|
175
|
+
This applies to actions declared idempotent and actions not declared idempotent. Settlement uses the
|
|
176
|
+
stored reconciliation metadata or the legacy row's error as its message and reason, without running
|
|
177
|
+
the handler or an adapter. For a running invocation whose action resolves, `executeInvocation` reads
|
|
178
|
+
each adapter step's row before the pending-handoff shortcut, stopping on recorded ambiguity or a
|
|
179
|
+
failed read. A failed read surfaces `RecoverableFinalizationError`, where base 939bcad returned the
|
|
180
|
+
parked result. The shortcut's condition is unchanged from base: a running invocation with
|
|
181
|
+
`pendingCompletion` and no lease owner returns early without requiring `pendingCompletion.parkedAt`.
|
|
182
|
+
If the action no longer resolves, the shortcut still returns the handoff before the unknown-action
|
|
183
|
+
failure. Completion callbacks that settle the invocation require parking, including after a deadline sweep.
|
|
184
|
+
|
|
185
|
+
`AdapterInvocationRecord.reconciliation` is an optional, redacted checkpoint preserving reason,
|
|
186
|
+
external reference, and the original `adapterEventSubject`. The subject stays out of the returned
|
|
187
|
+
`AdapterReconciliationOutcome`. Recovery uses this checkpoint without scanning `listEvents()`.
|
|
188
|
+
Without stored subject metadata it uses `{ subjectType: "AdapterInvocation", subjectId: adapterInvocationId }`;
|
|
189
|
+
the stable event ID deduplicates an event already written. Custom stores should persist the checkpoint
|
|
190
|
+
through adapter create, update, and read methods.
|
|
191
|
+
|
|
192
|
+
Ordinary adapter writes do not name the migration 8 column and remain compatible with schemas
|
|
193
|
+
lacking it. If a checkpoint write carrying `reconciliation` fails, fallback retries the row once
|
|
194
|
+
without that field, using legacy `status: "ambiguous"` plus `error`. Unknown outcomes therefore still
|
|
195
|
+
settle without migration 8, but without stored checkpoint metadata. Apply PostgreSQL migration 8
|
|
196
|
+
through `ensureSchema()` in controlled startup or migrations to retain that metadata. An error inside
|
|
197
|
+
an atomic PostgreSQL transaction first rolls it back; the legacy write runs outside that transaction
|
|
198
|
+
on the fallback path.
|
|
199
|
+
|
|
200
|
+
On a store without atomic support, if the verdict cannot be written at all, no durable write after the
|
|
201
|
+
call succeeds: neither verdict nor checkpoint is saved, as after a crash mid-call. A leftover `running`
|
|
202
|
+
step can then re-run on recovery for an idempotent action under the same `adapterInvocationId`
|
|
203
|
+
idempotency key, within the worker recovery budget. A reclaimed non-idempotent action with a
|
|
204
|
+
running row without a matching durable handoff settles that step as unknown. A succeeded row or
|
|
205
|
+
durable handoff instead ends `reconciliation_required` as an applied effect. Adapters should be
|
|
206
|
+
idempotent or carry that stable key. The same exposure applies if two atomic attempts roll back
|
|
207
|
+
and the fallback verdict cannot be written.
|
|
208
|
+
|
|
209
|
+
Once an invocation has a durable `succeeded` adapter row or accepted handoff, errors in Host
|
|
210
|
+
store reads or writes raise `RecoverableFinalizationError` instead of ending it `failed`.
|
|
211
|
+
This includes later adapter checkpoints, obligation reads, and the completion write in both
|
|
212
|
+
`before_adapters` and `after_adapters`. Recovery also protects Host store operations before
|
|
213
|
+
and around the handler when an earlier provider effect is recorded. Authorization, policy and handler callbacks
|
|
214
|
+
retain their own failure semantics. Idempotent recovery repairs recorded success/handoff evidence
|
|
215
|
+
without calling that step's provider. Non-idempotent interruption ends `reconciliation_required`
|
|
216
|
+
instead of repairing the invocation. Lease loss retains its fencing error.
|
|
217
|
+
|
|
218
|
+
If neither record is durable, Host retries the failed handoff or marker write once. After a
|
|
219
|
+
second failure it reuses degraded atomic finalization where available, otherwise writes the
|
|
220
|
+
`reconciliation_required` verdict first, then attempts the succeeded marker with safe output,
|
|
221
|
+
`ExternalOperationAccepted` when applicable, and `AdapterInvocationSucceeded`. Each fallback
|
|
222
|
+
evidence write is attempted independently; the first error raises `RecoverableFinalizationError`
|
|
223
|
+
after the remaining writes are attempted. One exception keeps its 7.x behaviour: when
|
|
224
|
+
unsatisfied obligations end a handed-off invocation `reconciliation_required` and the
|
|
225
|
+
reconciliation-log append fails, Host still returns the verdict. A terminal verdict can retain
|
|
226
|
+
missing evidence. If even the
|
|
227
|
+
verdict cannot be saved, Host throws `RecoverableFinalizationError`; the row can remain `running`.
|
|
228
|
+
Recovery may replay that unrecorded outcome for an idempotent action under its stable
|
|
229
|
+
`adapterInvocationId`, within the worker budget. A reclaimed non-idempotent action instead
|
|
230
|
+
ends `reconciliation_required`. A durable handoff takes precedence over its running row and is
|
|
231
|
+
recorded as an applied effect with its external reference. A running row without such a handoff
|
|
232
|
+
is settled as unknown. A succeeded row also ends `reconciliation_required`. With no rows or only failed rows, the existing
|
|
233
|
+
non-idempotent interruption gates still fail the invocation.
|
|
234
|
+
|
|
235
|
+
For idempotent recovery, the handler still runs before the adapter loop. The loop reads each
|
|
236
|
+
checkpoint before `getInput()` or adapter lookup. Succeeded rows and durable handoffs use stored
|
|
237
|
+
adapter type, operation, vendor, input and output; a falsy new input or removed adapter does not
|
|
238
|
+
skip evidence repair. Stable event IDs deduplicate replayed accepted and success events; newly
|
|
239
|
+
repaired success events include `recovered: true` and stored input/output when available.
|
|
240
|
+
|
|
241
|
+
Recovery looks up the stable attestation ID. If present, Host reuses it and reapplies its
|
|
242
|
+
`satisfiesObligations` updates without calling the extractor. If absent and stored output exists,
|
|
243
|
+
Host calls `extractExecutionAttestation` with stored input/output and checkpoint `createdAt` and
|
|
244
|
+
`updatedAt` as start/completion times. The extractor may run again after an interrupted append;
|
|
245
|
+
it must return deterministic evidence for the same input. A handoff checkpoint with no output,
|
|
246
|
+
or withheld output, skips extraction. Repair errors remain recoverable within the budget.
|
|
247
|
+
|
|
248
|
+
Known limits: a later step's definite provider failure or pre-call refusal can still end the
|
|
249
|
+
invocation `failed` after an earlier step succeeded. Authorization denials, ordinary handler
|
|
250
|
+
failures, validation/state-guard rejection, and unsatisfied policy obligations on recovery retain
|
|
251
|
+
their existing terminal outcomes. These are decisions or application failures, not Host store
|
|
252
|
+
errors. Idempotent recovery can call a provider again for an unrecorded running outcome, or for a
|
|
253
|
+
definite failed row whose invocation failure verdict was lost. The handler still re-runs before
|
|
254
|
+
the adapter loop. Saga compensation is unchanged. Withheld-output and refused-handoff
|
|
255
|
+
reconciliation retain the degraded pattern. Adapter-row fencing and overdue-sweep/late-callback
|
|
256
|
+
rules are described below.
|
|
257
|
+
|
|
258
|
+
### Conditional adapter-row writes (8.0.0, issue #79)
|
|
259
|
+
|
|
260
|
+
`PlatformHostStore.writeAdapterInvocation(input)` is optional. Memory and PostgreSQL implement it;
|
|
261
|
+
custom stores can keep their existing methods. Without it, Host retains unchecked adapter creates
|
|
262
|
+
and updates, so a reclaimed worker can overwrite a newer worker's adapter checkpoint.
|
|
263
|
+
|
|
264
|
+
The input carries `scope` (tenant, space, invocation), a create or update `mutation`, a `condition`,
|
|
265
|
+
and optional `expectedStatuses`. A lease condition requires a `running` parent with the supplied
|
|
266
|
+
`leaseOwner` and `leaseToken`. A verdict condition requires the exact parent `status` and `error`
|
|
267
|
+
written by this execution. Host also captures the handoff row ID as
|
|
268
|
+
`completionAdapterInvocationId`, so `externalCompletionDigest` guards that row's callback output:
|
|
269
|
+
an absent digest requires no recorded completion; a supplied digest requires that same completion.
|
|
270
|
+
Other adapter rows retain the status/error and expected-row-status checks without the digest check.
|
|
271
|
+
Omitting the captured row ID retains the conservative digest check on adapter writes. Absent error
|
|
272
|
+
matches no error. Both parent and adapter must match scope.
|
|
273
|
+
A create must also match the record's scope and refuses a duplicate ID. The method returns `false`
|
|
274
|
+
when its conditions fail. Conditional updates reject `succeeded` or `ambiguous` to `running` or
|
|
275
|
+
`failed`, including when callers omit `expectedStatuses`.
|
|
276
|
+
|
|
277
|
+
Host captures the supplied execution credentials before loading the invocation. It uses them for
|
|
278
|
+
fenced invocation writes and leased adapter creation, retry preparation, success, failure, and
|
|
279
|
+
ambiguity checkpoints. A refused lease or row-status check raises `RecoverableFinalizationError`.
|
|
280
|
+
This prevents stale checkpoint writes; adapters still need stable idempotency keys because a lease
|
|
281
|
+
cannot cancel a provider call already in flight. Inline writes use the verdict condition after
|
|
282
|
+
this execution writes its own verdict; earlier unleased adapter writes retain their existing path.
|
|
283
|
+
|
|
284
|
+
Verdict-first fallbacks use the settled verdict condition after the terminal write releases the
|
|
285
|
+
lease. Their expected row status is the last state observed or written by that execution. A changed
|
|
286
|
+
parent status/error or row refuses the adapter update; a changed completion also refuses a write
|
|
287
|
+
to the captured handoff row. These refusals emit
|
|
288
|
+
`invocation.adapter_evidence_refused` (`fabric.platform.invocation.adapter_evidence_refused`) through
|
|
289
|
+
the telemetry hook. The adapter row stays untouched, with no unchecked retry. Host still attempts
|
|
290
|
+
`AdapterInvocationAmbiguous` and reconciliation evidence; withheld-output and refused-handoff
|
|
291
|
+
success evidence may already have been appended before the refused marker write. A refusal is not
|
|
292
|
+
an evidence-collector error. Thrown store errors are retained while each independent fallback
|
|
293
|
+
write is attempted, then the first error is raised as `RecoverableFinalizationError`.
|
|
294
|
+
|
|
295
|
+
An inline writer preserving a sweep verdict does not adopt it as this execution's own verdict.
|
|
296
|
+
Its adapter evidence keeps the unchecked inline path; conditional settled-parent enrichment is
|
|
297
|
+
used only for a verdict this execution actually wrote. The inline guard reads the current row
|
|
298
|
+
only when the execution snapshot or its patch carries `pendingCompletion`.
|
|
299
|
+
|
|
300
|
+
Budget exhaustion saves its plain verdict before reading evidence. It then enriches the parent
|
|
301
|
+
through optional `updateSettledActionInvocation`, comparing that execution's exact status, error,
|
|
302
|
+
and completion snapshot. Parent enrichment joins the independent evidence collector, alongside
|
|
303
|
+
adapter checkpoints, events and reconciliation records. Refusal emits
|
|
304
|
+
`invocation.adapter_evidence_refused` with `evidence: "parent_enrichment"` and is not an error;
|
|
305
|
+
a missing `updateSettledActionInvocation` skips only parent enrichment. Evidence writes still run.
|
|
306
|
+
A thrown store error is collected and raised as `RecoverableFinalizationError` after the remaining
|
|
307
|
+
writes are attempted. The two optional store methods are expected together. Successful enrichment
|
|
308
|
+
stores the error and `adapterReconciliation` before returning them; skipped or refused enrichment
|
|
309
|
+
can leave replay with only the plain verdict. Stores omitting `writeAdapterInvocation` retain
|
|
310
|
+
unchecked adapter writes. A retained handoff's reference is included in the initial budget verdict.
|
|
311
|
+
|
|
312
|
+
Subclasses of the built-in stores that override `createAdapterInvocation` or
|
|
313
|
+
`updateAdapterInvocation` for encryption or auditing must also override `writeAdapterInvocation`.
|
|
314
|
+
Checked writes can bypass the unchecked methods. Alternatively, explicitly disable the capability
|
|
315
|
+
and accept the weaker stale-worker guarantee. Parent annotations similarly have their own
|
|
316
|
+
`updateSettledActionInvocation` seam.
|
|
317
|
+
|
|
318
|
+
Atomic finalizers require transaction-scoped `writeAdapterInvocation` when the store advertises the
|
|
319
|
+
capability and execution holds a lease. A missing member selects verdict-first fallback before
|
|
320
|
+
transaction evidence writes. PostgreSQL locks the matching parent with `FOR UPDATE` before the
|
|
321
|
+
adapter mutation in one SQL statement. Memory rechecks buffered lease and adapter predicates at
|
|
322
|
+
commit and excludes claims and other conditional writes while applying the commit. A commit-time
|
|
323
|
+
conflict rolls back and raises `RecoverableFinalizationError`. No schema migration is added by this
|
|
324
|
+
capability; it uses the existing parent lease and verdict columns.
|
|
325
|
+
|
|
326
|
+
External-completion adapter writes use the callback's transaction-scoped `updateAdapterInvocation`
|
|
327
|
+
when available, sharing its invocation lock. These are callback writes rather than worker-lease
|
|
328
|
+
writes; legacy transactions without that member retain the outer-store write. The sweep retains
|
|
329
|
+
its transaction-scoped reconciliation append and parked/overdue settlement rules.
|
|
330
|
+
|
|
331
|
+
### Bounded worker recovery (8.0.0)
|
|
332
|
+
|
|
333
|
+
`recoveryMaxAttempts` is an optional positive integer. Its effective default is
|
|
334
|
+
`max(5, retryableExecutionMaxAttempts)`; `Infinity` is rejected. Both caps count the same worker
|
|
335
|
+
claims. Construction rejects an explicit `recoveryMaxAttempts` below
|
|
336
|
+
`retryableExecutionMaxAttempts`. The governance cap stops transient pre-policy deferrals at its
|
|
337
|
+
cap; the recovery budget stops resumed execution above its cap. With the defaults, governance
|
|
338
|
+
retry exhaustion therefore happens on claim 5 before recovery exhaustion could happen on claim 6.
|
|
339
|
+
|
|
340
|
+
Host checks a resumed leased `running` invocation immediately after lease ownership, before
|
|
341
|
+
normal action resolution, ambiguity scans, authorization, the handler, or adapter work. When
|
|
342
|
+
`attemptCount > recoveryMaxAttempts`, it settles `reconciliation_required` with
|
|
343
|
+
`recovery attempts exhausted (n/max)`. It writes that verdict before resolving the action or
|
|
344
|
+
reading adapter evidence, releasing the lease. A running row without a matching durable handoff
|
|
345
|
+
receives unknown-outcome settlement. A recorded ambiguous row enriches the parent error and
|
|
346
|
+
`adapterReconciliation`, as well as adapter/event evidence, using its stored reconciliation evidence
|
|
347
|
+
and a conditional parent update. A durable handoff takes precedence over that step's running row.
|
|
348
|
+
It is recorded as an applied effect with its external reference in the message and, on governance
|
|
349
|
+
stores, the reconciliation record. It is not marked ambiguous solely because its marker was lost.
|
|
350
|
+
If resolution or row reads throw, or no relevant row exists, the plain verdict already written
|
|
351
|
+
stands. Enrichment failure can retain an evidence gap. Transition telemetry is emitted once when
|
|
352
|
+
the status changes. If the verdict is unwritable, Host throws `RecoverableFinalizationError`;
|
|
353
|
+
later claims continue but do not execute mutation code.
|
|
354
|
+
|
|
355
|
+
The counter counts worker claims, including the first run, rather than provider calls or completed
|
|
356
|
+
recoveries. With the default of 5: the first run plus four recoveries; the sixth claim settles.
|
|
357
|
+
A raised governance cap raises the implicit recovery budget with it. Inline/direct execution does
|
|
358
|
+
not increment the counter. Approval acceptance sets it to at least 1 without incrementing it on
|
|
359
|
+
resume. Approval resume holds a temporary lease and uses the existing count: a count equal to the
|
|
360
|
+
budget is admitted, while a count above the budget settles reconciliation before the handler.
|
|
361
|
+
Approval does not reset an exhausted budget. A worker dying before reaching the check consumes a
|
|
362
|
+
claim without calling a provider at that point. Parked handoffs have no lease, are not claimed by
|
|
363
|
+
workers, and are outside this budget. The existing parked shortcut and ambiguity scan remain in
|
|
364
|
+
place. `retryableExecutionMaxAttempts` still limits pre-policy deferrals. No new store method is
|
|
365
|
+
required.
|
|
366
|
+
|
|
367
|
+
**Inline and direct callers.** Inline `submitAction()` without a dispatcher and direct
|
|
368
|
+
`executeInvocation()` can now reject with an error whose `name` is
|
|
369
|
+
`"RecoverableFinalizationError"` where 7.x returned `failed`. The class is not exported. Without a
|
|
370
|
+
lease, the invocation stays `running` and workers do not claim it. Before a handoff is durable,
|
|
371
|
+
call `executeInvocation()` again yourself and cap your own retries. Once a handoff is durable,
|
|
372
|
+
re-running `executeInvocation()` on the unleased running row returns without resuming execution
|
|
373
|
+
(it first settles any step already recorded `ambiguous`).
|
|
374
|
+
If execution stopped before parking, callbacks are refused with `still_executing` until a sweep.
|
|
375
|
+
Declare `completionDeadlineMs` for actions that hand off so the sweep surfaces the row, then
|
|
376
|
+
reconcile it per the [external-completion runbook](../../docs/operations/runbooks.md#overdue-handoffs).
|
|
377
|
+
The leased worker budget does not bound unleased calls. A
|
|
378
|
+
previously leased direct call retains its lease until expiry and can also be recovered by a
|
|
379
|
+
worker. Terminal degraded verdicts can retain evidence gaps and are not automatically repaired.
|
|
380
|
+
|
|
381
|
+
A transaction implementation may rethrow the handler's error wrapped in its own error; Host
|
|
382
|
+
still treats a failure inside the handler callback as a handler outcome.
|
|
383
|
+
|
|
384
|
+
**Known limit.** If a later step's adapter row is created but its attempt-preparation write keeps
|
|
385
|
+
failing, the row stays `running` although the provider was not called. For an action not declared
|
|
386
|
+
idempotent, the next claim settles that step as an unknown provider outcome.
|
|
387
|
+
|
|
388
|
+
|
|
389
|
+
A deadline sweep retains the handoff and sets `pendingCompletion.overdueAt` with its
|
|
390
|
+
`reconciliation_required` verdict. A late callback settles that verdict only when the handoff also
|
|
391
|
+
has `parkedAt`, proving Host reached parking after its remaining steps and finalization. An unparked
|
|
392
|
+
handoff stays in reconciliation even if later-step ambiguity evidence is missing; the callback's
|
|
393
|
+
redacted result and outcome are recorded as evidence for an operator without applying the action's
|
|
394
|
+
result schema. This evidence-only behavior applies to invocations already
|
|
395
|
+
`reconciliation_required`; callbacks for unparked `running` invocations are refused with
|
|
396
|
+
`still_executing`. Any parked row without `overdueAt`, including a 7.x row, still uses the old
|
|
397
|
+
`"did not complete by"` error substring. That compatibility path is weaker: an unrelated error
|
|
398
|
+
containing the substring on a parked row lacking the marker can also authorize settlement.
|
|
399
|
+
|
|
400
|
+
Circuit-open refusals and deadline expiry before execution retain `failed` semantics because
|
|
401
|
+
the refused attempt made no provider call. A `prepareAttempt` store error retains its prior failure
|
|
402
|
+
behavior only while the invocation has no durable success or handoff. With an earlier recorded
|
|
403
|
+
effect, an exhausted Host-store error raises `RecoverableFinalizationError`. Returned `permanent_failure`
|
|
404
|
+
and exhausted `transient_failure` outcomes, plus exhausted non-timeout adapter throws, also retain
|
|
405
|
+
`failed` semantics and their existing events, unless reading any field of the reply throws: Host
|
|
406
|
+
copies the reply before interpreting it, so such a reply takes the unreadable-reply path to
|
|
407
|
+
reconciliation. Adapters report definite non-effects with
|
|
408
|
+
`permanent_failure` or `transient_failure`; no additional definite-failure flag is needed.
|
|
100
409
|
|
|
101
410
|
### Adapter circuit breakers
|
|
102
411
|
|
|
@@ -210,6 +519,10 @@ in action parameters or workflow history.
|
|
|
210
519
|
Production stores can implement `AtomicMutationPlatformHostStore.transactionWithEvents`. The
|
|
211
520
|
transaction-scoped `db`, event sequence, event append, event listing, and invocation update methods
|
|
212
521
|
must all use the same database transaction.
|
|
522
|
+
The optional transaction-scoped adapter-update and `appendExternalReconciliation` operations
|
|
523
|
+
extend this unit of work to degraded adapter finalization. Leased finalizers use
|
|
524
|
+
`writeAdapterInvocation` when the store advertises it, otherwise `updateAdapterInvocation`,
|
|
525
|
+
as described under redaction below.
|
|
213
526
|
|
|
214
527
|
`PostgresPlatformHostStore` enables this capability when constructed with a
|
|
215
528
|
`PostgresPlatformHostTransactionProvider`. The application owns that narrow binder because only the
|
|
@@ -222,7 +535,8 @@ With `transactionWithEvents`:
|
|
|
222
535
|
- `after_adapters`: declared domain events, the Host-generated `StateTransitioned` fact, and
|
|
223
536
|
invocation finalization commit together after required adapter steps succeed or accept a
|
|
224
537
|
parked external handoff. Handler writes have already committed separately. A finalization
|
|
225
|
-
failure leaves the invocation recoverable
|
|
538
|
+
failure leaves the invocation recoverable within `recoveryMaxAttempts`; recovery repairs evidence
|
|
539
|
+
for succeeded adapter checkpoints without repeating those calls.
|
|
226
540
|
|
|
227
541
|
Stores without `transactionWithEvents` retain the legacy non-atomic boundary for domain writes,
|
|
228
542
|
events, and invocation persistence. A partial append or finalization failure can leave some
|
|
@@ -298,7 +612,7 @@ Three optional `PlatformHostOptions` hooks cover other Host-recorded values.
|
|
|
298
612
|
| --- | --- |
|
|
299
613
|
| `redactAdapterOutput(actionId, adapterType, operation, output)` | Once after a successful adapter result is converted to output, before handoff handling. The adapter row, `AdapterInvocationSucceeded` payload, and `extractExecutionAttestation` input use the redacted output. |
|
|
300
614
|
| `redactErrorText(context, message)` | At the origin of caught handler/execution errors, adapter result errors and ambiguous-outcome reasons, and completion errors. The invocation/adapter errors, reconciliation messages and reasons, completion evidence, failure events, and returned error reuse that text. |
|
|
301
|
-
| `redactExternalCompletionResult(actionId, provider, result)` | After a completion matches its invocation, before
|
|
615
|
+
| `redactExternalCompletionResult(actionId, provider, result)` | After a completion matches its invocation, before persistence and digesting; successful settling callbacks also merge and validate against the result schema. See [External completion](#external-completion). |
|
|
302
616
|
|
|
303
617
|
`ErrorTextRedactionContext` is exported. It contains `actionId` and `source` (`handler`, `adapter`,
|
|
304
618
|
`external_completion`, or `execution`); adapter errors also supply `adapterType`, `operation`, and
|
|
@@ -314,29 +628,63 @@ failures refuse with non-retryable `redaction_failed` before writes, allowing a
|
|
|
314
628
|
callback after the redactor is repaired.
|
|
315
629
|
|
|
316
630
|
If `redactAdapterOutput` throws or returns null, an array, or a non-object value, the external effect
|
|
317
|
-
has already happened. Host
|
|
318
|
-
|
|
319
|
-
|
|
320
|
-
|
|
321
|
-
|
|
322
|
-
|
|
323
|
-
|
|
324
|
-
|
|
325
|
-
|
|
326
|
-
|
|
327
|
-
|
|
328
|
-
|
|
329
|
-
|
|
330
|
-
|
|
331
|
-
|
|
332
|
-
|
|
631
|
+
has already happened. Host withholds output and records a `reconciliation_required` verdict. Its
|
|
632
|
+
safe evidence includes the retained handoff's `ExternalOperationAccepted`,
|
|
633
|
+
`AdapterInvocationSucceeded` with redacted input and `outputWithheld: true` without output, and
|
|
634
|
+
fixed-text reconciliation evidence when the store supports governance. The events use the normal
|
|
635
|
+
step deduplication keys. A first accepted reference is retained as `pendingCompletion`, so its
|
|
636
|
+
callback can still land as evidence. References come from the original adapter result.
|
|
637
|
+
|
|
638
|
+
For withheld output and refused handoffs (immediate completion or a second handoff), Host uses
|
|
639
|
+
`transactionWithEvents` when the transaction supplies `updateAdapterInvocation`,
|
|
640
|
+
`updateLeasedActionInvocation` for a leased invocation, and `appendExternalReconciliation` for
|
|
641
|
+
withheld output or refused handoffs on a governance-capable store. Leased finalization also needs
|
|
642
|
+
transaction-scoped `writeAdapterInvocation` when the store advertises it. These optional members must
|
|
643
|
+
share the event/verdict transaction; the adapter update must enforce the store method's supplied
|
|
644
|
+
tenant, space, and invocation scope. Both built-in stores provide them when configured with a
|
|
645
|
+
transaction provider. Custom stores must implement them to get this atomic behavior.
|
|
646
|
+
|
|
647
|
+
When atomic finalization succeeds, evidence events, any withheld-output or refused-handoff
|
|
648
|
+
reconciliation entry, the adapter's `succeeded` marker, and the terminal `reconciliation_required`
|
|
649
|
+
verdict commit together. Host writes evidence and
|
|
650
|
+
the marker before the fenced verdict, which releases the lease, then emits status telemetry after
|
|
651
|
+
commit. It retries the whole transaction once on a non-lease error. If the retry also fails with a
|
|
652
|
+
non-lease error, Host selects the verdict-first fallback described below. Lease loss is not retried
|
|
653
|
+
or converted to fallback; it raises `RecoverableFinalizationError`. Atomic rollback leaves the
|
|
654
|
+
invocation and adapter row `running` until a retry or fallback commits. If lease loss or an unavailable
|
|
655
|
+
fallback verdict prevents finalization, the invocation remains reclaimable and the adapter can run
|
|
656
|
+
again under the same `adapterInvocationId` idempotency key for idempotent actions within the
|
|
657
|
+
worker recovery budget. Non-idempotent reclaim inspects its checkpoints instead of replaying a call.
|
|
658
|
+
|
|
659
|
+
Without the required transaction members, or after two non-lease transaction failures, Host keeps
|
|
660
|
+
the verdict-first order: the `reconciliation_required` verdict precedes evidence and the adapter marker.
|
|
661
|
+
It retries the verdict write once, except on lease loss. After the verdict is durable, Host attempts
|
|
662
|
+
each evidence write independently: a failed event, reconciliation append, or adapter checkpoint does
|
|
663
|
+
not skip the remaining writes. Host raises the first evidence failure as `RecoverableFinalizationError`
|
|
664
|
+
only after those attempts finish. This includes unknown-outcome settlement and post-success degraded
|
|
665
|
+
finalization. Failed writes can still leave a terminal verdict with missing evidence; terminal replay
|
|
666
|
+
does not repair that evidence gap. If both verdict-write attempts fail, the adapter can run again on
|
|
667
|
+
recovery for an idempotent action within the worker recovery budget, the same exposure as a crash mid-step.
|
|
668
|
+
A reclaimed non-idempotent action settles a running checkpoint as unknown.
|
|
669
|
+
|
|
670
|
+
A refused handoff's provider reference is included in the step's `AdapterInvocationSucceeded` event
|
|
671
|
+
even when explicit adapter output omits it. Capable atomic transactions append its reconciliation
|
|
672
|
+
record with the step's evidence. The fallback independently attempts the reference-bearing event,
|
|
673
|
+
the governance reconciliation append, the outer `recordCompletionReconciliation` append, and the
|
|
674
|
+
adapter marker. Both reconciliation appends use the same stable id to deduplicate repeated attempts.
|
|
675
|
+
A failure of either append is reported after the remaining fallback evidence writes are attempted.
|
|
676
|
+
A non-sweep verdict may receive a log-unavailable error note; a preserved sweep verdict keeps its error.
|
|
677
|
+
After successful atomic finalization, the outer append retains its existing tolerated-failure behavior.
|
|
678
|
+
If its annotation write fails, Host raises `RecoverableFinalizationError` while keeping the committed
|
|
679
|
+
verdict and adapter marker intact; terminal replay retains the verdict.
|
|
333
680
|
|
|
334
681
|
`extractExecutionAttestation` receives the redacted output, so an extractor that needs a stripped
|
|
335
|
-
field can fail after the external effect
|
|
682
|
+
field can fail after the external effect. Host leaves a durable success checkpoint recoverable
|
|
683
|
+
within `recoveryMaxAttempts`; repair may call the extractor again.
|
|
336
684
|
|
|
337
685
|
`redactActionParameters` and `redactAdapterInput` behavior is unchanged. Independently of hook
|
|
338
|
-
configuration, a
|
|
339
|
-
|
|
686
|
+
configuration, a refused fenced invocation write inside an adapter step rethrows. With
|
|
687
|
+
`writeAdapterInvocation`, stale adapter updates also refuse before marking the new owner's row failed. Refused-handoff ordering and audit events also
|
|
340
688
|
change without the new hooks.
|
|
341
689
|
|
|
342
690
|
Applications own the safety of domain events/results, policy and HITL evidence, explicit
|
|
@@ -540,13 +888,30 @@ can filter it. It does not need an `emitsEvents` declaration from the action.
|
|
|
540
888
|
For `before_adapters`, the transition is appended with the handler's domain events. For
|
|
541
889
|
`after_adapters`, it is deferred to finalization after required adapter steps succeed or
|
|
542
890
|
accept a parked external handoff, immediately after domain events. Adapter failure withholds
|
|
543
|
-
both domain events and the transition, leaving
|
|
544
|
-
invocation even though handler writes may have committed.
|
|
891
|
+
both domain events and the transition, leaving the built-in stores' ledger-derived entity state
|
|
892
|
+
unchanged by that invocation even though handler writes may have committed. Custom stores that
|
|
893
|
+
read domain rows can already observe the target state at this point. With `transactionWithEvents`, the
|
|
545
894
|
transition shares the transaction used for domain events in the selected phase; otherwise it
|
|
546
895
|
follows the same legacy non-atomic boundary as domain events.
|
|
547
896
|
|
|
548
|
-
|
|
549
|
-
|
|
897
|
+
For idempotent `after_adapters` actions, Host saves `pendingStateTransition` on the invocation.
|
|
898
|
+
It contains `{ entityType, entityId, fromState, toState }` from the validated transition.
|
|
899
|
+
Custom stores must persist and return this optional invocation-record field through
|
|
900
|
+
`updateActionInvocation`, including the transaction-scoped method. With a mutation transaction,
|
|
901
|
+
Host writes the field after the handler succeeds and result/events validate, just before returning
|
|
902
|
+
from the transaction callback. This keeps the invocation row lock out of the handler run while the
|
|
903
|
+
saved transition commits or rolls back with the domain writes. The `transaction`-only fallback saves
|
|
904
|
+
it separately before the handler. Non-idempotent actions do not write this field. It is retained as
|
|
905
|
+
evidence after completion. PostgreSQL `ensureSchema()` adds `pending_state_transition jsonb`.
|
|
906
|
+
|
|
907
|
+
On an admitted same-invocation replay, Host uses the saved `fromState` when the saved entity type,
|
|
908
|
+
entity ID, and target match the current binding and state. The fact follows the usual event phase
|
|
909
|
+
and stable `state:transition` ID, so built-in stores deduplicate an already appended event.
|
|
910
|
+
Invocations started before upgrading, or stores that omit this evidence, retain replay acceptance
|
|
911
|
+
without a re-emitted transition. Mismatched evidence also suppresses replay emission.
|
|
912
|
+
|
|
913
|
+
Invalid transitions, failed handlers, replay acceptance without matching saved evidence, and
|
|
914
|
+
bindings without a non-empty entity ID or target do not produce a Host transition fact. If the handler
|
|
550
915
|
returns its own `StateTransitioned` with the bound entity's subject type and ID, Host suppresses
|
|
551
916
|
its generated fact in either event phase. Applications using that workaround should remove it
|
|
552
917
|
to use Host emission.
|
|
@@ -562,8 +927,10 @@ binding's target; otherwise Host's later fact sets the recorded state to the tar
|
|
|
562
927
|
|
|
563
928
|
Configure `redactExternalCompletionResult` to withhold sensitive completion fields. Host applies it
|
|
564
929
|
once to a supplied result (including a result attached to a failed callback). The redacted value
|
|
565
|
-
feeds `externalCompletion.result`; for a completed outcome it also feeds the adapter output
|
|
566
|
-
`ExternalOperationCompleted` payload
|
|
930
|
+
feeds `externalCompletion.result`; for a completed outcome it also feeds the adapter output and
|
|
931
|
+
`ExternalOperationCompleted` payload. Only a settling successful callback merges it into the
|
|
932
|
+
invocation result and validates that merge against the action's result schema. A non-settling
|
|
933
|
+
callback returns the stored invocation result and keeps the callback result in its evidence. A supplied
|
|
567
934
|
`completion.error` passes through `redactErrorText` before recording, including when attached to a
|
|
568
935
|
completed outcome. A missing error uses Host's fixed failure message, which bypasses the hook.
|
|
569
936
|
|
|
@@ -610,13 +977,55 @@ refused. A repeated identical completion is absorbed, identity being a digest of
|
|
|
610
977
|
reference, outcome, result, error, and evidence. A contradictory one, a flipped outcome or the same
|
|
611
978
|
outcome with different evidence, moves the invocation to
|
|
612
979
|
`reconciliation_required`, keeps the first outcome on record, logs the second, and emits
|
|
613
|
-
`ExternalOperationContradicted`.
|
|
614
|
-
|
|
615
|
-
|
|
980
|
+
`ExternalOperationContradicted`. With `transactionWithEvents` and transaction-scoped
|
|
981
|
+
`getActionInvocationForUpdate`, the callback read and verdict write share a lock. The
|
|
982
|
+
successful settling callback's redacted result is merged with the invocation result, stripped of
|
|
983
|
+
private fields, and validated against the action's result schema before writes. A non-settling
|
|
984
|
+
callback records its redacted result/outcome as evidence without schema validation or merging it
|
|
985
|
+
into the invocation result.
|
|
616
986
|
An invocation that ended for another reason while a handoff was pending keeps its status and records
|
|
617
987
|
the callback as evidence. Declare `completionDeadlineMs` on `long-running` actions;
|
|
618
|
-
`reconcileOverdueCompletions()` moves
|
|
619
|
-
|
|
988
|
+
`reconcileOverdueCompletions()` moves running, unleased, due handoffs to reconciliation and retains
|
|
989
|
+
the handoff with `overdueAt` set to the sweep's current time. The worker cycle runs the sweep before
|
|
990
|
+
claiming work. A callback can settle a parked running invocation, even past its deadline before the sweep. From
|
|
991
|
+
reconciliation, settlement requires the parked, swept handoff or the substring fallback described above;
|
|
992
|
+
a swept unparked handoff retains its verdict and records the callback in `externalCompletion`,
|
|
993
|
+
adapter evidence, and a completion/failure event. It clears the pending handoff without claiming
|
|
994
|
+
that the invocation's remaining steps ran. The markerless parked-row fallback is described above.
|
|
995
|
+
|
|
996
|
+
For each sweep candidate, Host re-reads status, lease ownership, handoff identity
|
|
997
|
+
(`adapterInvocationId` and `externalReference`), due time, and absence of `overdueAt`. With
|
|
998
|
+
`transactionWithEvents` and `getActionInvocationForUpdate`, this check and the verdict/marker write
|
|
999
|
+
share the row lock. Reconciliation evidence joins that transaction when it exposes
|
|
1000
|
+
`appendExternalReconciliation`; otherwise evidence is written on the outer store and can survive a
|
|
1001
|
+
rolled-back verdict. A custom transaction exposing `getActionInvocationForUpdate` without
|
|
1002
|
+
`appendExternalReconciliation` makes the sweep append on the outer store while holding the
|
|
1003
|
+
invocation row lock. If the reconciliation table references that row through a foreign key, the
|
|
1004
|
+
outer append can block waiting for the held lock. Implement both members in the same transaction.
|
|
1005
|
+
Stores without a transaction lock use an immediate re-read, but retain a race
|
|
1006
|
+
between that read and the write: a concurrent callback or execution can still be overwritten.
|
|
1007
|
+
Custom stores must preserve and reload the optional `overdueAt` Date in the handoff JSON; PostgreSQL
|
|
1008
|
+
hydrates it without a new migration.
|
|
1009
|
+
|
|
1010
|
+
Inline writes bypass the extra guard read and transaction when neither the loaded invocation nor
|
|
1011
|
+
its patch carries `pendingCompletion`. Ordinary inline completion retains its base store cost.
|
|
1012
|
+
Writing a handoff into a running invocation from anything other than its executing step is not
|
|
1013
|
+
supported; an execution without that handoff in its snapshot does not guard against such imports.
|
|
1014
|
+
Inline execution preserves a stored `reconciliation_required` verdict while committing independent
|
|
1015
|
+
adapter checkpoints, events, and reconciliation records, including references from refused handoffs.
|
|
1016
|
+
If execution reaches parking after the sweep, it records the handler result and `parkedAt` on the
|
|
1017
|
+
same handoff, retaining the sweep's status, error, and `overdueAt`. Successful `after_adapters`
|
|
1018
|
+
finalization also commits its deferred domain events: the handler and remaining adapters have
|
|
1019
|
+
succeeded. Inline callers receive the latest
|
|
1020
|
+
record observed by the preserving writer, merged with its evidence update, after these writes
|
|
1021
|
+
succeed; there is no post-attempt read. Leased execution retains `RecoverableFinalizationError`
|
|
1022
|
+
on fencing failure. The transaction lock serializes the check with the sweep where available; stores
|
|
1023
|
+
without it retain a read/write race. This does not cancel an adapter in flight or undo earlier
|
|
1024
|
+
domain writes. Verdict-first fallbacks attempt independent evidence writes before raising the first
|
|
1025
|
+
failure as `RecoverableFinalizationError`. Reconciliation-log failures leave the sweep error unchanged. Concurrent inline executions of the
|
|
1026
|
+
same invocation can still raise `RecoverableFinalizationError` with `Invocation is no longer running`
|
|
1027
|
+
when another execution finalized the row first. This error is internal; callers classifying it can
|
|
1028
|
+
match its `name` and inspect the stored invocation before deciding how to recover.
|
|
620
1029
|
|
|
621
1030
|
The handoff is persisted the moment it is accepted, before the adapter step is marked done, so a
|
|
622
1031
|
worker that dies between the two leaves the handoff and not the marker. A custom store has four
|
|
@@ -655,7 +1064,8 @@ Host passes `AdapterInvocationScope` from the store-loaded action invocation to
|
|
|
655
1064
|
must match `tenantId`, `spaceId`, and `actionInvocationId` when scope is supplied. A mismatch returns
|
|
656
1065
|
`undefined` on read and leaves the row unchanged on update (a not-found error is permitted).
|
|
657
1066
|
The trailing argument is optional to preserve existing custom-store and direct-call compatibility;
|
|
658
|
-
calls without it retain ID-only behavior in the shipped stores.
|
|
1067
|
+
calls without it retain ID-only behavior in the shipped stores. When `writeAdapterInvocation` is present, the write probe also checks its scoped create/update
|
|
1068
|
+
refusals and an admitted update in our scope. The two adapter isolation checks
|
|
659
1069
|
fail for stores that ignore scope. Deployments running `runTenantIsolationChecks` against a custom
|
|
660
1070
|
store will see new failures until that store honours the argument.
|
|
661
1071
|
|