@fabricorg/webhook-delivery 0.1.3 → 0.2.1

package/CHANGELOG.md

@@ -1,5 +1,40 @@
 # @fabricorg/webhook-delivery
 
+## 0.2.1
+
+### Patch Changes
+
+- 71d306f: Republish workspace dependents against the action finalization contracts. Finalizing actions require the new Host protocol and atomic transaction capabilities; compatible durable preparation supports checkpoint replay under the Host recovery safeguards.
+
+  The compiler now flags adding or removing the finalization protocol as a breaking action-contract difference, so compatibility review includes the new lifecycle requirement.
+
+- f32aae6: Republish Host dependents for durable finalization checkpoint recovery. Checkpoint writers require migration 10; unknown adapter outcomes and incompatible preparation retain effect-aware refusal.
+- 3d24330: Permit bounded leased retries of non-idempotent actions after retryable pre-effect governance errors. The additive deferredBeforeEffectAttempt record field proves only the immediately preceding claim deferred before handler or adapter effects. Missing or stale proof retains interruption failure. Marker-write failure retains terminal failure where persistence succeeds; lease loss preserves fencing errors. Approval resume, inline execution, exhausted budgets, and errors after the governance window retain their existing refusal semantics. Workspace dependents receive compatible patch releases.
+
+  PostgreSQL migration 9 (`pre_effect_governance_deferral`) adds the nullable marker column. Run `ensureSchema()` before deploying Hosts that may defer non-idempotent actions. Invocation updates that do not carry the marker omit its column, preserving writes on schemas without migration 9.
+
+## 0.2.0
+
+### Minor Changes
+
+- ac87d29: Add atomic, scoped `redactEventRecords` to remove persisted payload and admission facts, fence outstanding workers, and scrub late inserts using durable erasure markers. PostgreSQL requires schema migration 4 via `ensureSchema()` before the new version serves any traffic: all job and intent inserts read the new marker table. The migration also indexes scoped event lookups; that index is built without `CONCURRENTLY`, so on an existing deployment all intent writes wait until the build finishes, for a time that grows with the intents table.
+
+  Compatibility: applications may supply custom `WebhookDeliveryStore` implementations through the public port; those implementations break and must add this method. Wire format and existing method signatures are unchanged. Jobs expose an additive `erased` flag; undecided subscriptions receive the additive `event_erased` denial code without calling admission. Admission ports may also return that code. Markers are retained indefinitely because accepted future-dated events invalidate timestamp-only marker expiry. Do not call `redactEventRecords` until every instance runs this version: older instances ignore markers and the `erased` flag, so they can write the payload back, deliver it, or ask admission with cleared facts. Repeating redaction scrubs rows they wrote but cannot recall deliveries; no-op repeats preserve fences. Requests already handed to transport cannot be recalled, and an attempt begun before erasure can still send its old body afterward. Its receipt can record the outcome, but settlement at the advanced fence is refused; the pending intent can be re-sent metadata-only under at-least-once delivery. Audit attempts and evidence are preserved.
+
+- 828a579: Add opt-in governed historical replay with `replay_events`, metadata-only `WebhookReplayRequested`, immutable request receipts, retained per-event outcomes, current-subscription reads and `runWebhookReplayCycle()`. Replay snapshots retained webhook projections, checks current eligibility and admission, and serializes pair settlement with erasure and the retention horizon. Ordinary fan-out completion stays final for its historical selection.
+
+  Deploy PostgreSQL migration 5 through `ensureSchema()` before enabling replay. All relevant writers, erasure handlers and purgers must support this contract before enablement; the migration lock does not certify mixed-version behavior. The new tables and indexes are created without `CONCURRENTLY`, so source writes can wait during index creation.
+
+  Compatibility: replay requires explicit module configuration and separate capabilities; existing custom base stores and insert signatures are unchanged. New action-name, event-type and refusal-code union members may affect exhaustive consumer switches. Replay receipts and work items contain metadata only and are never removed by this package; terminal outcomes live on the items. Items carry bounded acceptance-time interval/filter context so processing does not read or scan request membership. Replay honors the shared fan-out worker defaults and callback semantics, including `dlv_` UUIDs, batch size 25 and configured retry jitter/custom delays. Because both workers now share that option validation, `runWebhookFanOutCycle` given two or more invalid options may report a different one first; a single invalid option reports the same error as before.
+
+  Limits: replay covers retained webhook projections, not the canonical ledger or discarded fields. Completed legacy jobs with an absent actor cannot replay to subscriptions requiring identity. Over-cap requests are refused without partial acceptance; same-timestamp density can exceed the cap even for a narrow range. Replay never creates a second intent for an existing pair and never overrides a denial. Dispatch remains at-least-once. Subscription changes after the worker's last read can race insertion; subscription and delivery stores need not share a transaction.
+
+### Patch Changes
+
+- d69dd7a: README: recommend Node 22.23.4, 24.22.1, or 26.12.1 or newer in the same release line for the
+  Brotli decoder fix in CVE-2026-111497, and note that Node 20 is end-of-life with no fixed release.
+  The `engines` range is unchanged (`>=20`).
+
 ## 0.1.3
 
 ### Patch Changes

package/README.md

@@ -14,7 +14,8 @@ pnpm add @fabricorg/webhook-delivery@^0.1.0 @fabricorg/platform@^1.7.0 @fabricor
   submission. Every operator action declares the check, so against an older Host the actions
   refuse at execution as `admission_not_enforced` ([Secrets and parameters](#secrets-and-parameters)).
   `@fabricorg/platform` is imported for types only.
-- **Runtime.** Node.js 20 or newer. The package ships ESM and CommonJS builds with two entry
+- **Runtime.** Node.js 20 or newer; run 22.23.4, 24.22.1, or 26.12.1 or newer in the same release line in production for
+  the Brotli decoder fix in [CVE-2026-111497](https://nodejs.org/en/blog/vulnerability/october-2026-security-releases); Node 20 is end-of-life and has no fixed release. The package ships ESM and CommonJS builds with two entry
   points: `@fabricorg/webhook-delivery` and `@fabricorg/webhook-delivery/node` (the Node transport
   and DNS lookup). It has no runtime dependencies; the PostgreSQL stores take an injected pool
   such as `pg.Pool`.
@@ -175,14 +176,85 @@ need a horizon), and per-record retention clocks (two clocks for one decision).
 | Fan-out job (holds the actor source for admission) | terminal and older than the horizon | purge |
 | Intent and its tombstone | terminal and older than the horizon | purge |
 | Denial and its tombstone | older than the horizon | purge |
+| Event erasure markers | retained indefinitely (future-dated inserts are accepted) | none |
 | Eligibility revisions | never removed | none |
 | Delivery configuration versions | never removed | none |
+| Replay request receipts and work items (including terminal outcomes) | never removed by the package (audit/idempotency) | application retention policy |
 | Attempt log, requeue evidence | never removed by the package (audit) | application retention policy |
 
 The rejection is visible at the store, but the publisher (PR 6) decides what Host sees. A
 `past_horizon` from `insertFanOutJob` should fail the publish with a fixed code, so the Host
 outbox record dead-letters rather than counting as delivered.
 
+### Event erasure
+
+`redactEventRecords({ tenantId, spaceId, eventIds }, options)` atomically clears
+job payloads (`null`), projection flags (`false`) and admission facts (`{}`), and
+replaces intent bodies with metadata-only JSON using the shared serializer. It
+covers pending and terminal records. It keeps metadata and the identifier-only
+actor. Input accepts 1–100 distinct identifiers, using the identifier validation
+below; empty lists and duplicates are refused. Results count newly marked events
+and rows whose content changed. A repeat scrubs payload reintroduced by older writers;
+a no-op repeat returns zero counts and preserves fences and leases. Jobs carry the
+additive `erased` flag. After the tombstone check, fan-out denies undecided pairs with
+`event_erased` without calling admission. Decided pairs keep their metadata-only
+intents (`tests/fan-out.test.ts`, “erased events”). `event_erased` is an additive
+admission denial code; application admission ports may return it too.
+
+For changed rows, the job fence and pending intent fences advance, and their leases are cleared for
+immediate reclaim, matching actor enrichment. Old fan-out writes return `lease_lost`;
+old delivery `beginAttempt` returns `{ begun: false }`. A request already handed to
+transport before commit cannot be recalled. The fence check linearizes at
+`beginAttempt`: if begin commits first, its receipt remains valid. A PostgreSQL
+response may arrive after redaction commits, so this store-only port cannot promise
+that such an already-authorized handoff occurs before the erasure commit. Stronger
+transport exclusion requires coordination beyond this port. Status, attempt budgets, availability,
+attempt logs, outcomes, denials and requeue evidence are preserved. Requeue and actor
+enrichment keep the redacted fields. These operations do not erase the application's
+event store or deliveries already received by subscribers. An in-flight attempt can
+still append its outcome using its receipt, but settlement at the old fence is
+refused. Its pending intent can then be sent again with the metadata-only body,
+consistent with at-least-once delivery.
+
+A scoped marker is recorded even when no job exists. Inserts check it inside their
+atomic step and scrub the new draft/body. PostgreSQL migration 4 adds
+`webhook_event_erasures` and an intent index on `(tenant_id, space_id, event_id)`.
+Run `ensureSchema()` before the new version serves any traffic: job and intent
+inserts now read the marker table even when redaction is unused. The migration builds
+the intent index without `CONCURRENTLY`, so on an existing deployment every intent
+write (claims, attempts, settlements, inserts, cancellations) waits until the build
+finishes, for a time that grows with the intents table. Plan the upgrade for that
+pause. Redaction takes
+the horizon EXCLUSIVE lock first, then reads the clock and locks matching jobs in
+`event_id` order and intents in `delivery_id` order using set-based queries.
+Cancellation also locks intents in `delivery_id` order. Claims use ordered
+`FOR UPDATE SKIP LOCKED`, which does not wait on held rows. Ordinary horizon-checked
+writers continue to use SHARE. The live “check-to-write serialization” erasure cases
+exercise both lock orders against inserts.
+
+Do not call `redactEventRecords` until every instance runs this version. Older
+instances ignore markers and the `erased` flag: they can write the payload back after
+an erasure, deliver it before any repair, and ask admission with the cleared facts,
+which is the fail-open path `event_erased` closes. Repeating redaction scrubs rows
+an older instance wrote (`tests/store-contract.ts`, “repeat erasure repairs legacy
+rows”, which runs against PostgreSQL), but it cannot recall what was already delivered.
+Custom `WebhookDeliveryStore` implementations break at the type boundary and must
+implement the new method and its atomicity, marker and fail-closed semantics.
+
+Markers are retained indefinitely. The proposed timestamp expiry proof would require
+`event.recordedAt <= redaction time`, but the store accepts future-dated jobs (shared
+contract: “job claim respects availableAt, lease expiry and limit, oldest first”).
+A pre-recorded marker could otherwise expire before that event's horizon. Purge does
+not remove markers or count them in `removed`. Applications should budget this storage
+until a future contract supplies a safe upper bound on event time.
+
+The shared `tests/store-contract.ts` erasure cases cover scoped late inserts, terminal
+and pending data, stale fences, idempotence, and validation. Attempts, outcomes, denials
+and requeue evidence contain identifiers, timestamps and closed diagnostic fields,
+not payload or admission facts (`delivery-store-core.ts` input readers and evidence
+builders). Actor evidence retains identifiers; this API is payload erasure, not removal
+of identifying metadata.
+
 ### Identifiers and stored fields
 
 Every identifier a store takes is refused as `input_invalid` if it is empty, longer than 256
@@ -319,9 +391,9 @@ writer a clock that has not crossed the horizon and the purge one that has, so o
 floor can refuse the writer, which is the case the lock exists for. The tests then check the
 outcome:
 
-- A replay paused after reading the cutoff races a purge. The replay finds the job
+- Host outbox re-publication paused after reading the cutoff races a purge. It finds the job
   (`already_present`), then the purge removes it.
-- A purge paused before its commit races a replay. The replay waits, then sees the raised floor
+- A purge paused before its commit races Host outbox re-publication. The insert waits, then sees the raised floor
   (`past_horizon`).
 - A late intent insert races a purge. It cannot re-create the purged intent or its tombstone.
 - A requeue races a purge that has raised the floor past the record. The requeue is refused
@@ -786,8 +858,8 @@ to `dlv_` plus a random UUID.
 
 **Decision.** A job decides an event's eligibility once, after the event's eligibility has
 settled, and the job is complete when `completeFanOutJob` succeeds. Eligibility is settled when
-the worker's clock is at or past `recordedAt + eligibilitySettleMs`. From completion on, no
-eligibility change adds or removes an intent or denial for that event.
+the worker's clock is at or past `recordedAt + eligibilitySettleMs`. Ordinary fan-out completion stays final for its historical eligibility selection. A governed
+historical replay request has its own fixed membership and can decide previously undecided pairs.
 
 **What it rests on.** Two assumptions about clocks that the package cannot enforce:
 
@@ -811,8 +883,9 @@ backs each:
 | A revision written after an event settled does not contain its `recordedAt` | both clock assumptions above, plus a new revision starting at the store's clock and never before the previous one (`mathMax(nowMs, current.effectiveFromMs)` in the memory store's `#transition`; the `WebhookSubscriptionWriter` contract for other stores) | "a pattern change after recordedAt does not remove the event …", "… does not add the event …" (both run with one clock, so they do not test the assumptions) |
 | A late revision does not reopen a completed job | claims take only `pending` jobs; a job requeue accepts only dead letters | "a revision stamped behind an already-settled event …" |
 | The revision is chosen by `recordedAt`, not by when the job runs | `recordedAt: job.event.recordedAt` passed to `listEligibilityRevisions` | the pattern change, pause and resume cases |
-| A completed job is not run again | claims take only `pending` jobs (`claimFrom`, `src/memory-delivery-store.ts`); a job requeue accepts only dead letters | "after completion no eligibility change adds or removes an intent …" |
-| A replay or bridge adds nothing | `insertFanOutJob` is insert-if-absent by `eventId`; intents and denials are insert-if-absent by (subscriptionId, eventId) | "replay: …", "concurrent relays: …", the bridge overlap cases |
+| Ordinary fan-out completion is final for its historical selection | claims take only `pending` jobs (`claimFrom`, `src/memory-delivery-store.ts`); a job requeue accepts only dead letters | ordinary fan-out test "after completion no eligibility change adds or removes an intent …" |
+| Governed replay can decide previously undecided pairs without reopening the source job | `createReplayRequest` fixes separate membership; `settleReplayJob` preserves existing pair tombstones | replay contract "historical event for a later subscription …", "existing pair denial is retained …" |
+| Re-publishing through the Host outbox or bridge adds no second source job | `insertFanOutJob` is insert-if-absent by `eventId`; intents and denials are insert-if-absent by (subscriptionId, eventId) | "replay: …", "concurrent relays: …", the bridge overlap cases |
 | A worker that lost its lease adds nothing | the parent fence check in `insertDeliveryIntent` / `recordAdmissionDenial`, and `return` on `lease_lost` in `processJob` | "concurrent workers: …" |
 | Purged records stay purged | the replay horizon floor (see [Replay horizon](#replay-horizon-r5)) | the store contract's race cases |
 
@@ -828,7 +901,9 @@ first, so a pattern change neither removes the event from a subscription that ma
 to one that did not. A pause written after `recordedAt` stops delivery, not fan-out: the governed
 action's `cancelPendingIntents` and the delivery cycle's pre-handoff `readSubscription` check
 ([Delivery](#delivery)) act on the intent. A pause effective before `recordedAt` makes the revision paused, so the
-event gets no intent or denial, and a later resume does not resurrect it.
+event gets no intent or denial in ordinary fan-out; a later resume does not change that historical
+selection. A new governed replay request after resume may decide still-undecided pairs. Remaining
+items of a request accepted before a pause are skipped under its original active-interval rule.
 
 **Bridge overlap.** Covered above: the first insert of an `eventId` wins and every later one is a
 no-op, before and after the job completes.
@@ -837,8 +912,8 @@ no-op, before and after the job completes.
 dead-lettered with a diagnostic code: a visible failure that the fan-out-job requeue (PR 9)
 recovers. That is R1's "visible recoverable failure".
 
-**Rejected.** Completing without waiting and sweeping later for late-effective revisions: a sweep
-would add intents after completion, so completion would mean nothing, and could not withdraw an
+**Rejected.** Completing ordinary fan-out without waiting and automatically sweeping later for
+late-effective revisions: that would undermine its historical selection boundary and could not withdraw an
 intent a late revision excludes. Pinning eligibility at Host's commit: that is in-transaction
 fan-out, which ADR 0018 rejects. Settling on Host's clock: it is not the clock that stamps
 revisions.
@@ -965,7 +1040,9 @@ letter, not an endless loop.
 
 ### Pause and resume are enforced at dispatch
 
-ADR 0018 decision 3: pause cancels pending intents, and resume does not resurrect them. The
+ADR 0018 decision 3: pause cancels pending intents, and resume does not resurrect those existing
+intents. Governed replay also preserves existing pair decisions; a new request after resume may
+decide previously undecided pairs, while pre-pause requests retain their original active interval. The
 property the package holds is stronger than "the pause's cleanup ran": once a pause or delete
 revision is committed, no intent bound to an eligibility revision before it is handed to the
 transport after a dispatch check has seen that revision. That holds whether the action's
@@ -1315,6 +1392,7 @@ The namespace defaults to `webhook-delivery` (option `namespace`). Every action
 | `rotate_secret_emergency` | `subscriptionId`, `secretRef`, `expectedVersion?` | `replaceDeliveryConfiguration` | `WebhookSecretRotated` (`mode: "emergency"`, …, `previousSecretRetainUntil: null`) |
 | `requeue_delivery_intent` | `deliveryId`, `expectedFence` | `requeueDeliveryIntent` | `WebhookDeliveryIntentRequeued` (`deliveryId`, `subscriptionId`, `eventId`, `priorFence`, `priorAttemptCount`, `processingDeadline`) |
 | `requeue_fan_out_job` | `eventId`, `expectedFence` | `requeueFanOutJob` | `WebhookFanOutJobRequeued` (`eventId`, `priorFence`, `priorAttemptCount`, `processingDeadline`) |
+| `replay_events` | `subscriptionId`, `from`, `to`, optional `eventTypePatterns` | opt-in `createReplayRequest` | `WebhookReplayRequested` (range, count, acceptance and deadline metadata) |
 | `enrich_fan_out_job_actor` | `eventId`, `expectedFence` | `enrichFanOutJobActor` | `WebhookFanOutJobActorEnriched` (`eventId`, `priorFence`, `priorAttemptCount`, `processingDeadline`) |
 
 Events carry ids, numbers, patterns and timestamps only: no destination URL, no secret reference,
@@ -1698,3 +1776,88 @@ package's tests with `node:crypto` only. They are not part of the packed package
 Ports are passed to an instance, not registered in module scope. The Boundary Guard's module-state
 scan (G5) rejects module-scope `let` bindings, `Map` and `Set` registries and import-time
 environment reads in this package (`boundary/tiers.json`, `moduleStateScan`).
+
+
+## Historical replay
+
+Enable replay explicitly at the composition root:
+
+```ts
+const module = createWebhookDeliveryModule({
+  ...operatorOptions,
+  replay: { store: deliveries, subscriptions, maxEvents: 10_000, processingLifetimeMs: 3_600_000 },
+});
+```
+
+The built-in memory and PostgreSQL stores implement `WebhookReplayStore`; both subscription
+stores implement `WebhookReplaySubscriptionReader.readReplaySubscription`. These are separate
+capabilities: existing custom base ports and ordinary fan-out method checks are unchanged. Missing
+enabled methods fail module construction. Without configuration, `replay_events` refuses
+`replay_not_supported` before asking the operator policy. The authorization input optionally includes
+normalized `replay` range/filter metadata.
+
+Submit `<namespace>.replay_events` through Platform Host with `subscriptionId`, canonical ISO
+`from` (inclusive), `to` (exclusive), and optional `eventTypePatterns`. The range must be nonempty,
+inside the store's authoritative horizon and end no later than its clock. Patterns use the existing
+exact, prefix and wildcard grammar. Both the requested filter and acceptance-time subscription
+patterns must match each candidate; package-owned events are excluded. Scope and requester come
+from trusted Host context. The operator policy still requires literal `true`.
+
+Default `maxEvents` is 10,000; default `processingLifetimeMs` is one hour. These composition settings
+must be positive safe integers. Requests over the cap are refused with `replay_limit_exceeded`,
+without a receipt or partial work. Many events can share one timestamp and type, so narrowing the
+range need not make it fit. No pagination across changing source membership is implied.
+
+Deploy PostgreSQL migration 5 with `ensureSchema()` before enabling replay. Upgrade relevant
+writers, erasure handlers and purgers before enablement. Migration locking alone does not establish
+mixed-version behavioral compatibility. The migration adds receipt/item tables and source-range,
+claim and listing indexes; index creation can block writes while it runs.
+
+Acceptance atomically fixes membership from **retained webhook projections**, not the canonical
+ledger. Events never published/bridged or already purged are absent. Payload and facts are not copied
+to replay records. Each item carries its acceptance-time `observedStatusSinceRevision`, normalized
+request `eventTypePatterns` (default `["*"]`) and `observedEventTypePatterns`. These pattern lists are
+bounded by the existing 64-pattern limit. The worker reads and validates only that item context,
+without reading, copying or scanning the receipt's `selected` membership per item. A receipt exists
+even for zero matches. Reusing the trusted `actionInvocationId`
+returns the original receipt before mutable checks; conflicting action parameters refuse
+`replay_request_conflict`. A later publication cannot expand that receipt. `WebhookReplayRequested`
+means durable acceptance, not delivery. The webhook transaction and Host event append are separate;
+a lost response or Host append failure can leave accepted work without that event. As with requeue,
+the webhook store cannot fence overlapping Host attempts using Host's lease.
+
+Run `runWebhookReplayCycle()` with the replay capability, current subscription reader, ordinary
+delivery store, admission, payload policy, clock and worker identity. Worker options shared with
+fan-out use the same resolver: default batch size 25, lease 60,000 ms, and `dlv_` plus UUID delivery
+IDs. A custom `retryDelay` is honored; otherwise pass `random` for the configured schedule's jitter.
+The delivery-ID callback receives a frozen argument without a receiver, and must return a string.
+`eligibilitySettleMs` is omitted: replay uses fixed membership and current eligibility rather than
+waiting for the historical fan-out selection boundary. It shares actor-requirement,
+matching, admission normalization and rendering behavior with ordinary fan-out. Existing pair
+markers bypass admission; replay never creates a second intent for an existing pair and never
+overrides a denial. Dispatch remains at-least-once using the normal delivery worker, current
+destination/configuration checks, authorization, signing and attempt receipts.
+
+The worker enforces the acceptance-time active interval. Pause/resume ends that interval; deletion,
+inactivity or narrowed patterns skip remaining items without permanently denying the pair. Pattern
+broadening does not expand membership. Subscription changes after the worker's last read can race
+insertion because subscription and delivery stores can be independent. Dispatch retains its existing
+pause/resume checks; it does not recheck event patterns.
+
+Admission uses the original event actor, not the requesting operator. Suppressed/legacy absence
+remains unavailable when identity is required. In particular, completed legacy source jobs cannot
+be enriched by the existing action. Replay reapplies current payload policy and scope predicates to
+retained projected data; it cannot restore discarded fields or promise chronological HTTP delivery.
+
+Settlement checks replay ownership, deadline, source scope/fence, horizon, erasure marker and pair
+uniqueness atomically. A changed source requires fresh evaluation. An erased undecided pair gets
+`event_erased`. Original event `recordedAt` anchors the horizon; admission near its edge is best
+effort. Purge can remove a terminal source while replay remains pending: expiry is reported from
+the stored anchor; a missing source inside the horizon is visibly dead-lettered as `source_missing`.
+Intent delivery receives its ordinary finite deadline. Reclaims do not extend replay's persisted
+deadline. Unknown settlement writes end the current pass without a contradictory follow-up write.
+
+Inspect `readReplayRequest()` and scoped, event-ID-paged `listReplayItems()`. Each terminal item
+keeps its outcome once: intent plus delivery ID, denial plus code, already decided, skip plus reason,
+horizon expiry, or dead letter plus diagnostic. Evaluated revision is retained for pair decisions.
+Receipts and items are never removed by package purge; operators own their eventual audit retention.