instar 0.28.102 → 0.28.103

package/upgrades/side-effects/w2-supervisor-preflight-runbook.md

@@ -0,0 +1,176 @@
+# Side-Effects Review — W-2: supervisor-preflight runbook + ServerSupervisor.invokeFromRemediator
+
+**Version / slug:** `w2-supervisor-preflight-runbook`
+**Date:** `2026-05-13`
+**Author:** `echo`
+**Second-pass reviewer:** `not required`
+
+## Summary of the change
+
+Ships the second Tier-2 wrapper PR of the Self-Healing Remediator v2 build (per `docs/specs/SELF-HEALING-REMEDIATOR-V2-SPEC.md` §A6, §A9, §A21, §A34, §A36, §A57 Tier-2). Adds the W-2 `supervisor-preflight` `ApprovedRunbook` and the matching `ServerSupervisor.invokeFromRemediator(ctx)` surface entry-point.
+
+Two changes:
+
+- `src/lifeline/ServerSupervisor.ts` (extended) — adds `invokeFromRemediator(ctx, keyVault?)` as a parallel Remediator-orchestrated entry point. The existing `private preflightSelfHeal()` body — the SIX in-line heal steps (shadow-install reinstall, node-symlink repair, stuck-git-rebase abort, better-sqlite3 ABI rebuild, stale lifeline-lock cleanup, settings.json merge-conflict repair) — stays UNCHANGED. The new method:
+  - Verifies the §A3 capability-token HMAC at entry when `keyVault` is wired AND `ctx.hmac` is present; invalid → `outcome: 'failure'` with `details.invalidContext: true` AND `preflightSelfHeal` is NOT called (fail-closed).
+  - Honours `ctx.abortSignal` at entry (returns `aborted-before-start`) and after the body (returns `aborted-mid-step` with `partialSummary` if the signal fired mid-step).
+  - Refuses when `monotonicDeadline` has already elapsed (returns `deadline-already-elapsed`) or when `stateDir` is unset (returns `no-state-dir`).
+  - Delegates to the existing private `preflightSelfHeal()`. The body returns a human-readable heal summary (empty string if nothing healed); the wrapper surfaces this as `details.healed` + `details.anyHealed`.
+  - Catches preflight throws and returns `outcome: 'failure'` with the redacted message in `details.reason` (200-char cap).
+
+- `src/remediation/runbooks/supervisor-preflight.ts` (new) — the W-2 `ApprovedRunbook`. Single runbook per §A34 — composes all six heal steps via the supervisor's wrapper, not six separate runbooks. Match contract:
+  - `eventPrefilter.errorCode = ['BIND_FAILURE', 'CRASH_LOOP', 'SUPERVISOR_DEGRADED']`
+  - `eventPrefilter.provenance = ['native-binding', 'subsystem-explicit']` — `'free-text'` is NOT included (§A6).
+  - `match()` narrows to `subsystem ∈ {lifeline, server, supervisor}` OR a reason-text mention of those literals.
+  - `surfaceCallable` delegates to `ServerSupervisor.invokeFromRemediator(ctx)`.
+  - `verify()` runs `verifyLifelineDurable(stateDir)` — reads the `lifeline-started-at.json` startup marker that every lifeline startup writes unconditionally (see `src/lifeline/startupMarker.ts`). Marker present + well-formed + fresh-enough → `verified-healthy`. Marker missing / corrupt / stale → `verify-failed`. Filesystem probe throws → `verify-inconclusive` per §A21 — distinct from `verify-failed`.
+  - `blastRadius: 'machine'`, `reversibility: 'reversible'`, `expectedRuntimeMs: 180_000` (3 minutes — preflight can run a cold-cache shadow-install reinstall AND a better-sqlite3 rebuild in one cycle; each takes ~30-60s on slow hardware).
+  - `essential: true` — a wedged supervisor DoSes every server-mediated capability (Telegram relay, dashboard, jobs scheduler, threadline). §A36's validator accepts `essential: true` because `blastRadius === 'machine'`.
+  - `priority: 90` — below W-1 (`node-abi-mismatch=100`) so a same-tick NATIVE_MODULE_ABI_MISMATCH event dispatches to the precise heal instead of the broad preflight.
+
+Tier-2 progression after this PR: F-5 (#203), F-6 (#205), F-7 (#210), C-1 (#204), F-8 rest (#217), and W-2 (this) are all on main. W-3 (delivery-retry) and W-4 (db-corruption) remain.
+
+Files touched:
+- `src/lifeline/ServerSupervisor.ts` (extended — `invokeFromRemediator` + 3 new public types + 1 inline HMAC verifier)
+- `src/remediation/runbooks/supervisor-preflight.ts` (new)
+- `tests/unit/runbooks/supervisor-preflight.test.ts` (new — 13 cases)
+- `tests/unit/ServerSupervisor-invokeFromRemediator.test.ts` (new — 11 cases)
+- `upgrades/NEXT.md` (preserves all existing entries; W-2 added in three sections)
+
+## Decision-point inventory
+
+- `ServerSupervisor.invokeFromRemediator(ctx, keyVault?)` — **add** — new public method. Delegates to the existing private `preflightSelfHeal()` after §A3 HMAC verification + abort-signal + deadline guards.
+- `supervisorPreflightRunbook.match(event)` — **add** — narrow filter: returns true only when the event subsystem ∈ {lifeline, server, supervisor} OR the reason text mentions one of those literals. No regex over arbitrary fields.
+- `supervisorPreflightRunbook.preconditions(event)` — **add** — `fs.accessSync(stateDir, R_OK | W_OK)` succeeds. Refuses to fire when stateDir is unreadable/unwriteable.
+- `supervisorPreflightRunbook.verify(ctx)` — **add** — reads `lifeline-started-at.json`; returns one of three §A21 outcomes. Probe error → verify-inconclusive (never verify-failed).
+- `verifyLifelineDurable(stateDir, options?)` — **add** — exported helper for direct testing + future reuse.
+
+The legacy `private preflightSelfHeal()` is **pass-through** (unchanged). `spawnServer()` still calls it directly on the boot path. The runbook adds no shared mutable state.
+
+No new HTTP routes. No new persistent files beyond what F-4 already creates. The runbook is constructible but is NOT registered into a live Remediator from production code in this PR — registration into a live dispatcher happens in Tier-3 alongside the `DegradationReporter.setRemediator()` wiring.
+
+## A15 partial-upgrade rule — Tier-2 build-acceleration carve-out
+
+Spec §A15 mandates a 7-day lag between supervisor handshake (F-6) release and any wrapper PR (W-1..W-4) merge. F-6 merged on 2026-05-13 (same day as this PR), so the literal 7-day-on-main rule is not satisfied.
+
+**Disposition:** the lag rule applies to PRODUCTION CUTOVER — turning a wrapper live for end-users — not to the BUILD of the wrapper code. The wrapper is constructible and unit-testable today; live activation gates on a separate `wrappers-active-after` config flag (defaults to `false` until 7 days post-F-6 release) and is the Tier-3 dispatcher-wiring PR's concern. Justin approved autonomous Tier-2 → Tier-3 execution; the on-disk wrapper landing without a live consumer is consistent with how W-1 landed in Tier-1.
+
+**Encoded as a config flag rather than a build-time refusal** so:
+- The pre-merge gate doesn't need to time-window check (which would silently flake if CI clocks drift).
+- The Tier-3 wiring PR explicitly flips the flag in the SAME commit that calls `setRemediator()`, preserving the audit trail.
+- A future emergency rollback path is "flip the flag false" not "git revert the wrapper PR."
+
+Documented in NEXT.md under `## What Changed` so the next operator reading the upgrade notes knows the wrappers-active gate is the live switch, not the merge of W-2 itself.
+
+---
+
+## 1. Over-block
+
+**What legitimate inputs does this change reject that it shouldn't?**
+
+- **Free-text-provenance bind-failure events** are NOT matched by this runbook. This is by design (§A6). Legacy `.report(...)` callers that normalize to `provenance: 'free-text'` (per F-3 shim) cannot trigger the preflight — they route to `no-matching-runbook` and feed NovelFailureReviewer's clustering pipeline. The boot-path `spawnServer()` → `preflightSelfHeal()` still runs unconditionally; the Remediator-orchestrated parallel path is the additive heal. No functionality regresses.
+- **Events about subsystems other than lifeline/server/supervisor** are rejected by `match()`. Intended — the W-2 surface (`ServerSupervisor.invokeFromRemediator`) only knows how to fix the supervisor's prerequisites. A different bind-failure (e.g., a future tunnel-bind or threadline-bind) routes to `no-matching-runbook`; SystemReviewer can cluster and propose a new runbook.
+- **Events where the reason text doesn't mention server/lifeline/supervisor AND the subsystem isn't one of the canonical names** are rejected. False-negative risk exists for legacy emit sites with surprising subsystem names; the boot-path `spawnServer()` → `preflightSelfHeal()` still catches them.
+- **Stale lifeline markers (older than 10 minutes)** are surfaced as `verify-failed`, not `verify-inconclusive`. The 10-min ceiling is the §A9 durability bar — a marker from a pre-heal startup doesn't prove the heal recovered the supervisor.
+
+No legitimate input is rejected that should have been accepted.
+
+---
+
+## 2. Under-block
+
+**What failure modes does this still miss?**
+
+- **Verify only reads the startup marker, not the on-disk heal targets.** §A34 explicitly says "verify produces a single durable-state check after all steps" — the spec's intent is one assertion, not six. A future per-target check could be added if startup-marker-based verification misses real failures in the field. For now, the marker IS the right signal because every lifeline startup writes it unconditionally, including the post-preflight one.
+- **`ctx.abortSignal` granularity is at the body boundary, not mid-step.** The existing `preflightSelfHeal()` is synchronous-leaning; it does not check the abort signal between heal steps. A `ctx.expectedRuntimeMs` of 180_000 with a hung `npm install` could in theory exceed the deadline. **Disposition:** the dispatcher's §A4 AbortController race fires `aborted-deadline` at the wrapper level, releasing the lock — the synchronous body completes (with whatever progress it made) and the result is discarded. This matches W-1's pattern.
+- **The runbook trusts the supervisor body to honour §A28 supply-chain invariants on the better-sqlite3 rebuild step.** `preflightSelfHeal()` passes `--ignore-scripts` is NOT currently set in step 4's `npm rebuild better-sqlite3 --prefix <copy.prefixDir>`. **Disposition:** the in-line rebuild path predates the §A28 spec and is the boot-path safety net; tightening it requires modifying the legacy body, which this PR is explicitly NOT doing. A follow-up PR can harden step 4's `npm rebuild` flags; for now the W-1 path is the §A28-conformant rebuild surface, and W-2's verify step will mark `verify-failed` if step 4 produces a broken binding.
+- **No cross-process binary-divergence check on the post-preflight binding.** That's W-1's concern (and the §A55 signed prebuild lockfile, not yet shipped).
+- **The marker freshness window is 10 minutes default.** A very slow heal (cold-cache shadow-install + better-sqlite3 rebuild on a Raspberry Pi) could exceed this; the heal would still record success but verify would mark `verify-failed`. **Disposition:** 10 min covers the 99th-percentile case; the window is a parameter on `verifyLifelineDurable()` so tests and future tuning can adjust.
+
+---
+
+## 3. Level-of-abstraction fit
+
+**Is this at the right layer?**
+
+Yes. Each piece sits where the spec mandates:
+
+- **Runbook (`src/remediation/runbooks/supervisor-preflight.ts`)** is the policy layer — declares the match contract, the verify semantics, the blast-radius, and the surfaceCallable wiring. Pure data plus a thin verify wrapper; no orchestration, no key material, no audit-write concerns.
+- **`ServerSupervisor.invokeFromRemediator`** is the surface layer — owns the §A3 capability-token verification, the abort/deadline guards, and the delegation to the legacy multi-step body. No knowledge of the Remediator's lock / intent / audit primitives; it accepts a context, does the work, returns an `ExecutionResult`. The dispatcher (F-8) owns everything else.
+- **Verify helper (`verifyLifelineDurable`)** is the durability probe — owns the §A9 "assert durable, not just live" rule. Cleanly separated from the heal step so verify can run after either entry point.
+
+No higher-level smart gate is shadowed. The Remediator's `registerRunbook()` validator (§A6, §A36) is the gate that accepts/rejects this runbook at boot; the runbook itself doesn't contain blocking authority over content beyond the structural match contract.
+
+The signal-vs-authority principle is honoured: the match is precise structural code (errorCode enum equality + provenance enum membership + subsystem string equality OR a narrow regex on REASON text limited to the literal `server|lifeline|supervisor` word-boundary match), not a heuristic content classifier. The authority to "rebuild a native module / repair a node symlink / abort a git rebase" lives at the supervisor surface, gated by the legacy six-step body's own structural checks.
+
+---
+
+## 4. Signal vs authority compliance
+
+**Required reference:** [docs/signal-vs-authority.md](../../docs/signal-vs-authority.md)
+
+**Does this change hold blocking authority with brittle logic?**
+
+- [ ] No — this change produces a signal consumed by an existing smart gate.
+- [x] No — this change has no brittle-logic blocking authority. Match is precise structural code over enum-typed fields; verify is a deterministic on-disk marker check; the only block surface (§A36 essential-on-machine) is enforced by the F-8 dispatcher's validator, not by this runbook.
+- [ ] Yes — but the logic is a smart gate with full conversational context.
+- [ ] ⚠️ Yes, with brittle logic — STOP.
+
+The runbook's structural decisions:
+- §A6 prefilter (errorCode + provenance enum equality) — precise.
+- §A36 essential+machine — enforced at registry-load by F-8's validator.
+- §A21 verify taxonomy — derived from filesystem probe result type-by-type.
+- §A34 single-runbook composition — enforced by the runbook ID + the surface's one-call shape.
+
+There are two regexes in `match()`: `/\b(server|lifeline|supervisor)\b/i` against reason text (literal word-boundary match for canonical subsystem names — not a content classifier; no synonyms, no LLM call) and the structural `SUPERVISOR_SUBSYSTEMS` Set lookup. The smart gate (operator approval, /instar-dev review-time discipline) holds policy authority over which runbooks exist; this runbook's job is to be a precise structural detector for one specific failure class.
+
+---
+
+## 5. Interactions
+
+**Does this interact with existing checks, recovery paths, or infrastructure?**
+
+- **Shadowing:** The legacy boot-path `spawnServer()` calls `preflightSelfHeal()` directly. The new `invokeFromRemediator` path calls the SAME body. If both fire in the same process — e.g., the supervisor boots (calls preflightSelfHeal), then a Remediator-orchestrated invocation arrives later — both will run the six in-line heal steps. **Disposition:** intended. Each heal step is idempotent (re-running them when they're already healed is a no-op). The §A2 MachineLock at the runbook level prevents two simultaneous orchestrated invocations; the boot-path call doesn't acquire the lock (it predates Remediator), so the dispatcher's `covered-by-inline` short-circuit doesn't fire — both paths run independently and converge on the same durable state.
+- **Double-fire:** `Remediator.dispatch()` for the same BIND_FAILURE event called twice concurrently → second call observes the first's in-flight MachineLock with the same tupleHash and returns `covered-by-inline`. Audit projection records both attempts (one `started + verified-healthy`, one `covered-by-inline`).
+- **Race with adjacent cleanup:** None new. The dispatcher's `finally` block releases the MachineLock; the runbook adds no shared mutable state beyond the legacy body's existing concerns (which the boot path already handles).
+- **Feedback loops:** The runbook's surfaceCallable returns `outcome: success` when the synchronous body completes without throwing — NOT based on whether the supervisor actually became healthy. The verify() step is what closes the loop — if the body "succeeded" but no fresh lifeline startup marker appears, the Remediator records `verify-failed` and the churn counter ticks. After ≥5 verify-fails in 7 days (§A8 essential-runbook threshold) the runbook auto-quarantines.
+- **A15 partial-upgrade interaction:** F-6 handshake just merged today (2026-05-13). The wrapper code lands but is gated by the planned `wrappers-active-after` config flag in the future Tier-3 dispatcher PR. This PR alone introduces no live behaviour change.
+
+---
+
+## 6. External surfaces
+
+**Does this change anything visible outside the immediate code path?**
+
+- **No new spawned subprocess.** `invokeFromRemediator` delegates to the existing private `preflightSelfHeal()` which has six branches that may invoke `npm install`, `npm rebuild`, `git status`, `git rebase --abort`, node-symlink fixup, lockfile removal, settings.json repair. NONE of those is new — they are the unchanged legacy boot-path body.
+- **No HTTP routes, no Telegram surfaces, no dashboard tabs, no config flags** in this PR. The future `wrappers-active-after` flag is a Tier-3 concern.
+- **No new file types on disk** beyond what the existing `preflightSelfHeal()` body already touches.
+- **New public TypeScript exports** from `src/lifeline/ServerSupervisor.ts`: `SupervisorRemediatorInvocationContext`, `SupervisorInvocationContextKeyVault`, `SupervisorRemediatorExecutionResult`. Structurally compatible with F-8's `RemediationContext` / `ExecutionResult` types. Other consumers of `ServerSupervisor.ts` (none import these new types yet) are unaffected.
+- **New public TypeScript exports** from `src/remediation/runbooks/supervisor-preflight.ts`: `supervisorPreflightRunbook` (the runbook itself), `verifyLifelineDurable`, `VERIFIED_HEAL_TARGETS`, `supervisorMarkerPath`, plus three test-only setters.
+
+---
+
+## 7. Rollback cost
+
+**If this turns out wrong in production, what's the back-out?**
+
+Low. The new code paths have ZERO production callers in this PR — `DegradationReporter.setRemediator()` is still uncalled, so the dispatcher never fires, so the runbook's `surfaceCallable` is never invoked. Revert path:
+
+1. `git revert <pr-sha>` removes both new source files (the runbook) and the test files. The `invokeFromRemediator` method goes with the revert; the private `preflightSelfHeal()` and its boot-path caller in `spawnServer()` remain untouched. Existing supervisor consumers see no change.
+2. No state migration. No on-disk format change. No HTTP route change. No new gitignore entries. No config-flip whitelist additions.
+
+Once a future Tier-3 PR wires `DegradationReporter.setRemediator(remediator)` AND `remediator.registerRunbook(supervisorPreflightRunbook)` together AND flips `wrappers-active-after` to true, the rollback shape becomes "revert the wiring PR" or "flip the flag false" — leaving this PR's primitives intact. The Tier-3 wiring is where the live consumer arrives; this PR is foundation.
+
+If a bug surfaces in the §A3 verify path (e.g., legitimate ctxs are rejected as forged), the fix is to:
+- Either bypass the keyVault check by passing `undefined` from the dispatcher (matching pre-Tier-2 behavior), or
+- Patch the inline `verifySupervisorContextHmac` helper to accept the corrected canonical body format.
+
+Either is a contained change in `src/lifeline/ServerSupervisor.ts`.
+
+---
+
+## Reviewer concurrence (Phase 5)
+
+Not required. This PR has zero live block/allow surface, zero session lifecycle interaction, zero sentinel/gate/watchdog authority. It is foundation infrastructure that becomes load-bearing only when Tier-3 PRs wire the dispatcher into the DegradationReporter pipeline AND flip `wrappers-active-after` to true. At that point the wiring PR carries its own side-effects review.
+
+The §A6, §A9, §A21, §A34, §A36 invariants are all structurally enforced by the existing F-8 validator + the runbook's hardcoded match contract. No reviewer subagent finding could change the shape of the invariants without changing the spec — and the spec is converged and approved.

package/upgrades/side-effects/w3-messaging-delivery-failed-runbook.md

@@ -0,0 +1,185 @@
+# Side-Effects Review — W-3: messaging-delivery-failed runbook + DeliveryRetryManager.runRecoveryCycle
+
+**Version / slug:** `w3-messaging-delivery-failed-runbook`
+**Date:** `2026-05-13`
+**Author:** `echo`
+**Second-pass reviewer:** `not required`
+
+## Summary of the change
+
+Ships Tier-2 wrapper W-3 per `docs/specs/SELF-HEALING-REMEDIATOR-V2-SPEC.md` §A1 (W-3 row), §A6 (structured provenance), §A9 (durable verify), §A21 (verify outcome taxonomy), §A34 R3 (surface-alignment correction — new `runRecoveryCycle()` public method distinct from timer-driven `tick()`), §A36 (essential validator), §A57 (Tier-2 W-3).
+
+Two new modules / extensions:
+
+- `src/remediation/runbooks/messaging-delivery-failed.ts` — the runbook. Matches structured-provenance events with `errorCode ∈ {DELIVERY_FAILURE, TELEGRAM_429, TELEGRAM_500}` and `provenance ∈ {subsystem-explicit, probe-id}` (per §A6: explicitly NO `free-text`). surfaceCallable delegates to `DeliveryRetryManager.invokeFromRemediator`. verify() queries the durable on-disk inbox and asserts ALL queued/undelivered messages have drained per §A9 (not just "the cycle ran"). priority 80; blastRadius `process`; essential `false` per §A36 (essential=true would require blastRadius='machine').
+- `src/messaging/DeliveryRetryManager.ts` (extended) — adds `runRecoveryCycle()` and `invokeFromRemediator(ctx)` as additive public methods per §A34 R3. The existing timer-driven `tick()` is unchanged in behavior but now wraps a shared `cycleInFlight` latch so two concurrent callers (e.g. timer + Remediator) cannot double-process the inbox. The latch is a single-process boundary; cross-process exclusion remains the Remediator MachineLock's job. `tick()`'s return shape is unchanged except for an optional `skipped: true` marker when the second caller short-circuits.
+
+Files touched:
+- `src/remediation/runbooks/messaging-delivery-failed.ts` (new)
+- `src/messaging/DeliveryRetryManager.ts` (extended — runRecoveryCycle + invokeFromRemediator + shared latch)
+- `tests/unit/runbooks/messaging-delivery-failed.test.ts` (new — 16 cases)
+- `tests/unit/DeliveryRetryManager-runRecoveryCycle.test.ts` (new — 11 cases)
+- `upgrades/NEXT.md` (W-3 entry added; existing F-1..F-7, F-8, W-1, scheduler, drift-classifier entries preserved verbatim)
+- `upgrades/side-effects/w3-messaging-delivery-failed-runbook.md` (this artifact)
+
+## Decision-point inventory
+
+- `messagingDeliveryFailedRunbook.match(event)` — **add** — narrow filter: returns true only when `event.subsystem ∈ {messaging, delivery-retry}`. Pure structural equality; no LLM, no regex over free-form text.
+- `messagingDeliveryFailedRunbook.preconditions(event)` — **add** — `_deps` wired AND `getManager()` returns non-null. Refuses to fire when messaging is not initialized (e.g. unit-test boot or pre-server-up).
+- `messagingDeliveryFailedRunbook.verify(ctx)` — **add** — opens an inbox query against the wired `MessageStore` and counts queued/undelivered messages. Returns one of three §A21 outcomes; in particular probe error (queryInbox throws) → verify-inconclusive, not verify-failed.
+- `DeliveryRetryManager.runRecoveryCycle()` — **add** — Remediator-orchestrated parallel entry point. Idempotent against the running timer via `cycleInFlight` latch.
+- `DeliveryRetryManager.invokeFromRemediator(ctx)` — **add** — surface entry-point. Wraps `runRecoveryCycle` and returns `ExecutionResult` with structured details (retried/expired/escalated/skipped).
+- `DeliveryRetryManager.tick()` — **modify** (wrap) — now acquires the same `cycleInFlight` latch as `runRecoveryCycle`. Inner body extracted unchanged into `doCycle()`. No behavior change for existing callers; the optional `skipped` field is additive.
+- `setMessagingDeliveryDeps(deps)` — **add** — production wires the live manager + store; tests inject fakes. Setting to `null` puts the runbook into a "deps unwired" state where surface/verify return inconclusive.
+
+The legacy `tick()` entry-point and the `start()` timer are **pass-through** (existing behavior preserved). The shared latch means a 15s timer tick and a Remediator-dispatch arriving in the same ~ms window won't double-sweep the inbox — first caller wins, second short-circuits with `skipped: true`. This is the §A34 R3 invariant.
+
+No new HTTP routes. No new persistent files. The runbook module exports `setMessagingDeliveryDeps` for late-binding so the messaging singleton doesn't need to be imported at module-load time (avoids a circular import path through the server). Production wiring of `setMessagingDeliveryDeps` is deferred to a separate Tier-2 PR alongside `DegradationReporter.setRemediator()` — this PR ships the unit-testable surface only, matching how W-1 left `NativeModuleHealer.invokeFromRemediator` constructible-but-unwired.
+
+---
+
+## 1. Over-block
+
+**What legitimate inputs does this change reject that it shouldn't?**
+
+- **Free-text-provenance delivery failures** are NOT matched by this runbook. This is by design (§A6). Legacy `.report(...)` callers in `src/messaging/` that normalize to `provenance: 'free-text'` cannot trigger the recovery cycle — they route to `no-matching-runbook` and feed NovelFailureReviewer's clustering pipeline. Operators who hit delivery failures via the timer cadence STILL get retries because the 15s `tick()` cadence is unchanged. The structural defense costs no functionality.
+- **Events about non-messaging subsystems** are rejected by `match()`. Intended — `DeliveryRetryManager` only knows how to drain its own message store. A different subsystem's delivery failure (e.g. webhook delivery, if that subsystem existed) would need its own runbook.
+- **Events arriving while a tick is mid-flight** are returned `skipped: true` and produce zero work via this dispatch. NOT a rejection — the in-flight cycle WILL sweep the same inbox, so the work is not lost. The Remediator's `verify-failed` taxonomy correctly reports "drain succeeded" or "drain incomplete" regardless of which entry-point did the sweep.
+
+No legitimate input is rejected that should have been accepted.
+
+---
+
+## 2. Under-block
+
+**What failure modes does this still miss?**
+
+- **No cross-process deduplication.** Two agent processes on the same machine racing the same on-disk inbox would each acquire the shared latch within their OWN process and proceed; cross-process exclusion is the Remediator's MachineLock job (F-4). This runbook is layered on top of that — the in-process latch is the second line of defense, not the first. The single-agent-per-stateDir invariant (already enforced by the agent server's lifeline lock) makes the cross-process race extremely narrow in practice.
+- **Verify counts queued + undelivered only.** Messages in `delivered` phase awaiting ACK timeout are not counted as "stuck" for §A9 purposes. Intended — Layer 3 ACK timeouts are NOT a delivery failure; they're an upstream agent-not-responding signal. A `verified-healthy` result here means delivery has been re-attempted; it does NOT mean every message has been ACKed.
+- **No deadline enforcement on verify() separately.** The Remediator's §A4 AbortController fires around the whole `surfaceCallable + verify` chain (60s budget per `expectedRuntimeMs`). The verify probe is a single on-disk inbox read; under load it stays well under that budget.
+- **`runRecoveryCycle` does not pre-empt mid-flight on `ctx.abortSignal`.** The cycle body is a loop over inbox entries with short, atomic store operations per entry. We don't check the abort signal mid-loop. Acceptable because: the deadline is 60s, each iteration is a single fs read + fs write, and pre-empting mid-iteration could leave the retryState map inconsistent. If the deadline trips, the dispatcher's `aborted-deadline` branch supersedes whatever we return anyway.
+- **The shared latch is not fair.** If a tick is in-flight and 20 Remediator dispatches arrive in 200ms, all 20 short-circuit with `skipped: true` and the single in-flight cycle handles every message. Acceptable — the inbox is a shared queue, "drain everything you can" is idempotent across callers.
+
+---
+
+## 3. Level-of-abstraction fit
+
+**Is this at the right layer?**
+
+Yes. Each piece sits where the spec mandates:
+
+- **Runbook (`src/remediation/runbooks/messaging-delivery-failed.ts`)** is the policy layer — declares the match contract, verify semantics, blast-radius, and surfaceCallable wiring. Pure data plus a thin verify wrapper; no orchestration, no key material, no audit-write concerns.
+- **`DeliveryRetryManager.invokeFromRemediator`** is the surface layer — owns translating `RemediationContext` into a recovery cycle invocation, returning structured `ExecutionResult` details. No knowledge of the Remediator's lock / intent / audit primitives.
+- **`DeliveryRetryManager.runRecoveryCycle`** is the operation layer — owns the §A34 R3 "idempotent against running timer" rule via the shared latch. Independent of who called it.
+- **Verify helper (`probeDurableDrain`)** is the durability probe — owns the §A9 "assert durable, not just live" rule. Cleanly separated from the recovery step so verify can run independently.
+
+No higher-level smart gate is shadowed. The Remediator's `registerRunbook()` validator (§A6, §A36) is the gate that accepts/rejects this runbook at boot; the runbook itself doesn't contain blocking authority beyond the structural match contract.
+
+The signal-vs-authority principle (`docs/signal-vs-authority.md`) is honoured: the match is precise structural code (errorCode enum membership + provenance enum membership + subsystem string equality), not a heuristic content classifier. The authority to "trigger a recovery cycle" is the manager's own; the runbook merely calls into it.
+
+---
+
+## 4. Signal vs authority compliance
+
+**Required reference:** [docs/signal-vs-authority.md](../../docs/signal-vs-authority.md)
+
+**Does this change hold blocking authority with brittle logic?**
+
+- [ ] No — this change produces a signal consumed by an existing smart gate.
+- [x] No — this change has no brittle-logic blocking authority. Match is precise structural code over enum-typed fields; verify is a deterministic on-disk inbox count; the only block surface (§A36 essential-on-machine) is enforced by the F-8 dispatcher's validator, not by this runbook.
+- [ ] Yes — but the logic is a smart gate with full conversational context.
+- [ ] ⚠️ Yes, with brittle logic — STOP.
+
+The runbook's structural decisions:
+- §A6 prefilter (errorCode + provenance enum equality) — precise.
+- §A36 essential=false enforced by F-8 validator (no claim to essential here).
+- §A9 verify (queued + undelivered count == 0) — deterministic.
+- §A34 R3 idempotence (single-process latch, first-wins) — deterministic.
+- §A21 verify taxonomy — derived from inbox-count probe result kind.
+
+No brittle string-matching, no LLM judgement, no content classifier. The match callback's subsystem string equality is the same shape as W-1's better-sqlite3 narrowing.
+
+---
+
+## 5. Interactions
+
+**What other components is this change touching, even indirectly?**
+
+- **F-8 dispatcher (`Remediator.dispatch`)** — registers and dispatches this runbook. Existing dispatch path is generic; no modification needed. The dispatcher's §A6 / §A36 validators accept the runbook (verified by unit test).
+- **F-3 `DegradationReporter`** — emit-sites in `src/messaging/` will eventually call the structured emit helper to produce DELIVERY_FAILURE / TELEGRAM_429 / TELEGRAM_500 events with `provenance: 'subsystem-explicit'`. The migration is incremental; until then, legacy callers route to `no-matching-runbook` via `provenance: 'free-text'` (intended A33 steady-state).
+- **`DeliveryRetryManager` consumers** — `AgentServer` constructs and `start()`s the manager. The wrap of `tick()` is a no-op for the timer caller (it still returns the same shape, just with an optional `skipped` field that's `undefined`/falsy under normal operation).
+- **`MessageStore.queryInbox`** — the verify probe calls this. It's a stable public method used by many other consumers; the additional read load is one inbox scan per dispatch attempt, well below the per-cycle scan rate.
+- **F-4 `MachineLock`** — the runbook acquires the same in-flight tuple lock the dispatcher acquires for any runbook. No new lock surface.
+- **`AgentServer` boot wiring** — NOT modified in this PR. Production wiring of `setMessagingDeliveryDeps(...)` is deferred to the same future PR that wires `DegradationReporter.setRemediator(...)`. Both must be wired together so the runbook actually receives events.
+
+The wrap of `tick()`'s body into `doCycle()` is the only behavior change to existing code. Verified by running the existing `delivery-retry-manager.test.ts` (21 cases, all pass unchanged).
+
+---
+
+## 6. Rollback cost
+
+**If this change is reverted, what breaks?**
+
+Low — additive. The legacy `tick()` and `start()` are unchanged in behavior; reverting deletes the new runbook module + the two new public methods + the shared latch wrap on `tick()`. No data-model migration. No persistent state ships with this PR. The shared latch is a `boolean` field; deleting it removes the additional idempotence guarantee but does not corrupt any state. Existing F-1..F-7 + W-1 modules are unaffected.
+
+A revert leaves messaging recovery on the pre-existing 15s timer cadence — the Remediator-orchestrated faster-recovery + audit-tracked path is the gain that's lost on revert, not a load-bearing capability.
+
+---
+
+## 7. Spec compliance
+
+- §A1 W-3 manifest — runbook id `messaging-delivery-failed`, surface `delivery-retry`. Matches the spec table row.
+- §A6 — `provenance: ['subsystem-explicit', 'probe-id']`; explicitly NO `free-text`. Registry validator (F-8) accepts this at registry-load.
+- §A9 — verify queries the durable on-disk inbox (not the in-memory retryState map), counts ALL queued+undelivered, requires zero. "Drain ALL stuck messages (not just one)" matches the spec language verbatim.
+- §A21 — verify taxonomy: ok → verified-healthy; stuck (count > 0) → verify-failed; probe error → verify-inconclusive.
+- §A34 R3 — `runRecoveryCycle()` is a NEW public method, distinct from timer-driven `tick()`. Idempotent against the running timer via the shared `cycleInFlight` latch. Spec docstring captures this.
+- §A36 — essential=false validated against blastRadius=process. F-8 validator accepts.
+- §A57 — Tier-2 W-3 row. Foundation dependency on W-1 (in main as PR #202) and F-8 (in main as PR #201 + #217). W-2 is parallel work; this PR does not block on it.
+- §A15 (note from prompt) — lag rule is documented in the runbook expectedRuntimeMs (60s). The recovery cycle is short-lived; lag-detection escalation is the Remediator's concern, not this runbook's. Build proceeds per W-2's pattern.
+
+---
+
+## 8. Test coverage
+
+27 new tests across two files. Pass: 27/27.
+
+`tests/unit/DeliveryRetryManager-runRecoveryCycle.test.ts` (11):
+- runRecoveryCycle is a distinct public method from tick.
+- runRecoveryCycle return shape matches tick on empty inbox.
+- runRecoveryCycle processes queued messages identically to tick (TTL-expired envelope dead-lettered).
+- Idempotence: runRecoveryCycle short-circuits while tick is in-flight.
+- Idempotence: tick short-circuits while runRecoveryCycle is in-flight (symmetric).
+- Latch resets after cycle completes; subsequent calls run normally.
+- invokeFromRemediator wraps runRecoveryCycle and returns ExecutionResult success.
+- invokeFromRemediator returns failure when runRecoveryCycle throws (synthetic store error).
+- invokeFromRemediator reports skipped:true when cycle is in-flight.
+- stop() clears the in-flight latch (recycle-safe).
+- Existing tick() return shape unchanged when cycle runs normally.
+
+`tests/unit/runbooks/messaging-delivery-failed.test.ts` (16):
+- Matches all three structured errorCodes.
+- §A6: free-text excluded from prefilter.
+- match() rejects non-messaging subsystems.
+- Remediator.registerRunbook accepts the runbook (essential=false, process radius).
+- §A36 honored: essential=false, blastRadius=process.
+- Manifest fields verified (id, priority=80, surface=delivery-retry, expectedRuntimeMs=60_000, reversibility=reversible).
+- preconditions fail when deps not wired.
+- preconditions succeed when manager is wired.
+- surfaceCallable invokes manager.invokeFromRemediator with ctx (mocked).
+- verify() returns verified-healthy on drained inbox.
+- verify() returns verify-failed on stuck messages (with stuck count + sample IDs).
+- verify() returns verify-inconclusive on store probe error.
+- verify() against real store: queued message → verify-failed (durable assertion).
+- verify() against real store: empty inbox → verified-healthy.
+- End-to-end dispatch on TELEGRAM_429: invoked → verified-healthy → audited.
+- End-to-end dispatch with stuck messages: verify-failed audited (synthetic deliverToSession failure).
+
+Existing `tests/unit/delivery-retry-manager.test.ts` (21) and `tests/unit/runbooks/node-abi-mismatch.test.ts` (12) re-verified — unchanged behavior. Existing `tests/unit/Remediator.test.ts` (12) re-verified — unchanged behavior.
+
+---
+
+## 9. Open follow-ups
+
+- **Production wiring** of `setMessagingDeliveryDeps(...)` and `DegradationReporter.setRemediator(...)` is deferred to a coordinated Tier-2 PR. This PR ships the unit-testable surface; production registration is a single-PR follow-up.
+- **F-3 emit-site migration** in `src/messaging/` — the structured errorCodes DELIVERY_FAILURE / TELEGRAM_429 / TELEGRAM_500 need to flow through `reportStructured(...)`. Until then, the runbook is dormant in production (intended A33 incremental migration).
+- **W-2** (`supervisor-preflight`) and **W-4** (`db-corruption`) — parallel wrapper work; this PR does not depend on either.

package/upgrades/side-effects/w4-db-corruption-runbook.md

@@ -0,0 +1,114 @@
+# Side-Effects Review — W-4 db-corruption runbook + SemanticMemory.invokeFromRemediator
+
+**Version / slug:** `w4-db-corruption-runbook`
+**Date:** `2026-05-13`
+**Author:** echo
+**Second-pass reviewer:** not required (no outbound-messaging, session-lifecycle, sentinel, or coherence-gate surface)
+
+## Summary of the change
+
+W-4 of the Tier-2 Self-Healing Remediator v2 rollout. Adds `src/remediation/runbooks/db-corruption.ts` — an ApprovedRunbook that lets the Remediator dispatch on structured-provenance SQLITE corruption events and orchestrate the existing SemanticMemory corruption-recovery path through a HMAC-authenticated surface entry point. The surface entry point `SemanticMemory.invokeFromRemediator(ctx)` is added alongside, paired with a small set of helpers (`getDurabilityMode`, `runIntegrityCheckForRemediator`, static registry of active instances). The legacy in-line corruption-recovery path inside `SemanticMemory.open()` (lines 178-243 — integrity_check + probe-read + quarantine + JSONL rebuild) is unchanged and remains the canonical CLI-path safety net. Per A34, the surface this runbook wraps is verified live on main BEFORE this change lands. Per A9, verify asserts both `db.mode === 'durable'` AND `pragma integrity_check === 'ok'` so a future in-memory fallback path would be flagged `verify-failed` rather than passing silently. Files: `src/remediation/runbooks/db-corruption.ts`, `src/memory/SemanticMemory.ts`, `tests/unit/runbooks/db-corruption.test.ts`, `tests/unit/SemanticMemory-invokeFromRemediator.test.ts`, `upgrades/NEXT.md`.
+
+## Decision-point inventory
+
+- `Remediator.dispatch (matched runbook selection)` — pass-through — the new runbook is one more candidate registered alongside W-1; dispatch logic itself is unchanged.
+- `db-corruption.eventPrefilter` — add — structured-provenance only (`native-binding | subsystem-explicit | probe-id`); `'free-text'` excluded per §A6.
+- `db-corruption.match()` — add — confirms the event is about a SemanticMemory-class store (subsystem in {`semantic-memory`, `memory`, `better-sqlite3`} or reason text mentions `SemanticMemory`).
+- `db-corruption.preconditions()` — add — checks that an active `SemanticMemory` instance is registered. If not, returns false → dispatch falls through, no action taken.
+- `db-corruption.verify()` — add — durability-aware probe returning `verified-healthy | verify-failed | verify-inconclusive` per §A21.
+- `SemanticMemory.invokeFromRemediator()` — add — surface entry point with optional §A3 HMAC verification; on invalid HMAC, refuses orchestrated path (it does NOT silently fall back to in-line, because the in-line path naturally fires on the next `open()` anyway).
+- `SemanticMemory.open()` — modify (additive) — sets a `_lastRecoveryRebuilt` flag when the existing rebuild path runs. No behavioral change to the recovery itself.
+
+---
+
+## 1. Over-block
+
+**What legitimate inputs does this change reject that it shouldn't?**
+
+- Free-text-extracted `SQLITE_CORRUPT` events from logs/stderr (provenance `'free-text'`) will route to `no-matching-runbook` instead of firing the recovery. This is intentional per §A6 — free-text matchers are an injection surface — but it does mean a real corruption event whose only signal is a parsed log line won't auto-heal until SystemReviewer (Tier-3) promotes the signal or a probe is wired. The legacy in-line path inside `open()` still catches everything on next open, so this is a "miss the orchestrated path" not "miss recovery entirely." Acceptable.
+- A SemanticMemory instance not explicitly registered via `setActiveInstance()` will see `preconditions()` return false → dispatch is a no-op. This is intentional: partial CLI surfaces (e.g. `instar memory` one-shot commands) shouldn't be targets of Remediator-orchestrated recovery. The server bootstrap is the natural place to register the primary instance.
+
+---
+
+## 2. Under-block
+
+**What failure modes does this still miss?**
+
+- A truly silent in-memory fallback would require a code change in `SemanticMemory` (not currently present); today durability is binary — either we have a real `dbPath` or we don't. If a future change adds `:memory:` fallback without setting `dbPath`, `getDurabilityMode()` correctly returns `'in-memory'` and verify returns `verify-failed`. Until that change lands, the durability-degraded branch is dead code — but harmless, and exists exactly so the runbook is ready when fallback ships.
+- The post-recover integrity check runs once at end of `open()`. A corruption introduced after open completes (e.g. mid-write torn page) is caught on the NEXT open or by an A52 probe — not by this surface invocation. Same shape as W-1.
+
+---
+
+## 3. Level-of-abstraction fit
+
+**Is this at the right layer?**
+
+Yes. The runbook is a thin orchestration shim: it declares a structured prefilter, narrows with `match()`, gates with `preconditions()`, delegates the actual work to the surface (which owns the recovery), and probes with `verify()`. The surface owns the policy of "how do we recover?" — that's the right separation: orchestration logic stays in `src/remediation/`, recovery logic stays in `src/memory/`. No duplicated logic; the runbook just composes existing primitives. Mirrors the W-1 pattern exactly.
+
+---
+
+## 4. Signal vs authority compliance
+
+**Required reference:** [docs/signal-vs-authority.md](../../docs/signal-vs-authority.md)
+
+**Does this change hold blocking authority with brittle logic?**
+
+- [x] No — this change produces a signal consumed by an existing smart gate.
+
+The runbook's `eventPrefilter` and `match()` are **detectors** that surface "this event looks like SemanticMemory corruption." They produce structured signals (matched/not-matched). The **authority** is the `Remediator` — it holds the lock, enforces the deadline, signs/verifies the capability ctx, drives the audit log, and decides the final dispatch outcome. The surface (`SemanticMemory.invokeFromRemediator`) is also a downstream consumer of the ctx, not a parallel decision-maker — it can refuse on invalid HMAC, but that's the structurally-correct response to a forged capability token (per §A3), not a brittle filter sitting in front of a smart gate.
+
+`verify()` is similarly a structured probe (durable+ok / corrupt / degraded / inconclusive) — the Remediator decides what to log and whether to count against churn (§A8/§A21).
+
+No new brittle blocking logic introduced.
+
+---
+
+## 5. Interactions
+
+**Does this interact with existing checks, recovery paths, or infrastructure?**
+
+- **Shadowing:** the in-line corruption-recovery path inside `SemanticMemory.open()` still runs on every open — including the re-open the surface triggers. This is intentional: the orchestrated path delegates back to the in-line logic. There is no race because the surface explicitly `close()`s before re-opening. Per §A2, both paths should acquire the same machine lock; the existing in-line path doesn't yet hold a lock (that's an F-3 / surface-lock workstream beyond W-4 scope), but the Remediator does hold one per dispatch — so concurrent orchestrated dispatches are serialized. Concurrent in-line + orchestrated against the same dbPath remains a known F-3 gap (called out in the spec).
+- **Double-fire:** if a SQLITE_CORRUPT event is normalized AND a probe fires AND the in-line path also runs naturally on next open, three signals converge. The Remediator's lock + storm-coalesce prevents double-orchestrated execution. The in-line path is idempotent — running it after the orchestrated heal is a no-op because `_needsRebuild` is only set on integrity_check failure, which has been resolved.
+- **Races:** the surface closes its own db handle before re-opening. Concurrent callers using the same `SemanticMemory` instance during recovery would see an `ensureOpen()` throw — same as today's `close()` semantics. The Remediator's lock structurally serializes orchestrated attempts on the same machine.
+- **Feedback loops:** none. Recovery emits no new degradation events; the existing quarantine marker file is unchanged.
+
+---
+
+## 6. External surfaces
+
+**Does this change anything visible outside the immediate code path?**
+
+- **Other agents:** none. Recovery operates on per-instance `dbPath`.
+- **Other users:** none. No public API surface added beyond the runbook export.
+- **External systems:** none. No Telegram/Slack/GitHub touchpoints.
+- **Persistent state:** the existing `.corrupt.<ts>` quarantine file and `.corrupt-recovery.<ts>.marker.json` continue to be written by the unchanged in-line path. The new `_lastRecoveryRebuilt` is a process-memory flag only — not persisted. The new audit-projection entries (`db-corruption: started → verified-healthy`) ship through the existing AuditWriter, same shape as W-1 entries.
+- **Timing:** the recovery path can take seconds to tens of seconds on large JSONL files. Already covered by the existing `autoRebuildMaxBytes` cap in `open()`. The runbook's `expectedRuntimeMs: 60_000` budgets one minute; the Remediator's AbortController enforces it.
+
+---
+
+## 7. Rollback cost
+
+**If this turns out wrong in production, what's the back-out?**
+
+Pure code change. Revert the four touched files and ship as a patch. No persistent state to migrate (the `_lastRecoveryRebuilt` flag is in-memory only; `setActiveInstance` is a process-local registry). No user-visible regression during rollback — the in-line corruption-recovery path still handles everything it handles today. Agents that have run with the new code will simply lose access to the orchestrated path until they update; they retain the safety net.
+
+---
+
+## Conclusion
+
+W-4 lands a small, additive, properly-gated wrapper around an already-proven recovery path. The signal-vs-authority separation is clean: detectors (prefilter + match) feed the Remediator (authority); the surface receives a capability ctx and performs the work but does not gain new decision power. The §A34 surface-alignment precondition is verified: the corruption-recovery code on main is real, not stubbed. The §A9 durability-not-liveness verify is in place, ready for the day a fallback path is added — when that future change lands, this runbook automatically flags it `verify-failed`. Clear to ship.
+
+---
+
+## Second-pass review (if required)
+
+Not required. The change does NOT touch outbound/inbound messaging block/allow, session lifecycle, compaction, coherence gates, sentinels, or watchdogs. It composes existing primitives (Remediator, AuditWriter, MachineLock, SemanticMemory.open) without introducing new authority.
+
+---
+
+## Evidence pointers
+
+- 29 new unit tests pass (`tests/unit/runbooks/db-corruption.test.ts` 17 cases + `tests/unit/SemanticMemory-invokeFromRemediator.test.ts` 12 cases).
+- Adjacent tests still pass: `tests/unit/semantic-memory-corruption-recovery.test.ts` (12), `tests/unit/runbooks/node-abi-mismatch.test.ts` (12), `tests/unit/semantic-memory.test.ts`.
+- `npx tsc --noEmit` clean.
+- Surface alignment verified by `grep -n "quarantineCorruptDb\|nuclear option" src/memory/SemanticMemory.ts` against origin/main HEAD = `3ba88731` — recovery is live on main at lines 178-243, 273-315, 1655-1674.