instar 1.3.231 → 1.3.233

package/upgrades/1.3.233.md

@@ -0,0 +1,36 @@
+# Upgrade Guide — vNEXT
+
+<!-- assembled-by: assemble-next-md -->
+<!-- bump: minor -->
+
+## What Changed
+
+The Correction & Preference Learning Sentinel now survives an LLM throttle without losing corrections.
+
+Previously, the capture hop distilled each detected correction through a rate-limited LLM queue behind the account-global circuit breaker. When that distill call was rejected for a capacity reason — daily spend cap reached, interactive-reserve breach, an aborted background call, or the breaker being open — `CorrectionCaptureLoop` dropped the capture silently with no retry and no backlog. On a production agent under sustained throttling that meant Layer 0 kept *detecting* corrections while the `CorrectionLedger` stayed permanently EMPTY, because every distill attempt hit the breaker and was discarded.
+
+This release adds `CorrectionCaptureBacklog` — a bounded, durable SQLite store at `correction-capture-backlog.db`. When a distill is rejected for a capacity reason, the already-pre-scrubbed capture is PERSISTED to the backlog instead of dropped. A new `isCapacityThrow` classifier distinguishes a capacity rejection (worth retrying later) from a genuine fault (a malformed envelope still drops, preserving the old behavior). A new audit decision `distill-backlogged` records the persist.
+
+A new `drainBacklog` function works the backlog down off the message hot path. It is triggered two ways: opportunistically right after a live capture's distill SUCCEEDS (proof the LLM has headroom *now*), and by a slow 5-minute periodic sweep. The drainer is breaker-gated — it consults `llmCircuitAvailable()` (a pure, non-probe-consuming read) and never drains while the breaker is open, since that would just re-fail every entry and re-trip the circuit. For each claimed entry it rebuilds the same distill prompt, records a real learning into the `CorrectionLedger`, then DELETES the backlog row (`markDistilled`); a failed attempt calls `bumpAttempt`, which drops the entry once it exceeds its retry budget.
+
+Retention is bounded two ways: a max-entries cap (oldest evicted on overflow) and a TTL (stale entries pruned before each drain). The store persists ONLY the pre-scrubbed turns — the same secret scrub that guards the distill prompt runs before anything is written, plus a defensive re-scrub on enqueue — and no API route ever serves backlog contents. It is on by default whenever the feature is enabled (pure resilience) and is disabled by setting its max-entries to zero, which restores the old drop-on-throttle behavior. All four new dials backfill into existing agents via `applyDefaults` deep-merge.
+
+The whole path is fail-open and signal-only: a backlog or drain error can never block, delay, or throw into message delivery.
+
+## What to Tell Your User
+
+- **Corrections survive a busy stretch now**: "When you correct me about the same thing and I happen to be rate-limited at that moment, I no longer just lose it. I tuck the correction away safely and learn from it a little later once I have headroom — so a busy period can't make me quietly forget what you taught me. It only ever keeps a private, scrubbed note of the lesson, never your raw words, and it tidies those away as soon as they're learned. This whole thing only runs when correction learning is switched on, and I can turn it off for you if you ever want the old behavior back."
+
+## Summary of New Capabilities
+
+| Capability | How to Use |
+|-----------|-----------|
+| Durable retry of throttled corrections | Automatic when correction learning is enabled |
+| Disable the backlog (old drop behavior) | Set the correction-learning capture-backlog max-entries to 0 |
+| See learned corrections (the ledger the backlog feeds) | GET /corrections |
+
+## Evidence
+
+- **Reported live**: on a production agent with correction learning enabled, Layer 0 was detecting corrections but GET /corrections returned an empty ledger. Root cause confirmed in `src/monitoring/CorrectionCaptureLoop.ts` (~line 204): the distill catch block dropped every capture on any LlmQueue throw, and the agent's account was sustained-throttled, so every distill hit the open circuit breaker and was discarded.
+- **After (code path), full round-trip proven**: `tests/e2e/correction-capture-backlog-lifecycle.test.ts` drives a learning-signal capture whose distill is rate-limited → asserts decision `distill-backlogged` and a backlog count of 1 (NOT dropped); asserts the drain is SKIPPED while the breaker is open (entry retained); then flips the breaker closed, drains, and asserts the distilled record is recorded into the ledger, the backlog row is deleted, and the record is observable on the real GET /corrections route with only its scrubbed summary served (raw learning never crosses HTTP).
+- **Unit coverage**: `tests/unit/CorrectionCaptureBacklog.test.ts` (11 cases: enqueue, max-entries eviction oldest-first, near-identical dedupe, claimBatch ordering + min-retry-gap, markDistilled, bumpAttempt-drops-at-maxRetries, TTL prune, pre-scrubbed-only persistence, fail-open). `tests/unit/CorrectionCaptureBacklog-drain.test.ts` (12 cases: the four-shape isCapacityThrow classifier, capacity-throw → backlog vs non-capacity → drop, no-backlog-wired old drop, backlog-fault fallback, drain breaker-gated/recorded/bumped/pruned/fail-open). Wiring-integrity in `tests/unit/correction-learning-wiring-integrity.test.ts` (backlog constructed iff enabled+maxEntries>0; a rate-limited capture lands in the backlog without throwing into the hook; drain skipped while breaker open). Integration round-trip in `tests/integration/corrections-routes.test.ts`. All green: 11 + 12 + 9 + 13 + 2 = the new/updated suites pass with zero failures.

package/upgrades/side-effects/correction-capture-backlog.md

@@ -0,0 +1,137 @@
+# Side-Effects Review — correction capture-backlog with retry
+
+**Version / slug:** `correction-capture-backlog`
+**Date:** `2026-06-02`
+**Author:** `echo`
+**Second-pass reviewer:** `pending`
+
+## Summary of the change
+
+Adds `src/monitoring/CorrectionCaptureBacklog.ts` — a bounded, durable SQLite store
+that persists pre-scrubbed captures the Correction & Preference Learning Sentinel
+could not distill because the LLM was rate-limited (daily-cap / reserve-breach /
+aborted / circuit-breaker-open). `CorrectionCaptureLoop.captureAndDistill` now, in
+its distill catch block, classifies the rejection via the new `isCapacityThrow`: a
+CAPACITY rejection enqueues the pre-scrubbed capture to the backlog (audit decision
+`distill-backlogged`) instead of dropping it; a genuine fault still drops
+(`distill-dropped`). A new `drainBacklog` distills backlogged entries into the
+`CorrectionLedger` off the hot path, breaker-gated on `llmCircuitAvailable()`,
+triggered after a successful live distill and by a 5-minute periodic sweep. Four
+config dials (`captureBacklogMaxEntries` default 200, `captureBacklogTtlHours`
+default 24, `captureBacklogDrainPerTick` default 5, `captureBacklogMaxRetries`
+default 3) backfill via `applyDefaults`; maxEntries=0 disables the backlog.
+
+## Decision-point inventory
+
+- `captureAndDistill: distill threw — backlog or drop?` (CorrectionCaptureLoop.ts) —
+  **new branch** — a capacity throw now persists (was: always drop). A non-capacity
+  throw is unchanged (drops). Wrapped in try/catch → a backlog fault falls back to drop.
+- `drainBacklog: is the LLM available?` (CorrectionCaptureLoop.ts) — **new gate** —
+  skips the whole drain while the breaker is open; re-checks between entries.
+- `server.ts: construct the backlog?` — **new gate** — constructed IFF the feature is
+  enabled AND `captureBacklogMaxEntries > 0`. Else null (old drop behavior).
+- `CorrectionCaptureBacklog.enqueue: at the cap?` — **new** — evicts oldest on overflow;
+  dedupes a near-identical entry instead of inserting a duplicate.
+
+---
+
+## 1. Over-block
+
+**No block/allow surface — over-block not applicable.** Nothing in this change can
+reject, block, delay, or gate a message or an operation. The backlog/drain only
+move an internal observe-only record (a distilled correction) from a deferred state
+to the ledger. The worst over-reach is a redundant LLM distill call on an entry that
+would have been noise — bounded by `captureBacklogDrainPerTick` (5) per sweep and by
+the per-sentinel daily LLM cap (the drain shares the same `correctionLlmQueue`
+budget, so it cannot exceed the feature's existing spend ceiling).
+
+---
+
+## 2. Under-block
+
+**What does it still miss?** `isCapacityThrow` is conservative — an unrecognized
+error string is treated as a non-capacity fault and DROPS (preserving old behavior),
+so a novel provider rate-limit phrasing not in the matcher would still be dropped
+rather than backlogged. That is the safe direction (no false persistence). A capture
+also still drops if the backlog enqueue itself faults (disk full) — again fail-open,
+no worse than today. Entries that exhaust `captureBacklogMaxRetries` or age past the
+TTL are intentionally discarded (bounded retention), so a persistently-un-distillable
+capture is eventually dropped by design.
+
+---
+
+## 3. Level-of-abstraction fit
+
+Correct layer. The backlog mirrors `CorrectionLedger`'s own SQLite discipline (WAL,
+prune-in-transaction, indexes, fail-open, `NativeModuleHealer` open, `SqliteRegistry`
+close-on-exit) and lives beside it in `src/monitoring/`. The drain reuses the existing
+`buildDistillPrompt` / `parseDistillEnvelope` rather than inventing a second distill
+path, and rides the existing account-global `LlmCircuitBreaker` via its pure
+`llmCircuitAvailable()` read rather than adding a parallel availability notion.
+
+---
+
+## 4. Signal vs authority compliance
+
+**Required reference:** [docs/signal-vs-authority.md](../../docs/signal-vs-authority.md)
+
+- [x] No authority added — this change produces and defers a SIGNAL (a distilled
+  correction record) consumed by the existing `CorrectionAnalyzer` / `CorrectionLoopDriver`,
+  which own every downstream decision. The backlog cannot mint a proposal, write a
+  preference, post feedback, or change policy; it only delays then completes the same
+  ledger write the hot path would have done. The `CorrectionLoopDriver`'s
+  by-construction authority guard (no proposal-mint, no memory-write) is untouched.
+
+---
+
+## 5. Interactions
+
+- **Shadowing:** none — the backlog is a NEW db file (`correction-capture-backlog.db`),
+  separate from `correction-ledger.db`; no existing query path changes.
+- **LLM budget:** the drain shares the per-sentinel `correctionLlmQueue` daily cap, so
+  it competes with (never exceeds) the existing distill budget — and it only runs when
+  `llmCircuitAvailable()` is true, so it cannot pile onto an already-throttled account.
+- **Circuit breaker:** the drain is a pure consumer of the breaker's read state; it does
+  not trip, reset, or consume a half-open probe slot.
+- **Multi-machine:** the db is per-stateDir (per machine), like the ledger; no mesh
+  coordination needed (the worst case is each machine independently distilling its own
+  backlogged captures, which dedupe-collapse in the ledger by `dedupeKey`).
+
+---
+
+## 6. Privacy / data exposure
+
+**This is the load-bearing dimension and the central design constraint.**
+
+- **Persist-pre-scrubbed-only posture.** The backlog persists ONLY the
+  already-pre-scrubbed turns. The §3.3 deterministic `scrubSecrets` runs inside
+  `buildDistillPrompt` BEFORE the catch that enqueues, and `enqueue` re-scrubs every
+  turn defensively (belt-and-suspenders) so even a mis-wired caller cannot write raw
+  text. Unit-proven: a `ghp_…` token in a turn is `gh***_REDACTED` on disk
+  (`CorrectionCaptureBacklog.test.ts` + `CorrectionCaptureBacklog-drain.test.ts`).
+- **Bounded retention.** Max-entries cap (oldest evicted) AND a TTL (stale pruned),
+  and every entry is DELETED the instant it distills (`markDistilled`) or exhausts its
+  retries (`bumpAttempt`). It is a bounded, time-limited extension of the existing
+  ephemeral look-back ring — not a new long-lived data sink.
+- **No new exposure surface.** No API route serves backlog row content; the scrubbed
+  turn text never crosses HTTP. The only observable is a count. The E2E pins that the
+  drained record surfaces on GET /corrections with ONLY its scrubbed summary (raw
+  learning absent from the payload), exactly like the existing ledger guarantee.
+
+---
+
+## 7. Failure modes / rollback
+
+- **Fail-open everywhere.** Every backlog method swallows storage errors and returns a
+  safe default; `captureAndDistill`'s backlog enqueue is wrapped in try/catch and falls
+  back to the old drop; `drainBacklog` is fully detached (`void`/async), single-flight
+  guarded, and its top-level catch can never throw into a seam. A backlog fault degrades
+  to exactly the pre-change behavior (drop-on-throttle).
+- **Rollback.** Per-agent: set `monitoring.correctionLearning.captureBacklogMaxEntries`
+  to 0 — the backlog is not constructed and the loop reverts to the old drop. Whole
+  feature: the existing `monitoring.correctionLearning.enabled` dark gate. No schema
+  migration is destructive (a new db file); deleting `correction-capture-backlog.db`
+  is safe and just discards any pending retries.
+- **Boot cost.** The store opens an empty WAL db with two small indexes — no boot-time
+  full-scan (the `TokenLedger` backfill-boot-hang class is explicitly avoided; there is
+  no startup scan, only on-demand claim/prune).

package/upgrades/side-effects/mcp-process-reaper.md

@@ -0,0 +1,58 @@
+# Side-Effects Review — MCP-Process Reaper (Option B)
+
+**Version / slug:** `mcp-process-reaper`
+**Date:** `2026-06-04`
+**Author:** `echo`
+**Second-pass reviewer:** `recommended before dryRun:false is set on any agent (it kills processes)`
+
+## Summary of the change
+
+A new `McpProcessReaper` (separate sibling, NOT a change to `OrphanProcessReaper` /
+`ReapGuard` / `ReapAuthority`) reaps leaked MCP-server children (playwright-mcp /
+mcp-remote / instar-mcp-stdio) whose owning tmux session is dead/stale or fully
+orphaned. Ships OFF + dry-run fleet-wide; the `developmentAgent` gate enables it on
+echo but leaves `dryRun:true`, so it only observes + audits would-reap. Read-only
+report at `GET /processes/mcp-reaper`.
+
+## Decision-point inventory
+
+One decision point: the per-proc keep/reap classifier (`classifyMcpProcess`). It is
+pure and unit-tested on both sides of every boundary. The kill action is gated
+behind `killsEnabled = enabled && !dryRun`.
+
+## 1. Over-block (what legitimate things could it wrongly KILL?)
+
+The risk is killing an MCP server that a live session still needs. Mitigations,
+each unit-tested:
+- A proc under a **live/tracked** session is KEPT regardless of age (`session-live`)
+  — a long-running autonomous session keeps its old MCP servers.
+- A proc under an **external (non-instar)** tmux session is KEPT (`external-session`)
+  — the reaper never touches the user's own processes.
+- A proc **younger than `minAgeMs` (2h)** is KEPT — a starting session's children are
+  never candidates.
+- A **failed evaluation** KEEPS (never reap on error).
+- Only **three exact signatures** match — never a broad node/npm match.
+- Ships **dark + dry-run**; even on the dev agent it only observes. Kills require an
+  explicit `dryRun:false`.
+- `maxReapsPerPass` (25) bounds the blast radius per pass.
+
+## 2. Under-block (what leaks does it still MISS?)
+
+- It does not prevent NEW leaks — that is Option A (tree-kill on session teardown),
+  noted as a follow-up. B is the backstop that cleans existing + future unclean deaths.
+- A leaked MCP proc whose dead session's tmux pane somehow still lingers AND is not
+  recognized as an instar session would be treated as external (KEPT). This is the
+  safe failure direction (favor false-negatives).
+- Non-MCP leaked processes are out of scope by design (precise allow-list).
+
+## 3. Reversibility
+
+SIGTERM on an already-orphaned helper is low-risk (its session is gone). The reaper
+itself is fully reversible via config (`enabled:false`). No persistent state is
+mutated except the append-only audit log.
+
+## 4. Blast radius on the existing reaper
+
+**Zero.** This is a separate class; the shared `OrphanProcessReaper` / `ReapGuard` /
+`ReapAuthority` / session-reap authority path is untouched (the same Option-B
+isolation used for #722).