instar 1.3.110 → 1.3.112

package/upgrades/NEXT.eli16.md

@@ -2,78 +2,87 @@
 
 ## What this change is
 
-In JavaScript/TypeScript, you can write `try { ... } catch {}` — a block
-that says "do this, and if anything goes wrong, throw the error away
-and continue silently." It's three characters of body and zero
-explanation, and it's a really cheap shortcut that almost always means
-"I don't want to think about this right now."
-
-The problem: when an error gets thrown away silently, you don't find
-out about it from a stack trace or a log line. You find out about it
-when something downstream breaks, often hours later, often from a user.
-
-The worst recent example was the **PromptGate $452 incident**. There
-was a 5-second loop checking whether the LLM had hit its rate limit.
-Inside that loop, a bare `catch {}` was swallowing every actual
-rate-limit error. The loop kept burning credits at full speed —
-hundreds of times per minute, sometimes for hours — because nothing
-ever surfaced the problem. By the time someone noticed, we'd burned
-$452 we shouldn't have.
-
-This PR adds a unit-test lint that refuses any bare `catch {}` in
-production code (`src/**/*.ts`, excluding tests and code-generation
-hosts) unless it carries a comment annotation explaining WHY the
-silent swallow is safe.
+The post-mortem this morning named five recurring bug classes. Four of
+the five PRs landed today (#542, #545, #550, #551, #552) each closed
+one specific incident — silent 403s, missing wiring, watch-exit
+merges, bare catches. This is the fifth and final post-mortem PR, and
+it closes the broader pattern that produced them:
+
+**Tested on fresh state, not real-world state.**
+
+That class is the one where the test suite passes because it uses
+small, fresh, just-created fixtures, but the actual deployed agent has
+a 202MB token-ledger.db / an externalized secret config / a
+wrong-ABI binary / 8 concurrent jobs — and the boot path breaks in
+production despite tests being green. Every fix-shape PR I named in
+the post-mortem analysis had at least one test that PASSED on fresh
+state and would have FAILED on real state.
+
+This PR adds a new test category called `tests/real-world-state/`
+specifically for that class. It sits alongside `unit/`,
+`integration/`, and `e2e/` as a peer.
 
 ## What already exists
 
-- An older lint called `no-silent-fallbacks.test.ts` that catches a
-  related pattern: catches that return a degraded value (`return
-  null`, `return []`, etc.).
-- The `@silent-fallback-ok` comment annotation, already used in
-  several places (`TrustRecovery.ts`, `SyncOrchestrator.ts`, etc.).
-- `DegradationReporter` for emitting properly-tracked degradation
-  signals.
+- The three existing test categories. None of them load real-shaped
+  state — they all use small fresh fixtures (`tests/fixtures/`).
+- Some adjacent code that COULD be tested at real-world state (the
+  SecretMigrator + SecretStore have unit tests but their merge layer
+  was never exercised end-to-end against the externalized shape until
+  this PR).
 
 ## What's new
 
-- A new lint test `tests/unit/no-empty-catch-blocks.test.ts` that
-  scans for truly empty `catch{}` blocks and fails on any new ones.
-- The seven existing bare `catch{}` sites are annotated in this same
-  PR — five in `src/paste/PasteManager.ts` (file cleanup that's
-  genuinely safe to silently fail) and three elsewhere. Each carries
-  a one-line rationale.
-- A focused regression check on `src/core/PromptGate.ts` — the file
-  that gave the post-mortem its poster-child incident. It can never
-  silently regress.
-- The lint's ratchet baseline starts at zero. New bare `catch{}` in
-  unannotated form fails the unit suite at commit time.
+- A new directory `tests/real-world-state/`.
+- A small framework helper (`_framework.ts`) with two things:
+  - A two-tier system. Small/fast fixtures run on every PR. Big/slow
+    fixtures (multi-100MB DBs, concurrency at scale) are gated on
+    `INSTAR_REAL_WORLD_BIG=1` env, default off, so CI cost stays
+    bounded.
+  - A `makeAgentFixture()` helper that gives each test a real-shape
+    on-disk agent home (`projectDir` + `.instar/`) for setting up the
+    scenario.
+- The first scenario: `externalized-config-boot.test.ts`. Five tests
+  that target the #542 incident class — making sure `loadConfig()`
+  correctly merges the real authToken back from the secret store when
+  the on-disk config holds the placeholder. This is the in-process
+  Node side of the bug; the existing PR #542 tests cover only the
+  shell-script side.
+- Vitest config updated to include the new directory.
 
 ## What you need to decide
 
-Nothing. Lint-only. No runtime change, no config, no fleet migration.
+Nothing. Test-only change. No runtime code modified. CI cost negligible
+(the PR-tier fixture is tiny; nightly tier is opt-in).
 
 ## How to verify it worked after deploy
 
-Try writing a function with a bare `catch{}` in `src/`. Stage it. The
-pre-commit gate's test step will fail with a clear message naming
-PromptGate and the post-mortem. Fix: add `// @silent-fallback-ok —
-<why>` above the catch, OR put a real body inside it.
+In CI, the new tests will appear in the unit-shard results as
+`[real-world-state:pr] externalized-config-boot — ...`. The
+`[real-world-state:nightly]` blocks (when added in future PRs) will
+appear as "skipped (set INSTAR_REAL_WORLD_BIG=1 to run)".
+
+Locally: `npm test` runs the PR tier. `INSTAR_REAL_WORLD_BIG=1 npm test`
+runs everything.
 
 ## Why this matters more than it might look
 
-This is the fifth post-mortem PR landing in roughly four hours. Each
-one closes a recurring bug class, not just one incident:
-
-- #542 — silent-403 on secret-externalization (config-reader didn't
-  survive the security upgrade).
-- #545 — failure-learning loop wiring + migration-parity tests.
-- #550 — failure-learning git reads opt into the source-tree read
-  escape hatch.
-- #551 — `gh pr merge` refused on red checks (the watch-exit-merge
-  class).
-- THIS — bare `catch {}` refused in production paths.
-
-Each of these patterns has cost real time + real money. Closing them
-at the structural level (a lint, a test, a hook) is much cheaper than
-relying on "we'll be careful next time."
+This is the framework for catching the broader class. Future PRs will
+add scenarios that target the other patterns I named in the
+post-mortem:
+
+- Multi-100MB token-ledger.db boot (catches the #534 class).
+- Wrong-ABI better-sqlite3 binary swap (catches #539).
+- Concurrent-job restart-during-tick (catches the class behind
+  several silent-stop incidents).
+
+Each future scenario follows the same `describeAtTier(...)` shape as
+the first one. Adding the next one is small (~30 minutes of test
+writing); designing the framework was the hard part and it's done.
+
+This is the LAST recommended fix from the post-mortem. Six PRs in <7
+hours total. The fix-rate of 19% over the last 14 days won't drop
+overnight, but each of these closes a class structurally — which means
+the rate drops the next time someone tries to ship one of those
+patterns again, not because they remember to be careful but because
+the lint or the test or the gate refuses to let them.

package/upgrades/side-effects/real-world-state-fixture-framework.md

@@ -0,0 +1,137 @@
+# Side-effects review — Real-World-State fixture framework + first scenario (post-mortem lever B)
+
+## What changed
+
+A new test category `tests/real-world-state/` for scenarios that exercise
+instar against state that LOOKS LIKE a real production agent — not the
+small fresh-fixture state the rest of the suite uses.
+
+- **`tests/real-world-state/_framework.ts`** — fixture-loader helpers,
+  tier system, agent-fixture builder.
+- **`tests/real-world-state/externalized-config-boot.test.ts`** — the
+  first scenario (5 tests). Targets the post-mortem's #542 incident
+  class: `loadConfig()` MUST merge the externalized authToken back from
+  the secret store, or every downstream consumer that doesn't go
+  through `loadConfig()` silently 403s.
+- **`vitest.config.ts`** — the new directory wired into the
+  `include` array (alongside unit / integration / e2e).
+
+The framework introduces a **two-tier system**:
+
+- **'pr' tier** — small fixtures (< ~5 MB, < ~30s setup). Runs on
+  every PR / every CI shard. The first scenario lives here.
+- **'nightly' tier** — large fixtures (multi-100MB DBs, generated
+  JSONL volumes, environment-specific shapes like wrong-ABI binaries).
+  Gated on `INSTAR_REAL_WORLD_BIG=1` env. Default OFF; default behavior
+  is `describe.skip` with a clear "skipped (set ... to run)" message so
+  the coverage gap is visible, not silently absent.
+
+Helper API:
+
+```ts
+import { describeAtTier, makeAgentFixture } from './_framework.js';
+
+describeAtTier('pr', 'scenario name', () => {
+  let fx;
+  afterEach(() => fx?.cleanup());
+  it('asserts X about Y', () => {
+    fx = makeAgentFixture();
+    // fx.projectDir + fx.stateDir are a real on-disk agent home shape
+    // … set up scenario, run code-under-test, assert
+  });
+});
+```
+
+## Why
+
+Per the 2026-05-29 pipeline post-mortem (PR #545), pattern #1 — "Tested
+on fresh state, not real-world state" — was the **largest** of the five
+named bug classes. Bugs in this class shipped in the last 2 weeks:
+- #534 (TokenLedger backfill bricked boot at 202MB; fixture was tiny).
+- #512 (TokenLedger schema-order died on pre-existing DB but passed on
+  fresh).
+- #509 (heal-then-heal failed; heal-once passed).
+- #501 (SleepWakeDetector false-wake under load).
+- #503 (PresenceProxy redundant after ack).
+- And the originator of this whole arc — #542 (silent-403 secret
+  externalization). The in-process Node side's `loadConfig()` wasn't
+  exercised against the externalized shape; only the SHELL hook/script
+  side was, by PR #542's tests.
+
+Lever B is the only post-mortem lever that closes the broader pattern
+rather than a specific incident shape. The other four (#542, #545,
+#550, #551, #552) each closed individual classes; this one builds the
+infrastructure to catch the class FAMILY going forward.
+
+## Risk surface
+
+- **Test-only change.** No runtime code modified. Existing test suites
+  unchanged.
+- **vitest.config.ts include list extended** — additive only; the three
+  existing entries (unit / integration / e2e) remain unchanged. CI
+  shards will pick up the new path automatically.
+- **PR-tier fixture is tiny.** The externalized-config-boot scenario
+  writes a ~1KB JSON file and runs the SecretMigrator + SecretStore
+  paths. Test runtime: ~200ms. CI impact: negligible.
+- **Nightly tier defaults OFF.** No big-fixture cost on PR runs until
+  someone deliberately opts in. The skip is loud (`describe.skip` with
+  a message), not silent — the coverage gap is visible in every CI
+  report.
+- **No fixture generation in this PR.** The first scenario uses
+  in-test fixture builders (write JSON file, run SecretMigrator). When
+  the nightly tier ships its first scenario (multi-100MB DB → catches
+  #534), it will need a `scripts/build-real-world-fixtures.mjs`
+  generator + a git-ignored `tests/fixtures/real-world-state/` cache.
+  Deferred to that PR.
+
+## Bug surfaces eliminated
+
+- A future regression in `mergeConfigWithSecrets` (or the call from
+  `Config.ts:loadConfig`) that re-introduces the placeholder-leak shape
+  fails the 5 new tests with clear messages. Verified destructive-
+  negative: disabling the merge call in `Config.ts` trips 4 of the 5
+  scenario tests with the exact failure modes the bug class produced.
+- The full-shape sub-test asserts that telegram token, telegram chatId,
+  dashboard PIN, AND tunnel token all merge — not just authToken. A
+  future externalization that adds a new secret field without wiring
+  the merge fails immediately.
+- The idempotency sub-test asserts that re-running pairing on an
+  already-externalized agent doesn't double-extract or lose state.
+
+## Testing
+
+- **5 tests, all green** on the current main:
+  - Disk shape sanity (post-pairing layout).
+  - `loadConfig()` returns merged real authToken (the headline check).
+  - `loadConfig()` never leaks the placeholder shape (regression).
+  - Full-shape merge (telegram + dashboard + tunnel + auth).
+  - Idempotency on re-run.
+- **Tier-gating verified both directions**. With `INSTAR_REAL_WORLD_BIG`
+  unset, `nightly`-tier blocks skip with a visible message; with it set
+  to `1`, they execute. Verified with a sentinel test that
+  intentionally fails — proves the gating respects the env.
+- **Destructive-negative verified**. Disabling
+  `mergeConfigWithSecrets()` in `Config.ts` fails 4 of 5 scenario tests
+  with the failure messages the bug class produced
+  (`{ secret: true }` returned as authToken, telegram token leaks the
+  placeholder, etc.).
+- **No regression in existing related tests** —
+  `secret-migrator.test.ts` (40), `secret-store.test.ts` (presumed
+  green), `config-secret-merge.test.ts` (4) all still pass.
+- `tsc --noEmit` clean.
+
+## Follow-ups
+
+- **First nightly-tier scenario** — multi-100MB token-ledger.db boot
+  test that would have caught #534. Needs a fixture generator
+  (`scripts/build-real-world-fixtures.mjs`) and a git-ignored cache
+  (`tests/fixtures/real-world-state/`). Separate PR.
+- **Wrong-ABI better-sqlite3 scenario** — catches #539's class. Needs
+  the framework to support "environment fixtures" (specific Node + ABI
+  combos), which is a bigger design step.
+- **Concurrency-at-scale scenarios** — multi-job + multi-session under
+  restart-during-tick. Bigger design step (probably needs a tmux
+  harness).
+
+This PR ships the FRAMEWORK + ONE scenario as a working backstop.
+Subsequent PRs add scenarios at the same incremental pace.

package/upgrades/side-effects/stopgate-cli-circuit-breaker.md

@@ -0,0 +1,62 @@
+# Side-effects review — UnjustifiedStopGate circuit breaker
+
+**Spec:** `docs/specs/STOPGATE-CLI-CIRCUIT-BREAKER-SPEC.md`
+**Change:** a self-circuit-breaker on the stop gate's LLM path so subscription
+agents (where a `claude -p` judgment call takes ~5-6s but the budget is ~2s) stop
+churning doomed subprocesses and flooding /health with per-stop degradations.
+**Class:** robustness fix (found by the 12h session's "load is suspect" investigation).
+
+## What changed
+
+- **`src/core/UnjustifiedStopGate.ts`** — new config `breakerThreshold` (default 3),
+  `breakerCooldownMs` (default 5 min), injectable `now`. New `breakerOpen` failure
+  kind + `breakerState()` telemetry. `evaluate()` short-circuits (no LLM call) while
+  the breaker is open; counts provider failures (timeout/llmUnavailable); resets on
+  a reachable provider.
+- **`src/core/StopGateDb.ts`** — `InvalidKind` gains `breakerOpen`.
+- **`src/server/routes.ts`** — `/internal/stop-gate/evaluate` fail-opens on
+  `breakerOpen` (as for any failure) but skips the failure DB record + the
+  DegradationReport for it (deliberate short-circuit, not an evaluation failure).
+
+## Blast radius
+
+- **Stop decisions:** unchanged. `breakerOpen` allows the stop exactly like a
+  `timeout` fail-open — the breaker can never make a stop decision worse, only the
+  fail-open faster + quieter. (This is what keeps the change low-risk despite the
+  stop gate being safety-adjacent.)
+- **Fast-provider (API-key) agents:** the breaker never opens (calls succeed) →
+  identical behaviour to today. Only slow/unavailable-provider agents see the
+  short-circuit.
+- **Stop-gate analytics:** `breakerOpen` is excluded from the failure rollup (it
+  is not an evaluation failure), so the gate's failure-rate metric is no longer
+  skewed by the very churn the breaker stops.
+- **/health:** stops flooding — at most `breakerThreshold` degradations per
+  cooldown window instead of one per stop event.
+- **Config / schema / migration:** none. Safe in-code defaults; no PostUpdateMigrator
+  entry. Operators may pass `breakerThreshold`/`breakerCooldownMs`/`clientTimeoutMs`
+  via the gate constructor if they want to tune it.
+
+## What could break (and why it doesn't)
+
+- **Breaker stuck open forever?** No — it half-opens after the cooldown and a
+  single reachable response closes it. A genuinely recovered provider self-heals.
+- **A transient blip opens it?** Only after `breakerThreshold` (3) CONSECUTIVE
+  failures; a single success in between resets the counter.
+- **Lost audit of fail-opens?** Real evaluation failures (timeout, malformed, etc.)
+  are still recorded + reported exactly as before; only the deliberate breaker
+  short-circuit is excluded.
+
+## Security
+
+No new external input / network / auth / fs surface. Pure in-memory breaker state.
+
+## Rollback
+
+Revert the commit, or set `breakerThreshold: 0` (disables the breaker). No state.
+
+## Tests
+
+`tests/unit/UnjustifiedStopGate-breaker.test.ts` (+5): opens after K + stops
+calling the provider; half-open retry after cooldown; reachable-provider reset;
+threshold=0 disables; real-timeout counts. Existing gate/route/db suites (42) green.
+`tsc` + lint clean.

package/upgrades/side-effects/throttle-settled-detection.md

@@ -0,0 +1,70 @@
+# Side-Effects Review — settled-throttle detection (RateLimitSentinel actually fires)
+
+**Version / slug:** `throttle-settled-detection`
+**Date:** `2026-05-30`
+**Author:** `echo`
+**Driving spec:** `docs/specs/rate-limit-sentinel.md` (§Settled-throttle detection — the live fix)
+**Second-pass reviewer:** `not required` (single-author behavioral fix to an already-shipped sentinel; no new external surface)
+
+## Summary of the change
+
+The RateLimitSentinel (shipped PR #528, live at v1.3.108) was built, wired, and
+enabled but fired **zero** times in the field — throttled sessions hung 5–10
+minutes until the 15-minute silence fallback. Root cause was the detection
+preconditions in `SessionWatchdog.checkRateLimited`, not the recovery machinery:
+it required a session be "cleanly idle, no active child processes, at a prompt,
+throttle string in the last 20 lines." Busy dev sessions never satisfy that —
+they have a live background shell/MCP process, and Claude Code's input box pushes
+the `API Error:` line out of the 20-line window.
+
+The fix replaces those gates with a **settled-output signal**: throttle string in
+a widened 45-line window + pane byte-identical across two consecutive watchdog
+polls (an actively-working session animates its spinner/elapsed-timer every tick,
+so a frozen pane is a reliable "turn ended, stuck" signal). No process-tree
+inspection, no at-prompt heuristic. After a recovery cycle escalates and clears
+(~30s), a still-stuck pane re-emits past the 60s cooldown → **unbounded retry**
+until the throttle lifts (the "never hang forever" guarantee Justin asked for).
+
+## Files touched
+
+- `src/monitoring/rateLimitDetection.ts` — `detectRateLimited` gains an optional
+  `captureLines` param (default 20, unchanged for existing callers); new
+  `RATE_LIMIT_SETTLED_CAPTURE_LINES=45`, `RATE_LIMIT_DEFAULT_SETTLE_MS=20000`,
+  `throttleSignature()`, and the pure clock-injected `evaluateThrottleSettle()`.
+- `src/monitoring/SessionWatchdog.ts` — `checkRateLimited` rewritten to the
+  settled-output decision; removed the active-child-process and at-prompt gates;
+  added `throttleSettle` tracking map + `rateLimitSettleMs` (config-read).
+- `src/core/types.ts` — `monitoring.watchdog.rateLimitSettleMs?: number`.
+- `src/commands/server.ts` — wire RateLimitSentinel lifecycle events
+  (detected/resuming/recovered/escalated) into `logs/sentinel-events.jsonl`.
+- `tests/unit/rate-limit-detection.test.ts` — +16 cases (widened-window
+  regression reproducing the incident; throttleSignature; evaluateThrottleSettle
+  both sides of every branch).
+- `tests/unit/SessionWatchdog-rate-limit-settle.test.ts` — new, 8 wiring cases.
+- `upgrades/NEXT.md`, `docs/specs/rate-limit-sentinel.md` (+eli16) — docs.
+
+## Decision-point inventory
+
+- **Stuck-vs-working discriminator** — *modify*. Was process-tree + at-prompt +
+  20-line window; is now byte-identical-pane across polls + 45-line window. Pure
+  function (`evaluateThrottleSettle`), no judgment, fully unit-tested.
+- **Widened capture window (45 lines)** — *new, bounded*. Only the watchdog
+  settled path uses it; `detectRateLimited`'s default stays 20 for every other
+  caller (SessionManager idle path unchanged). Old-throttle-scrolled-away and
+  usage-limit/mid-retry exclusions still hold at 45 lines (tested).
+- **Unbounded re-emit after escalation** — *behavioral, intentional*. Bounded by
+  the 60s watchdog cooldown + the sentinel's own dedupe; the throttle is
+  Anthropic's short-lived shared-capacity signal, so cycles are few in practice.
+  This is the load-bearing "never hang forever" property.
+- **Audit-log wiring** — *additive observability*. Best-effort `appendFileSync`
+  to the existing sentinel-events.jsonl; annotated `@silent-fallback-ok` (never
+  crashes the monitor path). No new route, no new external surface.
+- **`rateLimitSettleMs` config** — *additive, back-compat*. Optional, code
+  default 20000; configs without it are unaffected → no migration required.
+
+## Blast radius / rollback
+
+Pure monitoring-path change; no API routes, no schema, no state-file format
+change. The `RateLimitSentinel.enabled` master switch (existing) disables the
+whole path. Reverting this commit restores the prior gates verbatim. Existing
+121 rate-limit/watchdog tests pass unchanged (event contract preserved).