instar 1.3.110 → 1.3.111

package/upgrades/side-effects/real-world-state-fixture-framework.md

@@ -0,0 +1,137 @@
+# Side-effects review — Real-World-State fixture framework + first scenario (post-mortem lever B)
+
+## What changed
+
+A new test category `tests/real-world-state/` for scenarios that exercise
+instar against state that LOOKS LIKE a real production agent — not the
+small fresh-fixture state the rest of the suite uses.
+
+- **`tests/real-world-state/_framework.ts`** — fixture-loader helpers,
+  tier system, agent-fixture builder.
+- **`tests/real-world-state/externalized-config-boot.test.ts`** — the
+  first scenario (5 tests). Targets the post-mortem's #542 incident
+  class: `loadConfig()` MUST merge the externalized authToken back from
+  the secret store, or every downstream consumer that doesn't go
+  through `loadConfig()` silently 403s.
+- **`vitest.config.ts`** — the new directory wired into the
+  `include` array (alongside unit / integration / e2e).
+
+The framework introduces a **two-tier system**:
+
+- **'pr' tier** — small fixtures (< ~5 MB, < ~30s setup). Runs on
+  every PR / every CI shard. The first scenario lives here.
+- **'nightly' tier** — large fixtures (multi-100MB DBs, generated
+  JSONL volumes, environment-specific shapes like wrong-ABI binaries).
+  Gated on `INSTAR_REAL_WORLD_BIG=1` env. Default OFF; default behavior
+  is `describe.skip` with a clear "skipped (set ... to run)" message so
+  the coverage gap is visible, not silently absent.
+
+Helper API:
+
+```ts
+import { describeAtTier, makeAgentFixture } from './_framework.js';
+
+describeAtTier('pr', 'scenario name', () => {
+  let fx;
+  afterEach(() => fx?.cleanup());
+  it('asserts X about Y', () => {
+    fx = makeAgentFixture();
+    // fx.projectDir + fx.stateDir are a real on-disk agent home shape
+    // … set up scenario, run code-under-test, assert
+  });
+});
+```
+
+## Why
+
+Per the 2026-05-29 pipeline post-mortem (PR #545), pattern #1 — "Tested
+on fresh state, not real-world state" — was the **largest** of the five
+named bug classes. Bugs in this class shipped in the last 2 weeks:
+- #534 (TokenLedger backfill bricked boot at 202MB; fixture was tiny).
+- #512 (TokenLedger schema-order died on pre-existing DB but passed on
+  fresh).
+- #509 (heal-then-heal failed; heal-once passed).
+- #501 (SleepWakeDetector false-wake under load).
+- #503 (PresenceProxy redundant after ack).
+- And the originator of this whole arc — #542 (silent-403 secret
+  externalization). The in-process Node side's `loadConfig()` wasn't
+  exercised against the externalized shape; only the SHELL hook/script
+  side was, by PR #542's tests.
+
+Lever B is the only post-mortem lever that closes the broader pattern
+rather than a specific incident shape. The other four (#542, #545,
+#550, #551, #552) each closed individual classes; this one builds the
+infrastructure to catch the class FAMILY going forward.
+
+## Risk surface
+
+- **Test-only change.** No runtime code modified. Existing test suites
+  unchanged.
+- **vitest.config.ts include list extended** — additive only; the three
+  existing entries (unit / integration / e2e) remain unchanged. CI
+  shards will pick up the new path automatically.
+- **PR-tier fixture is tiny.** The externalized-config-boot scenario
+  writes a ~1KB JSON file and runs the SecretMigrator + SecretStore
+  paths. Test runtime: ~200ms. CI impact: negligible.
+- **Nightly tier defaults OFF.** No big-fixture cost on PR runs until
+  someone deliberately opts in. The skip is loud (`describe.skip` with
+  a message), not silent — the coverage gap is visible in every CI
+  report.
+- **No fixture generation in this PR.** The first scenario uses
+  in-test fixture builders (write JSON file, run SecretMigrator). When
+  the nightly tier ships its first scenario (multi-100MB DB → catches
+  #534), it will need a `scripts/build-real-world-fixtures.mjs`
+  generator + a git-ignored `tests/fixtures/real-world-state/` cache.
+  Deferred to that PR.
+
+## Bug surfaces eliminated
+
+- A future regression in `mergeConfigWithSecrets` (or the call from
+  `Config.ts:loadConfig`) that re-introduces the placeholder-leak shape
+  fails the 5 new tests with clear messages. Verified destructive-
+  negative: disabling the merge call in `Config.ts` trips 4 of the 5
+  scenario tests with the exact failure modes the bug class produced.
+- The full-shape sub-test asserts that telegram token, telegram chatId,
+  dashboard PIN, AND tunnel token all merge — not just authToken. A
+  future externalization that adds a new secret field without wiring
+  the merge fails immediately.
+- The idempotency sub-test asserts that re-running pairing on an
+  already-externalized agent doesn't double-extract or lose state.
+
+## Testing
+
+- **5 tests, all green** on the current main:
+  - Disk shape sanity (post-pairing layout).
+  - `loadConfig()` returns merged real authToken (the headline check).
+  - `loadConfig()` never leaks the placeholder shape (regression).
+  - Full-shape merge (telegram + dashboard + tunnel + auth).
+  - Idempotency on re-run.
+- **Tier-gating verified both directions**. With `INSTAR_REAL_WORLD_BIG`
+  unset, `nightly`-tier blocks skip with a visible message; with it set
+  to `1`, they execute. Verified with a sentinel test that
+  intentionally fails — proves the gating respects the env.
+- **Destructive-negative verified**. Disabling
+  `mergeConfigWithSecrets()` in `Config.ts` fails 4 of 5 scenario tests
+  with the failure messages the bug class produced
+  (`{ secret: true }` returned as authToken, telegram token leaks the
+  placeholder, etc.).
+- **No regression in existing related tests** —
+  `secret-migrator.test.ts` (40), `secret-store.test.ts` (presumed
+  green), `config-secret-merge.test.ts` (4) all still pass.
+- `tsc --noEmit` clean.
+
+## Follow-ups
+
+- **First nightly-tier scenario** — multi-100MB token-ledger.db boot
+  test that would have caught #534. Needs a fixture generator
+  (`scripts/build-real-world-fixtures.mjs`) and a git-ignored cache
+  (`tests/fixtures/real-world-state/`). Separate PR.
+- **Wrong-ABI better-sqlite3 scenario** — catches #539's class. Needs
+  the framework to support "environment fixtures" (specific Node + ABI
+  combos), which is a bigger design step.
+- **Concurrency-at-scale scenarios** — multi-job + multi-session under
+  restart-during-tick. Bigger design step (probably needs a tmux
+  harness).
+
+This PR ships the FRAMEWORK + ONE scenario as a working backstop.
+Subsequent PRs add scenarios at the same incremental pace.

package/upgrades/side-effects/throttle-settled-detection.md

@@ -0,0 +1,70 @@
+# Side-Effects Review — settled-throttle detection (RateLimitSentinel actually fires)
+
+**Version / slug:** `throttle-settled-detection`
+**Date:** `2026-05-30`
+**Author:** `echo`
+**Driving spec:** `docs/specs/rate-limit-sentinel.md` (§Settled-throttle detection — the live fix)
+**Second-pass reviewer:** `not required` (single-author behavioral fix to an already-shipped sentinel; no new external surface)
+
+## Summary of the change
+
+The RateLimitSentinel (shipped PR #528, live at v1.3.108) was built, wired, and
+enabled but fired **zero** times in the field — throttled sessions hung 5–10
+minutes until the 15-minute silence fallback. Root cause was the detection
+preconditions in `SessionWatchdog.checkRateLimited`, not the recovery machinery:
+it required a session be "cleanly idle, no active child processes, at a prompt,
+throttle string in the last 20 lines." Busy dev sessions never satisfy that —
+they have a live background shell/MCP process, and Claude Code's input box pushes
+the `API Error:` line out of the 20-line window.
+
+The fix replaces those gates with a **settled-output signal**: throttle string in
+a widened 45-line window + pane byte-identical across two consecutive watchdog
+polls (an actively-working session animates its spinner/elapsed-timer every tick,
+so a frozen pane is a reliable "turn ended, stuck" signal). No process-tree
+inspection, no at-prompt heuristic. After a recovery cycle escalates and clears
+(~30s), a still-stuck pane re-emits past the 60s cooldown → **unbounded retry**
+until the throttle lifts (the "never hang forever" guarantee Justin asked for).
+
+## Files touched
+
+- `src/monitoring/rateLimitDetection.ts` — `detectRateLimited` gains an optional
+  `captureLines` param (default 20, unchanged for existing callers); new
+  `RATE_LIMIT_SETTLED_CAPTURE_LINES=45`, `RATE_LIMIT_DEFAULT_SETTLE_MS=20000`,
+  `throttleSignature()`, and the pure clock-injected `evaluateThrottleSettle()`.
+- `src/monitoring/SessionWatchdog.ts` — `checkRateLimited` rewritten to the
+  settled-output decision; removed the active-child-process and at-prompt gates;
+  added `throttleSettle` tracking map + `rateLimitSettleMs` (config-read).
+- `src/core/types.ts` — `monitoring.watchdog.rateLimitSettleMs?: number`.
+- `src/commands/server.ts` — wire RateLimitSentinel lifecycle events
+  (detected/resuming/recovered/escalated) into `logs/sentinel-events.jsonl`.
+- `tests/unit/rate-limit-detection.test.ts` — +16 cases (widened-window
+  regression reproducing the incident; throttleSignature; evaluateThrottleSettle
+  both sides of every branch).
+- `tests/unit/SessionWatchdog-rate-limit-settle.test.ts` — new, 8 wiring cases.
+- `upgrades/NEXT.md`, `docs/specs/rate-limit-sentinel.md` (+eli16) — docs.
+
+## Decision-point inventory
+
+- **Stuck-vs-working discriminator** — *modify*. Was process-tree + at-prompt +
+  20-line window; is now byte-identical-pane across polls + 45-line window. Pure
+  function (`evaluateThrottleSettle`), no judgment, fully unit-tested.
+- **Widened capture window (45 lines)** — *new, bounded*. Only the watchdog
+  settled path uses it; `detectRateLimited`'s default stays 20 for every other
+  caller (SessionManager idle path unchanged). Old-throttle-scrolled-away and
+  usage-limit/mid-retry exclusions still hold at 45 lines (tested).
+- **Unbounded re-emit after escalation** — *behavioral, intentional*. Bounded by
+  the 60s watchdog cooldown + the sentinel's own dedupe; the throttle is
+  Anthropic's short-lived shared-capacity signal, so cycles are few in practice.
+  This is the load-bearing "never hang forever" property.
+- **Audit-log wiring** — *additive observability*. Best-effort `appendFileSync`
+  to the existing sentinel-events.jsonl; annotated `@silent-fallback-ok` (never
+  crashes the monitor path). No new route, no new external surface.
+- **`rateLimitSettleMs` config** — *additive, back-compat*. Optional, code
+  default 20000; configs without it are unaffected → no migration required.
+
+## Blast radius / rollback
+
+Pure monitoring-path change; no API routes, no schema, no state-file format
+change. The `RateLimitSentinel.enabled` master switch (existing) disables the
+whole path. Reverting this commit restores the prior gates verbatim. Existing
+121 rate-limit/watchdog tests pass unchanged (event contract preserved).