instar 1.3.62 → 1.3.63

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (57) hide show
  1. package/dist/commands/server.d.ts.map +1 -1
  2. package/dist/commands/server.js +132 -8
  3. package/dist/commands/server.js.map +1 -1
  4. package/dist/config/ConfigDefaults.d.ts.map +1 -1
  5. package/dist/config/ConfigDefaults.js +20 -0
  6. package/dist/config/ConfigDefaults.js.map +1 -1
  7. package/dist/core/PostUpdateMigrator.d.ts.map +1 -1
  8. package/dist/core/PostUpdateMigrator.js +22 -0
  9. package/dist/core/PostUpdateMigrator.js.map +1 -1
  10. package/dist/core/ReapGuard.d.ts +79 -0
  11. package/dist/core/ReapGuard.d.ts.map +1 -0
  12. package/dist/core/ReapGuard.js +114 -0
  13. package/dist/core/ReapGuard.js.map +1 -0
  14. package/dist/core/SessionLivenessOracle.d.ts +104 -0
  15. package/dist/core/SessionLivenessOracle.d.ts.map +1 -0
  16. package/dist/core/SessionLivenessOracle.js +201 -0
  17. package/dist/core/SessionLivenessOracle.js.map +1 -0
  18. package/dist/core/SessionManager.d.ts +90 -10
  19. package/dist/core/SessionManager.d.ts.map +1 -1
  20. package/dist/core/SessionManager.js +200 -40
  21. package/dist/core/SessionManager.js.map +1 -1
  22. package/dist/core/types.d.ts +30 -0
  23. package/dist/core/types.d.ts.map +1 -1
  24. package/dist/core/types.js.map +1 -1
  25. package/dist/monitoring/ReapLog.d.ts +51 -0
  26. package/dist/monitoring/ReapLog.d.ts.map +1 -0
  27. package/dist/monitoring/ReapLog.js +82 -0
  28. package/dist/monitoring/ReapLog.js.map +1 -0
  29. package/dist/monitoring/ReapNotifier.d.ts +66 -0
  30. package/dist/monitoring/ReapNotifier.d.ts.map +1 -0
  31. package/dist/monitoring/ReapNotifier.js +112 -0
  32. package/dist/monitoring/ReapNotifier.js.map +1 -0
  33. package/dist/monitoring/SessionReaper.d.ts +4 -0
  34. package/dist/monitoring/SessionReaper.d.ts.map +1 -1
  35. package/dist/monitoring/SessionReaper.js +19 -42
  36. package/dist/monitoring/SessionReaper.js.map +1 -1
  37. package/dist/monitoring/StaleSessionBackstop.d.ts +89 -0
  38. package/dist/monitoring/StaleSessionBackstop.d.ts.map +1 -0
  39. package/dist/monitoring/StaleSessionBackstop.js +176 -0
  40. package/dist/monitoring/StaleSessionBackstop.js.map +1 -0
  41. package/dist/scaffold/templates.d.ts.map +1 -1
  42. package/dist/scaffold/templates.js +5 -0
  43. package/dist/scaffold/templates.js.map +1 -1
  44. package/dist/server/AgentServer.d.ts +3 -0
  45. package/dist/server/AgentServer.d.ts.map +1 -1
  46. package/dist/server/AgentServer.js +1 -0
  47. package/dist/server/AgentServer.js.map +1 -1
  48. package/dist/server/routes.d.ts +1 -0
  49. package/dist/server/routes.d.ts.map +1 -1
  50. package/dist/server/routes.js +33 -11
  51. package/dist/server/routes.js.map +1 -1
  52. package/package.json +1 -1
  53. package/src/data/builtin-manifest.json +64 -64
  54. package/src/scaffold/templates.ts +5 -0
  55. package/upgrades/1.3.62.md +43 -45
  56. package/upgrades/1.3.63.md +87 -0
  57. package/upgrades/side-effects/unified-session-lifecycle-robustness.md +241 -0
@@ -4,6 +4,8 @@
4
4
 
5
5
  ## What Changed
6
6
 
7
+ ### ContextWedgeSentinel — detect + (optionally) recover the thinking-block-400 wedge
8
+
7
9
  **A new way sessions die is now detected and (optionally) auto-recovered.** When
8
10
  a tool call is cancelled inside a *parallel* tool batch while extended thinking
9
11
  is on, Claude Code corrupts the thinking block on the latest assistant turn —
@@ -33,57 +35,53 @@ and rides the Graduated Feature Rollout track. The sentinel also feeds the
33
35
  SessionReaper kill-veto so the reaper never reaps a session mid-recovery. New
34
36
  config kill switch `monitoring.contextWedgeSentinel.enabled` (default true).
35
37
 
38
+ ### Unified session-lifecycle robustness — Phase 1 (ReapAuthority + reap-log + backstop)
39
+
40
+ Phase 1 of **unified session-lifecycle robustness** (spec `docs/specs/unified-session-lifecycle-robustness.md`, converged + approved). The trigger was a real incident: at a busy server boot the startup "cleanup crew" treated a slow-to-answer tmux session as *dead* and wiped 9 of 9 live sessions — silently. The audit that followed found eight different code paths that can end a session, each re-inventing "alive or dead?" its own way, several of them able to mistake slow/busy for dead and none of them telling the user. This release builds the single authority + safety backbone that the rest of the killers will move onto.
41
+
42
+ What landed:
43
+
44
+ - **One authority, not eight.** `terminateSession()` is now the sole kill chokepoint. Every autonomous shutoff routes through it and can only *request* a kill — the authority holds the safety checks: a compare-and-set live-status guard, the `protected` set, an **awake-machine-only lease gate** (a standby never reaps another machine's sessions), and a mandatory **KEEP-guard** consult (the same positive-evidence checks the careful SessionReaper uses — relay-lease, pending-injection, recent-user-message, active subagent/process, etc.). An explicit **operator** kill (stamped only by the Bearer-authed HTTP route) bypasses those gates so a human "kill" always happens. The first three killers — boot purge, the age/5-hour cutoff, and the idle-zombie killer — are funneled through it.
45
+ - **A reap is never silent.** A new coalescing notifier surfaces "your session was shut down — <reason>" so a session can't just vanish. Routine kill-to-respawn bounces and your own operator kills stay quiet; a burst collapses into ONE consolidated message instead of spamming. Default **on** (`monitoring.reapNotify`).
46
+ - **A reap-log you can read.** Every shutoff *and* every refused/skipped shutoff (protected / not-lease-holder / a KEEP-hold / in-flight) is recorded as one JSON line, served read-only at `GET /sessions/reap-log`. "Where did my session go?" now always has an answer.
47
+ - **Nothing is unconditionally immortal — and nothing can lock you out.** A signal-only backstop watches for a session that fakes work (heartbeat-byte / tight-loop) or is stuck unverifiable, and after a threshold raises ONE deduped Attention item asking *you* to decide — it never auto-kills. Long-unverifiable sessions are excluded from the absolute spawn cap so a fleet of stuck panes can't lock you out of starting new work.
48
+
49
+ All changes are additive and only make killing *more* conservative; the boot-purge fix already turns the original 9-of-9 wipe into 0 false kills. Phase 2 (watchdog / orphan / recovery / wake-reaper onto the authority) and Phase 3 (quota soft-check + session-label-follows-rename) follow.
50
+
36
51
  ## What to Tell Your User
37
52
 
38
53
  - I can now spot a specific way a session silently dies — it gets stuck on a
39
54
  "can't modify the thinking block" error and fails instantly on every message
40
55
  while still looking busy — which used to leave you seeing only standby replies.
41
- - By default I just notice it and log it (nothing is killed). You can opt in to
42
- having me automatically restart a stuck session cleanly; when that's on I throw
43
- away the corrupted conversation and start fresh so it can't get re-stuck.
44
- - Nothing you see changes by default. If you ever want the auto-restart on, it's
45
- one config switch, and it's on a track that will nag toward on-by-default once
46
- it's proven itself.
56
+ By default I just notice and log it; you can opt in to having me restart a
57
+ stuck session cleanly (I throw away the corrupted conversation so it can't get
58
+ re-stuck).
59
+ - Your sessions also won't silently disappear anymore. The startup cleanup no
60
+ longer mistakes a slow-to-answer session for a dead one, and if anything does
61
+ get shut down you get told why — with a reap-log page recording every shutoff
62
+ and every refused shutoff.
63
+ - Every automatic cleanup now goes through one careful gatekeeper that refuses to
64
+ end a session that might be working, and only the "awake" machine ever reaps.
65
+ When you kill a session it still happens immediately.
66
+ - Nothing can get wedged-but-immortal or quietly fill your slots: instead of ever
67
+ auto-killing a stuck session, the agent raises a single "this looks stuck —
68
+ investigate or force-kill?" item for you.
47
69
 
48
- ## Evidence
70
+ ## Summary of New Capabilities
49
71
 
50
- **Reproduction (live, 2026-05-28).** The `echo-instar-exo` session (topic 13481,
51
- the `mm-proof-build` worktree) fired a parallel Bash batch; one call
52
- (`rm -rf …/echo…`) was blocked by the source-tree guard, which cancelled the
53
- whole batch. With extended thinking on, that corrupted the latest assistant
54
- turn's thinking block.
55
-
56
- **Observed before.** Every subsequent inbound Telegram message produced, in the
57
- tmux pane, an instant:
58
- ```
59
- ⎿ API Error: 400 messages.9.content.20: `thinking` or `redacted_thinking` blocks
60
- in the latest assistant message cannot be modified.
61
- ✻ Cooked for 0s
62
- ```
63
- The user sent "How is this looking?" twice over ~30 minutes and received only
64
- sentinel/standby replies — the session was a corpse, and the silence + socket
65
- sentinels never fired (output never went quiet; no disconnect string).
66
-
67
- **Observed after.** With ContextWedgeSentinel: the wedge is detected as the live
68
- tail and written to `logs/sentinel-events.jsonl` (`kind:detected` →
69
- `kind:escalated` in detect-only mode, or `kind:recovered` when autoRecovery is
70
- on). The e2e test (`tests/e2e/context-wedge-sentinel-lifecycle.test.ts`) drives
71
- the production assembly and reads those exact rows back from disk; the live
72
- recovery clears the topic's resume UUID so the respawn starts fresh (verified by
73
- `SessionRefresh.test.ts` asserting the kill→clear→respawn order). The original
74
- wedged session was recovered manually by killing it (the next message spawned
75
- clean), confirming the fresh-respawn approach before it was automated.
72
+ | Capability | How to Use |
73
+ |-----------|-----------|
74
+ | ContextWedgeSentinel (thinking-block-400 detection) | Automatic + default-on housekeeping; audits to `logs/sentinel-events.jsonl` |
75
+ | Opt-in wedge auto-recovery | `{"monitoring": {"contextWedgeSentinel": {"autoRecovery": {"enabled": true, "dryRun": false}}}}` (Graduated Rollout track) |
76
+ | SessionRefresh `fresh` mode | Internal — kill + respawn that clears the topic's resume UUID (no re-wedge loop) |
77
+ | Single ReapAuthority (all autonomous kills funnel through `terminateSession`) | Automatic — boot-purge, age-kill, and idle-zombie now route through it with the KEEP-guard + lease gate |
78
+ | Reap notice ("your session was shut down — <reason>") | On by default; silence it with `{"monitoring": {"reapNotify": {"enabled": false}}}` |
79
+ | Reap-log | `curl -H "Authorization: Bearer $AUTH" "http://localhost:4040/sessions/reap-log?limit=50"` (read-only) |
80
+ | Unkillability backstop (escalate, never auto-kill) | Automatic + signal-only; tune via `monitoring.staleBackstop` |
81
+ | Spawn-cap exclusion for long-unverifiable sessions | Automatic — they no longer count toward the absolute `maxSessions × 3` cap |
76
82
 
77
- ## Summary of New Capabilities
83
+ ## Evidence
84
+
85
+ **ContextWedgeSentinel.** Reproduced live 2026-05-28: a parallel Bash batch with one guard-blocked call corrupted the latest assistant thinking block; every subsequent message fast-failed with the `400 … cannot be modified` error while the silence + socket sentinels stayed quiet. After: the wedge is detected as the live tail and written to `logs/sentinel-events.jsonl`; the e2e test drives the production assembly and reads those rows back; the live recovery clears the topic's resume UUID so the respawn starts fresh.
78
86
 
79
- - **ContextWedgeSentinel** — detects the thinking-block-400 fast-fail wedge as a
80
- non-progressing session tail (45s confirm window); audits every transition to
81
- `logs/sentinel-events.jsonl`.
82
- - **SessionRefresh `fresh` mode** — kill + respawn that clears the topic's resume
83
- UUID, so recovery never reloads a corrupted transcript (no re-wedge loop).
84
- - **Opt-in auto-recovery** — `monitoring.contextWedgeSentinel.autoRecovery`
85
- (default off + dry-run), on the Graduated Feature Rollout track; promotion to
86
- default-on propagates fleet-wide on update with no migration.
87
- - Escalation (when a confirmed wedge isn't auto-recovered) routes through the
88
- existing tone-gated SentinelNotifier, gated by `sentinelTelegramEscalation`
89
- (default off, coalesced).
87
+ **Session-lifecycle Phase 1.** 123 unit + 10 integration + 3 e2e green: ReapGuard (15), ReapNotifier (10), ReapLog (6), StaleSessionBackstop (10), terminate-CAS (9), liveness-oracle (15), boot-purge death-spiral reproduction (the 2026-05-27 9-of-9 incident → 0 false purges), reap-log route (integration + e2e "feature is alive" 200-not-503), and a full event-path integration (autonomous reap → log + one notice; recovery-bounce/operator silent; relay-lease/standby/protected refused-and-survive). `tsc --noEmit` clean. Migration parity: `reapNotify` + `staleBackstop` defaults flow to existing agents via ConfigDefaults; the reap-log Agent-Awareness section ships in both the CLAUDE.md template and `migrateClaudeMd`. Spec converged over 3 code-grounded review rounds (`docs/specs/reports/unified-session-lifecycle-robustness-convergence.md`); side-effects review: `upgrades/side-effects/unified-session-lifecycle-robustness.md`. Honest scope note: the boot-purge false-purge and the notice/log/survival paths are reproduced at the unit + real-event-path + real-AgentServer tiers; a live "boot a server, slow tmux, watch a Telegram notice land" pass was not performed in this environment.
@@ -0,0 +1,87 @@
1
+ # Upgrade Guide — vNEXT
2
+
3
+ <!-- bump: minor -->
4
+
5
+ ## What Changed
6
+
7
+ ### ContextWedgeSentinel — detect + (optionally) recover the thinking-block-400 wedge
8
+
9
+ **A new way sessions die is now detected and (optionally) auto-recovered.** When
10
+ a tool call is cancelled inside a *parallel* tool batch while extended thinking
11
+ is on, Claude Code corrupts the thinking block on the latest assistant turn —
12
+ and from then on the Anthropic API rejects every resume with
13
+ `400 … thinking blocks in the latest assistant message cannot be modified`. The
14
+ session fast-fails instantly on every message ("Cooked for 0s"): permanently
15
+ dead, yet still emitting output. Because it's not *silent* and hasn't lost its
16
+ socket, neither the ActiveWorkSilenceSentinel nor the SocketDisconnectSentinel
17
+ catches it — the user just sees standby/sentinel replies forever while the real
18
+ session is a corpse. (Live incident: a session on a real topic fast-failed every
19
+ "How is this looking?" for ~30 minutes.)
20
+
21
+ The new **ContextWedgeSentinel** (a 4th member of the silently-stopped family)
22
+ recognizes this exact wedge: it matches the error as the *live session tail* and
23
+ waits a 45s confirm window so a session merely *discussing* the bug is never
24
+ flagged. A gentle nudge can't fix this one (re-engaging re-sends the corrupted
25
+ turn), so recovery is a **fresh respawn** — kill the session and clear the
26
+ topic's saved resume UUID so the bridge spawns a brand-new conversation instead
27
+ of `--resume`-ing the corrupted transcript (which would just re-wedge on the
28
+ next message — the central correctness property). This reuses the existing
29
+ rate-guarded `SessionRefresh` via a new `fresh` mode.
30
+
31
+ Detection + audit are **default-ON housekeeping** (they write to
32
+ `logs/sentinel-events.jsonl` and kill nothing). The destructive auto-respawn is
33
+ **opt-in** (`monitoring.contextWedgeSentinel.autoRecovery`, default off + dry-run)
34
+ and rides the Graduated Feature Rollout track. The sentinel also feeds the
35
+ SessionReaper kill-veto so the reaper never reaps a session mid-recovery. New
36
+ config kill switch `monitoring.contextWedgeSentinel.enabled` (default true).
37
+
38
+ ### Unified session-lifecycle robustness — Phase 1 (ReapAuthority + reap-log + backstop)
39
+
40
+ Phase 1 of **unified session-lifecycle robustness** (spec `docs/specs/unified-session-lifecycle-robustness.md`, converged + approved). The trigger was a real incident: at a busy server boot the startup "cleanup crew" treated a slow-to-answer tmux session as *dead* and wiped 9 of 9 live sessions — silently. The audit that followed found eight different code paths that can end a session, each re-inventing "alive or dead?" its own way, several of them able to mistake slow/busy for dead and none of them telling the user. This release builds the single authority + safety backbone that the rest of the killers will move onto.
41
+
42
+ What landed:
43
+
44
+ - **One authority, not eight.** `terminateSession()` is now the sole kill chokepoint. Every autonomous shutoff routes through it and can only *request* a kill — the authority holds the safety checks: a compare-and-set live-status guard, the `protected` set, an **awake-machine-only lease gate** (a standby never reaps another machine's sessions), and a mandatory **KEEP-guard** consult (the same positive-evidence checks the careful SessionReaper uses — relay-lease, pending-injection, recent-user-message, active subagent/process, etc.). An explicit **operator** kill (stamped only by the Bearer-authed HTTP route) bypasses those gates so a human "kill" always happens. The first three killers — boot purge, the age/5-hour cutoff, and the idle-zombie killer — are funneled through it.
45
+ - **A reap is never silent.** A new coalescing notifier surfaces "your session was shut down — <reason>" so a session can't just vanish. Routine kill-to-respawn bounces and your own operator kills stay quiet; a burst collapses into ONE consolidated message instead of spamming. Default **on** (`monitoring.reapNotify`).
46
+ - **A reap-log you can read.** Every shutoff *and* every refused/skipped shutoff (protected / not-lease-holder / a KEEP-hold / in-flight) is recorded as one JSON line, served read-only at `GET /sessions/reap-log`. "Where did my session go?" now always has an answer.
47
+ - **Nothing is unconditionally immortal — and nothing can lock you out.** A signal-only backstop watches for a session that fakes work (heartbeat-byte / tight-loop) or is stuck unverifiable, and after a threshold raises ONE deduped Attention item asking *you* to decide — it never auto-kills. Long-unverifiable sessions are excluded from the absolute spawn cap so a fleet of stuck panes can't lock you out of starting new work.
48
+
49
+ All changes are additive and only make killing *more* conservative; the boot-purge fix already turns the original 9-of-9 wipe into 0 false kills. Phase 2 (watchdog / orphan / recovery / wake-reaper onto the authority) and Phase 3 (quota soft-check + session-label-follows-rename) follow.
50
+
51
+ ## What to Tell Your User
52
+
53
+ - I can now spot a specific way a session silently dies — it gets stuck on a
54
+ "can't modify the thinking block" error and fails instantly on every message
55
+ while still looking busy — which used to leave you seeing only standby replies.
56
+ By default I just notice and log it; you can opt in to having me restart a
57
+ stuck session cleanly (I throw away the corrupted conversation so it can't get
58
+ re-stuck).
59
+ - Your sessions also won't silently disappear anymore. The startup cleanup no
60
+ longer mistakes a slow-to-answer session for a dead one, and if anything does
61
+ get shut down you get told why — with a reap-log page recording every shutoff
62
+ and every refused shutoff.
63
+ - Every automatic cleanup now goes through one careful gatekeeper that refuses to
64
+ end a session that might be working, and only the "awake" machine ever reaps.
65
+ When you kill a session it still happens immediately.
66
+ - Nothing can get wedged-but-immortal or quietly fill your slots: instead of ever
67
+ auto-killing a stuck session, the agent raises a single "this looks stuck —
68
+ investigate or force-kill?" item for you.
69
+
70
+ ## Summary of New Capabilities
71
+
72
+ | Capability | How to Use |
73
+ |-----------|-----------|
74
+ | ContextWedgeSentinel (thinking-block-400 detection) | Automatic + default-on housekeeping; audits to `logs/sentinel-events.jsonl` |
75
+ | Opt-in wedge auto-recovery | `{"monitoring": {"contextWedgeSentinel": {"autoRecovery": {"enabled": true, "dryRun": false}}}}` (Graduated Rollout track) |
76
+ | SessionRefresh `fresh` mode | Internal — kill + respawn that clears the topic's resume UUID (no re-wedge loop) |
77
+ | Single ReapAuthority (all autonomous kills funnel through `terminateSession`) | Automatic — boot-purge, age-kill, and idle-zombie now route through it with the KEEP-guard + lease gate |
78
+ | Reap notice ("your session was shut down — <reason>") | On by default; silence it with `{"monitoring": {"reapNotify": {"enabled": false}}}` |
79
+ | Reap-log | `curl -H "Authorization: Bearer $AUTH" "http://localhost:4040/sessions/reap-log?limit=50"` (read-only) |
80
+ | Unkillability backstop (escalate, never auto-kill) | Automatic + signal-only; tune via `monitoring.staleBackstop` |
81
+ | Spawn-cap exclusion for long-unverifiable sessions | Automatic — they no longer count toward the absolute `maxSessions × 3` cap |
82
+
83
+ ## Evidence
84
+
85
+ **ContextWedgeSentinel.** Reproduced live 2026-05-28: a parallel Bash batch with one guard-blocked call corrupted the latest assistant thinking block; every subsequent message fast-failed with the `400 … cannot be modified` error while the silence + socket sentinels stayed quiet. After: the wedge is detected as the live tail and written to `logs/sentinel-events.jsonl`; the e2e test drives the production assembly and reads those rows back; the live recovery clears the topic's resume UUID so the respawn starts fresh.
86
+
87
+ **Session-lifecycle Phase 1.** 123 unit + 10 integration + 3 e2e green: ReapGuard (15), ReapNotifier (10), ReapLog (6), StaleSessionBackstop (10), terminate-CAS (9), liveness-oracle (15), boot-purge death-spiral reproduction (the 2026-05-27 9-of-9 incident → 0 false purges), reap-log route (integration + e2e "feature is alive" 200-not-503), and a full event-path integration (autonomous reap → log + one notice; recovery-bounce/operator silent; relay-lease/standby/protected refused-and-survive). `tsc --noEmit` clean. Migration parity: `reapNotify` + `staleBackstop` defaults flow to existing agents via ConfigDefaults; the reap-log Agent-Awareness section ships in both the CLAUDE.md template and `migrateClaudeMd`. Spec converged over 3 code-grounded review rounds (`docs/specs/reports/unified-session-lifecycle-robustness-convergence.md`); side-effects review: `upgrades/side-effects/unified-session-lifecycle-robustness.md`. Honest scope note: the boot-purge false-purge and the notice/log/survival paths are reproduced at the unit + real-event-path + real-AgentServer tiers; a live "boot a server, slow tmux, watch a Telegram notice land" pass was not performed in this environment.
@@ -0,0 +1,241 @@
1
+ # Side-Effects Review — Unified Session-Lifecycle Robustness (implementation)
2
+
3
+ Spec: `docs/specs/unified-session-lifecycle-robustness.md` (converged 3 iterations, approved by Justin
4
+ topic 2169). Full adversarial/standards review in
5
+ `docs/specs/reports/unified-session-lifecycle-robustness-convergence.md`. This artifact tracks the
6
+ side-effects of the *implementation* commits as they land.
7
+
8
+ ## Commit 1 — P1 SessionLivenessOracle + boot-purge fix
9
+
10
+ **Files:**
11
+ - `src/core/SessionLivenessOracle.ts` (new) — tri-state liveness oracle.
12
+ - `src/core/SessionManager.ts` — lazy oracle field + `setLivenessOracle()` DI seam; `purgeDeadSessions()`
13
+ rewired to use the oracle (purge only on `dead`, keep on `indeterminate`/`alive`).
14
+ - `src/core/types.ts` — added optional `liveness?: Partial<SessionLivenessOracleConfig>` to
15
+ `SessionManagerConfig`.
16
+ - `tests/unit/session-liveness-oracle.test.ts` (new, 15 tests) — incident fix (timeout→indeterminate),
17
+ exact-id match, cache, retry, boot-cap, coalescing, config floors.
18
+ - `tests/unit/death-spiral-fixes.test.ts` — purge suite rewritten to oracle semantics + the 2026-05-27
19
+ incident reproduction (timing-out tmux at boot → 0 purges).
20
+
21
+ **What changes behaviorally:** the boot purge no longer treats a slow/timing-out/unreachable
22
+ `tmux` probe as "dead." It resolves liveness from a single `tmux list-sessions` and purges ONLY when
23
+ the server is reachable AND the exact session id is absent. A timeout/unreachable result is
24
+ `indeterminate` → the session is KEPT and re-verified on the next monitoring tick.
25
+
26
+ **Over-block risk (a live session wrongly purged):** eliminated for the timeout case — the root cause
27
+ of the incident. Residual: a name-collision could in theory hide a live session, but matching is
28
+ exact-full-id against `list-sessions` output, not prefix/substring.
29
+
30
+ **Under-block risk (a dead session lingers):** bounded — a genuinely dead session absent from a
31
+ reachable `list-sessions` is still purged immediately. Only the *unverifiable* case lingers, and only
32
+ until the next tick (cheap). The §P5 backstop (later commit) escalates a permanently-unverifiable
33
+ session to the Attention queue so it can never leak forever or fill the spawn cap.
34
+
35
+ **Death-spiral (the original purge's reason for existing):** preserved and improved. The oracle is
36
+ async (never `execFileSync` on boot), resolves the whole set from ONE `list-sessions`, retries once
37
+ with backoff, and is bounded by a total boot-cap (default 8s) — on cap it returns `indeterminate`
38
+ rather than blocking boot. So conservatism does not re-introduce startup-blocking latency.
39
+
40
+ **Signal vs authority:** the oracle is pure-data (tier0) — it only *reports* alive/dead/indeterminate;
41
+ it never kills. The kill decision stays with `terminateSession()` (the authority, wired in a later
42
+ commit). No new kill authority is introduced here.
43
+
44
+ **Migration parity:** the oracle is server-side TS (ships with the server, no migrator needed). The new
45
+ `liveness` config block is optional (`Partial`), defaults applied in-code via `DEFAULT_LIVENESS_CONFIG`;
46
+ a `migrateConfig` entry + startup validation (`validateLivenessConfig`, rejects a sub-floor timeout)
47
+ land with the config-wiring commit.
48
+
49
+ **Rollback:** additive. Reverting the `purgeDeadSessions` change restores the old behavior; the new
50
+ module is unreferenced if the field/getter are removed. The DI seam (`setLivenessOracle`) is test-only
51
+ surface with no production caller.
52
+
53
+ **Tests:** 15 (oracle) + 14 (purge/death-spiral) green; typecheck clean. Reproduce-before-claim for
54
+ the live boot scenario is owed before declaring the incident fixed end-to-end (E2E tier, later commit).
55
+
56
+ ## Commit 2 — P2 ReapGuard (stateless KEEP-guards extracted from SessionReaper)
57
+
58
+ **Files:**
59
+ - `src/core/ReapGuard.ts` (new) — `reapBlockedReason(session)` over the stateless guards: protected,
60
+ spawn-grace (parameterized minAgeMs), recovery-in-flight, pending-injection, relay-lease,
61
+ recent-user-message, open-commitment, active-subagent, structural-long-work, active-process,
62
+ main-process-uninspectable/active. Cheap-first ordering; safe-by-default (a throwing signal →
63
+ KEEP 'guard-error', never reap).
64
+ - `src/monitoring/SessionReaper.ts` — `evaluate()` now calls the shared guard first, then layers its
65
+ STATEFUL checks (transcript-growth via per-instance `obs`, positive-idle via captured frame). Guard
66
+ built in the constructor from the reaper's deps + cfg.
67
+ - `tests/unit/reap-guard.test.ts` (new, 15 tests) — both sides of every guard, cannot-inspect→KEEP,
68
+ cheap-first ordering, throwing-signal→KEEP.
69
+ - `tests/unit/session-reaper.test.ts` — one label assertion updated ('eval-error'→'guard-error') for
70
+ the throwing-stateless-signal case; the KEEP-never-reap behavior is unchanged and still asserted.
71
+
72
+ **Parity:** SessionReaper's 30 existing tests all pass post-extraction — the `keptBy` reasons are
73
+ identical for every extracted guard (the spec's required wiring/parity check). Only the diagnostic
74
+ label for a *throwing* stateless signal moved (the guard now catches it as 'guard-error' rather than
75
+ letting it propagate to the reaper's outer 'eval-error' catch — a safer default for the shared guard;
76
+ the reaper's outer catch remains live for throws in the stateful checks).
77
+
78
+ **Behavioral change:** none for the reaper (parity). The guard is not yet consulted by
79
+ `terminateSession` — that wiring is P0 (next commit). So no other killer's behavior changes yet; this
80
+ commit is a pure, parity-verified extraction.
81
+
82
+ **Signal vs authority:** the guard is a pure predicate (no kill power) — it only *reports* a KEEP
83
+ reason. The authority (terminateSession) decides; wired in P0.
84
+
85
+ **Rollback:** additive. Reverting restores the inlined guards in `evaluate()`; `ReapGuard.ts` is
86
+ unreferenced if the reaper's guard field + call are removed.
87
+
88
+ **Tests:** 15 (guard) + 30 (reaper parity) + 4 (wiring) green; typecheck clean.
89
+
90
+ ## Commit — P0 ReapAuthority gates + funnel killers #1/#2/#3
91
+
92
+ **Files:**
93
+ - `src/core/SessionManager.ts` — `terminateSession` gains `origin`/`disposition`/`knownDead` opts, a
94
+ lease-holder gate, the mandatory ReapGuard consult, and a single `sessionReaped` emission;
95
+ `setReapGuard()`/`setAwakeChecker()` DI seams. Killers funneled: #1 boot-purge (`knownDead:true`,
96
+ lease-gated), #2 age-limit (inline kill → `terminateSession('age-limit')`), #3 idle-zombie (explicit
97
+ `disposition:'terminal'`).
98
+ - `src/server/routes.ts` — `DELETE /sessions/:id` is now async and routes through
99
+ `terminateSession(origin:'operator')` instead of the raw `killSession`.
100
+ - `src/commands/server.ts` — shared `reapGuardDeps` back BOTH the SessionReaper and the authority's
101
+ ReapGuard (built `minAgeMs:0`); `setAwakeChecker` wires the lease gate (single-machine ⇒ always awake).
102
+
103
+ **Implementation decisions with side-effects (reviewed):**
104
+ 1. **Operator kill bypasses `protected`.** The `protected` check moved *inside* the autonomous-only
105
+ gate block, so an `origin:'operator'` kill is no longer blocked on protected sessions. Risk: an
106
+ operator can now kill a protected (e.g. lifeline) session via the dashboard. Mitigation/justification:
107
+ this *preserves* the prior behavior — the old `DELETE` route used `killSession`, which killed
108
+ unconditionally — and matches the spec's "an explicit human kill must always happen". Autonomous
109
+ reapers remain fully blocked on protected.
110
+ 2. **`knownDead` bypass for boot-purge.** A `dead` verdict skips the KEEP-guard. Risk: a mis-verdict
111
+ would skip protection. Mitigation: the oracle returns `dead` ONLY on tmux-server-reachable +
112
+ exact-full-id-absent, never on timeout/unreachable/unknown — so `knownDead` is only ever passed for a
113
+ provably-gone session, which has no liveness to protect. Without the bypass, the guard's
114
+ liveness-blind topic-state KEEPs (recent-user-message / open-commitment) would pin a dead record in
115
+ the running list and re-create the boot death-spiral.
116
+ 3. **`DELETE` route now emits `beforeSessionKill`/`sessionComplete`/`sessionReaped`.** The old
117
+ `killSession` path historically emitted none of these. Effect: resume-UUID listeners now fire on an
118
+ operator kill (beneficial — a manually-killed topic session can be resumed), and the operator kill
119
+ lands in the reap-log (P4). No double-emit: `killSession` is no longer on this path.
120
+
121
+ **Signal vs authority:** unchanged — killers *request*, the authority decides. The only callers that
122
+ may mint `origin:'operator'` are Bearer-authed HTTP routes; in-process killers default to `autonomous`
123
+ and so always pass through the gates.
124
+
125
+ **Re-entrancy:** the guard is consulted before the in-flight (`terminating`) lock is acquired, so the
126
+ authority never misreads its own lock as a KEEP (spec §P0 ordering).
127
+
128
+ **Rollback:** additive per killer; reverting restores the inline kills. P0 gates default to the prior
129
+ behavior when `setReapGuard`/`setAwakeChecker` are unset (tests/standalone) — no guard consult, treated
130
+ as awake.
131
+
132
+ **Tests:** terminate (9) + reaper (30) + guard (15) + oracle (15) + timeout (4+6) + async-monitor (6) +
133
+ lifecycle integration (6) green; typecheck clean. One brittle source-string test updated to the new
134
+ age-kill log + funnel contract.
135
+
136
+ ## Commit — P3 reap-notify seam + P4 reap-log + Agent-Awareness + migration
137
+
138
+ **Files:**
139
+ - `src/monitoring/ReapNotifier.ts` (new) — single coalescing `sessionReaped` listener. Silent on
140
+ `recovery-bounce` and `origin:'operator'`; isolated reap → bound topic (or lifeline); burst within
141
+ the window → ONE consolidated lifeline message stating the EXACT total (count tracked separately from
142
+ the bounded detail buffer, so an overflow never under-reports). User-controlled names/reasons wrapped
143
+ as literal inline-code spans (inner backticks neutralized) so the downstream formatter never renders
144
+ them as markup.
145
+ - `src/monitoring/ReapLog.ts` (new) — append-only JSONL audit at `logs/reap-log.jsonl`; records BOTH
146
+ reaps (`sessionReaped`) and refused/skipped terminates (`reapBlocked`); JSON-encoded (no concat →
147
+ no newline injection); read tolerates corrupt lines.
148
+ - `src/server/routes.ts` — `GET /sessions/reap-log` (Bearer-auth via router middleware, read-only,
149
+ `?limit` capped at 1000, default 200).
150
+ - `src/server/AgentServer.ts` — `reapLog` option + ctx wire (mirrors `sessionReaper`).
151
+ - `src/commands/server.ts` — ReapLog + ReapNotifier built and the `sessionReaped`/`reapBlocked`
152
+ listeners + `setAwakeChecker` wired BEFORE the boot purge, so boot reaps are lease-gated, logged, and
153
+ notified. (The KEEP-guard is wired later, after its tracker deps exist — safe: boot-purge bypasses it
154
+ via `knownDead`, and no monitorTick kill can fire in the first seconds.)
155
+ - `src/core/types.ts` + `src/config/ConfigDefaults.ts` — `monitoring.reapNotify {enabled, coalesceWindowMs}`,
156
+ default ON.
157
+ - `src/scaffold/templates.ts` (Agent-Awareness) + `src/core/PostUpdateMigrator.migrateClaudeMd` (existing
158
+ agents get the Reap-Log section) + `migrateConfig` via ConfigDefaults (existing agents get the
159
+ `reapNotify` default automatically).
160
+
161
+ **Notify default = ON (deliberate, differs from sentinelTelegramEscalation).** The silently-stopped
162
+ sentinel escalation defaults OFF (post-2026-05-22 flood). The reap-notify defaults ON because the
163
+ incident this whole spec answers is *silent disappearance* — the user explicitly asked to be told. The
164
+ flood risk is bounded by: (a) coalescing a burst into one message, (b) staying silent on the common
165
+ recovery-bounce + operator paths, (c) SUMMARY tier (quiet-hours aware). A single config flag disables it.
166
+
167
+ **Discoverability:** `/sessions/reap-log` lives under the already-classified `sessions` prefix
168
+ (operator/dashboard-only in CapabilityIndex) — no new prefix, no lint change. Agent-awareness is carried
169
+ by the CLAUDE.md template + migrateClaudeMd instead (the template IS the agent's awareness).
170
+
171
+ **Ordering side-effect (reviewed):** the listeners attach pre-boot-purge; the guard wires post. The only
172
+ guard consumers besides boot-purge are monitorTick #2/#3, which require multi-hour age / 15+m observed
173
+ idle — unreachable in the sub-second guard-unset window. Documented inline.
174
+
175
+ **Tests:** ReapNotifier (10: silent dispositions/origins, isolated bound/unbound routing, burst
176
+ coalescing, overflow exact-count, auto-timer, malicious-name literalization, unreachable-channel drop)
177
+ + ReapLog (6: empty, reaped/skipped fields, newline-injection→valid-JSON, tail, corrupt-line tolerance)
178
+ green; typecheck clean. Route integration + e2e land in the test-phase commit.
179
+
180
+ ## Commit — P5 unkillability backstop
181
+
182
+ **Files:**
183
+ - `src/monitoring/StaleSessionBackstop.ts` (new) — signal-only. Per-tick forward-progress check; after
184
+ M min of no-forward-progress OR N consecutive indeterminate probes, raises ONE per-episode-deduped
185
+ Attention item (never auto-kills). Forward progress = meaningful transcript advance (≥floor AND new
186
+ tail — guards the heartbeat/loop case) OR main-process CPU OR a prompt/idle-state change. A
187
+ control-plane-unreachable tick raises ONE global item, not one per session.
188
+ - `src/core/SessionManager.ts` — `markLongIndeterminate()` + a `longIndeterminateSessions` set excluded
189
+ from the ABSOLUTE `maxSessions × 3` spawn cap (so unverifiable panes can't lock out spawning);
190
+ `probeLivenessBatch()` resolves tri-state liveness + reachability from ONE oracle snapshot.
191
+ - `src/core/types.ts` + `src/config/ConfigDefaults.ts` — `monitoring.staleBackstop`, default ON;
192
+ propagates to existing agents via ConfigDefaults.
193
+ - `src/commands/server.ts` — backstop wired after the SessionReaper; snapshot built from probeTranscript
194
+ (+ tail-hash read so a heartbeat byte isn't "progress"), captureOutput frame-hash, hasActiveProcesses;
195
+ Attention via makeAttentionPoster.
196
+
197
+ **Never-auto-kills (structural):** the backstop's deps surface has NO terminate/kill function — it
198
+ physically cannot end a session. Asserted by a unit test. It only observes and asks.
199
+
200
+ **Oracle interaction (reviewed):** with the committed single-snapshot oracle, a session is only
201
+ `indeterminate` when the whole snapshot is non-authoritative (server unreachable). So the per-session
202
+ indeterminate-streak path is currently reached only under global unreachability (handled by the global
203
+ item); the live, reachable path is the M-minute no-forward-progress escalation for ALIVE-but-faking
204
+ sessions. The per-session indeterminate code is kept (defensive — correct if the oracle later adds
205
+ individual re-probes that return indeterminate while reachable).
206
+
207
+ **Spawn-cap exclusion:** long-indeterminate sessions are removed from the ABSOLUTE-cap count only; they
208
+ still count toward the soft scheduler cap. Cleared the moment a session is verifiable again.
209
+
210
+ **Tests:** StaleSessionBackstop (10: M-min escalation, no-re-raise-per-episode, heartbeat-not-progress,
211
+ meaningful-advance/CPU/prompt-change progress, per-episode re-raise after recovery, global-unreachable
212
+ dedup, long-indeterminate flag set+cleared, never-kills structural) green; typecheck clean.
213
+
214
+ ## Commit — Phase 1 test tiers (integration + e2e)
215
+
216
+ **Files:**
217
+ - `tests/integration/reap-log-route.test.ts` (new, 4) — GET /sessions/reap-log through the real
218
+ createRoutes pipeline: 503-when-unwired, 200 with reaped+skipped entries, ?limit tail, read-only
219
+ (POST/DELETE → 404).
220
+ - `tests/integration/session-lifecycle-reap-wiring.test.ts` (new, 6) — the real SessionManager
221
+ terminateSession authority wired (as server.ts does) to ReapGuard + ReapNotifier + ReapLog, asserting
222
+ through the real emit path: autonomous terminal reap → log + exactly one notice; recovery-bounce →
223
+ logged-but-silent; operator → logged-silent + bypasses guard+protected; relay-lease KEEP → refused +
224
+ survives + logged 'skipped'; standby → refused + survives; protected → refused + survives.
225
+ - `tests/e2e/reap-log-lifecycle.test.ts` (new, 3) — boots the REAL AgentServer: GET /sessions/reap-log
226
+ is alive (200 not 503), surfaces recorded entries, requires Bearer, read-only.
227
+
228
+ **Reproduce-before-claim status (honest):** the boot-purge false-purge is reproduced at the unit tier
229
+ (death-spiral-fixes — the 2026-05-27 9-of-9 incident now yields 0 purges); "one terminal reap → exactly
230
+ one notice" and "guarded/standby/protected survive" are reproduced through the real SessionManager event
231
+ path (integration); the route is proven alive on the real AgentServer (e2e). A full live boot with a
232
+ deliberately-slowed tmux + a real Telegram notice landing was NOT performed in this environment — the
233
+ test-tier reproductions stand in for it, and that gap is stated rather than papered over.
234
+
235
+ ## Commit — annotate intentional silent catches (no-silent-fallbacks ratchet)
236
+
237
+ Two new intentional silent catches are annotated `@silent-fallback-ok` so the
238
+ no-silent-fallbacks ratchet doesn't count them: `ReapLog.read` (no log file ⇒
239
+ empty list is correct, not degraded) and the §P5 transcript-tail read (an
240
+ unreadable tail ⇒ no meaningful-advance signal this tick; treated as ambiguous,
241
+ never as progress). Behavior unchanged.