instar 1.3.62 → 1.3.63
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/commands/server.d.ts.map +1 -1
- package/dist/commands/server.js +132 -8
- package/dist/commands/server.js.map +1 -1
- package/dist/config/ConfigDefaults.d.ts.map +1 -1
- package/dist/config/ConfigDefaults.js +20 -0
- package/dist/config/ConfigDefaults.js.map +1 -1
- package/dist/core/PostUpdateMigrator.d.ts.map +1 -1
- package/dist/core/PostUpdateMigrator.js +22 -0
- package/dist/core/PostUpdateMigrator.js.map +1 -1
- package/dist/core/ReapGuard.d.ts +79 -0
- package/dist/core/ReapGuard.d.ts.map +1 -0
- package/dist/core/ReapGuard.js +114 -0
- package/dist/core/ReapGuard.js.map +1 -0
- package/dist/core/SessionLivenessOracle.d.ts +104 -0
- package/dist/core/SessionLivenessOracle.d.ts.map +1 -0
- package/dist/core/SessionLivenessOracle.js +201 -0
- package/dist/core/SessionLivenessOracle.js.map +1 -0
- package/dist/core/SessionManager.d.ts +90 -10
- package/dist/core/SessionManager.d.ts.map +1 -1
- package/dist/core/SessionManager.js +200 -40
- package/dist/core/SessionManager.js.map +1 -1
- package/dist/core/types.d.ts +30 -0
- package/dist/core/types.d.ts.map +1 -1
- package/dist/core/types.js.map +1 -1
- package/dist/monitoring/ReapLog.d.ts +51 -0
- package/dist/monitoring/ReapLog.d.ts.map +1 -0
- package/dist/monitoring/ReapLog.js +82 -0
- package/dist/monitoring/ReapLog.js.map +1 -0
- package/dist/monitoring/ReapNotifier.d.ts +66 -0
- package/dist/monitoring/ReapNotifier.d.ts.map +1 -0
- package/dist/monitoring/ReapNotifier.js +112 -0
- package/dist/monitoring/ReapNotifier.js.map +1 -0
- package/dist/monitoring/SessionReaper.d.ts +4 -0
- package/dist/monitoring/SessionReaper.d.ts.map +1 -1
- package/dist/monitoring/SessionReaper.js +19 -42
- package/dist/monitoring/SessionReaper.js.map +1 -1
- package/dist/monitoring/StaleSessionBackstop.d.ts +89 -0
- package/dist/monitoring/StaleSessionBackstop.d.ts.map +1 -0
- package/dist/monitoring/StaleSessionBackstop.js +176 -0
- package/dist/monitoring/StaleSessionBackstop.js.map +1 -0
- package/dist/scaffold/templates.d.ts.map +1 -1
- package/dist/scaffold/templates.js +5 -0
- package/dist/scaffold/templates.js.map +1 -1
- package/dist/server/AgentServer.d.ts +3 -0
- package/dist/server/AgentServer.d.ts.map +1 -1
- package/dist/server/AgentServer.js +1 -0
- package/dist/server/AgentServer.js.map +1 -1
- package/dist/server/routes.d.ts +1 -0
- package/dist/server/routes.d.ts.map +1 -1
- package/dist/server/routes.js +33 -11
- package/dist/server/routes.js.map +1 -1
- package/package.json +1 -1
- package/src/data/builtin-manifest.json +64 -64
- package/src/scaffold/templates.ts +5 -0
- package/upgrades/1.3.62.md +43 -45
- package/upgrades/1.3.63.md +87 -0
- package/upgrades/side-effects/unified-session-lifecycle-robustness.md +241 -0
package/upgrades/1.3.62.md
CHANGED
|
@@ -4,6 +4,8 @@
|
|
|
4
4
|
|
|
5
5
|
## What Changed
|
|
6
6
|
|
|
7
|
+
### ContextWedgeSentinel — detect + (optionally) recover the thinking-block-400 wedge
|
|
8
|
+
|
|
7
9
|
**A new way sessions die is now detected and (optionally) auto-recovered.** When
|
|
8
10
|
a tool call is cancelled inside a *parallel* tool batch while extended thinking
|
|
9
11
|
is on, Claude Code corrupts the thinking block on the latest assistant turn —
|
|
@@ -33,57 +35,53 @@ and rides the Graduated Feature Rollout track. The sentinel also feeds the
|
|
|
33
35
|
SessionReaper kill-veto so the reaper never reaps a session mid-recovery. New
|
|
34
36
|
config kill switch `monitoring.contextWedgeSentinel.enabled` (default true).
|
|
35
37
|
|
|
38
|
+
### Unified session-lifecycle robustness — Phase 1 (ReapAuthority + reap-log + backstop)
|
|
39
|
+
|
|
40
|
+
Phase 1 of **unified session-lifecycle robustness** (spec `docs/specs/unified-session-lifecycle-robustness.md`, converged + approved). The trigger was a real incident: at a busy server boot the startup "cleanup crew" treated a slow-to-answer tmux session as *dead* and wiped 9 of 9 live sessions — silently. The audit that followed found eight different code paths that can end a session, each re-inventing "alive or dead?" its own way, several of them able to mistake slow/busy for dead and none of them telling the user. This release builds the single authority + safety backbone that the rest of the killers will move onto.
|
|
41
|
+
|
|
42
|
+
What landed:
|
|
43
|
+
|
|
44
|
+
- **One authority, not eight.** `terminateSession()` is now the sole kill chokepoint. Every autonomous shutoff routes through it and can only *request* a kill — the authority holds the safety checks: a compare-and-set live-status guard, the `protected` set, an **awake-machine-only lease gate** (a standby never reaps another machine's sessions), and a mandatory **KEEP-guard** consult (the same positive-evidence checks the careful SessionReaper uses — relay-lease, pending-injection, recent-user-message, active subagent/process, etc.). An explicit **operator** kill (stamped only by the Bearer-authed HTTP route) bypasses those gates so a human "kill" always happens. The first three killers — boot purge, the age/5-hour cutoff, and the idle-zombie killer — are funneled through it.
|
|
45
|
+
- **A reap is never silent.** A new coalescing notifier surfaces "your session was shut down — <reason>" so a session can't just vanish. Routine kill-to-respawn bounces and your own operator kills stay quiet; a burst collapses into ONE consolidated message instead of spamming. Default **on** (`monitoring.reapNotify`).
|
|
46
|
+
- **A reap-log you can read.** Every shutoff *and* every refused/skipped shutoff (protected / not-lease-holder / a KEEP-hold / in-flight) is recorded as one JSON line, served read-only at `GET /sessions/reap-log`. "Where did my session go?" now always has an answer.
|
|
47
|
+
- **Nothing is unconditionally immortal — and nothing can lock you out.** A signal-only backstop watches for a session that fakes work (heartbeat-byte / tight-loop) or is stuck unverifiable, and after a threshold raises ONE deduped Attention item asking *you* to decide — it never auto-kills. Long-unverifiable sessions are excluded from the absolute spawn cap so a fleet of stuck panes can't lock you out of starting new work.
|
|
48
|
+
|
|
49
|
+
All changes are additive and only make killing *more* conservative; the boot-purge fix already turns the original 9-of-9 wipe into 0 false kills. Phase 2 (watchdog / orphan / recovery / wake-reaper onto the authority) and Phase 3 (quota soft-check + session-label-follows-rename) follow.
|
|
50
|
+
|
|
36
51
|
## What to Tell Your User
|
|
37
52
|
|
|
38
53
|
- I can now spot a specific way a session silently dies — it gets stuck on a
|
|
39
54
|
"can't modify the thinking block" error and fails instantly on every message
|
|
40
55
|
while still looking busy — which used to leave you seeing only standby replies.
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
-
|
|
45
|
-
|
|
46
|
-
|
|
56
|
+
By default I just notice and log it; you can opt in to having me restart a
|
|
57
|
+
stuck session cleanly (I throw away the corrupted conversation so it can't get
|
|
58
|
+
re-stuck).
|
|
59
|
+
- Your sessions also won't silently disappear anymore. The startup cleanup no
|
|
60
|
+
longer mistakes a slow-to-answer session for a dead one, and if anything does
|
|
61
|
+
get shut down you get told why — with a reap-log page recording every shutoff
|
|
62
|
+
and every refused shutoff.
|
|
63
|
+
- Every automatic cleanup now goes through one careful gatekeeper that refuses to
|
|
64
|
+
end a session that might be working, and only the "awake" machine ever reaps.
|
|
65
|
+
When you kill a session it still happens immediately.
|
|
66
|
+
- Nothing can get wedged-but-immortal or quietly fill your slots: instead of ever
|
|
67
|
+
auto-killing a stuck session, the agent raises a single "this looks stuck —
|
|
68
|
+
investigate or force-kill?" item for you.
|
|
47
69
|
|
|
48
|
-
##
|
|
70
|
+
## Summary of New Capabilities
|
|
49
71
|
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
(
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
in the latest assistant message cannot be modified.
|
|
61
|
-
✻ Cooked for 0s
|
|
62
|
-
```
|
|
63
|
-
The user sent "How is this looking?" twice over ~30 minutes and received only
|
|
64
|
-
sentinel/standby replies — the session was a corpse, and the silence + socket
|
|
65
|
-
sentinels never fired (output never went quiet; no disconnect string).
|
|
66
|
-
|
|
67
|
-
**Observed after.** With ContextWedgeSentinel: the wedge is detected as the live
|
|
68
|
-
tail and written to `logs/sentinel-events.jsonl` (`kind:detected` →
|
|
69
|
-
`kind:escalated` in detect-only mode, or `kind:recovered` when autoRecovery is
|
|
70
|
-
on). The e2e test (`tests/e2e/context-wedge-sentinel-lifecycle.test.ts`) drives
|
|
71
|
-
the production assembly and reads those exact rows back from disk; the live
|
|
72
|
-
recovery clears the topic's resume UUID so the respawn starts fresh (verified by
|
|
73
|
-
`SessionRefresh.test.ts` asserting the kill→clear→respawn order). The original
|
|
74
|
-
wedged session was recovered manually by killing it (the next message spawned
|
|
75
|
-
clean), confirming the fresh-respawn approach before it was automated.
|
|
72
|
+
| Capability | How to Use |
|
|
73
|
+
|-----------|-----------|
|
|
74
|
+
| ContextWedgeSentinel (thinking-block-400 detection) | Automatic + default-on housekeeping; audits to `logs/sentinel-events.jsonl` |
|
|
75
|
+
| Opt-in wedge auto-recovery | `{"monitoring": {"contextWedgeSentinel": {"autoRecovery": {"enabled": true, "dryRun": false}}}}` (Graduated Rollout track) |
|
|
76
|
+
| SessionRefresh `fresh` mode | Internal — kill + respawn that clears the topic's resume UUID (no re-wedge loop) |
|
|
77
|
+
| Single ReapAuthority (all autonomous kills funnel through `terminateSession`) | Automatic — boot-purge, age-kill, and idle-zombie now route through it with the KEEP-guard + lease gate |
|
|
78
|
+
| Reap notice ("your session was shut down — <reason>") | On by default; silence it with `{"monitoring": {"reapNotify": {"enabled": false}}}` |
|
|
79
|
+
| Reap-log | `curl -H "Authorization: Bearer $AUTH" "http://localhost:4040/sessions/reap-log?limit=50"` (read-only) |
|
|
80
|
+
| Unkillability backstop (escalate, never auto-kill) | Automatic + signal-only; tune via `monitoring.staleBackstop` |
|
|
81
|
+
| Spawn-cap exclusion for long-unverifiable sessions | Automatic — they no longer count toward the absolute `maxSessions × 3` cap |
|
|
76
82
|
|
|
77
|
-
##
|
|
83
|
+
## Evidence
|
|
84
|
+
|
|
85
|
+
**ContextWedgeSentinel.** Reproduced live 2026-05-28: a parallel Bash batch with one guard-blocked call corrupted the latest assistant thinking block; every subsequent message fast-failed with the `400 … cannot be modified` error while the silence + socket sentinels stayed quiet. After: the wedge is detected as the live tail and written to `logs/sentinel-events.jsonl`; the e2e test drives the production assembly and reads those rows back; the live recovery clears the topic's resume UUID so the respawn starts fresh.
|
|
78
86
|
|
|
79
|
-
-
|
|
80
|
-
non-progressing session tail (45s confirm window); audits every transition to
|
|
81
|
-
`logs/sentinel-events.jsonl`.
|
|
82
|
-
- **SessionRefresh `fresh` mode** — kill + respawn that clears the topic's resume
|
|
83
|
-
UUID, so recovery never reloads a corrupted transcript (no re-wedge loop).
|
|
84
|
-
- **Opt-in auto-recovery** — `monitoring.contextWedgeSentinel.autoRecovery`
|
|
85
|
-
(default off + dry-run), on the Graduated Feature Rollout track; promotion to
|
|
86
|
-
default-on propagates fleet-wide on update with no migration.
|
|
87
|
-
- Escalation (when a confirmed wedge isn't auto-recovered) routes through the
|
|
88
|
-
existing tone-gated SentinelNotifier, gated by `sentinelTelegramEscalation`
|
|
89
|
-
(default off, coalesced).
|
|
87
|
+
**Session-lifecycle Phase 1.** 123 unit + 10 integration + 3 e2e green: ReapGuard (15), ReapNotifier (10), ReapLog (6), StaleSessionBackstop (10), terminate-CAS (9), liveness-oracle (15), boot-purge death-spiral reproduction (the 2026-05-27 9-of-9 incident → 0 false purges), reap-log route (integration + e2e "feature is alive" 200-not-503), and a full event-path integration (autonomous reap → log + one notice; recovery-bounce/operator silent; relay-lease/standby/protected refused-and-survive). `tsc --noEmit` clean. Migration parity: `reapNotify` + `staleBackstop` defaults flow to existing agents via ConfigDefaults; the reap-log Agent-Awareness section ships in both the CLAUDE.md template and `migrateClaudeMd`. Spec converged over 3 code-grounded review rounds (`docs/specs/reports/unified-session-lifecycle-robustness-convergence.md`); side-effects review: `upgrades/side-effects/unified-session-lifecycle-robustness.md`. Honest scope note: the boot-purge false-purge and the notice/log/survival paths are reproduced at the unit + real-event-path + real-AgentServer tiers; a live "boot a server, slow tmux, watch a Telegram notice land" pass was not performed in this environment.
|
|
@@ -0,0 +1,87 @@
|
|
|
1
|
+
# Upgrade Guide — vNEXT
|
|
2
|
+
|
|
3
|
+
<!-- bump: minor -->
|
|
4
|
+
|
|
5
|
+
## What Changed
|
|
6
|
+
|
|
7
|
+
### ContextWedgeSentinel — detect + (optionally) recover the thinking-block-400 wedge
|
|
8
|
+
|
|
9
|
+
**A new way sessions die is now detected and (optionally) auto-recovered.** When
|
|
10
|
+
a tool call is cancelled inside a *parallel* tool batch while extended thinking
|
|
11
|
+
is on, Claude Code corrupts the thinking block on the latest assistant turn —
|
|
12
|
+
and from then on the Anthropic API rejects every resume with
|
|
13
|
+
`400 … thinking blocks in the latest assistant message cannot be modified`. The
|
|
14
|
+
session fast-fails instantly on every message ("Cooked for 0s"): permanently
|
|
15
|
+
dead, yet still emitting output. Because it's not *silent* and hasn't lost its
|
|
16
|
+
socket, neither the ActiveWorkSilenceSentinel nor the SocketDisconnectSentinel
|
|
17
|
+
catches it — the user just sees standby/sentinel replies forever while the real
|
|
18
|
+
session is a corpse. (Live incident: a session on a real topic fast-failed every
|
|
19
|
+
"How is this looking?" for ~30 minutes.)
|
|
20
|
+
|
|
21
|
+
The new **ContextWedgeSentinel** (a 4th member of the silently-stopped family)
|
|
22
|
+
recognizes this exact wedge: it matches the error as the *live session tail* and
|
|
23
|
+
waits a 45s confirm window so a session merely *discussing* the bug is never
|
|
24
|
+
flagged. A gentle nudge can't fix this one (re-engaging re-sends the corrupted
|
|
25
|
+
turn), so recovery is a **fresh respawn** — kill the session and clear the
|
|
26
|
+
topic's saved resume UUID so the bridge spawns a brand-new conversation instead
|
|
27
|
+
of `--resume`-ing the corrupted transcript (which would just re-wedge on the
|
|
28
|
+
next message — the central correctness property). This reuses the existing
|
|
29
|
+
rate-guarded `SessionRefresh` via a new `fresh` mode.
|
|
30
|
+
|
|
31
|
+
Detection + audit are **default-ON housekeeping** (they write to
|
|
32
|
+
`logs/sentinel-events.jsonl` and kill nothing). The destructive auto-respawn is
|
|
33
|
+
**opt-in** (`monitoring.contextWedgeSentinel.autoRecovery`, default off + dry-run)
|
|
34
|
+
and rides the Graduated Feature Rollout track. The sentinel also feeds the
|
|
35
|
+
SessionReaper kill-veto so the reaper never reaps a session mid-recovery. New
|
|
36
|
+
config kill switch `monitoring.contextWedgeSentinel.enabled` (default true).
|
|
37
|
+
|
|
38
|
+
### Unified session-lifecycle robustness — Phase 1 (ReapAuthority + reap-log + backstop)
|
|
39
|
+
|
|
40
|
+
Phase 1 of **unified session-lifecycle robustness** (spec `docs/specs/unified-session-lifecycle-robustness.md`, converged + approved). The trigger was a real incident: at a busy server boot the startup "cleanup crew" treated a slow-to-answer tmux session as *dead* and wiped 9 of 9 live sessions — silently. The audit that followed found eight different code paths that can end a session, each re-inventing "alive or dead?" its own way, several of them able to mistake slow/busy for dead and none of them telling the user. This release builds the single authority + safety backbone that the rest of the killers will move onto.
|
|
41
|
+
|
|
42
|
+
What landed:
|
|
43
|
+
|
|
44
|
+
- **One authority, not eight.** `terminateSession()` is now the sole kill chokepoint. Every autonomous shutoff routes through it and can only *request* a kill — the authority holds the safety checks: a compare-and-set live-status guard, the `protected` set, an **awake-machine-only lease gate** (a standby never reaps another machine's sessions), and a mandatory **KEEP-guard** consult (the same positive-evidence checks the careful SessionReaper uses — relay-lease, pending-injection, recent-user-message, active subagent/process, etc.). An explicit **operator** kill (stamped only by the Bearer-authed HTTP route) bypasses those gates so a human "kill" always happens. The first three killers — boot purge, the age/5-hour cutoff, and the idle-zombie killer — are funneled through it.
|
|
45
|
+
- **A reap is never silent.** A new coalescing notifier surfaces "your session was shut down — <reason>" so a session can't just vanish. Routine kill-to-respawn bounces and your own operator kills stay quiet; a burst collapses into ONE consolidated message instead of spamming. Default **on** (`monitoring.reapNotify`).
|
|
46
|
+
- **A reap-log you can read.** Every shutoff *and* every refused/skipped shutoff (protected / not-lease-holder / a KEEP-hold / in-flight) is recorded as one JSON line, served read-only at `GET /sessions/reap-log`. "Where did my session go?" now always has an answer.
|
|
47
|
+
- **Nothing is unconditionally immortal — and nothing can lock you out.** A signal-only backstop watches for a session that fakes work (heartbeat-byte / tight-loop) or is stuck unverifiable, and after a threshold raises ONE deduped Attention item asking *you* to decide — it never auto-kills. Long-unverifiable sessions are excluded from the absolute spawn cap so a fleet of stuck panes can't lock you out of starting new work.
|
|
48
|
+
|
|
49
|
+
All changes are additive and only make killing *more* conservative; the boot-purge fix already turns the original 9-of-9 wipe into 0 false kills. Phase 2 (watchdog / orphan / recovery / wake-reaper onto the authority) and Phase 3 (quota soft-check + session-label-follows-rename) follow.
|
|
50
|
+
|
|
51
|
+
## What to Tell Your User
|
|
52
|
+
|
|
53
|
+
- I can now spot a specific way a session silently dies — it gets stuck on a
|
|
54
|
+
"can't modify the thinking block" error and fails instantly on every message
|
|
55
|
+
while still looking busy — which used to leave you seeing only standby replies.
|
|
56
|
+
By default I just notice and log it; you can opt in to having me restart a
|
|
57
|
+
stuck session cleanly (I throw away the corrupted conversation so it can't get
|
|
58
|
+
re-stuck).
|
|
59
|
+
- Your sessions also won't silently disappear anymore. The startup cleanup no
|
|
60
|
+
longer mistakes a slow-to-answer session for a dead one, and if anything does
|
|
61
|
+
get shut down you get told why — with a reap-log page recording every shutoff
|
|
62
|
+
and every refused shutoff.
|
|
63
|
+
- Every automatic cleanup now goes through one careful gatekeeper that refuses to
|
|
64
|
+
end a session that might be working, and only the "awake" machine ever reaps.
|
|
65
|
+
When you kill a session it still happens immediately.
|
|
66
|
+
- Nothing can get wedged-but-immortal or quietly fill your slots: instead of ever
|
|
67
|
+
auto-killing a stuck session, the agent raises a single "this looks stuck —
|
|
68
|
+
investigate or force-kill?" item for you.
|
|
69
|
+
|
|
70
|
+
## Summary of New Capabilities
|
|
71
|
+
|
|
72
|
+
| Capability | How to Use |
|
|
73
|
+
|-----------|-----------|
|
|
74
|
+
| ContextWedgeSentinel (thinking-block-400 detection) | Automatic + default-on housekeeping; audits to `logs/sentinel-events.jsonl` |
|
|
75
|
+
| Opt-in wedge auto-recovery | `{"monitoring": {"contextWedgeSentinel": {"autoRecovery": {"enabled": true, "dryRun": false}}}}` (Graduated Rollout track) |
|
|
76
|
+
| SessionRefresh `fresh` mode | Internal — kill + respawn that clears the topic's resume UUID (no re-wedge loop) |
|
|
77
|
+
| Single ReapAuthority (all autonomous kills funnel through `terminateSession`) | Automatic — boot-purge, age-kill, and idle-zombie now route through it with the KEEP-guard + lease gate |
|
|
78
|
+
| Reap notice ("your session was shut down — <reason>") | On by default; silence it with `{"monitoring": {"reapNotify": {"enabled": false}}}` |
|
|
79
|
+
| Reap-log | `curl -H "Authorization: Bearer $AUTH" "http://localhost:4040/sessions/reap-log?limit=50"` (read-only) |
|
|
80
|
+
| Unkillability backstop (escalate, never auto-kill) | Automatic + signal-only; tune via `monitoring.staleBackstop` |
|
|
81
|
+
| Spawn-cap exclusion for long-unverifiable sessions | Automatic — they no longer count toward the absolute `maxSessions × 3` cap |
|
|
82
|
+
|
|
83
|
+
## Evidence
|
|
84
|
+
|
|
85
|
+
**ContextWedgeSentinel.** Reproduced live 2026-05-28: a parallel Bash batch with one guard-blocked call corrupted the latest assistant thinking block; every subsequent message fast-failed with the `400 … cannot be modified` error while the silence + socket sentinels stayed quiet. After: the wedge is detected as the live tail and written to `logs/sentinel-events.jsonl`; the e2e test drives the production assembly and reads those rows back; the live recovery clears the topic's resume UUID so the respawn starts fresh.
|
|
86
|
+
|
|
87
|
+
**Session-lifecycle Phase 1.** 123 unit + 10 integration + 3 e2e green: ReapGuard (15), ReapNotifier (10), ReapLog (6), StaleSessionBackstop (10), terminate-CAS (9), liveness-oracle (15), boot-purge death-spiral reproduction (the 2026-05-27 9-of-9 incident → 0 false purges), reap-log route (integration + e2e "feature is alive" 200-not-503), and a full event-path integration (autonomous reap → log + one notice; recovery-bounce/operator silent; relay-lease/standby/protected refused-and-survive). `tsc --noEmit` clean. Migration parity: `reapNotify` + `staleBackstop` defaults flow to existing agents via ConfigDefaults; the reap-log Agent-Awareness section ships in both the CLAUDE.md template and `migrateClaudeMd`. Spec converged over 3 code-grounded review rounds (`docs/specs/reports/unified-session-lifecycle-robustness-convergence.md`); side-effects review: `upgrades/side-effects/unified-session-lifecycle-robustness.md`. Honest scope note: the boot-purge false-purge and the notice/log/survival paths are reproduced at the unit + real-event-path + real-AgentServer tiers; a live "boot a server, slow tmux, watch a Telegram notice land" pass was not performed in this environment.
|
|
@@ -0,0 +1,241 @@
|
|
|
1
|
+
# Side-Effects Review — Unified Session-Lifecycle Robustness (implementation)
|
|
2
|
+
|
|
3
|
+
Spec: `docs/specs/unified-session-lifecycle-robustness.md` (converged 3 iterations, approved by Justin
|
|
4
|
+
topic 2169). Full adversarial/standards review in
|
|
5
|
+
`docs/specs/reports/unified-session-lifecycle-robustness-convergence.md`. This artifact tracks the
|
|
6
|
+
side-effects of the *implementation* commits as they land.
|
|
7
|
+
|
|
8
|
+
## Commit 1 — P1 SessionLivenessOracle + boot-purge fix
|
|
9
|
+
|
|
10
|
+
**Files:**
|
|
11
|
+
- `src/core/SessionLivenessOracle.ts` (new) — tri-state liveness oracle.
|
|
12
|
+
- `src/core/SessionManager.ts` — lazy oracle field + `setLivenessOracle()` DI seam; `purgeDeadSessions()`
|
|
13
|
+
rewired to use the oracle (purge only on `dead`, keep on `indeterminate`/`alive`).
|
|
14
|
+
- `src/core/types.ts` — added optional `liveness?: Partial<SessionLivenessOracleConfig>` to
|
|
15
|
+
`SessionManagerConfig`.
|
|
16
|
+
- `tests/unit/session-liveness-oracle.test.ts` (new, 15 tests) — incident fix (timeout→indeterminate),
|
|
17
|
+
exact-id match, cache, retry, boot-cap, coalescing, config floors.
|
|
18
|
+
- `tests/unit/death-spiral-fixes.test.ts` — purge suite rewritten to oracle semantics + the 2026-05-27
|
|
19
|
+
incident reproduction (timing-out tmux at boot → 0 purges).
|
|
20
|
+
|
|
21
|
+
**What changes behaviorally:** the boot purge no longer treats a slow/timing-out/unreachable
|
|
22
|
+
`tmux` probe as "dead." It resolves liveness from a single `tmux list-sessions` and purges ONLY when
|
|
23
|
+
the server is reachable AND the exact session id is absent. A timeout/unreachable result is
|
|
24
|
+
`indeterminate` → the session is KEPT and re-verified on the next monitoring tick.
|
|
25
|
+
|
|
26
|
+
**Over-block risk (a live session wrongly purged):** eliminated for the timeout case — the root cause
|
|
27
|
+
of the incident. Residual: a name-collision could in theory hide a live session, but matching is
|
|
28
|
+
exact-full-id against `list-sessions` output, not prefix/substring.
|
|
29
|
+
|
|
30
|
+
**Under-block risk (a dead session lingers):** bounded — a genuinely dead session absent from a
|
|
31
|
+
reachable `list-sessions` is still purged immediately. Only the *unverifiable* case lingers, and only
|
|
32
|
+
until the next tick (cheap). The §P5 backstop (later commit) escalates a permanently-unverifiable
|
|
33
|
+
session to the Attention queue so it can never leak forever or fill the spawn cap.
|
|
34
|
+
|
|
35
|
+
**Death-spiral (the original purge's reason for existing):** preserved and improved. The oracle is
|
|
36
|
+
async (never `execFileSync` on boot), resolves the whole set from ONE `list-sessions`, retries once
|
|
37
|
+
with backoff, and is bounded by a total boot-cap (default 8s) — on cap it returns `indeterminate`
|
|
38
|
+
rather than blocking boot. So conservatism does not re-introduce startup-blocking latency.
|
|
39
|
+
|
|
40
|
+
**Signal vs authority:** the oracle is pure-data (tier0) — it only *reports* alive/dead/indeterminate;
|
|
41
|
+
it never kills. The kill decision stays with `terminateSession()` (the authority, wired in a later
|
|
42
|
+
commit). No new kill authority is introduced here.
|
|
43
|
+
|
|
44
|
+
**Migration parity:** the oracle is server-side TS (ships with the server, no migrator needed). The new
|
|
45
|
+
`liveness` config block is optional (`Partial`), defaults applied in-code via `DEFAULT_LIVENESS_CONFIG`;
|
|
46
|
+
a `migrateConfig` entry + startup validation (`validateLivenessConfig`, rejects a sub-floor timeout)
|
|
47
|
+
land with the config-wiring commit.
|
|
48
|
+
|
|
49
|
+
**Rollback:** additive. Reverting the `purgeDeadSessions` change restores the old behavior; the new
|
|
50
|
+
module is unreferenced if the field/getter are removed. The DI seam (`setLivenessOracle`) is test-only
|
|
51
|
+
surface with no production caller.
|
|
52
|
+
|
|
53
|
+
**Tests:** 15 (oracle) + 14 (purge/death-spiral) green; typecheck clean. Reproduce-before-claim for
|
|
54
|
+
the live boot scenario is owed before declaring the incident fixed end-to-end (E2E tier, later commit).
|
|
55
|
+
|
|
56
|
+
## Commit 2 — P2 ReapGuard (stateless KEEP-guards extracted from SessionReaper)
|
|
57
|
+
|
|
58
|
+
**Files:**
|
|
59
|
+
- `src/core/ReapGuard.ts` (new) — `reapBlockedReason(session)` over the stateless guards: protected,
|
|
60
|
+
spawn-grace (parameterized minAgeMs), recovery-in-flight, pending-injection, relay-lease,
|
|
61
|
+
recent-user-message, open-commitment, active-subagent, structural-long-work, active-process,
|
|
62
|
+
main-process-uninspectable/active. Cheap-first ordering; safe-by-default (a throwing signal →
|
|
63
|
+
KEEP 'guard-error', never reap).
|
|
64
|
+
- `src/monitoring/SessionReaper.ts` — `evaluate()` now calls the shared guard first, then layers its
|
|
65
|
+
STATEFUL checks (transcript-growth via per-instance `obs`, positive-idle via captured frame). Guard
|
|
66
|
+
built in the constructor from the reaper's deps + cfg.
|
|
67
|
+
- `tests/unit/reap-guard.test.ts` (new, 15 tests) — both sides of every guard, cannot-inspect→KEEP,
|
|
68
|
+
cheap-first ordering, throwing-signal→KEEP.
|
|
69
|
+
- `tests/unit/session-reaper.test.ts` — one label assertion updated ('eval-error'→'guard-error') for
|
|
70
|
+
the throwing-stateless-signal case; the KEEP-never-reap behavior is unchanged and still asserted.
|
|
71
|
+
|
|
72
|
+
**Parity:** SessionReaper's 30 existing tests all pass post-extraction — the `keptBy` reasons are
|
|
73
|
+
identical for every extracted guard (the spec's required wiring/parity check). Only the diagnostic
|
|
74
|
+
label for a *throwing* stateless signal moved (the guard now catches it as 'guard-error' rather than
|
|
75
|
+
letting it propagate to the reaper's outer 'eval-error' catch — a safer default for the shared guard;
|
|
76
|
+
the reaper's outer catch remains live for throws in the stateful checks).
|
|
77
|
+
|
|
78
|
+
**Behavioral change:** none for the reaper (parity). The guard is not yet consulted by
|
|
79
|
+
`terminateSession` — that wiring is P0 (next commit). So no other killer's behavior changes yet; this
|
|
80
|
+
commit is a pure, parity-verified extraction.
|
|
81
|
+
|
|
82
|
+
**Signal vs authority:** the guard is a pure predicate (no kill power) — it only *reports* a KEEP
|
|
83
|
+
reason. The authority (terminateSession) decides; wired in P0.
|
|
84
|
+
|
|
85
|
+
**Rollback:** additive. Reverting restores the inlined guards in `evaluate()`; `ReapGuard.ts` is
|
|
86
|
+
unreferenced if the reaper's guard field + call are removed.
|
|
87
|
+
|
|
88
|
+
**Tests:** 15 (guard) + 30 (reaper parity) + 4 (wiring) green; typecheck clean.
|
|
89
|
+
|
|
90
|
+
## Commit — P0 ReapAuthority gates + funnel killers #1/#2/#3
|
|
91
|
+
|
|
92
|
+
**Files:**
|
|
93
|
+
- `src/core/SessionManager.ts` — `terminateSession` gains `origin`/`disposition`/`knownDead` opts, a
|
|
94
|
+
lease-holder gate, the mandatory ReapGuard consult, and a single `sessionReaped` emission;
|
|
95
|
+
`setReapGuard()`/`setAwakeChecker()` DI seams. Killers funneled: #1 boot-purge (`knownDead:true`,
|
|
96
|
+
lease-gated), #2 age-limit (inline kill → `terminateSession('age-limit')`), #3 idle-zombie (explicit
|
|
97
|
+
`disposition:'terminal'`).
|
|
98
|
+
- `src/server/routes.ts` — `DELETE /sessions/:id` is now async and routes through
|
|
99
|
+
`terminateSession(origin:'operator')` instead of the raw `killSession`.
|
|
100
|
+
- `src/commands/server.ts` — shared `reapGuardDeps` back BOTH the SessionReaper and the authority's
|
|
101
|
+
ReapGuard (built `minAgeMs:0`); `setAwakeChecker` wires the lease gate (single-machine ⇒ always awake).
|
|
102
|
+
|
|
103
|
+
**Implementation decisions with side-effects (reviewed):**
|
|
104
|
+
1. **Operator kill bypasses `protected`.** The `protected` check moved *inside* the autonomous-only
|
|
105
|
+
gate block, so an `origin:'operator'` kill is no longer blocked on protected sessions. Risk: an
|
|
106
|
+
operator can now kill a protected (e.g. lifeline) session via the dashboard. Mitigation/justification:
|
|
107
|
+
this *preserves* the prior behavior — the old `DELETE` route used `killSession`, which killed
|
|
108
|
+
unconditionally — and matches the spec's "an explicit human kill must always happen". Autonomous
|
|
109
|
+
reapers remain fully blocked on protected.
|
|
110
|
+
2. **`knownDead` bypass for boot-purge.** A `dead` verdict skips the KEEP-guard. Risk: a mis-verdict
|
|
111
|
+
would skip protection. Mitigation: the oracle returns `dead` ONLY on tmux-server-reachable +
|
|
112
|
+
exact-full-id-absent, never on timeout/unreachable/unknown — so `knownDead` is only ever passed for a
|
|
113
|
+
provably-gone session, which has no liveness to protect. Without the bypass, the guard's
|
|
114
|
+
liveness-blind topic-state KEEPs (recent-user-message / open-commitment) would pin a dead record in
|
|
115
|
+
the running list and re-create the boot death-spiral.
|
|
116
|
+
3. **`DELETE` route now emits `beforeSessionKill`/`sessionComplete`/`sessionReaped`.** The old
|
|
117
|
+
`killSession` path historically emitted none of these. Effect: resume-UUID listeners now fire on an
|
|
118
|
+
operator kill (beneficial — a manually-killed topic session can be resumed), and the operator kill
|
|
119
|
+
lands in the reap-log (P4). No double-emit: `killSession` is no longer on this path.
|
|
120
|
+
|
|
121
|
+
**Signal vs authority:** unchanged — killers *request*, the authority decides. The only callers that
|
|
122
|
+
may mint `origin:'operator'` are Bearer-authed HTTP routes; in-process killers default to `autonomous`
|
|
123
|
+
and so always pass through the gates.
|
|
124
|
+
|
|
125
|
+
**Re-entrancy:** the guard is consulted before the in-flight (`terminating`) lock is acquired, so the
|
|
126
|
+
authority never misreads its own lock as a KEEP (spec §P0 ordering).
|
|
127
|
+
|
|
128
|
+
**Rollback:** additive per killer; reverting restores the inline kills. P0 gates default to the prior
|
|
129
|
+
behavior when `setReapGuard`/`setAwakeChecker` are unset (tests/standalone) — no guard consult, treated
|
|
130
|
+
as awake.
|
|
131
|
+
|
|
132
|
+
**Tests:** terminate (9) + reaper (30) + guard (15) + oracle (15) + timeout (4+6) + async-monitor (6) +
|
|
133
|
+
lifecycle integration (6) green; typecheck clean. One brittle source-string test updated to the new
|
|
134
|
+
age-kill log + funnel contract.
|
|
135
|
+
|
|
136
|
+
## Commit — P3 reap-notify seam + P4 reap-log + Agent-Awareness + migration
|
|
137
|
+
|
|
138
|
+
**Files:**
|
|
139
|
+
- `src/monitoring/ReapNotifier.ts` (new) — single coalescing `sessionReaped` listener. Silent on
|
|
140
|
+
`recovery-bounce` and `origin:'operator'`; isolated reap → bound topic (or lifeline); burst within
|
|
141
|
+
the window → ONE consolidated lifeline message stating the EXACT total (count tracked separately from
|
|
142
|
+
the bounded detail buffer, so an overflow never under-reports). User-controlled names/reasons wrapped
|
|
143
|
+
as literal inline-code spans (inner backticks neutralized) so the downstream formatter never renders
|
|
144
|
+
them as markup.
|
|
145
|
+
- `src/monitoring/ReapLog.ts` (new) — append-only JSONL audit at `logs/reap-log.jsonl`; records BOTH
|
|
146
|
+
reaps (`sessionReaped`) and refused/skipped terminates (`reapBlocked`); JSON-encoded (no concat →
|
|
147
|
+
no newline injection); read tolerates corrupt lines.
|
|
148
|
+
- `src/server/routes.ts` — `GET /sessions/reap-log` (Bearer-auth via router middleware, read-only,
|
|
149
|
+
`?limit` capped at 1000, default 200).
|
|
150
|
+
- `src/server/AgentServer.ts` — `reapLog` option + ctx wire (mirrors `sessionReaper`).
|
|
151
|
+
- `src/commands/server.ts` — ReapLog + ReapNotifier built and the `sessionReaped`/`reapBlocked`
|
|
152
|
+
listeners + `setAwakeChecker` wired BEFORE the boot purge, so boot reaps are lease-gated, logged, and
|
|
153
|
+
notified. (The KEEP-guard is wired later, after its tracker deps exist — safe: boot-purge bypasses it
|
|
154
|
+
via `knownDead`, and no monitorTick kill can fire in the first seconds.)
|
|
155
|
+
- `src/core/types.ts` + `src/config/ConfigDefaults.ts` — `monitoring.reapNotify {enabled, coalesceWindowMs}`,
|
|
156
|
+
default ON.
|
|
157
|
+
- `src/scaffold/templates.ts` (Agent-Awareness) + `src/core/PostUpdateMigrator.migrateClaudeMd` (existing
|
|
158
|
+
agents get the Reap-Log section) + `migrateConfig` via ConfigDefaults (existing agents get the
|
|
159
|
+
`reapNotify` default automatically).
|
|
160
|
+
|
|
161
|
+
**Notify default = ON (deliberate, differs from sentinelTelegramEscalation).** The silently-stopped
|
|
162
|
+
sentinel escalation defaults OFF (post-2026-05-22 flood). The reap-notify defaults ON because the
|
|
163
|
+
incident this whole spec answers is *silent disappearance* — the user explicitly asked to be told. The
|
|
164
|
+
flood risk is bounded by: (a) coalescing a burst into one message, (b) staying silent on the common
|
|
165
|
+
recovery-bounce + operator paths, (c) SUMMARY tier (quiet-hours aware). A single config flag disables it.
|
|
166
|
+
|
|
167
|
+
**Discoverability:** `/sessions/reap-log` lives under the already-classified `sessions` prefix
|
|
168
|
+
(operator/dashboard-only in CapabilityIndex) — no new prefix, no lint change. Agent-awareness is carried
|
|
169
|
+
by the CLAUDE.md template + migrateClaudeMd instead (the template IS the agent's awareness).
|
|
170
|
+
|
|
171
|
+
**Ordering side-effect (reviewed):** the listeners attach pre-boot-purge; the guard wires post. The only
|
|
172
|
+
guard consumers besides boot-purge are monitorTick #2/#3, which require multi-hour age / 15+m observed
|
|
173
|
+
idle — unreachable in the sub-second guard-unset window. Documented inline.
|
|
174
|
+
|
|
175
|
+
**Tests:** ReapNotifier (10: silent dispositions/origins, isolated bound/unbound routing, burst
|
|
176
|
+
coalescing, overflow exact-count, auto-timer, malicious-name literalization, unreachable-channel drop)
|
|
177
|
+
+ ReapLog (6: empty, reaped/skipped fields, newline-injection→valid-JSON, tail, corrupt-line tolerance)
|
|
178
|
+
green; typecheck clean. Route integration + e2e land in the test-phase commit.
|
|
179
|
+
|
|
180
|
+
## Commit — P5 unkillability backstop
|
|
181
|
+
|
|
182
|
+
**Files:**
|
|
183
|
+
- `src/monitoring/StaleSessionBackstop.ts` (new) — signal-only. Per-tick forward-progress check; after
|
|
184
|
+
M min of no-forward-progress OR N consecutive indeterminate probes, raises ONE per-episode-deduped
|
|
185
|
+
Attention item (never auto-kills). Forward progress = meaningful transcript advance (≥floor AND new
|
|
186
|
+
tail — guards the heartbeat/loop case) OR main-process CPU OR a prompt/idle-state change. A
|
|
187
|
+
control-plane-unreachable tick raises ONE global item, not one per session.
|
|
188
|
+
- `src/core/SessionManager.ts` — `markLongIndeterminate()` + a `longIndeterminateSessions` set excluded
|
|
189
|
+
from the ABSOLUTE `maxSessions × 3` spawn cap (so unverifiable panes can't lock out spawning);
|
|
190
|
+
`probeLivenessBatch()` resolves tri-state liveness + reachability from ONE oracle snapshot.
|
|
191
|
+
- `src/core/types.ts` + `src/config/ConfigDefaults.ts` — `monitoring.staleBackstop`, default ON;
|
|
192
|
+
propagates to existing agents via ConfigDefaults.
|
|
193
|
+
- `src/commands/server.ts` — backstop wired after the SessionReaper; snapshot built from probeTranscript
|
|
194
|
+
(+ tail-hash read so a heartbeat byte isn't "progress"), captureOutput frame-hash, hasActiveProcesses;
|
|
195
|
+
Attention via makeAttentionPoster.
|
|
196
|
+
|
|
197
|
+
**Never-auto-kills (structural):** the backstop's deps surface has NO terminate/kill function — it
|
|
198
|
+
physically cannot end a session. Asserted by a unit test. It only observes and asks.
|
|
199
|
+
|
|
200
|
+
**Oracle interaction (reviewed):** with the committed single-snapshot oracle, a session is only
|
|
201
|
+
`indeterminate` when the whole snapshot is non-authoritative (server unreachable). So the per-session
|
|
202
|
+
indeterminate-streak path is currently reached only under global unreachability (handled by the global
|
|
203
|
+
item); the live, reachable path is the M-minute no-forward-progress escalation for ALIVE-but-faking
|
|
204
|
+
sessions. The per-session indeterminate code is kept (defensive — correct if the oracle later adds
|
|
205
|
+
individual re-probes that return indeterminate while reachable).
|
|
206
|
+
|
|
207
|
+
**Spawn-cap exclusion:** long-indeterminate sessions are removed from the ABSOLUTE-cap count only; they
|
|
208
|
+
still count toward the soft scheduler cap. Cleared the moment a session is verifiable again.
|
|
209
|
+
|
|
210
|
+
**Tests:** StaleSessionBackstop (10: M-min escalation, no-re-raise-per-episode, heartbeat-not-progress,
|
|
211
|
+
meaningful-advance/CPU/prompt-change progress, per-episode re-raise after recovery, global-unreachable
|
|
212
|
+
dedup, long-indeterminate flag set+cleared, never-kills structural) green; typecheck clean.
|
|
213
|
+
|
|
214
|
+
## Commit — Phase 1 test tiers (integration + e2e)
|
|
215
|
+
|
|
216
|
+
**Files:**
|
|
217
|
+
- `tests/integration/reap-log-route.test.ts` (new, 4) — GET /sessions/reap-log through the real
|
|
218
|
+
createRoutes pipeline: 503-when-unwired, 200 with reaped+skipped entries, ?limit tail, read-only
|
|
219
|
+
(POST/DELETE → 404).
|
|
220
|
+
- `tests/integration/session-lifecycle-reap-wiring.test.ts` (new, 6) — the real SessionManager
|
|
221
|
+
terminateSession authority wired (as server.ts does) to ReapGuard + ReapNotifier + ReapLog, asserting
|
|
222
|
+
through the real emit path: autonomous terminal reap → log + exactly one notice; recovery-bounce →
|
|
223
|
+
logged-but-silent; operator → logged-silent + bypasses guard+protected; relay-lease KEEP → refused +
|
|
224
|
+
survives + logged 'skipped'; standby → refused + survives; protected → refused + survives.
|
|
225
|
+
- `tests/e2e/reap-log-lifecycle.test.ts` (new, 3) — boots the REAL AgentServer: GET /sessions/reap-log
|
|
226
|
+
is alive (200 not 503), surfaces recorded entries, requires Bearer, read-only.
|
|
227
|
+
|
|
228
|
+
**Reproduce-before-claim status (honest):** the boot-purge false-purge is reproduced at the unit tier
|
|
229
|
+
(death-spiral-fixes — the 2026-05-27 9-of-9 incident now yields 0 purges); "one terminal reap → exactly
|
|
230
|
+
one notice" and "guarded/standby/protected survive" are reproduced through the real SessionManager event
|
|
231
|
+
path (integration); the route is proven alive on the real AgentServer (e2e). A full live boot with a
|
|
232
|
+
deliberately-slowed tmux + a real Telegram notice landing was NOT performed in this environment — the
|
|
233
|
+
test-tier reproductions stand in for it, and that gap is stated rather than papered over.
|
|
234
|
+
|
|
235
|
+
## Commit — annotate intentional silent catches (no-silent-fallbacks ratchet)
|
|
236
|
+
|
|
237
|
+
Two new intentional silent catches are annotated `@silent-fallback-ok` so the
|
|
238
|
+
no-silent-fallbacks ratchet doesn't count them: `ReapLog.read` (no log file ⇒
|
|
239
|
+
empty list is correct, not degraded) and the §P5 transcript-tail read (an
|
|
240
|
+
unreadable tail ⇒ no meaningful-advance signal this tick; treated as ambiguous,
|
|
241
|
+
never as progress). Behavior unchanged.
|