instar 1.3.66 → 1.3.68
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/commands/server.d.ts.map +1 -1
- package/dist/commands/server.js +34 -5
- package/dist/commands/server.js.map +1 -1
- package/dist/core/SessionManager.d.ts +21 -0
- package/dist/core/SessionManager.d.ts.map +1 -1
- package/dist/core/SessionManager.js +48 -1
- package/dist/core/SessionManager.js.map +1 -1
- package/dist/core/SleepWakeDetector.d.ts +13 -0
- package/dist/core/SleepWakeDetector.d.ts.map +1 -1
- package/dist/core/SleepWakeDetector.js +26 -0
- package/dist/core/SleepWakeDetector.js.map +1 -1
- package/dist/messaging/TelegramAdapter.d.ts +12 -0
- package/dist/messaging/TelegramAdapter.d.ts.map +1 -1
- package/dist/messaging/TelegramAdapter.js +29 -0
- package/dist/messaging/TelegramAdapter.js.map +1 -1
- package/dist/monitoring/OrphanProcessReaper.d.ts +11 -0
- package/dist/monitoring/OrphanProcessReaper.d.ts.map +1 -1
- package/dist/monitoring/OrphanProcessReaper.js +63 -7
- package/dist/monitoring/OrphanProcessReaper.js.map +1 -1
- package/dist/monitoring/QuotaManager.d.ts.map +1 -1
- package/dist/monitoring/QuotaManager.js +59 -0
- package/dist/monitoring/QuotaManager.js.map +1 -1
- package/dist/monitoring/SessionMigrator.d.ts +37 -0
- package/dist/monitoring/SessionMigrator.d.ts.map +1 -1
- package/dist/monitoring/SessionMigrator.js +49 -1
- package/dist/monitoring/SessionMigrator.js.map +1 -1
- package/dist/monitoring/SessionRecovery.d.ts +31 -2
- package/dist/monitoring/SessionRecovery.d.ts.map +1 -1
- package/dist/monitoring/SessionRecovery.js +46 -4
- package/dist/monitoring/SessionRecovery.js.map +1 -1
- package/dist/monitoring/SessionWatchdog.d.ts +0 -1
- package/dist/monitoring/SessionWatchdog.d.ts.map +1 -1
- package/dist/monitoring/SessionWatchdog.js +36 -12
- package/dist/monitoring/SessionWatchdog.js.map +1 -1
- package/dist/scheduler/JobScheduler.d.ts +16 -2
- package/dist/scheduler/JobScheduler.d.ts.map +1 -1
- package/dist/scheduler/JobScheduler.js +50 -5
- package/dist/scheduler/JobScheduler.js.map +1 -1
- package/package.json +1 -1
- package/src/data/builtin-manifest.json +5 -5
- package/upgrades/1.3.65.md +97 -0
- package/upgrades/1.3.67.md +164 -0
- package/upgrades/1.3.68.md +71 -0
- package/upgrades/side-effects/unified-session-lifecycle-robustness.md +191 -0
|
@@ -0,0 +1,164 @@
|
|
|
1
|
+
# Upgrade Guide — vNEXT
|
|
2
|
+
|
|
3
|
+
<!-- bump: patch -->
|
|
4
|
+
|
|
5
|
+
## What Changed
|
|
6
|
+
|
|
7
|
+
**Fixed a runaway credit-burn bug: LLM call loops no longer keep hammering the
|
|
8
|
+
provider after you hit a usage or spend limit.**
|
|
9
|
+
|
|
10
|
+
Instar runs small background LLM calls to watch your sessions (for example, the
|
|
11
|
+
per-tick check that detects when a session is blocked on a prompt). Until now,
|
|
12
|
+
if one of those calls came back with a usage/rate/spend-limit error, the loop
|
|
13
|
+
swallowed the error and tried again on the very next tick — with no backoff. If
|
|
14
|
+
your account had auto-reload turned on, each retry refueled and re-burned. A
|
|
15
|
+
real agent in the wild burned $452 of $455 in usage credits this way over a
|
|
16
|
+
couple of days before anyone noticed.
|
|
17
|
+
|
|
18
|
+
There is now an **account-global circuit breaker** in front of every internal
|
|
19
|
+
LLM call. The moment the provider reports a usage/rate/spend limit, the breaker
|
|
20
|
+
opens and *all* background LLM-backed work pauses — without spawning another
|
|
21
|
+
`claude` subprocess, so it costs nothing while paused. After a cool-down window
|
|
22
|
+
(15 minutes by default) it sends exactly one quiet probe; if the limit has
|
|
23
|
+
lifted it closes and resumes automatically, and if not it waits another window.
|
|
24
|
+
It is wired at the single provider-construction chokepoint, so every feature —
|
|
25
|
+
current and future — is covered with no per-feature work.
|
|
26
|
+
|
|
27
|
+
This is reactive (it listens to the provider's own "you're over limit" signal)
|
|
28
|
+
and complements the existing volume-based burn-detection, which reacts to
|
|
29
|
+
statistical token-share over a longer window.
|
|
30
|
+
|
|
31
|
+
## What to Tell Your User
|
|
32
|
+
|
|
33
|
+
Your agent can no longer burn through your credits by repeatedly calling the
|
|
34
|
+
model after you've hit a usage or spend limit. The instant the provider says
|
|
35
|
+
you're over your limit, background model work pauses on its own, costs nothing
|
|
36
|
+
while paused, and quietly resumes once the limit lifts. This is on by default —
|
|
37
|
+
no setup needed. If you ever want to tune the cool-down window or turn it off,
|
|
38
|
+
there's an optional circuit-breaker setting in your config, but the safe
|
|
39
|
+
defaults are designed to just work.
|
|
40
|
+
|
|
41
|
+
## Summary of New Capabilities
|
|
42
|
+
|
|
43
|
+
| Capability | How to Use |
|
|
44
|
+
|-----------|-----------|
|
|
45
|
+
| Account-global LLM rate-limit circuit breaker | Automatic. When the provider returns a usage/rate/spend-limit error, all background LLM-backed work pauses (no subprocess spawned) and self-heals via a single probe after the cool-down. |
|
|
46
|
+
| Tunable cool-down / kill switch | Optional `intelligence.circuitBreaker` in `.instar/config.json`: `enabled` (default true) and `openMs` (default 900000 = 15 min). Absent config uses the safe defaults, so existing agents are protected with no changes. |
|
|
47
|
+
|
|
48
|
+
## Evidence
|
|
49
|
+
|
|
50
|
+
**Reproduction (the incident pattern), reproduced as an automated test:** a fake
|
|
51
|
+
`claude` binary that records each spawn to an on-disk counter and exits with
|
|
52
|
+
"Claude AI usage limit reached" on stderr.
|
|
53
|
+
`tests/integration/llm-circuit-breaker-chokepoint.test.ts` builds two providers
|
|
54
|
+
through the real `buildIntelligenceProvider` factory and drives them:
|
|
55
|
+
|
|
56
|
+
- Before fix (old behaviour): every `evaluate()` call spawns the binary →
|
|
57
|
+
unbounded spawns while limited (this is exactly the $452 burn).
|
|
58
|
+
- After fix (observed in the test): the first call spawns once and trips the
|
|
59
|
+
breaker; the next 5 calls — and a call through a *second, independently built*
|
|
60
|
+
provider — all reject with `LlmCircuitOpenError` and the spawn counter stays
|
|
61
|
+
at **1**. Zero additional subprocesses, proving the burn is stopped and the
|
|
62
|
+
breaker is account-global.
|
|
63
|
+
|
|
64
|
+
39 new circuit-breaker tests pass (state machine both-sides-of-boundary, the
|
|
65
|
+
rate-limit classifier on true-positive and unrelated-error strings, and the
|
|
66
|
+
chokepoint wiring). `tsc --noEmit` clean. The server log emits
|
|
67
|
+
`[llm-circuit] OPEN: …` / `[llm-circuit] closing: …` lines on each transition.
|
|
68
|
+
|
|
69
|
+
---
|
|
70
|
+
|
|
71
|
+
**Phase 2 of the unified session-lifecycle robustness work (spec
|
|
72
|
+
`docs/specs/unified-session-lifecycle-robustness.md`).** The four remaining
|
|
73
|
+
autonomous session killers — the SessionWatchdog, OrphanProcessReaper,
|
|
74
|
+
SessionRecovery (kill-to-respawn), and the scheduler's wake-reaper — are now
|
|
75
|
+
funneled through the single ReapAuthority that shipped in Phase 1. Each one
|
|
76
|
+
gains the careful KEEP-checks and the awake-machine lease gate the previous
|
|
77
|
+
killer-of-the-week didn't have, plus structural fixes to a few long-standing
|
|
78
|
+
near-misses.
|
|
79
|
+
|
|
80
|
+
What landed:
|
|
81
|
+
|
|
82
|
+
- **#5 SessionWatchdog.** The final escalation level (the session-kill that
|
|
83
|
+
fires after Ctrl+C / SIGTERM / SIGKILL all failed to free a stuck child) now
|
|
84
|
+
routes through `terminateSession('watchdog-stuck', terminal/killed)` instead
|
|
85
|
+
of a raw `tmux kill-session`. If the authority's KEEP-guards refuse the kill
|
|
86
|
+
(a relay lease, an active subagent, a recent user message, etc.), the
|
|
87
|
+
watchdog STANDS DOWN — clears its escalation state, logs the exact refusal
|
|
88
|
+
reason, and lets the §P5 backstop own the operator-decision escalation.
|
|
89
|
+
- **#6 OrphanProcessReaper.** Three changes. (a) Orphan classification is now
|
|
90
|
+
EXACT-id membership in `listKnownTmuxSessions()`, not project-prefix
|
|
91
|
+
startsWith — so a user-created `tmux new -s the-portal-myhand` is `external`,
|
|
92
|
+
not orphan, and the reaper cannot false-reap a user pane. (b) The 60-minute
|
|
93
|
+
age threshold is necessary, not sufficient: a new `processHasActiveChildren`
|
|
94
|
+
work-check (pgrep -P) vetoes the kill when the orphan still has running
|
|
95
|
+
children, and the deferred path logs a `Kept orphan PID … work check vetoed
|
|
96
|
+
reap` action. (c) When the orphan's tmux name matches a currently-tracked
|
|
97
|
+
session, the kill goes through the ReapAuthority for the lifecycle events +
|
|
98
|
+
reap-log entry.
|
|
99
|
+
- **#8 SessionRecovery.** A single shared `killForRecovery()` helper consults
|
|
100
|
+
`hasActiveProcesses` before every kill-to-respawn — if the pane is producing
|
|
101
|
+
work, the JSONL stall/crash reading is unreliable and the recovery defers
|
|
102
|
+
with a structured `deferred-still-working` result. All four recovery paths
|
|
103
|
+
(stall, context-exhaustion, crash, error-loop) route through this helper.
|
|
104
|
+
The actual kill goes through `terminateSession('session-recovery',
|
|
105
|
+
recovery-bounce, bypassRecoveryFlag:true)` so the §P3 "shut down" notice
|
|
106
|
+
stays silent on a bounce (which is not a disappearance) and the reap-log
|
|
107
|
+
records the recovery-bounce disposition. `bypassRecoveryFlag` is narrowly
|
|
108
|
+
scoped to the `recovery-in-flight` KEEP-guard reason — all other KEEP-
|
|
109
|
+
guards still apply, so a session mid-conversation isn't killed-to-respawn
|
|
110
|
+
under the cover of "recovery."
|
|
111
|
+
- **#9 wake-reaper.** Three changes. (a) The threshold-check now uses
|
|
112
|
+
CUMULATIVE wall-time-asleep during the run (the new
|
|
113
|
+
`SleepWakeDetector.getCumulativeSleepMsBetween()`), not the single last
|
|
114
|
+
`sleepDurationSeconds` event — a job that spanned multiple sleeps was
|
|
115
|
+
previously credited only the last sleep and reaped early. (b) The P1/P2 work
|
|
116
|
+
gate keeps a session whose process is still producing work, regardless of
|
|
117
|
+
clock. (c) The kill routes through `terminateSession('wake-reaper',
|
|
118
|
+
terminal/killed)` when tracked.
|
|
119
|
+
|
|
120
|
+
All four killers now share the same brain — the structural guarantee the
|
|
121
|
+
spec's audit named when it found "is this session alive/dead/stuck/working?"
|
|
122
|
+
answered ad-hoc in eight different places.
|
|
123
|
+
|
|
124
|
+
## What to Tell Your User
|
|
125
|
+
|
|
126
|
+
- Every part of the agent that can shut a session down — the stuck-child
|
|
127
|
+
watchdog, the orphan-process cleanup, the crash-recovery, and the after-
|
|
128
|
+
sleep cleanup — now goes through the same careful gatekeeper that the boot
|
|
129
|
+
cleanup uses. None of them can quietly kill a session that might still be
|
|
130
|
+
working.
|
|
131
|
+
- The watchdog's last-resort session kill, the crash-recovery's restart, and
|
|
132
|
+
the after-sleep cleanup all show up in the reap-log with the reason, so the
|
|
133
|
+
"where did my session go?" answer is always there.
|
|
134
|
+
- A crash-recovery restart (kill + respawn fresh) stays quiet — it's a bounce,
|
|
135
|
+
not a disappearance — but it's still logged.
|
|
136
|
+
- The orphan-process cleanup can no longer accidentally classify a tmux pane
|
|
137
|
+
you started yourself as one of mine, and it will defer the kill if the
|
|
138
|
+
orphan still has active child processes.
|
|
139
|
+
- After your machine sleeps and wakes, the job cleanup correctly subtracts the
|
|
140
|
+
total sleep time (across multiple sleeps if needed) before deciding a job
|
|
141
|
+
ran too long, so a job that was paused while you closed the lid isn't
|
|
142
|
+
marked stuck.
|
|
143
|
+
|
|
144
|
+
## Summary of New Capabilities
|
|
145
|
+
|
|
146
|
+
| Capability | How to Use |
|
|
147
|
+
|-----------|-----------|
|
|
148
|
+
| Every autonomous killer routes through the single ReapAuthority | Automatic — the watchdog, orphan reaper, crash-recovery, and wake-reaper all funnel through `terminateSession`. |
|
|
149
|
+
| Recovery-bounce disposition stays silent in the "shut down" notice | Automatic — a kill-to-respawn doesn't pretend a session vanished; it still lands in the reap-log. |
|
|
150
|
+
| Orphan reaper: exact-id classification + work-check + ReapAuthority routing | Automatic — `listKnownTmuxSessions()` replaces the prefix match; `processHasActiveChildren` vetoes reaping a working orphan. |
|
|
151
|
+
| Wake-reaper: cumulative-sleep math + P1/P2 work gate | Automatic — multi-sleep runs no longer reaped early; an actively-working session is kept regardless of clock. |
|
|
152
|
+
|
|
153
|
+
## Evidence
|
|
154
|
+
|
|
155
|
+
- All Phase 1 unit/integration/e2e tests remain green, plus 14 new Phase-2
|
|
156
|
+
wiring contracts (SessionWatchdog → ReapAuthority routing + KEEP stand-down,
|
|
157
|
+
OrphanProcessReaper exact-id + work-check + P0, SessionRecovery
|
|
158
|
+
killForRecovery + recovery-bounce + bypassRecoveryFlag scope, wake-reaper
|
|
159
|
+
cumulative-sleep + P1/P2 + P0), 6 new SleepWakeDetector cumulative-sleep
|
|
160
|
+
math tests, and the existing JobScheduler reaper suite (7) updated to the
|
|
161
|
+
new async + effective-elapsed contract.
|
|
162
|
+
- `tsc --noEmit` clean. Existing SessionWatchdog (58), OrphanProcessReaper
|
|
163
|
+
(9), SessionRecovery (9), terminate-CAS (9), session-timeout/activity-aware
|
|
164
|
+
(4+6) tests all still pass.
|
|
@@ -0,0 +1,71 @@
|
|
|
1
|
+
# Upgrade Guide — vNEXT
|
|
2
|
+
|
|
3
|
+
<!-- bump: patch -->
|
|
4
|
+
|
|
5
|
+
## What Changed
|
|
6
|
+
|
|
7
|
+
**Phase 3 of the unified session-lifecycle robustness work (spec
|
|
8
|
+
`docs/specs/unified-session-lifecycle-robustness.md`) — the last two items.**
|
|
9
|
+
With Phase 1 and Phase 2 already on main, this completes the converged design:
|
|
10
|
+
every autonomous killer shares the same brain, and the small UX bonus you
|
|
11
|
+
asked for is in.
|
|
12
|
+
|
|
13
|
+
What landed:
|
|
14
|
+
|
|
15
|
+
- **#7 quota soft-check (bounded).** The quota-driven session migrator (the
|
|
16
|
+
thing that halts running sessions to switch accounts when you're running out
|
|
17
|
+
of weekly/5-hour budget) used to send Ctrl+C, wait `gracePeriodMs`, and then
|
|
18
|
+
force-kill anyone still alive. Now: when current usage is at or below a
|
|
19
|
+
configurable ceiling (`softCheckMaxUsagePercent`, default 95%), an
|
|
20
|
+
active build or autonomous run gets ONE extra Ctrl+C grace round before
|
|
21
|
+
force-kill. ABOVE the ceiling the soft check is **disabled** so quota's
|
|
22
|
+
final authority is never undermined; if current usage can't be read,
|
|
23
|
+
it's treated as 100% (fail-closed — no grace without proof we're below
|
|
24
|
+
the ceiling). The force-kill itself now goes through the single
|
|
25
|
+
ReapAuthority (`terminateSession('quota-shed', terminal/killed)`) so the
|
|
26
|
+
"your session was shut down" notice fires + the reap-log records the
|
|
27
|
+
`quota-shed` reason. A new structured `quota-force-kill-decision` event
|
|
28
|
+
publishes the inputs for a future Tier-1 supervisor.
|
|
29
|
+
- **Bonus — session label follows topic rename.** When you rename a Telegram
|
|
30
|
+
forum topic, the bound session's display `name` updates to match. The
|
|
31
|
+
operational identity (`tmuxSession` key + `id` UUID) is never touched —
|
|
32
|
+
every internal lookup keeps working. The handler fires ONLY on a true
|
|
33
|
+
rename event (`forum_topic_edited`), not on initial-capture or topic
|
|
34
|
+
creation. The renamed value flows through the same §P3 sanitization
|
|
35
|
+
(literal code spans) wherever it surfaces in user-facing notices, so a
|
|
36
|
+
malicious name can't inject markup.
|
|
37
|
+
|
|
38
|
+
This closes the spec's Phase 3 scope. After this, all eight autonomous
|
|
39
|
+
killers (boot purge, age-kill, idle-zombie, watchdog, orphan reaper,
|
|
40
|
+
SessionRecovery, wake-reaper, quota-shed) share the same authority + KEEP-
|
|
41
|
+
guard + lease gate + reap-log + sessionReaped event.
|
|
42
|
+
|
|
43
|
+
## What to Tell Your User
|
|
44
|
+
|
|
45
|
+
- When you're running low on hours, the quota cleanup is gentler: a
|
|
46
|
+
session that's actively building or running an autonomous task gets one
|
|
47
|
+
more polite Ctrl+C before being shut down — unless we're already very
|
|
48
|
+
near the limit, in which case quota's final authority still kicks in
|
|
49
|
+
immediately. Either way, the shutdown shows up in the reap-log with the
|
|
50
|
+
reason.
|
|
51
|
+
- When you rename a Telegram topic, the session label updates to match
|
|
52
|
+
what you renamed it. The agent's plumbing keeps working — only the
|
|
53
|
+
display name changes.
|
|
54
|
+
|
|
55
|
+
## Summary of New Capabilities
|
|
56
|
+
|
|
57
|
+
| Capability | How to Use |
|
|
58
|
+
|-----------|-----------|
|
|
59
|
+
| Bounded quota soft-check (one extra Ctrl+C grace for working sessions) | Automatic. Tunable via `monitoring.sessionMigration.thresholds.softCheckEnabled` / `softCheckMaxUsagePercent` (default 95) / `softCheckExtraGraceMs` (default = `gracePeriodMs`). |
|
|
60
|
+
| Quota force-kill via ReapAuthority + `quota-force-kill-decision` event | Automatic — surfaces "your session was shut down — quota-shed" + lands in the reap-log; a future Tier-1 LLM supervisor can subscribe to the decision event. |
|
|
61
|
+
| Session label follows Telegram topic rename | Automatic — rename a forum topic, the bound session's display name follows. |
|
|
62
|
+
|
|
63
|
+
## Evidence
|
|
64
|
+
|
|
65
|
+
- Unit: SessionManager rename (5) + Phase-3 wiring (9) green; SessionMigrator
|
|
66
|
+
existing (37) still passes; SessionWatchdog (58), OrphanProcessReaper (9),
|
|
67
|
+
SessionRecovery (9), terminate-CAS (9), JobScheduler reaper (7),
|
|
68
|
+
SleepWakeDetector cumulative (6) — all still green.
|
|
69
|
+
- `tsc --noEmit` clean.
|
|
70
|
+
- Side-effects review updated per-commit:
|
|
71
|
+
`upgrades/side-effects/unified-session-lifecycle-robustness.md`.
|
|
@@ -239,3 +239,194 @@ no-silent-fallbacks ratchet doesn't count them: `ReapLog.read` (no log file ⇒
|
|
|
239
239
|
empty list is correct, not degraded) and the §P5 transcript-tail read (an
|
|
240
240
|
unreadable tail ⇒ no meaningful-advance signal this tick; treated as ambiguous,
|
|
241
241
|
never as progress). Behavior unchanged.
|
|
242
|
+
|
|
243
|
+
## Phase 2 — Commit #5: SessionWatchdog → ReapAuthority
|
|
244
|
+
|
|
245
|
+
**Files:** `src/monitoring/SessionWatchdog.ts`,
|
|
246
|
+
`tests/unit/session-lifecycle-phase-2-wiring.test.ts` (new).
|
|
247
|
+
|
|
248
|
+
`handleEscalation` is now `async`; the final `EscalationLevel.KillSession` no
|
|
249
|
+
longer calls a raw `tmux kill-session` — it resolves the instar session by
|
|
250
|
+
`tmuxSession` and routes through
|
|
251
|
+
`sessionManager.terminateSession(id, 'watchdog-stuck', { disposition: 'terminal', finalStatus: 'killed' })`.
|
|
252
|
+
The authority's mandatory KEEP-guard + lease gate + protected-set check now
|
|
253
|
+
apply: a session the guards would have kept survives a buggy watchdog. The raw
|
|
254
|
+
`killTmuxSession` method is deleted; the LLM gate + stdin guard + signal
|
|
255
|
+
escalations (CtrlC / SigTerm / SigKill) are unchanged.
|
|
256
|
+
|
|
257
|
+
**Stand-down on KEEP (anti-thrash):** if the authority refuses the kill
|
|
258
|
+
(`protected` / `not-lease-holder` / a KEEP-guard hold / `in-flight`), the
|
|
259
|
+
watchdog logs the exact reason, records a `kept (<reason>)` intervention, and
|
|
260
|
+
**clears escalationState**. It does not re-escalate against a guarded session
|
|
261
|
+
every tick — the §P5 backstop owns operator-decision escalation for a
|
|
262
|
+
persistently-stale session.
|
|
263
|
+
|
|
264
|
+
**Tests:** existing SessionWatchdog tests (58) green; new wiring contract (3) green; typecheck clean.
|
|
265
|
+
|
|
266
|
+
**Rollback:** revert this commit; the raw `killTmuxSession` is the only thing
|
|
267
|
+
deleted and is recoverable from history.
|
|
268
|
+
|
|
269
|
+
## Phase 2 — Commit #6: OrphanProcessReaper → exact-id + work-check + ReapAuthority
|
|
270
|
+
|
|
271
|
+
**Files:** `src/monitoring/OrphanProcessReaper.ts`, `src/core/SessionManager.ts`,
|
|
272
|
+
`tests/unit/session-lifecycle-phase-2-wiring.test.ts`.
|
|
273
|
+
|
|
274
|
+
Three spec-faithful changes:
|
|
275
|
+
1. **Exact-id classification.** The legacy `tmuxSession.startsWith(this.projectPrefix)`
|
|
276
|
+
substring match is replaced by EXACT membership in
|
|
277
|
+
`SessionManager.listKnownTmuxSessions()` — the new method returns every tmux
|
|
278
|
+
name instar has ever tracked (any status). A user-created tmux session that
|
|
279
|
+
happens to share the project prefix is now classified `external`, not
|
|
280
|
+
`instar-orphan`, so the reaper cannot false-reap a user pane.
|
|
281
|
+
2. **60-min age = necessary, not sufficient.** Before any orphan kill the new
|
|
282
|
+
`processHasActiveChildren(pid)` helper (`pgrep -P`) is consulted; a positive
|
|
283
|
+
result vetoes the reap and surfaces a `Kept orphan PID … work check vetoed
|
|
284
|
+
reap` action. An uninspectable result returns `false` (allowed to proceed)
|
|
285
|
+
to preserve prior behavior — this gate is a soft veto layered on top of the
|
|
286
|
+
age gate, never an escalation.
|
|
287
|
+
3. **Route through P0 when applicable.** When the orphan's tmuxSession matches a
|
|
288
|
+
CURRENTLY-tracked session (a scan/refresh race), the kill goes through
|
|
289
|
+
`terminateSession(id, 'orphan-reap', { disposition:'terminal', finalStatus:'killed' })`
|
|
290
|
+
so the authority's KEEP-guard + lease gate + reap-log entry + `sessionReaped`
|
|
291
|
+
emission apply. Otherwise the tracked record is already terminal and the raw
|
|
292
|
+
tmux-kill cleans up the stray pane.
|
|
293
|
+
|
|
294
|
+
The server session remains excluded unconditionally.
|
|
295
|
+
|
|
296
|
+
**Tests:** OrphanProcessReaper (9) + Phase-2 wiring (6) green; typecheck clean.
|
|
297
|
+
|
|
298
|
+
**Rollback:** revert this commit. `listKnownTmuxSessions()` becomes unused but harmless.
|
|
299
|
+
|
|
300
|
+
## Phase 2 — Commit #8: SessionRecovery → P1/P2 cross-check + recovery-bounce disposition
|
|
301
|
+
|
|
302
|
+
**Files:** `src/monitoring/SessionRecovery.ts`, `src/core/SessionManager.ts`,
|
|
303
|
+
`src/commands/server.ts`, `tests/unit/session-lifecycle-phase-2-wiring.test.ts`.
|
|
304
|
+
|
|
305
|
+
Three spec-faithful changes:
|
|
306
|
+
1. **P1/P2 cross-check** — a new shared `killForRecovery(name)` helper inside
|
|
307
|
+
`SessionRecovery` consults `deps.hasActiveProcesses` BEFORE any kill. When
|
|
308
|
+
the tmux pane has active child processes, the recovery DEFERS this attempt
|
|
309
|
+
and returns a structured `deferred-still-working` result with the correct
|
|
310
|
+
`failureType` (stall / context_exhaustion / crash / error_loop). All four
|
|
311
|
+
recovery paths route through this helper — `deps.killSession` is no longer
|
|
312
|
+
called directly from any recovery method.
|
|
313
|
+
2. **`disposition:'recovery-bounce'`** — the dep's `killSession` is rewired in
|
|
314
|
+
server.ts to route through
|
|
315
|
+
`sessionManager.terminateSession(id, 'session-recovery', { disposition:
|
|
316
|
+
'recovery-bounce', finalStatus: 'killed', bypassRecoveryFlag: true })`. The
|
|
317
|
+
§P3 notifier is silent on recovery-bounce, so the user isn't told "shut
|
|
318
|
+
down" when the bounce immediately respawns; the reap-log still records it.
|
|
319
|
+
3. **`bypassRecoveryFlag` (scoped)** — terminateSession gains an opt that
|
|
320
|
+
skips ONLY the `recovery-in-flight` KEEP-guard reason (not the whole guard).
|
|
321
|
+
Without it the recovery would refuse to kill its OWN in-flight session.
|
|
322
|
+
Other KEEP-guards (active subagent, recent-user-message, etc.) still apply
|
|
323
|
+
so a session mid-conversation is not killed-to-respawn under the cover of
|
|
324
|
+
"recovery."
|
|
325
|
+
|
|
326
|
+
`SessionRecoveryDeps.killSession` is now `() => void | Promise<void>` so the
|
|
327
|
+
implementation can await terminateSession.
|
|
328
|
+
|
|
329
|
+
**Tests:** SessionRecovery (9) + terminate-CAS (9) + Phase-2 wiring (10) green; typecheck clean.
|
|
330
|
+
|
|
331
|
+
**Rollback:** revert this commit. `bypassRecoveryFlag` becomes unused but harmless.
|
|
332
|
+
|
|
333
|
+
## Phase 2 — Commit #9: wake-reaper → P1/P2 + cumulative sleep + ReapAuthority
|
|
334
|
+
|
|
335
|
+
**Files:** `src/scheduler/JobScheduler.ts`, `src/core/SleepWakeDetector.ts`,
|
|
336
|
+
`src/commands/server.ts`, `tests/unit/JobScheduler-reaper.test.ts`,
|
|
337
|
+
`tests/unit/sleep-wake-cumulative.test.ts` (new),
|
|
338
|
+
`tests/unit/session-lifecycle-phase-2-wiring.test.ts`.
|
|
339
|
+
|
|
340
|
+
Three spec-faithful changes:
|
|
341
|
+
1. **Cumulative sleep (SE-8).** `SleepWakeDetector.getCumulativeSleepMsBetween(startMs, endMs)`
|
|
342
|
+
sums the overlap of every recorded sleep window with the query range. The
|
|
343
|
+
wake-reaper now reads `cumulativeSleepProvider(runStartMs, now)` per run and
|
|
344
|
+
uses `effectiveElapsed = elapsedMs − cumulativeSleep` against its threshold —
|
|
345
|
+
a run that spanned multiple sleeps is no longer reaped early because the old
|
|
346
|
+
code credited only the single last `sleepDurationSeconds` event.
|
|
347
|
+
2. **P1/P2 gate.** Before any kill, `sessionManager.hasActiveProcesses(sessionName)`
|
|
348
|
+
is consulted; on `true` the run is KEPT regardless of clock (spec: "a
|
|
349
|
+
progressing process is KEEP regardless of clock"). The deferred run is
|
|
350
|
+
counted as `skipped` and a structured log line names the reason.
|
|
351
|
+
3. **Route through P0.** When a stuck-run session is currently tracked, the
|
|
352
|
+
kill goes through
|
|
353
|
+
`terminateSession(id, 'wake-reaper', { disposition:'terminal', finalStatus:'killed' })`
|
|
354
|
+
— the authority's protected-set + lease gate + reap-log + `sessionReaped`
|
|
355
|
+
emission apply. Untracked tmux panes fall back to the raw `killSession`.
|
|
356
|
+
|
|
357
|
+
`reapStuckRuns` is now async; the timeout error message now names the effective-
|
|
358
|
+
elapsed + cumulative-sleep subtraction instead of the single last sleep event.
|
|
359
|
+
|
|
360
|
+
**Tests:** JobScheduler reaper (7, updated to async + new error contract) +
|
|
361
|
+
SleepWakeDetector cumulative (6, new) + Phase-2 wiring (14) green; typecheck clean.
|
|
362
|
+
|
|
363
|
+
**Rollback:** revert this commit. `getCumulativeSleepMsBetween` becomes unused
|
|
364
|
+
but harmless.
|
|
365
|
+
|
|
366
|
+
## Phase 3 — Commit #7: quota soft-check (bounded, force-kill via ReapAuthority)
|
|
367
|
+
|
|
368
|
+
**Files:** `src/monitoring/SessionMigrator.ts`, `src/monitoring/QuotaManager.ts`,
|
|
369
|
+
`tests/unit/session-lifecycle-phase-3-wiring.test.ts` (new).
|
|
370
|
+
|
|
371
|
+
The quota migrator's force-kill path now goes through the single ReapAuthority,
|
|
372
|
+
and grants a bounded extra-grace round to working sessions — without ever
|
|
373
|
+
letting that grace push real usage to 100%/lockout.
|
|
374
|
+
|
|
375
|
+
1. **Bounded soft-check (SE-9).** New `MigrationThresholds`:
|
|
376
|
+
`softCheckEnabled` (default true), `softCheckMaxUsagePercent` (default 95),
|
|
377
|
+
`softCheckExtraGraceMs` (default = `gracePeriodMs`). The kill loop computes
|
|
378
|
+
`softCheckActive = softEnabled && currentUsagePct ≤ softCeilingPct`. **Above**
|
|
379
|
+
the ceiling the soft check is **disabled** — quota's final authority cannot
|
|
380
|
+
be undermined when usage is already near 100%. When unknown,
|
|
381
|
+
`quotaUsagePercent` falls back to **100** (fail-closed posture: no grace
|
|
382
|
+
without proof we're below the ceiling).
|
|
383
|
+
2. **One extra Ctrl+C grace round.** When the soft check is active AND
|
|
384
|
+
`isBuildOrAutonomousActive()` returns true, the session gets ONE more C-c +
|
|
385
|
+
`softCheckExtraGraceMs` wait before force-kill. The current implementation
|
|
386
|
+
uses a coarse signal — `state/build/build-state.json` fresh OR any
|
|
387
|
+
`autonomous/*.local.md` fresh under 30 min — which trades per-topic
|
|
388
|
+
precision for simplicity (the ceiling backstop bounds the worst case).
|
|
389
|
+
3. **Route through ReapAuthority.** The force-kill goes through
|
|
390
|
+
`deps.terminateSession(id, 'quota-shed', { disposition: 'terminal',
|
|
391
|
+
finalStatus: 'killed' })` so the §P3 notifier surfaces "your session was
|
|
392
|
+
shut down — quota-shed" and the reap-log records it. Falls back to
|
|
393
|
+
`deps.killSession` only if `terminateSession` is unwired (older agents).
|
|
394
|
+
4. **Tier-1 supervision seam.** A new `quota-force-kill-decision` event is
|
|
395
|
+
emitted with `{ tmuxSession, sessionId, jobSlug, currentUsagePct,
|
|
396
|
+
softCheckActive, workingSoftCheckFired }` so a future Haiku-wrapping
|
|
397
|
+
supervisor can validate the policy decision. The event is observability,
|
|
398
|
+
not a gate — the kill still happens.
|
|
399
|
+
|
|
400
|
+
**Tests:** session-migrator existing (37) + Phase-3 wiring (6) green; typecheck clean.
|
|
401
|
+
|
|
402
|
+
**Rollback:** revert this commit. The new thresholds default to safe values and
|
|
403
|
+
the migrator dep stays optional, so older callers are unaffected.
|
|
404
|
+
|
|
405
|
+
## Phase 3 — Commit (bonus): session label follows topic rename
|
|
406
|
+
|
|
407
|
+
**Files:** `src/core/SessionManager.ts`, `src/messaging/TelegramAdapter.ts`,
|
|
408
|
+
`src/commands/server.ts`, `tests/unit/session-rename-by-tmux.test.ts` (new),
|
|
409
|
+
`tests/unit/session-lifecycle-phase-3-wiring.test.ts`.
|
|
410
|
+
|
|
411
|
+
When the user renames a Telegram forum topic, the bound session's DISPLAY name
|
|
412
|
+
follows the rename — but the operational identity (`tmuxSession` key + `id`
|
|
413
|
+
UUID) stays intact, as the spec requires.
|
|
414
|
+
|
|
415
|
+
1. **`SessionManager.renameSessionByTmux(tmuxSession, newName)`** — updates
|
|
416
|
+
`session.name` ONLY. NEVER touches `tmuxSession` (the tmux key + every
|
|
417
|
+
internal lookup) or `id` (state lookups). Idempotent; safe on unknown
|
|
418
|
+
tmuxSession; rejects empty / whitespace-only / non-string names.
|
|
419
|
+
2. **`TelegramAdapter.setTopicRenamedHandler`** — wires a fire-and-forget
|
|
420
|
+
callback fired ONLY on a true rename (`forum_topic_edited` present AND the
|
|
421
|
+
name changed). Initial-capture and topic-creation cases do not fire.
|
|
422
|
+
3. **server.ts wiring** — `setTopicRenamedHandler((topicId, newName) => {
|
|
423
|
+
sessionManager.renameSessionByTmux(telegram.getSessionForTopic(topicId), newName); })`.
|
|
424
|
+
|
|
425
|
+
Because the renamed value is user-controlled, it flows through the same §P3
|
|
426
|
+
sanitization (literal code spans) wherever it surfaces in user-facing notices.
|
|
427
|
+
|
|
428
|
+
**Tests:** SessionManager rename (5) + Phase-3 wiring (9, including the bonus
|
|
429
|
+
contracts) green; typecheck clean.
|
|
430
|
+
|
|
431
|
+
**Rollback:** revert this commit; the handler stays optional, so older callers
|
|
432
|
+
are unaffected.
|