instar 1.3.65 → 1.3.67
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/commands/server.d.ts.map +1 -1
- package/dist/commands/server.js +66 -5
- package/dist/commands/server.js.map +1 -1
- package/dist/core/SessionManager.d.ts +10 -0
- package/dist/core/SessionManager.d.ts.map +1 -1
- package/dist/core/SessionManager.js +23 -1
- package/dist/core/SessionManager.js.map +1 -1
- package/dist/core/SleepWakeDetector.d.ts +13 -0
- package/dist/core/SleepWakeDetector.d.ts.map +1 -1
- package/dist/core/SleepWakeDetector.js +26 -0
- package/dist/core/SleepWakeDetector.js.map +1 -1
- package/dist/core/types.d.ts +16 -0
- package/dist/core/types.d.ts.map +1 -1
- package/dist/core/types.js.map +1 -1
- package/dist/monitoring/CollaborationRedriveEngine.d.ts +95 -0
- package/dist/monitoring/CollaborationRedriveEngine.d.ts.map +1 -0
- package/dist/monitoring/CollaborationRedriveEngine.js +342 -0
- package/dist/monitoring/CollaborationRedriveEngine.js.map +1 -0
- package/dist/monitoring/CommitmentTracker.d.ts +13 -0
- package/dist/monitoring/CommitmentTracker.d.ts.map +1 -1
- package/dist/monitoring/CommitmentTracker.js +10 -0
- package/dist/monitoring/CommitmentTracker.js.map +1 -1
- package/dist/monitoring/OrphanProcessReaper.d.ts +11 -0
- package/dist/monitoring/OrphanProcessReaper.d.ts.map +1 -1
- package/dist/monitoring/OrphanProcessReaper.js +63 -7
- package/dist/monitoring/OrphanProcessReaper.js.map +1 -1
- package/dist/monitoring/SessionRecovery.d.ts +31 -2
- package/dist/monitoring/SessionRecovery.d.ts.map +1 -1
- package/dist/monitoring/SessionRecovery.js +46 -4
- package/dist/monitoring/SessionRecovery.js.map +1 -1
- package/dist/monitoring/SessionWatchdog.d.ts +0 -1
- package/dist/monitoring/SessionWatchdog.d.ts.map +1 -1
- package/dist/monitoring/SessionWatchdog.js +36 -12
- package/dist/monitoring/SessionWatchdog.js.map +1 -1
- package/dist/scheduler/JobScheduler.d.ts +16 -2
- package/dist/scheduler/JobScheduler.d.ts.map +1 -1
- package/dist/scheduler/JobScheduler.js +50 -5
- package/dist/scheduler/JobScheduler.js.map +1 -1
- package/package.json +1 -1
- package/src/data/builtin-manifest.json +5 -5
- package/upgrades/1.3.65.md +97 -0
- package/upgrades/1.3.66.md +28 -0
- package/upgrades/1.3.67.md +164 -0
- package/upgrades/side-effects/collaboration-redrive-engine.md +36 -0
- package/upgrades/side-effects/unified-session-lifecycle-robustness.md +123 -0
package/upgrades/1.3.65.md
CHANGED
|
@@ -65,3 +65,100 @@ through the real `buildIntelligenceProvider` factory and drives them:
|
|
|
65
65
|
rate-limit classifier on true-positive and unrelated-error strings, and the
|
|
66
66
|
chokepoint wiring). `tsc --noEmit` clean. The server log emits
|
|
67
67
|
`[llm-circuit] OPEN: …` / `[llm-circuit] closing: …` lines on each transition.
|
|
68
|
+
|
|
69
|
+
---
|
|
70
|
+
|
|
71
|
+
**Phase 2 of the unified session-lifecycle robustness work (spec
|
|
72
|
+
`docs/specs/unified-session-lifecycle-robustness.md`).** The four remaining
|
|
73
|
+
autonomous session killers — the SessionWatchdog, OrphanProcessReaper,
|
|
74
|
+
SessionRecovery (kill-to-respawn), and the scheduler's wake-reaper — are now
|
|
75
|
+
funneled through the single ReapAuthority that shipped in Phase 1. Each one
|
|
76
|
+
gains the careful KEEP-checks and the awake-machine lease gate the previous
|
|
77
|
+
killer-of-the-week didn't have, plus structural fixes to a few long-standing
|
|
78
|
+
near-misses.
|
|
79
|
+
|
|
80
|
+
What landed:
|
|
81
|
+
|
|
82
|
+
- **#5 SessionWatchdog.** The final escalation level (the session-kill that
|
|
83
|
+
fires after Ctrl+C / SIGTERM / SIGKILL all failed to free a stuck child) now
|
|
84
|
+
routes through `terminateSession('watchdog-stuck', terminal/killed)` instead
|
|
85
|
+
of a raw `tmux kill-session`. If the authority's KEEP-guards refuse the kill
|
|
86
|
+
(a relay lease, an active subagent, a recent user message, etc.), the
|
|
87
|
+
watchdog STANDS DOWN — clears its escalation state, logs the exact refusal
|
|
88
|
+
reason, and lets the §P5 backstop own the operator-decision escalation.
|
|
89
|
+
- **#6 OrphanProcessReaper.** Three changes. (a) Orphan classification is now
|
|
90
|
+
EXACT-id membership in `listKnownTmuxSessions()`, not project-prefix
|
|
91
|
+
startsWith — so a user-created `tmux new -s the-portal-myhand` is `external`,
|
|
92
|
+
not orphan, and the reaper cannot false-reap a user pane. (b) The 60-minute
|
|
93
|
+
age threshold is necessary, not sufficient: a new `processHasActiveChildren`
|
|
94
|
+
work-check (pgrep -P) vetoes the kill when the orphan still has running
|
|
95
|
+
children, and the deferred path logs a `Kept orphan PID … work check vetoed
|
|
96
|
+
reap` action. (c) When the orphan's tmux name matches a currently-tracked
|
|
97
|
+
session, the kill goes through the ReapAuthority for the lifecycle events +
|
|
98
|
+
reap-log entry.
|
|
99
|
+
- **#8 SessionRecovery.** A single shared `killForRecovery()` helper consults
|
|
100
|
+
`hasActiveProcesses` before every kill-to-respawn — if the pane is producing
|
|
101
|
+
work, the JSONL stall/crash reading is unreliable and the recovery defers
|
|
102
|
+
with a structured `deferred-still-working` result. All four recovery paths
|
|
103
|
+
(stall, context-exhaustion, crash, error-loop) route through this helper.
|
|
104
|
+
The actual kill goes through `terminateSession('session-recovery',
|
|
105
|
+
recovery-bounce, bypassRecoveryFlag:true)` so the §P3 "shut down" notice
|
|
106
|
+
stays silent on a bounce (which is not a disappearance) and the reap-log
|
|
107
|
+
records the recovery-bounce disposition. `bypassRecoveryFlag` is narrowly
|
|
108
|
+
scoped to the `recovery-in-flight` KEEP-guard reason — all other KEEP-
|
|
109
|
+
guards still apply, so a session mid-conversation isn't killed-to-respawn
|
|
110
|
+
under the cover of "recovery."
|
|
111
|
+
- **#9 wake-reaper.** Three changes. (a) The threshold-check now uses
|
|
112
|
+
CUMULATIVE wall-time-asleep during the run (the new
|
|
113
|
+
`SleepWakeDetector.getCumulativeSleepMsBetween()`), not the single last
|
|
114
|
+
`sleepDurationSeconds` event — a job that spanned multiple sleeps was
|
|
115
|
+
previously credited only the last sleep and reaped early. (b) The P1/P2 work
|
|
116
|
+
gate keeps a session whose process is still producing work, regardless of
|
|
117
|
+
clock. (c) The kill routes through `terminateSession('wake-reaper',
|
|
118
|
+
terminal/killed)` when tracked.
|
|
119
|
+
|
|
120
|
+
All four killers now share the same brain — the structural guarantee the
|
|
121
|
+
spec's audit named when it found "is this session alive/dead/stuck/working?"
|
|
122
|
+
answered ad-hoc in eight different places.
|
|
123
|
+
|
|
124
|
+
## What to Tell Your User
|
|
125
|
+
|
|
126
|
+
- Every part of the agent that can shut a session down — the stuck-child
|
|
127
|
+
watchdog, the orphan-process cleanup, the crash-recovery, and the after-
|
|
128
|
+
sleep cleanup — now goes through the same careful gatekeeper that the boot
|
|
129
|
+
cleanup uses. None of them can quietly kill a session that might still be
|
|
130
|
+
working.
|
|
131
|
+
- The watchdog's last-resort session kill, the crash-recovery's restart, and
|
|
132
|
+
the after-sleep cleanup all show up in the reap-log with the reason, so the
|
|
133
|
+
"where did my session go?" answer is always there.
|
|
134
|
+
- A crash-recovery restart (kill + respawn fresh) stays quiet — it's a bounce,
|
|
135
|
+
not a disappearance — but it's still logged.
|
|
136
|
+
- The orphan-process cleanup can no longer accidentally classify a tmux pane
|
|
137
|
+
you started yourself as one of mine, and it will defer the kill if the
|
|
138
|
+
orphan still has active child processes.
|
|
139
|
+
- After your machine sleeps and wakes, the job cleanup correctly subtracts the
|
|
140
|
+
total sleep time (across multiple sleeps if needed) before deciding a job
|
|
141
|
+
ran too long, so a job that was paused while you closed the lid isn't
|
|
142
|
+
marked stuck.
|
|
143
|
+
|
|
144
|
+
## Summary of New Capabilities
|
|
145
|
+
|
|
146
|
+
| Capability | How to Use |
|
|
147
|
+
|-----------|-----------|
|
|
148
|
+
| Every autonomous killer routes through the single ReapAuthority | Automatic — the watchdog, orphan reaper, crash-recovery, and wake-reaper all funnel through `terminateSession`. |
|
|
149
|
+
| Recovery-bounce disposition stays silent in the "shut down" notice | Automatic — a kill-to-respawn doesn't pretend a session vanished; it still lands in the reap-log. |
|
|
150
|
+
| Orphan reaper: exact-id classification + work-check + ReapAuthority routing | Automatic — `listKnownTmuxSessions()` replaces the prefix match; `processHasActiveChildren` vetoes reaping a working orphan. |
|
|
151
|
+
| Wake-reaper: cumulative-sleep math + P1/P2 work gate | Automatic — multi-sleep runs no longer reaped early; an actively-working session is kept regardless of clock. |
|
|
152
|
+
|
|
153
|
+
## Evidence
|
|
154
|
+
|
|
155
|
+
- All Phase 1 unit/integration/e2e tests remain green, plus 14 new Phase-2
|
|
156
|
+
wiring contracts (SessionWatchdog → ReapAuthority routing + KEEP stand-down,
|
|
157
|
+
OrphanProcessReaper exact-id + work-check + P0, SessionRecovery
|
|
158
|
+
killForRecovery + recovery-bounce + bypassRecoveryFlag scope, wake-reaper
|
|
159
|
+
cumulative-sleep + P1/P2 + P0), 6 new SleepWakeDetector cumulative-sleep
|
|
160
|
+
math tests, and the existing JobScheduler reaper suite (7) updated to the
|
|
161
|
+
new async + effective-elapsed contract.
|
|
162
|
+
- `tsc --noEmit` clean. Existing SessionWatchdog (58), OrphanProcessReaper
|
|
163
|
+
(9), SessionRecovery (9), terminate-CAS (9), session-timeout/activity-aware
|
|
164
|
+
(4+6) tests all still pass.
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
# Upgrade Guide — vNEXT
|
|
2
|
+
|
|
3
|
+
<!-- bump: minor -->
|
|
4
|
+
|
|
5
|
+
## What Changed
|
|
6
|
+
|
|
7
|
+
Added `CollaborationRedriveEngine` — a small, off-by-default monitor that fills the one verified gap in cross-agent collaboration continuity: nothing today proactively re-engages a counterpart when **they** go silent on an unfinished objective. Spec `docs/specs/collaboration-redrive-on-counterpart-silence.md` (approved by Justin 2026-05-28 after a 2-round adversarial convergence). It builds on the already-shipped Threadline Conversation Keystone (`WarrantsReplyGate` loop-safety + `ConversationStore` continuity + `CompletionEvaluator` done-judgment) — it is **not** a rebuild of any of those.
|
|
8
|
+
|
|
9
|
+
The engine sweeps active `threadline-reply` commitments on its own 5-minute cadence (NOT the PromiseBeacon tick, which only schedules beacon-enabled commitments) and, for each unfinished objective whose counterpart has been silent past a threshold, sends one bounded peer nudge. The termination guarantee is a **durable, monotonic, reply-INDEPENDENT** per-commitment counter (`redriveCount` on the Commitment record) — a counterpart reply updates `lastReplyAt` but never touches the counter, closing the mutual-re-drive hole the round-1 adversarial review caught. After the per-commitment cap (default 2), the engine raises ONE Attention-queue item and goes terminal-quiet on that commitment. Independent per-peer 24h cap, engine-wide daily fuse, per-tick fuse, finite-timestamp validation, and skip-don't-increment on name→fingerprint resolution miss provide defence-in-depth.
|
|
10
|
+
|
|
11
|
+
## What to Tell Your User
|
|
12
|
+
|
|
13
|
+
- I can now nudge another agent once or twice if they go quiet on something we're working on together, then I shut up and surface it to you — never a loop.
|
|
14
|
+
- It ships **OFF**. To turn it on for me, flip the collaboration-redrive enabled setting in my config to true. The other tunables (silence threshold, max nudges, per-peer cap, daily fuse) all have safe defaults.
|
|
15
|
+
- It only triggers on commitments I already have for "I'm waiting on agent X to do thing Y", so it has no effect outside that surface.
|
|
16
|
+
|
|
17
|
+
## Summary of New Capabilities
|
|
18
|
+
|
|
19
|
+
| Capability | How to Use |
|
|
20
|
+
|-----------|-----------|
|
|
21
|
+
| Proactive peer re-drive on counterpart silence | Set `monitoring.collaborationRedrive.enabled: true` (defaults: 45-min silence threshold, max 2 nudges per objective, escalate to Attention queue then stop) |
|
|
22
|
+
|
|
23
|
+
## Evidence
|
|
24
|
+
|
|
25
|
+
- Tier-1 unit tests cover both sides of every eligibility boundary (terminal status, NaN/future timestamp, silence threshold, cap, spacing window, missing peer), the load-bearing reply-independent cap (the round-1 adversarial fix), restart-survival across a fresh `CommitmentTracker` load, name-unresolved skip-don't-increment, and disabled-mode no-op.
|
|
26
|
+
- New fields on `Commitment`: `redriveCount?: number`, `lastRedriveAt?: string`, `lastRedriveText?: string` — additive and optional; `loadStore()` backfills `redriveCount: 0` on existing rows alongside the existing `correctionCount`/`escalated` backfill.
|
|
27
|
+
- Side-effects review: `upgrades/side-effects/collaboration-redrive-engine.md`.
|
|
28
|
+
- Spec convergence: `docs/specs/reports/collaboration-redrive-convergence.md` (2 review rounds; round 1 caught a real mutual-re-drive hole reopening the ack-loop blast radius and was fixed by switching to a reply-independent durable cap; round 2 caught a fingerprint-resolution wiring bug and was fixed).
|
|
@@ -0,0 +1,164 @@
|
|
|
1
|
+
# Upgrade Guide — vNEXT
|
|
2
|
+
|
|
3
|
+
<!-- bump: patch -->
|
|
4
|
+
|
|
5
|
+
## What Changed
|
|
6
|
+
|
|
7
|
+
**Fixed a runaway credit-burn bug: LLM call loops no longer keep hammering the
|
|
8
|
+
provider after you hit a usage or spend limit.**
|
|
9
|
+
|
|
10
|
+
Instar runs small background LLM calls to watch your sessions (for example, the
|
|
11
|
+
per-tick check that detects when a session is blocked on a prompt). Until now,
|
|
12
|
+
if one of those calls came back with a usage/rate/spend-limit error, the loop
|
|
13
|
+
swallowed the error and tried again on the very next tick — with no backoff. If
|
|
14
|
+
your account had auto-reload turned on, each retry refueled and re-burned. A
|
|
15
|
+
real agent in the wild burned $452 of $455 in usage credits this way over a
|
|
16
|
+
couple of days before anyone noticed.
|
|
17
|
+
|
|
18
|
+
There is now an **account-global circuit breaker** in front of every internal
|
|
19
|
+
LLM call. The moment the provider reports a usage/rate/spend limit, the breaker
|
|
20
|
+
opens and *all* background LLM-backed work pauses — without spawning another
|
|
21
|
+
`claude` subprocess, so it costs nothing while paused. After a cool-down window
|
|
22
|
+
(15 minutes by default) it sends exactly one quiet probe; if the limit has
|
|
23
|
+
lifted it closes and resumes automatically, and if not it waits another window.
|
|
24
|
+
It is wired at the single provider-construction chokepoint, so every feature —
|
|
25
|
+
current and future — is covered with no per-feature work.
|
|
26
|
+
|
|
27
|
+
This is reactive (it listens to the provider's own "you're over limit" signal)
|
|
28
|
+
and complements the existing volume-based burn-detection, which reacts to
|
|
29
|
+
statistical token-share over a longer window.
|
|
30
|
+
|
|
31
|
+
## What to Tell Your User
|
|
32
|
+
|
|
33
|
+
Your agent can no longer burn through your credits by repeatedly calling the
|
|
34
|
+
model after you've hit a usage or spend limit. The instant the provider says
|
|
35
|
+
you're over your limit, background model work pauses on its own, costs nothing
|
|
36
|
+
while paused, and quietly resumes once the limit lifts. This is on by default —
|
|
37
|
+
no setup needed. If you ever want to tune the cool-down window or turn it off,
|
|
38
|
+
there's an optional circuit-breaker setting in your config, but the safe
|
|
39
|
+
defaults are designed to just work.
|
|
40
|
+
|
|
41
|
+
## Summary of New Capabilities
|
|
42
|
+
|
|
43
|
+
| Capability | How to Use |
|
|
44
|
+
|-----------|-----------|
|
|
45
|
+
| Account-global LLM rate-limit circuit breaker | Automatic. When the provider returns a usage/rate/spend-limit error, all background LLM-backed work pauses (no subprocess spawned) and self-heals via a single probe after the cool-down. |
|
|
46
|
+
| Tunable cool-down / kill switch | Optional `intelligence.circuitBreaker` in `.instar/config.json`: `enabled` (default true) and `openMs` (default 900000 = 15 min). Absent config uses the safe defaults, so existing agents are protected with no changes. |
|
|
47
|
+
|
|
48
|
+
## Evidence
|
|
49
|
+
|
|
50
|
+
**Reproduction (the incident pattern), reproduced as an automated test:** a fake
|
|
51
|
+
`claude` binary that records each spawn to an on-disk counter and exits with
|
|
52
|
+
"Claude AI usage limit reached" on stderr.
|
|
53
|
+
`tests/integration/llm-circuit-breaker-chokepoint.test.ts` builds two providers
|
|
54
|
+
through the real `buildIntelligenceProvider` factory and drives them:
|
|
55
|
+
|
|
56
|
+
- Before fix (old behaviour): every `evaluate()` call spawns the binary →
|
|
57
|
+
unbounded spawns while limited (this is exactly the $452 burn).
|
|
58
|
+
- After fix (observed in the test): the first call spawns once and trips the
|
|
59
|
+
breaker; the next 5 calls — and a call through a *second, independently built*
|
|
60
|
+
provider — all reject with `LlmCircuitOpenError` and the spawn counter stays
|
|
61
|
+
at **1**. Zero additional subprocesses, proving the burn is stopped and the
|
|
62
|
+
breaker is account-global.
|
|
63
|
+
|
|
64
|
+
39 new circuit-breaker tests pass (state machine both-sides-of-boundary, the
|
|
65
|
+
rate-limit classifier on true-positive and unrelated-error strings, and the
|
|
66
|
+
chokepoint wiring). `tsc --noEmit` clean. The server log emits
|
|
67
|
+
`[llm-circuit] OPEN: …` / `[llm-circuit] closing: …` lines on each transition.
|
|
68
|
+
|
|
69
|
+
---
|
|
70
|
+
|
|
71
|
+
**Phase 2 of the unified session-lifecycle robustness work (spec
|
|
72
|
+
`docs/specs/unified-session-lifecycle-robustness.md`).** The four remaining
|
|
73
|
+
autonomous session killers — the SessionWatchdog, OrphanProcessReaper,
|
|
74
|
+
SessionRecovery (kill-to-respawn), and the scheduler's wake-reaper — are now
|
|
75
|
+
funneled through the single ReapAuthority that shipped in Phase 1. Each one
|
|
76
|
+
gains the careful KEEP-checks and the awake-machine lease gate the previous
|
|
77
|
+
killer-of-the-week didn't have, plus structural fixes to a few long-standing
|
|
78
|
+
near-misses.
|
|
79
|
+
|
|
80
|
+
What landed:
|
|
81
|
+
|
|
82
|
+
- **#5 SessionWatchdog.** The final escalation level (the session-kill that
|
|
83
|
+
fires after Ctrl+C / SIGTERM / SIGKILL all failed to free a stuck child) now
|
|
84
|
+
routes through `terminateSession('watchdog-stuck', terminal/killed)` instead
|
|
85
|
+
of a raw `tmux kill-session`. If the authority's KEEP-guards refuse the kill
|
|
86
|
+
(a relay lease, an active subagent, a recent user message, etc.), the
|
|
87
|
+
watchdog STANDS DOWN — clears its escalation state, logs the exact refusal
|
|
88
|
+
reason, and lets the §P5 backstop own the operator-decision escalation.
|
|
89
|
+
- **#6 OrphanProcessReaper.** Three changes. (a) Orphan classification is now
|
|
90
|
+
EXACT-id membership in `listKnownTmuxSessions()`, not project-prefix
|
|
91
|
+
startsWith — so a user-created `tmux new -s the-portal-myhand` is `external`,
|
|
92
|
+
not orphan, and the reaper cannot false-reap a user pane. (b) The 60-minute
|
|
93
|
+
age threshold is necessary, not sufficient: a new `processHasActiveChildren`
|
|
94
|
+
work-check (pgrep -P) vetoes the kill when the orphan still has running
|
|
95
|
+
children, and the deferred path logs a `Kept orphan PID … work check vetoed
|
|
96
|
+
reap` action. (c) When the orphan's tmux name matches a currently-tracked
|
|
97
|
+
session, the kill goes through the ReapAuthority for the lifecycle events +
|
|
98
|
+
reap-log entry.
|
|
99
|
+
- **#8 SessionRecovery.** A single shared `killForRecovery()` helper consults
|
|
100
|
+
`hasActiveProcesses` before every kill-to-respawn — if the pane is producing
|
|
101
|
+
work, the JSONL stall/crash reading is unreliable and the recovery defers
|
|
102
|
+
with a structured `deferred-still-working` result. All four recovery paths
|
|
103
|
+
(stall, context-exhaustion, crash, error-loop) route through this helper.
|
|
104
|
+
The actual kill goes through `terminateSession('session-recovery',
|
|
105
|
+
recovery-bounce, bypassRecoveryFlag:true)` so the §P3 "shut down" notice
|
|
106
|
+
stays silent on a bounce (which is not a disappearance) and the reap-log
|
|
107
|
+
records the recovery-bounce disposition. `bypassRecoveryFlag` is narrowly
|
|
108
|
+
scoped to the `recovery-in-flight` KEEP-guard reason — all other KEEP-
|
|
109
|
+
guards still apply, so a session mid-conversation isn't killed-to-respawn
|
|
110
|
+
under the cover of "recovery."
|
|
111
|
+
- **#9 wake-reaper.** Three changes. (a) The threshold-check now uses
|
|
112
|
+
CUMULATIVE wall-time-asleep during the run (the new
|
|
113
|
+
`SleepWakeDetector.getCumulativeSleepMsBetween()`), not the single last
|
|
114
|
+
`sleepDurationSeconds` event — a job that spanned multiple sleeps was
|
|
115
|
+
previously credited only the last sleep and reaped early. (b) The P1/P2 work
|
|
116
|
+
gate keeps a session whose process is still producing work, regardless of
|
|
117
|
+
clock. (c) The kill routes through `terminateSession('wake-reaper',
|
|
118
|
+
terminal/killed)` when tracked.
|
|
119
|
+
|
|
120
|
+
All four killers now share the same brain — the structural guarantee the
|
|
121
|
+
spec's audit named when it found "is this session alive/dead/stuck/working?"
|
|
122
|
+
answered ad-hoc in eight different places.
|
|
123
|
+
|
|
124
|
+
## What to Tell Your User
|
|
125
|
+
|
|
126
|
+
- Every part of the agent that can shut a session down — the stuck-child
|
|
127
|
+
watchdog, the orphan-process cleanup, the crash-recovery, and the after-
|
|
128
|
+
sleep cleanup — now goes through the same careful gatekeeper that the boot
|
|
129
|
+
cleanup uses. None of them can quietly kill a session that might still be
|
|
130
|
+
working.
|
|
131
|
+
- The watchdog's last-resort session kill, the crash-recovery's restart, and
|
|
132
|
+
the after-sleep cleanup all show up in the reap-log with the reason, so the
|
|
133
|
+
"where did my session go?" answer is always there.
|
|
134
|
+
- A crash-recovery restart (kill + respawn fresh) stays quiet — it's a bounce,
|
|
135
|
+
not a disappearance — but it's still logged.
|
|
136
|
+
- The orphan-process cleanup can no longer accidentally classify a tmux pane
|
|
137
|
+
you started yourself as one of mine, and it will defer the kill if the
|
|
138
|
+
orphan still has active child processes.
|
|
139
|
+
- After your machine sleeps and wakes, the job cleanup correctly subtracts the
|
|
140
|
+
total sleep time (across multiple sleeps if needed) before deciding a job
|
|
141
|
+
ran too long, so a job that was paused while you closed the lid isn't
|
|
142
|
+
marked stuck.
|
|
143
|
+
|
|
144
|
+
## Summary of New Capabilities
|
|
145
|
+
|
|
146
|
+
| Capability | How to Use |
|
|
147
|
+
|-----------|-----------|
|
|
148
|
+
| Every autonomous killer routes through the single ReapAuthority | Automatic — the watchdog, orphan reaper, crash-recovery, and wake-reaper all funnel through `terminateSession`. |
|
|
149
|
+
| Recovery-bounce disposition stays silent in the "shut down" notice | Automatic — a kill-to-respawn doesn't pretend a session vanished; it still lands in the reap-log. |
|
|
150
|
+
| Orphan reaper: exact-id classification + work-check + ReapAuthority routing | Automatic — `listKnownTmuxSessions()` replaces the prefix match; `processHasActiveChildren` vetoes reaping a working orphan. |
|
|
151
|
+
| Wake-reaper: cumulative-sleep math + P1/P2 work gate | Automatic — multi-sleep runs no longer reaped early; an actively-working session is kept regardless of clock. |
|
|
152
|
+
|
|
153
|
+
## Evidence
|
|
154
|
+
|
|
155
|
+
- All Phase 1 unit/integration/e2e tests remain green, plus 14 new Phase-2
|
|
156
|
+
wiring contracts (SessionWatchdog → ReapAuthority routing + KEEP stand-down,
|
|
157
|
+
OrphanProcessReaper exact-id + work-check + P0, SessionRecovery
|
|
158
|
+
killForRecovery + recovery-bounce + bypassRecoveryFlag scope, wake-reaper
|
|
159
|
+
cumulative-sleep + P1/P2 + P0), 6 new SleepWakeDetector cumulative-sleep
|
|
160
|
+
math tests, and the existing JobScheduler reaper suite (7) updated to the
|
|
161
|
+
new async + effective-elapsed contract.
|
|
162
|
+
- `tsc --noEmit` clean. Existing SessionWatchdog (58), OrphanProcessReaper
|
|
163
|
+
(9), SessionRecovery (9), terminate-CAS (9), session-timeout/activity-aware
|
|
164
|
+
(4+6) tests all still pass.
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
# Side-Effects Review — CollaborationRedriveEngine
|
|
2
|
+
|
|
3
|
+
**Spec:** `docs/specs/collaboration-redrive-on-counterpart-silence.md` (approved 2026-05-28)
|
|
4
|
+
**Change:** add `src/monitoring/CollaborationRedriveEngine.ts`; add `redriveCount`/`lastRedriveAt`/`lastRedriveText` optional fields to `Commitment`; backfill in `loadStore()`; default new threadline-reply commitments with `redriveCount: 0`; wire construction into `src/commands/server.ts` after `completionEvaluator` (ships OFF).
|
|
5
|
+
|
|
6
|
+
## 1. Over-block / under-block
|
|
7
|
+
**Over-drive:** nudging a counterpart who is legitimately working. Mitigated by the `CompletionEvaluator` "objective-met" gate (auto-closes the commitment instead of nudging), the silence threshold, the per-commitment cap, the per-peer/daily/per-tick fuses, and (decoratively) the novelty tiebreaker. The cap biases toward UNDER-drive (stop + escalate), which is the safe direction.
|
|
8
|
+
**Under-drive:** never nudging → that is today's gap and the thing this fixes. The ship-OFF default is the strongest under-drive bias; we dogfood-on-Echo before any rollout.
|
|
9
|
+
|
|
10
|
+
## 2. Level-of-abstraction fit
|
|
11
|
+
Composes existing primitives — no new transport, no new persistence store. Reads/writes existing `Commitment` records via `mutate()`; sends via the same `threadlineRelayClient.sendPlaintext` the inbound auto-ack path uses; surfaces via the existing `CollaborationSurfacer.notify`; escalates via `telegram.createAttentionItem` (injected `raiseAttention`). The engine is a small monitor sibling to `PromiseBeacon` / `CommitmentTracker`.
|
|
12
|
+
|
|
13
|
+
## 3. Signal vs Authority
|
|
14
|
+
Stall + not-met are SIGNALS. The bounded nudge is a limited action with a hard cap. The OPERATOR holds terminal authority — when the cap is hit, the engine raises an Attention-queue item ("collaboration with <peer> stalled — your call") and goes silent. The engine cannot decide to keep going past the cap.
|
|
15
|
+
|
|
16
|
+
## 4. Interactions
|
|
17
|
+
- `CommitmentTracker`: adds optional fields; uses `getActive()` + `mutate()` + `markReplyArrived()` (already only-touches-`lastReplyAt`).
|
|
18
|
+
- `PromiseBeacon`: independent. The engine runs its OWN sweep so most threadline-reply commitments (which are NOT beacon-enabled) are still covered.
|
|
19
|
+
- `WarrantsReplyGate`: the peer's gate suppresses our nudge if it's pure-ack; the engine sends concrete questions/next-steps so it warrants reply (convergence-driving by construction).
|
|
20
|
+
- `CollaborationSurfacer`: visibility-only — silent hub posts; does not buzz the operator.
|
|
21
|
+
- Attention queue (`telegram.createAttentionItem`): used only at cap-hit + after N consecutive name-resolution misses.
|
|
22
|
+
|
|
23
|
+
## 5. Rollback
|
|
24
|
+
Ships OFF (`monitoring.collaborationRedrive.enabled: false`); flipping back to false fully disables. The new fields are additive/optional — no destructive data transform; the only "migration" is an idempotent backfill defaulting `redriveCount: 0` on existing rows (safe to remove — would just read back as `undefined`, which the engine treats as 0).
|
|
25
|
+
|
|
26
|
+
## 6. Data integrity
|
|
27
|
+
The only mutation to existing Commitment lifecycle semantics is the auto-close on objective-met (transitions `pending → delivered`), which is a documented terminal transition this engine is now authorised to make. All other writes are to the new additive fields. The cap and spacing live durably on the commitment record (via `mutate()`), not in PromiseBeacon hot-state, so they survive restart.
|
|
28
|
+
|
|
29
|
+
## 7. Failure modes
|
|
30
|
+
- **Runaway nudges (incl. mutual re-drive):** bounded by the durable reply-independent cap + per-peer 24h cap + engine-wide daily fuse + per-tick fuse. The cap is the ONLY termination guarantee; novelty is decorative. Tested.
|
|
31
|
+
- **Restart amnesia:** cap/spacing persisted to `commitments.json` via `mutate()`. Restart-survival test required and present.
|
|
32
|
+
- **Clock jump / sleep-wake burst:** per-tick fuse + `Number.isFinite` validation + injectable `now()`. NaN/future timestamps disqualify, fail-safe.
|
|
33
|
+
- **Peer ack-storm in response:** the peer's own `WarrantsReplyGate` suppresses inbound; our durable cap stops our side regardless of replies.
|
|
34
|
+
- **Wrong-peer send:** name→fingerprint resolved via `known-agents.json`; unresolved or ambiguous names skip + don't increment + escalate after N strikes. A send is never attempted against a guessed address.
|
|
35
|
+
- **Re-drive while operator is mid-conversation:** visibility surfaces silently to the Threadline hub; never the active parent topic. Cap-hit escalates to the Attention queue, which the operator opens on their own schedule.
|
|
36
|
+
- **Evaluator unreachable:** errs to "not met" (keep waiting); an evaluator exception never produces an uncounted nudge — but the engine does NOT nudge on evaluator-throw this tick (the evaluator failure short-circuits the eligible path).
|
|
@@ -239,3 +239,126 @@ no-silent-fallbacks ratchet doesn't count them: `ReapLog.read` (no log file ⇒
|
|
|
239
239
|
empty list is correct, not degraded) and the §P5 transcript-tail read (an
|
|
240
240
|
unreadable tail ⇒ no meaningful-advance signal this tick; treated as ambiguous,
|
|
241
241
|
never as progress). Behavior unchanged.
|
|
242
|
+
|
|
243
|
+
## Phase 2 — Commit #5: SessionWatchdog → ReapAuthority
|
|
244
|
+
|
|
245
|
+
**Files:** `src/monitoring/SessionWatchdog.ts`,
|
|
246
|
+
`tests/unit/session-lifecycle-phase-2-wiring.test.ts` (new).
|
|
247
|
+
|
|
248
|
+
`handleEscalation` is now `async`; the final `EscalationLevel.KillSession` no
|
|
249
|
+
longer calls a raw `tmux kill-session` — it resolves the instar session by
|
|
250
|
+
`tmuxSession` and routes through
|
|
251
|
+
`sessionManager.terminateSession(id, 'watchdog-stuck', { disposition: 'terminal', finalStatus: 'killed' })`.
|
|
252
|
+
The authority's mandatory KEEP-guard + lease gate + protected-set check now
|
|
253
|
+
apply: a session the guards would have kept survives a buggy watchdog. The raw
|
|
254
|
+
`killTmuxSession` method is deleted; the LLM gate + stdin guard + signal
|
|
255
|
+
escalations (CtrlC / SigTerm / SigKill) are unchanged.
|
|
256
|
+
|
|
257
|
+
**Stand-down on KEEP (anti-thrash):** if the authority refuses the kill
|
|
258
|
+
(`protected` / `not-lease-holder` / a KEEP-guard hold / `in-flight`), the
|
|
259
|
+
watchdog logs the exact reason, records a `kept (<reason>)` intervention, and
|
|
260
|
+
**clears escalationState**. It does not re-escalate against a guarded session
|
|
261
|
+
every tick — the §P5 backstop owns operator-decision escalation for a
|
|
262
|
+
persistently-stale session.
|
|
263
|
+
|
|
264
|
+
**Tests:** existing SessionWatchdog tests (58) green; new wiring contract (3) green; typecheck clean.
|
|
265
|
+
|
|
266
|
+
**Rollback:** revert this commit; the raw `killTmuxSession` is the only thing
|
|
267
|
+
deleted and is recoverable from history.
|
|
268
|
+
|
|
269
|
+
## Phase 2 — Commit #6: OrphanProcessReaper → exact-id + work-check + ReapAuthority
|
|
270
|
+
|
|
271
|
+
**Files:** `src/monitoring/OrphanProcessReaper.ts`, `src/core/SessionManager.ts`,
|
|
272
|
+
`tests/unit/session-lifecycle-phase-2-wiring.test.ts`.
|
|
273
|
+
|
|
274
|
+
Three spec-faithful changes:
|
|
275
|
+
1. **Exact-id classification.** The legacy `tmuxSession.startsWith(this.projectPrefix)`
|
|
276
|
+
substring match is replaced by EXACT membership in
|
|
277
|
+
`SessionManager.listKnownTmuxSessions()` — the new method returns every tmux
|
|
278
|
+
name instar has ever tracked (any status). A user-created tmux session that
|
|
279
|
+
happens to share the project prefix is now classified `external`, not
|
|
280
|
+
`instar-orphan`, so the reaper cannot false-reap a user pane.
|
|
281
|
+
2. **60-min age = necessary, not sufficient.** Before any orphan kill the new
|
|
282
|
+
`processHasActiveChildren(pid)` helper (`pgrep -P`) is consulted; a positive
|
|
283
|
+
result vetoes the reap and surfaces a `Kept orphan PID … work check vetoed
|
|
284
|
+
reap` action. An uninspectable result returns `false` (allowed to proceed)
|
|
285
|
+
to preserve prior behavior — this gate is a soft veto layered on top of the
|
|
286
|
+
age gate, never an escalation.
|
|
287
|
+
3. **Route through P0 when applicable.** When the orphan's tmuxSession matches a
|
|
288
|
+
CURRENTLY-tracked session (a scan/refresh race), the kill goes through
|
|
289
|
+
`terminateSession(id, 'orphan-reap', { disposition:'terminal', finalStatus:'killed' })`
|
|
290
|
+
so the authority's KEEP-guard + lease gate + reap-log entry + `sessionReaped`
|
|
291
|
+
emission apply. Otherwise the tracked record is already terminal and the raw
|
|
292
|
+
tmux-kill cleans up the stray pane.
|
|
293
|
+
|
|
294
|
+
The server session remains excluded unconditionally.
|
|
295
|
+
|
|
296
|
+
**Tests:** OrphanProcessReaper (9) + Phase-2 wiring (6) green; typecheck clean.
|
|
297
|
+
|
|
298
|
+
**Rollback:** revert this commit. `listKnownTmuxSessions()` becomes unused but harmless.
|
|
299
|
+
|
|
300
|
+
## Phase 2 — Commit #8: SessionRecovery → P1/P2 cross-check + recovery-bounce disposition
|
|
301
|
+
|
|
302
|
+
**Files:** `src/monitoring/SessionRecovery.ts`, `src/core/SessionManager.ts`,
|
|
303
|
+
`src/commands/server.ts`, `tests/unit/session-lifecycle-phase-2-wiring.test.ts`.
|
|
304
|
+
|
|
305
|
+
Three spec-faithful changes:
|
|
306
|
+
1. **P1/P2 cross-check** — a new shared `killForRecovery(name)` helper inside
|
|
307
|
+
`SessionRecovery` consults `deps.hasActiveProcesses` BEFORE any kill. When
|
|
308
|
+
the tmux pane has active child processes, the recovery DEFERS this attempt
|
|
309
|
+
and returns a structured `deferred-still-working` result with the correct
|
|
310
|
+
`failureType` (stall / context_exhaustion / crash / error_loop). All four
|
|
311
|
+
recovery paths route through this helper — `deps.killSession` is no longer
|
|
312
|
+
called directly from any recovery method.
|
|
313
|
+
2. **`disposition:'recovery-bounce'`** — the dep's `killSession` is rewired in
|
|
314
|
+
server.ts to route through
|
|
315
|
+
`sessionManager.terminateSession(id, 'session-recovery', { disposition:
|
|
316
|
+
'recovery-bounce', finalStatus: 'killed', bypassRecoveryFlag: true })`. The
|
|
317
|
+
§P3 notifier is silent on recovery-bounce, so the user isn't told "shut
|
|
318
|
+
down" when the bounce immediately respawns; the reap-log still records it.
|
|
319
|
+
3. **`bypassRecoveryFlag` (scoped)** — terminateSession gains an opt that
|
|
320
|
+
skips ONLY the `recovery-in-flight` KEEP-guard reason (not the whole guard).
|
|
321
|
+
Without it the recovery would refuse to kill its OWN in-flight session.
|
|
322
|
+
Other KEEP-guards (active subagent, recent-user-message, etc.) still apply
|
|
323
|
+
so a session mid-conversation is not killed-to-respawn under the cover of
|
|
324
|
+
"recovery."
|
|
325
|
+
|
|
326
|
+
`SessionRecoveryDeps.killSession` is now `() => void | Promise<void>` so the
|
|
327
|
+
implementation can await terminateSession.
|
|
328
|
+
|
|
329
|
+
**Tests:** SessionRecovery (9) + terminate-CAS (9) + Phase-2 wiring (10) green; typecheck clean.
|
|
330
|
+
|
|
331
|
+
**Rollback:** revert this commit. `bypassRecoveryFlag` becomes unused but harmless.
|
|
332
|
+
|
|
333
|
+
## Phase 2 — Commit #9: wake-reaper → P1/P2 + cumulative sleep + ReapAuthority
|
|
334
|
+
|
|
335
|
+
**Files:** `src/scheduler/JobScheduler.ts`, `src/core/SleepWakeDetector.ts`,
|
|
336
|
+
`src/commands/server.ts`, `tests/unit/JobScheduler-reaper.test.ts`,
|
|
337
|
+
`tests/unit/sleep-wake-cumulative.test.ts` (new),
|
|
338
|
+
`tests/unit/session-lifecycle-phase-2-wiring.test.ts`.
|
|
339
|
+
|
|
340
|
+
Three spec-faithful changes:
|
|
341
|
+
1. **Cumulative sleep (SE-8).** `SleepWakeDetector.getCumulativeSleepMsBetween(startMs, endMs)`
|
|
342
|
+
sums the overlap of every recorded sleep window with the query range. The
|
|
343
|
+
wake-reaper now reads `cumulativeSleepProvider(runStartMs, now)` per run and
|
|
344
|
+
uses `effectiveElapsed = elapsedMs − cumulativeSleep` against its threshold —
|
|
345
|
+
a run that spanned multiple sleeps is no longer reaped early because the old
|
|
346
|
+
code credited only the single last `sleepDurationSeconds` event.
|
|
347
|
+
2. **P1/P2 gate.** Before any kill, `sessionManager.hasActiveProcesses(sessionName)`
|
|
348
|
+
is consulted; on `true` the run is KEPT regardless of clock (spec: "a
|
|
349
|
+
progressing process is KEEP regardless of clock"). The deferred run is
|
|
350
|
+
counted as `skipped` and a structured log line names the reason.
|
|
351
|
+
3. **Route through P0.** When a stuck-run session is currently tracked, the
|
|
352
|
+
kill goes through
|
|
353
|
+
`terminateSession(id, 'wake-reaper', { disposition:'terminal', finalStatus:'killed' })`
|
|
354
|
+
— the authority's protected-set + lease gate + reap-log + `sessionReaped`
|
|
355
|
+
emission apply. Untracked tmux panes fall back to the raw `killSession`.
|
|
356
|
+
|
|
357
|
+
`reapStuckRuns` is now async; the timeout error message now names the effective-
|
|
358
|
+
elapsed + cumulative-sleep subtraction instead of the single last sleep event.
|
|
359
|
+
|
|
360
|
+
**Tests:** JobScheduler reaper (7, updated to async + new error contract) +
|
|
361
|
+
SleepWakeDetector cumulative (6, new) + Phase-2 wiring (14) green; typecheck clean.
|
|
362
|
+
|
|
363
|
+
**Rollback:** revert this commit. `getCumulativeSleepMsBetween` becomes unused
|
|
364
|
+
but harmless.
|