instar 0.28.91 → 0.28.93

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (32) hide show
  1. package/dist/commands/server.d.ts.map +1 -1
  2. package/dist/commands/server.js +9 -0
  3. package/dist/commands/server.js.map +1 -1
  4. package/dist/core/InitiativeTracker.d.ts +7 -2
  5. package/dist/core/InitiativeTracker.d.ts.map +1 -1
  6. package/dist/core/InitiativeTracker.js.map +1 -1
  7. package/dist/core/ProjectDriftChecker.d.ts +200 -0
  8. package/dist/core/ProjectDriftChecker.d.ts.map +1 -0
  9. package/dist/core/ProjectDriftChecker.js +487 -0
  10. package/dist/core/ProjectDriftChecker.js.map +1 -0
  11. package/dist/core/SessionManager.d.ts +29 -8
  12. package/dist/core/SessionManager.d.ts.map +1 -1
  13. package/dist/core/SessionManager.js +89 -28
  14. package/dist/core/SessionManager.js.map +1 -1
  15. package/dist/core/StuckInputSentinel.d.ts +160 -0
  16. package/dist/core/StuckInputSentinel.d.ts.map +1 -0
  17. package/dist/core/StuckInputSentinel.js +315 -0
  18. package/dist/core/StuckInputSentinel.js.map +1 -0
  19. package/dist/core/types.d.ts +40 -0
  20. package/dist/core/types.d.ts.map +1 -1
  21. package/dist/core/types.js.map +1 -1
  22. package/dist/lifeline/ServerSupervisor.d.ts.map +1 -1
  23. package/dist/lifeline/ServerSupervisor.js +8 -0
  24. package/dist/lifeline/ServerSupervisor.js.map +1 -1
  25. package/package.json +1 -1
  26. package/src/data/builtin-manifest.json +3 -3
  27. package/upgrades/0.28.92.md +107 -0
  28. package/upgrades/0.28.93.md +126 -0
  29. package/upgrades/side-effects/0.28.91.md +11 -0
  30. package/upgrades/side-effects/multishot-stuck-input-recovery.md +151 -0
  31. package/upgrades/side-effects/project-scope-phase1b-pr1.md +252 -0
  32. package/upgrades/side-effects/stuck-input-sentinel.md +137 -0
@@ -0,0 +1,151 @@
1
+ # Side-Effects Review — Multi-Shot Stuck-Input Recovery
2
+
3
+ ## Change Summary
4
+
5
+ `SessionManager.verifyInjection` upgrades from single-shot to multi-shot
6
+ polling with escalating recovery actions. When the recovery Enter is itself
7
+ eaten by the same paste-end race that caused the original stuck state, the
8
+ loop keeps trying — with different key combinations — until the marker
9
+ clears the `❯` prompt or the bounded schedule (4 attempts over 6.5s)
10
+ exhausts.
11
+
12
+ Three surface changes in `src/core/SessionManager.ts`:
13
+
14
+ 1. `verifyInjection` now polls at 500ms, 1500ms, 3500ms, 6500ms from
15
+ injection. Each tick checks pane state; stops as soon as the marker
16
+ clears the prompt. Single Degradation entry on the first recovery firing.
17
+ 2. `isMarkerStuckAtPrompt(pane, marker)` extracted as a named method so the
18
+ polling loop and tests share one detection rule.
19
+ 3. `fireStuckInputRecovery(tmuxSession, attempt)` escalates the recovery
20
+ action: attempts 0 and 1 send `Enter`; attempt 2 sends `C-m` (literal
21
+ carriage return — bypasses any Enter-specific consumer); attempt 3 sends
22
+ `Enter` + 150ms sleep + `Enter` (covers sub-second race windows).
23
+
24
+ ## Failure Mode Being Fixed
25
+
26
+ Reproduced live in `/Users/justin/.instar/agents/echo/logs/server.log`:
27
+
28
+ ```
29
+ 01:31:49 [LOG] Injected initial message into "echo-telegram-messages-pausing-in-input" (297 chars)
30
+ 01:33:49 [WARN] Injection stuck — marker "[telegram:7195] Your session j…" still at prompt. Resending Enter.
31
+ 01:33:49 [DEGRADATION] SessionManager.verifyInjection: Enter eaten by paste-end race
32
+ 01:34:15 [WARN] Injection stuck — marker "[telegram:7195] Your session j…" still at prompt. Resending Enter.
33
+ 01:34:15 [DEGRADATION] SessionManager.verifyInjection: Enter eaten by paste-end race
34
+ ```
35
+
36
+ Two observations:
37
+
38
+ - The async setTimeout(1500ms) fired 120 seconds after injection — the Node
39
+ event loop was blocked, likely by initial-message-driven session startup
40
+ work. Single-shot timing is unreliable under load.
41
+ - The recovery Enter itself failed (the second `Injection stuck` log line
42
+ for the same marker proves the first recovery did not unstick the input).
43
+
44
+ The single-shot design assumed both: (a) the timer fires near 1.5s and (b)
45
+ the recovery Enter always lands. Both assumptions break in practice.
46
+
47
+ ## Over-Block Risk
48
+
49
+ **Polling more times = more chances to send a spurious Enter.** The
50
+ detection gate is unchanged from the single-shot version: marker (first
51
+ 40 chars, stripped of leading whitespace, ≥8 chars) must appear on or
52
+ immediately after a line containing `❯`. Claude Code only renders `❯` on
53
+ the active input prompt row — not on transcript history — so the marker
54
+ cannot match a historic echo of the message. Each polling tick re-runs the
55
+ same gate; if the message has submitted, the gate refuses and recovery
56
+ does not fire.
57
+
58
+ Worst-case false-positive: marker is genuinely visible at `❯` because the
59
+ user is still typing the same prefix the agent just injected. Extremely
60
+ implausible (the marker is the message's first 40 chars, not a common
61
+ prefix). Impact if it happened: one or more extra Enters in the input
62
+ box. The user would see their typing submitted prematurely. Acceptable
63
+ trade for fixing the stuck-message bug.
64
+
65
+ The escalation to `C-m` and double-Enter applies only on attempts 2 and
66
+ 3 — after two consecutive ticks showed the marker still stuck at `❯`.
67
+ That's an additional safety margin against accidental escalation.
68
+
69
+ ## Under-Block Risk
70
+
71
+ If the polling schedule's last tick (6500ms) still shows the marker stuck
72
+ despite four recovery attempts, the loop exits without further action.
73
+ The existing recovery layers below still apply:
74
+
75
+ - `StallTriageNurse` fast-path nudges stuck inputs when its periodic
76
+ diagnosis pass runs.
77
+ - The `pasteRetried` mechanism handles the `[Pasted text #N]` placeholder
78
+ case.
79
+ - The idle-prompt zombie reaper eventually kills sessions with no output
80
+ changes past the configured threshold.
81
+
82
+ A session that survives all four ticks without recovery is almost certainly
83
+ not suffering the paste-end race — something else is wrong (tmux frozen,
84
+ terminal in a weird state, etc.) and continued Enter-hammering would not
85
+ help.
86
+
87
+ ## Level-of-Abstraction Fit
88
+
89
+ Same as the single-shot version: `verifyInjection`,
90
+ `isMarkerStuckAtPrompt`, and `fireStuckInputRecovery` are all private
91
+ methods on `SessionManager` and operate at the tmux-send-keys abstraction
92
+ layer. No new public API. No upward leakage of tmux details. Helpers are
93
+ extracted only because polling and detection are now distinct concerns
94
+ that benefit from named test boundaries.
95
+
96
+ ## Signal vs Authority Compliance
97
+
98
+ `verifyInjection` is a verification (signal-emitting) layer. It detects a
99
+ specific structural failure (`marker visible at ❯ after Enter was sent`)
100
+ and takes one bounded structural action per detection (one Enter / C-m /
101
+ Enter+sleep+Enter). It does not replace any higher gate; it is the only
102
+ component at this layer with both the context (the marker text and the
103
+ injection timing) and the authority (it owns the tmux session interaction)
104
+ to perform the recovery. There is no higher gate to defer to.
105
+
106
+ The escalation across attempts is a structural escalation in mitigation
107
+ strategy, not in authority. Each recovery attempt remains "send one of N
108
+ recovery key sequences to tmux" — same authority bound throughout.
109
+
110
+ ## Interactions With Existing Recovery Paths
111
+
112
+ | Layer | Trigger | Scope | Conflicts with multi-shot? |
113
+ |-------|---------|-------|----------------------------|
114
+ | `verifyInjection` polling (new) | After every `rawInject` | per-injection, bounded by 6.5s schedule | n/a — this is the new layer |
115
+ | `pasteRetried` | `[Pasted text #N]` visible at idle prompt | per-session lifetime | independent — different match string, fires from idle detector |
116
+ | `StallTriageNurse` fast-path | text at ❯ with no glyphs, before LLM | per-triage-pass | independent — runs from triage scheduler. Could fire AFTER verifyInjection's 6.5s window if the session is still stuck. Sending one more Enter at that point is harmless. |
117
+ | Zombie-kill timer | `idleMs > idlePromptKillMinutes` | global | unchanged — only fires if all prior layers missed and session sat for the configured minutes |
118
+
119
+ The four layers gate from fast-and-proactive (per-injection polling) to
120
+ slow-and-destructive (zombie kill). Adding more ticks to the proactive
121
+ layer is monotonically safer — it shifts more stuck-state recovery to the
122
+ fast path rather than waiting for the LLM-backed triage.
123
+
124
+ ## Rollback Cost
125
+
126
+ - Revert: restore the single `setTimeout` body from the prior version of
127
+ `verifyInjection`; delete `isMarkerStuckAtPrompt` and
128
+ `fireStuckInputRecovery`. Pure additive — no state, no schema, no
129
+ migration.
130
+ - Tests added: `tests/unit/session-multishot-recovery.test.ts` (12 tests).
131
+ Existing structural tests (`tests/unit/session-injection-verify.test.ts`)
132
+ were updated to assert the bounded-multi-shot invariants — 10 tests pass.
133
+ - Branch surface area: 1 source file modified, 1 test file modified, 1 test
134
+ file added.
135
+
136
+ ## Evidence
137
+
138
+ - Live failure: `/Users/justin/.instar/agents/echo/logs/server.log`
139
+ 01:31–01:34 — two stuck-then-eaten-recovery events on the same injection
140
+ marker.
141
+ - `tests/unit/session-multishot-recovery.test.ts` — 12 tests covering
142
+ multi-shot polling, escalation across attempts, bounded recovery count,
143
+ early-stop on submission detected, no-op for clean injections, no-op for
144
+ short markers, halt on tmux session disappearance, and detection-
145
+ heuristic edge cases.
146
+ - `tests/unit/session-injection-verify.test.ts` — 10 structural tests
147
+ asserting the new design contract (bounded retries, monotonic backoff
148
+ schedule, recovery-method escalation, early-stop predicate).
149
+ - No regression: `SessionManager-injection.test.ts`,
150
+ `paste-stuck-detection.test.ts`, `session-telegram-inject.test.ts`,
151
+ `stall-triage-typed-not-submitted.test.ts` all pass (25 tests).
@@ -0,0 +1,252 @@
1
+ # Side-Effects Review — project-scope Phase 1b PR 1 (Drift checker + supervisor git-creds fix)
2
+
3
+ **Version / slug:** `project-scope-phase1b-pr1`
4
+ **Date:** `2026-05-11`
5
+ **Author:** `echo`
6
+ **Second-pass reviewer:** `required (new LLM-mediated surface + new tmux-spawn flag)`
7
+
8
+ ## Summary of the change
9
+
10
+ First PR of Phase 1b for the project-scope feature. Ships the *signal*
11
+ producer that Phase 1.5's round-runner will consume — the drift
12
+ checker that asks an `IntelligenceProvider` whether a spec's stated
13
+ premises still hold against the files it depends on. By itself it is
14
+ inert: nothing wires it up yet. The Phase 1b PR 2 (round runner) is
15
+ the first consumer.
16
+
17
+ Tag-along: a one-line fix to the server supervisor that has been
18
+ producing a runaway "Server unhealthy" restart loop on every fresh
19
+ spawn since at least v0.28.87. Discovered during the Phase 1a gate
20
+ verification this evening. Fix lives in the same PR because it
21
+ unblocks the supervisor (and therefore unblocks every agent that runs
22
+ into the credential-prompt loop) and is one line of code.
23
+
24
+ Spec source: `docs/specs/PROJECT-SCOPE-SPEC.md` § Phase 1.4 (drift
25
+ checker). Supervisor fix is not spec'd — it's a defect repair.
26
+
27
+ New files:
28
+ - `src/core/ProjectDriftChecker.ts` (≈480 lines) — the verdict
29
+ producer. Path-jails every file via the existing `jailPath` helper,
30
+ reads spec + referenced files, enforces hard limits (count / per-file
31
+ bytes / per-file lines / total token budget), wraps content in
32
+ `<UNTRUSTED_SPEC_BODY>` / `<UNTRUSTED_FILE_CONTENT>` blocks, calls
33
+ the provider with a 30s timeout + 1 retry, parses + structurally
34
+ validates the response, re-verifies every citation against the bytes
35
+ on disk, downgrades to `manual-review-required` whenever anything
36
+ can't be trusted.
37
+ - `tests/unit/ProjectDriftChecker.test.ts` (45 tests) — covers path
38
+ jail (`../`, absolute, symlink escape), prompt-injection delimiters,
39
+ over-budget (count / bytes / lines / tokens), empty-spec,
40
+ deleted-files, partial-deleted-files, schema-fail (no JSON / bad
41
+ enum / non-string rationale / non-array citations), citation
42
+ verification (byteRange OOB, fabricated file, LLM-claimed excerpt
43
+ discarded), all-citations-fail downgrade, zero-citations no-downgrade,
44
+ timeout (both calls fail) + timeout-then-success retry, non-timeout
45
+ error bails immediately, code-fenced JSON tolerance, cacheKeyInputs
46
+ determinism across file-order and instability across file-content /
47
+ template-version / model-id changes.
48
+
49
+ Modified files:
50
+ - `src/core/types.ts` (+50/-2) — adds `DriftVerdict` and
51
+ `VerifiedCitation` types, adds `timeoutMs` to `IntelligenceOptions`.
52
+ - `src/core/InitiativeTracker.ts` (+5/-2) — types
53
+ `InitiativeRound.lastDriftVerdict` as `DriftVerdict` (was `unknown`,
54
+ flagged "typed in Phase 1b" in the source comment), adds two
55
+ companion fields `lastDriftVerdictAt` and
56
+ `lastDriftReferencedFileHashes` that the round-runner will populate
57
+ in PR 2.
58
+ - `src/lifeline/ServerSupervisor.ts` (+9/-1) — adds
59
+ `'-e', 'GIT_TERMINAL_PROMPT=0'` to the `tmux new-session` invocation
60
+ in `spawnServer()`. Comment block explains why (askpass fallback +
61
+ inherited tmux server env). Source-pattern test in
62
+ `tests/unit/server-supervisor-preflight.test.ts` (+25) asserts the
63
+ flag is present in the new-session args.
64
+
65
+ Nothing else moves.
66
+
67
+ ## Decision-point inventory
68
+
69
+ The drift checker IS a decision point — that's its purpose — but its
70
+ output is a **signal**, not authority. Round-runner (Phase 1b PR 2)
71
+ combines this signal with deterministic artifact checks before
72
+ deciding to start a round. The signal-vs-authority separation is
73
+ enforced both by the type system (the verdict is consumed only by
74
+ `ProjectRoundRunner` in PR 2) and by the spec.
75
+
76
+ - **Drift verdict** (`ProjectDriftChecker.run`) — **add** — produces
77
+ one of `no-drift | minor-drift | premise-violated |
78
+ manual-review-required`. The verdict alone is not enough to block
79
+ or start a round; the round-runner combines it with frontmatter
80
+ re-validation, PR/CI artifact checks, ownership, and ack state.
81
+ - **Path-jail rejection** (delegated to `jailPath`) — **reuse** —
82
+ inherits the same allow-list behavior the StageTransitionValidator
83
+ uses. Failure produces `manual-review-required`, never a soft pass.
84
+ - **Token-budget rejection** — **add** — over-budget input is rejected
85
+ with `manual-review-required(over-budget)` rather than silently
86
+ truncated. The spec explicitly forbids silent summarization to
87
+ prevent a half-evaluated drift verdict being trusted.
88
+ - **Citation re-verification** — **add** — every LLM-proposed citation
89
+ is independently re-read off disk. The LLM's claimed excerpt is
90
+ discarded. If the LLM produced citations and none survived
91
+ verification, the verdict is downgraded to
92
+ `manual-review-required(failed-citation-verification)` — defense
93
+ against fabricated evidence supporting a fabricated verdict.
94
+ - **Schema-fail downgrade** — **add** — any malformed LLM output
95
+ (non-JSON, bad enum, missing fields) produces
96
+ `manual-review-required(schema-fail)` rather than a best-guess.
97
+
98
+ Supervisor change has no decision-point semantics — it's an env-flag
99
+ addition that prevents git from prompting interactively. Drift is in
100
+ the BEHAVIOR of git-on-tmux, not in any agent decision.
101
+
102
+ ## Over-block vs under-block analysis
103
+
104
+ ### Drift checker
105
+
106
+ The verdict producer is at the right level of abstraction. It does
107
+ **not** read project records, it does **not** decide whether a round
108
+ may start, it does **not** persist anything. It takes an input
109
+ struct, returns a verdict struct. The round-runner (PR 2) holds all
110
+ authority. This matches the signal-vs-authority memory rule and the
111
+ spec.
112
+
113
+ Could it be over-blocking? The four "downgrade to
114
+ manual-review-required" paths (path-jail-fail, over-budget,
115
+ schema-fail, failed-citation-verification, timeout, no-provider) are
116
+ **deliberately conservative**. The spec is explicit: any failure mode
117
+ that means the verdict cannot be trusted MUST surface as user
118
+ attention rather than be treated as a soft pass. The cost of a
119
+ false-positive manual review is the user's eyeballs; the cost of a
120
+ soft pass is shipping something that doesn't match the code on disk.
121
+
122
+ Could it be under-blocking? The checker accepts a list of referenced
123
+ files from the caller — it doesn't *find* them itself by parsing the
124
+ spec. That's deliberate: parsing files-the-spec-claims-to-depend-on
125
+ is a job for the round-runner (or for spec frontmatter), not the
126
+ drift checker. Otherwise we'd have two places where spec parsing
127
+ lives, and they'd drift (irony noted).
128
+
129
+ The 50,000-token budget is a static cap rather than dynamically sized
130
+ to the provider. Caller bears the risk if they're using a provider
131
+ with a smaller context window — but in practice every supported
132
+ model has a window far bigger than 50k. Documented as a constant
133
+ that tests can override.
134
+
135
+ ### Supervisor fix
136
+
137
+ The flag only affects git operations performed inside the spawned
138
+ tmux session. It does NOT change:
139
+ - Whether git is invoked (the auto-pull / git-sync code is unchanged).
140
+ - Whether credentials are tried (osxkeychain credential helper still
141
+ runs first as configured).
142
+ - Whether non-interactive auth works (it does, as the same git pull
143
+ works from any shell that inherits the supervisor env).
144
+
145
+ It only blocks the *interactive fallback* — git's behavior when both
146
+ the credential helper AND `GIT_ASKPASS` have failed. Before this PR
147
+ that fallback hung indefinitely behind a TTY prompt. After this PR
148
+ git returns a normal "could not read Username" error and the bash
149
+ command exits, the supervisor logs a degradation, and the next
150
+ restart attempt proceeds normally.
151
+
152
+ Under-block risk: zero. The flag does not silence auth errors, only
153
+ the prompt. Misconfigured credentials still surface as a visible
154
+ failure; they just no longer wedge the supervisor.
155
+
156
+ Over-block risk: a user who *wants* git to prompt them interactively
157
+ when running `git pull` inside `tmux attach -t echo-server` will no
158
+ longer see the prompt — but that's already an unusual workflow on a
159
+ supervisor-managed tmux session (the user is supposed to use their
160
+ own shell for that). Acceptable trade-off; the alternative is the
161
+ loop.
162
+
163
+ ## Signal vs authority audit
164
+
165
+ Drift checker is a **signal source**. Three structural guarantees:
166
+
167
+ 1. The verdict is the **only** thing the checker returns. It does not
168
+ call into the InitiativeTracker, does not write to disk, does not
169
+ short-circuit any other path. Authority for "may this round start"
170
+ lives entirely in the round-runner (PR 2).
171
+ 2. The `manual-review-required` verdict is **distinct** from the
172
+ three "trustable" verdicts. Consumers cannot accidentally treat it
173
+ as a soft pass: the type system requires them to discriminate.
174
+ 3. Citation excerpts shown to the user / surfaced in the digest are
175
+ re-rendered from disk by the checker, **not** the LLM's claimed
176
+ text. A model that fabricates evidence cannot poison the digest
177
+ that the user reads.
178
+
179
+ Supervisor fix has no signal-vs-authority semantics — it's an env
180
+ flag.
181
+
182
+ ## Interactions with existing systems
183
+
184
+ - **IntelligenceProvider.** The checker is the third consumer of the
185
+ abstraction (after `DiscoveryEvaluator` and `MessageSentinel`).
186
+ Same call shape (`evaluate(prompt, options)`), same `'fast' |
187
+ 'balanced' | 'capable'` model tiers. Adds a per-call `timeoutMs`
188
+ to `IntelligenceOptions`; existing providers ignore the field
189
+ (signature is additive), and the checker enforces the timeout
190
+ externally via `Promise.race` regardless. No provider needs to be
191
+ updated.
192
+ - **InitiativeTracker.** The two new fields (`lastDriftVerdictAt`,
193
+ `lastDriftReferencedFileHashes`) are optional and unused until the
194
+ round-runner ships in PR 2. Records persisted today round-trip
195
+ through TaskFlow's `stateJson` blob; the new fields will appear on
196
+ read for any record they're written to and be silently absent for
197
+ records that predate them.
198
+ - **Server supervisor.** The tmux flag survives an existing tmux
199
+ server (per `man tmux`, `-e` is the per-session env that
200
+ process.env cannot reach once tmux is already running). The flag
201
+ is set on every new server-session spawn, including the recovery
202
+ path after a wake or a restart.
203
+ - **Existing routes / skills.** No route changes in this PR. The
204
+ `/projects/:id/drift-check` endpoint mentioned in the spec lands
205
+ with the round-runner in PR 2.
206
+
207
+ ## Rollback cost
208
+
209
+ Drift checker is inert if reverted — no callers, no migration to
210
+ unwind, no on-disk state. Reverting `src/core/types.ts` removes the
211
+ `DriftVerdict` export, which is a compile-time impact for any
212
+ downstream code that imports it — but no such code exists yet.
213
+
214
+ Supervisor fix is a one-line revert. The git-prompt-hang behavior
215
+ returns; mitigation is to set `GIT_TERMINAL_PROMPT=0` in the user's
216
+ launchd plist environment (which is also why the source comment
217
+ explains *why* the flag is needed: future authors will not
218
+ accidentally remove it thinking it's redundant).
219
+
220
+ ## What this PR explicitly defers
221
+
222
+ Per spec § Phase 1.4, these belong to drift but ship in Phase 1b
223
+ PR 2 alongside the round-runner:
224
+
225
+ - **Cache** keyed by `sha256(promptTemplateVersion + modelId +
226
+ specBodySha + sortedFileHashes)`, TTL 24h. The function that builds
227
+ the cache-key inputs is shipped here (`cacheKeyInputs`) so the
228
+ consumer in PR 2 doesn't fork the implementation; what's missing
229
+ is the store + lookup + TTL.
230
+ - **Mtime fast-path** — re-uses last cache entry if (specPath mtime,
231
+ referenced-file mtimes) all match, skipping the sha256 work.
232
+ - **Cost ledger** at `.instar/drift-spend-YYYY-MM-DD.jsonl` with
233
+ flock-protected pre-reservation and a $1/day per-agent ceiling.
234
+ - **`POST /projects/:id/drift-check` HTTP endpoint** (mutex-guarded
235
+ per-project).
236
+
237
+ Each of these is additive — the checker as shipped returns a correct
238
+ verdict on every call, just without rate-limit or memoization.
239
+
240
+ ## Verification
241
+
242
+ - `npm run lint` — passes (tsc + lint-no-direct-destructive).
243
+ - `npx vitest run tests/unit/ProjectDriftChecker.test.ts
244
+ tests/unit/server-supervisor-preflight.test.ts` — 50/50 pass.
245
+ - Manual review of the source against spec § Phase 1.4: the four
246
+ hardening properties (path jail, prompt delimiters, schema validation,
247
+ citation re-verification) are each backed by at least two test
248
+ cases. The token budget enforces the spec's "never silently
249
+ summarize" rule.
250
+ - Supervisor fix verified live: setting `GIT_TERMINAL_PROMPT=0` at
251
+ the tmux global env unblocked the runaway-restart loop on the
252
+ author's own server during this evening's gate verification.
@@ -0,0 +1,137 @@
1
+ # Side-Effects Review — Persistent Stuck-Input Sentinel
2
+
3
+ **Version / slug:** `stuck-input-sentinel`
4
+ **Date:** `2026-05-11`
5
+ **Author:** `echo`
6
+ **Second-pass reviewer:** `general-purpose subagent` *(required: sentinel terminology + session-lifecycle interaction)*
7
+
8
+ ## Summary of the change
9
+
10
+ Adds `src/core/StuckInputSentinel.ts`, a long-lived periodic worker that runs alongside `SessionManager.startMonitoring()`. Every `tickMs` (default 10s) it scans every running session, captures the bottom of the tmux pane, and decides whether the prompt holds unsubmitted text. If the same text persists across two ticks (≥20s with defaults) AND the pane shows no working indicator, it fires the same escalating recovery the in-process `verifyInjection` uses (`SessionManager.fireStuckInputRecovery`) — Enter, Enter, C-m, Enter+sleep+Enter — bounded at four attempts per stuck event. Each fire writes one row to `<stateDir>/stuck-input-events.jsonl` for operator audit and DegradationReporter consumption.
11
+
12
+ This is the durable backstop for PR #159 (v0.28.92). The in-process polling schedule (500/1500/3500/6500ms) sits entirely inside a single server process. A crash inside that 6.5s window — exactly what happened on echo 2026-05-11 during the better-sqlite3 ABI-rebuild loop that put the server in a 30+ minute restart cycle — drops every armed timer with no recovery. The sentinel re-attempts recovery from a fresh state on every restart, so the window of "stuck and undetected" is bounded by the tick interval rather than by the crash schedule.
13
+
14
+ **Files touched:**
15
+ - `src/core/StuckInputSentinel.ts` (new, 290 lines)
16
+ - `src/core/SessionManager.ts` (visibility change: `isMarkerStuckAtPrompt` and `fireStuckInputRecovery` `private` → public so the sentinel can reuse them without inheriting from SessionManager or duplicating logic)
17
+ - `src/commands/server.ts` (instantiate the sentinel after `sessionManager.startMonitoring()`; stop it during the shutdown handler)
18
+ - `tests/unit/StuckInputSentinel.test.ts` (new — 20 behavior tests)
19
+ - `tests/unit/session-injection-verify.test.ts` (regex regression: the previous test asserted the `private` keyword in the recovery helper definition; updated to match either visibility)
20
+
21
+ ## Decision-point inventory
22
+
23
+ - `StuckInputSentinel.evaluateSession` — **add** — decides "does this pane look stuck right now?" (text-at-prompt + no working indicator + persistent across ticks)
24
+ - `StuckInputSentinel.fireStuckInputRecovery` *(delegated)* — **pass-through** — reuses the existing `SessionManager.fireStuckInputRecovery` escalation; not a new authority surface, just a new caller.
25
+ - `SessionManager.isMarkerStuckAtPrompt` / `fireStuckInputRecovery` — **modify** — visibility relaxed (private → public). Behavior unchanged.
26
+
27
+ ---
28
+
29
+ ## 1. Over-block
30
+
31
+ **No block/allow surface — over-block not applicable.**
32
+
33
+ The sentinel does not gate or filter messages. Its single action surface is `tmux send-keys ... Enter` (or `C-m`) into a pane the SessionManager already owns. The "false positive" failure mode is "Enter pressed against an idle prompt that had no actual text" — and the sentinel's preconditions (extractPromptText returns non-empty AND minTicksBeforeFire elapsed) make this concrete shape essentially impossible: if `extractPromptText` returns empty, the sentinel returns early and never fires.
34
+
35
+ The closest analog to over-block is "Enter fired during a transient state that wasn't actually stuck." The `isPaneActivelyWorking` guard rejects that case: any pane showing a Claude Code footer activity hint (`esc to interrupt`, `ctrl+t to hide tasks`, `tokens · esc`) is skipped without firing. We deliberately **do not** key on line-start spinner glyphs (`✻ ✶` etc.) because past-tense markers like `✻ Brewed for 14m 11s` and `✻ Churned for 1m 16s` stick around as visible pane content long after the turn finished. Live reproduction on echo's 2026-05-11 stuck sessions (`echo-qalatra`, `echo-exploring-slack-integration`) showed both held a stale `Brewed`/`Churned` line while genuinely idle — keying on the glyph would have caused the sentinel to skip exactly the cases it's meant to recover. The footer hints, on the other hand, are structurally only rendered mid-turn, so they're the precise tell. The min-ticks-before-fire requirement (default 2 ticks, ≥20s) gives `verifyInjection`'s 6.5s window full priority on fresh injections — the sentinel never races with the in-process recovery on the same event.
36
+
37
+ ## 2. Under-block
38
+
39
+ **No block/allow surface — under-block not applicable.**
40
+
41
+ What the sentinel _misses_ is recoverable text that takes longer than `maxAttempts * tickMs` to recover (default ≥40s, all four Enter forms tried). After that, the per-session record marks itself `exhausted` and the sentinel stops firing on that exact prompt text until the user types something new. This is intentional: if four escalating recovery forms can't unstick a pane, more of the same won't help, and the user is now in a state where manual intervention is appropriate. The sentinel records the exhausted state to the events log so operators can see what was tried.
42
+
43
+ The other "miss": panes whose state genuinely needs something other than Enter (e.g., the pane is showing a `[Y/n]` confirmation prompt that ate the injection's Enter and is now waiting for `y`). Pressing Enter against a `[Y/n]` prompt typically takes the default — usually the safe choice. This is the same behavior `verifyInjection`'s recovery has today; not a regression.
44
+
45
+ ## 3. Level-of-abstraction fit
46
+
47
+ This belongs at the **session-lifecycle layer** (alongside `SessionManager`), not at the messaging layer (Telegram/Slack/iMessage adapters) and not at the per-injection layer (`verifyInjection`).
48
+
49
+ - Adapter layer is wrong because the bug isn't messaging-specific; any text that arrives at the tmux prompt and never submits is affected, regardless of how it got there.
50
+ - Per-injection layer is the existing `verifyInjection` — it owns the fast path (≤6.5s). The sentinel _complements_ it at the slow / restart-resilient path, not replaces it.
51
+ - Session-lifecycle layer is correct: the sentinel reads sessions from `SessionManager.listRunningSessions()`, captures via `SessionManager.captureOutput`, and reuses `fireStuckInputRecovery`. It owns one specific recovery concern — "did the prompt accept this Enter?" — and has no other responsibility.
52
+
53
+ A lower-level primitive worth flagging: the `extractPromptText` heuristic ("text after the last ❯") is brittle to Claude Code TUI changes. If a future Claude Code release replaces ❯ with a different prompt glyph, the sentinel goes silent (no over-fires, just no recoveries). That's acceptable behavior — the sentinel degrades to a no-op, and `verifyInjection`'s existing heuristic is updated alongside. Both heuristics live in code that's easy to find and change together (`isMarkerStuckAtPrompt` and `extractPromptText` are both in `src/core/`).
54
+
55
+ ---
56
+
57
+ ## 4. Signal vs authority compliance
58
+
59
+ **Required reference:** [docs/signal-vs-authority.md](../../docs/signal-vs-authority.md)
60
+
61
+ - [x] No — this change has no block/allow surface.
62
+
63
+ The sentinel is a **recovery detector with a bounded action surface**. It does not gate messages, does not block any flow, does not produce a 4xx response, does not refuse to dispatch. It produces one `tmux send-keys Enter` at most per tick per session, bounded at four attempts per stuck event, against a pane whose state is unambiguous (text-at-prompt + no working indicator + persistent).
64
+
65
+ The "Enter" action against a pane that turns out not to be stuck is provably benign: an empty Enter at an idle Claude Code prompt is a no-op. We're not in the "brittle detector with blocking authority" risk category — there is no blocking authority anywhere in this change.
66
+
67
+ ---
68
+
69
+ ## 5. Interactions
70
+
71
+ - **Shadowing of `verifyInjection`:** the sentinel's `minTicksBeforeFire = 2` (≥20s with default `tickMs = 10s`) sits well past `verifyInjection`'s last attempt at 6.5s, so the two never race on the same event. The sentinel takes over only after the in-process path either succeeded (prompt clear → sentinel never sees it as stuck) or exhausted (prompt still stuck after 6.5s → sentinel picks up at ~20s).
72
+ - **Shadowing of `SessionManager.monitorTick`:** independent setInterval, independent `tickInProgress` guard. They share `listRunningSessions` reads but each call is `O(running)` and the data is already cached on the SessionManager. No mutual write contention.
73
+ - **Double-fire on the same Enter:** would require `verifyInjection`'s schedule still ticking past the 20s mark, which the schedule does not (max attempt is at 6.5s). Cannot happen by construction.
74
+ - **Race with session-killing path:** if a session gets killed between `listRunningSessions()` and the per-session pane capture, `captureOutput` returns null (or throws); the sentinel swallows the error and moves on. The GC step at end of `tick()` drops the record so a respawned session with the same name starts clean.
75
+ - **Race with `purgeDeadSessions`:** purge runs once at boot before `startMonitoring`; the sentinel starts only after monitoring is up, so it never sees the pre-purge set.
76
+ - **Feedback loop with itself:** firing Enter against a non-stuck pane (false positive) appears at the next tick as a different (or empty) prompt text, which resets the per-session record. The sentinel cannot loop on its own action.
77
+
78
+ ---
79
+
80
+ ## 6. External surfaces
81
+
82
+ - **Other agents on the same machine:** none — the sentinel only sends keys to tmux sessions this agent's SessionManager owns. Cross-agent tmux interaction is impossible because tmux sessions are namespaced per-agent (`echo-…`, `dawn-…`).
83
+ - **Other users:** none — runtime addition with no API surface.
84
+ - **External systems:** none — `tmux send-keys` is a local IPC.
85
+ - **Persistent state:** new file `<stateDir>/stuck-input-events.jsonl`, append-only, one row per fire. No schema migration; consumers can ignore the file. The file is best-effort — failures to write don't halt the sentinel.
86
+ - **Timing/runtime conditions:** the sentinel's effectiveness depends on the server being up at least one `tickMs` past the stuck event. If the server crashes faster than `tickMs` *every* time after a stuck event, recovery never fires. This is a degenerate case — the underlying server crash is the bug, and other mechanisms (lifeline restart counter, manual operator intervention) catch it.
87
+
88
+ ---
89
+
90
+ ## 7. Rollback cost
91
+
92
+ **Pure code addition. Revert and ship as next patch.**
93
+
94
+ - No persistent state migration. The JSONL events log is append-only and can simply stop being written.
95
+ - No schema changes.
96
+ - No agent state repair. Disabling the sentinel mid-run loses no in-flight work — verifyInjection still handles the fast path.
97
+ - User-visible regression during rollback: zero. The sentinel only fires Enter against panes the user *wants* to recover; removing it returns to v0.28.92 behavior.
98
+
99
+ Worst case if the sentinel misbehaves in production: false-positive Enters against idle prompts (no-ops) or against `[Y/n]` confirmations (takes the default). Neither is destructive. A hot-fix that disables the sentinel via early-return at `start()` is a one-line change.
100
+
101
+ ---
102
+
103
+ ## Conclusion
104
+
105
+ This change closes the architectural hole in PR #159 (v0.28.92): in-process timers don't survive server restarts, and the bug the user reported on 2026-05-11 is _specifically_ that gap. The sentinel is decoupled from injection-time state — it observes panes purely from `captureOutput`, holds state in memory, and recovers from a fresh start on every server boot. The signal-vs-authority principle does not apply because there is no blocking surface; the action surface is one Enter keypress per attempt, bounded.
106
+
107
+ A second-pass review is required by the skill (sentinel terminology + session-lifecycle interaction). The reviewer should specifically check the `isPaneActivelyWorking` heuristic (does it correctly skip "✶ Running…" panes? does it correctly NOT skip a stale `✻ Brewed for…` past-tense marker?) and confirm that no race with `verifyInjection` can produce double-fires within the 6.5s window.
108
+
109
+ ---
110
+
111
+ ## Second-pass review (if required)
112
+
113
+ **Reviewer:** general-purpose subagent
114
+ **Independent read of the artifact: concur (with two tight-but-acceptable callouts)**
115
+
116
+ A. Verified clean. `evaluateSession` sets `consecutiveTicks=1` on first sight and returns; firing requires `consecutiveTicks >= minTicksBeforeFire` (=2), so the earliest fire is one full `tickMs` (10s) after first observation. That's safely past `verifyInjection`'s 6.5s window in wall-clock terms — even if a tick fires immediately after server start and observes a prompt that an injection just landed, the second observation is ~10s later, by which point `verifyInjection`'s last attempt has already resolved. (`src/core/StuckInputSentinel.ts:220-236`.)
117
+
118
+ B. *Originally flagged as tight-but-acceptable; resolved before commit.* After live-testing the sentinel against echo's 2026-05-11 stuck sessions (`echo-qalatra`, `echo-exploring-slack-integration`), both panes held a stale `✻ Brewed for…` / `✻ Churned for…` line — exactly the user-visible failure mode this change is meant to recover. The spinner-glyph branch was removed; `isPaneActivelyWorking` now keys solely on the footer activity hints (`esc to interrupt`, `ctrl+t to hide tasks`, `tokens · esc`), which Claude Code only renders mid-turn. Tests updated: the suite now explicitly asserts that stale Brewed/Churned lines are treated as idle (the previously-mis-flagged case) and that present-tense glyph lines are also ignored unless accompanied by a footer hint.
119
+
120
+ C. `extractPromptText` iterates bottom-up and picks the LAST `❯` line, which matches Claude Code's TUI invariant (active prompt at bottom). Historical `❯` glyphs higher in scrollback are correctly ignored. For 3+ line wrapped input, only the first wrapped line is read (`lines[i+1]`); this is sufficient as a stability fingerprint across ticks even if it doesn't capture the full wrapped body. Tight-but-acceptable callout: if Claude Code ever wraps such that `lines[i+1]` is whitespace and `lines[i+2]` holds the real content, the prompt looks empty and the sentinel returns null (silent no-op). Low-likelihood under current TUI behavior; flag for re-test on any Claude Code TUI bump.
121
+
122
+ D. Cross-restart unboundedness acknowledged in §6. Worst case is N+M Enters across a crash boundary (N before, M after); after the first successful Enter the prompt clears and subsequent Enters land on an idle pane (no-op). Cost bounded and benign.
123
+
124
+ E. `tick()` is fully synchronous; the `try / finally` at `StuckInputSentinel.ts:144-152` resets `tickInProgress=false` even if `tick()` throws. The inner `evaluateSession` also wraps tmux failures locally, so an error there won't escape past the outer finally. Clean.
125
+
126
+ F. Confirmed pure side-effect surface. The sentinel only delegates to `SessionManager.fireStuckInputRecovery`, which is exclusively `execFileSync(tmuxPath, ['send-keys', …])`. No message gate, no transport-layer block, no 4xx response path. Signal-vs-authority compliance holds.
127
+
128
+ Net: ship.
129
+
130
+ ---
131
+
132
+ ## Evidence pointers
133
+
134
+ - Live reproduction: echo's tmux sessions `echo-exploring-slack-integration`, `echo-qalatra`, `echo-threadline-dev` on 2026-05-11 — text-at-prompt with no spinner, no submission, after a server crash-loop window. None of these messages reached the corresponding Claude Code transcripts (`grep -l "start the rewrite now" ~/.claude/projects/-Users-justin--instar-agents-echo/*.jsonl` returns nothing).
135
+ - Unit tests: `tests/unit/StuckInputSentinel.test.ts` — 20 passing tests covering extraction, working-state detection, tick lifecycle, escalation order, max-attempts cap, prompt-change reset, prompt-clear reset, session-GC.
136
+ - Regression coverage: `tests/unit/session-injection-verify.test.ts`, `tests/unit/session-multishot-recovery.test.ts`, `tests/unit/SessionManager-injection.test.ts` all green after the visibility-relaxation regex update.
137
+ - Type-check: `tsc --noEmit -p .` clean.