instar 1.3.641 → 1.3.643

package/upgrades/1.3.643.md

@@ -0,0 +1,66 @@
+# Upgrade Guide — vNEXT
+
+<!-- assembled-by: assemble-next-md -->
+<!-- bump: minor -->
+
+## What Changed
+
+The instar server makes frequent calls to `tmux` (the terminal multiplexer that
+holds every agent session). Those calls were SYNCHRONOUS — they blocked the
+server's single event loop until tmux answered. All agents on one machine share
+ONE tmux server, so when it got slow under load a single call could freeze the
+whole server for ~15 seconds. Two user-visible things broke as a result: the
+dashboard's live connection dropped (it showed "disconnected"), and the ~0-CPU
+freeze was MISREAD as the machine going to sleep (a false sleep/wake event).
+
+This change makes the hot path non-blocking and adds three defenses — all
+dev-gated, meaning they run LIVE on development agents (where the bug was
+diagnosed) and ship DARK on the fleet for a graduated rollout:
+
+- The frequent tmux calls now run asynchronously with a hard timeout, and the
+  dashboard's and sessions' reads are served from a cache that can never hang on
+  tmux.
+- An in-flight-operation marker lets the sleep detector tell a real freeze apart
+  from a real sleep, so a slow tmux call is no longer misreported as sleep. The
+  lifeline reads the same marker cross-process, so it won't restart a
+  busy-but-alive server.
+- A signal-only "degraded tmux" guard watches for a persistently slow tmux server
+  and surfaces it on `/guards` — without ever killing anything.
+
+Crucially, a tmux call that times out is treated as INDETERMINATE (keep the
+session) — never as "the session is dead" — so a slow tmux can never cause a live
+session to be wrongly reaped.
+
+## What to Tell Your User
+
+For most agents nothing changes yet — this ships disabled on the fleet and is
+being soaked on development agents first. On a development agent the visible win
+is that the dashboard stops randomly showing "disconnected" and the agent stops
+occasionally misreporting itself as having gone to sleep — both were caused by a
+busy shared terminal manager freezing the server loop.
+
+## Summary of New Capabilities
+
+- Async, timeout-bounded tmux on the server hot path; dashboard + session reads
+  served from a non-blocking cache that can't hang on a slow tmux.
+- Tri-state tmux outcome (success / definitely-absent / indeterminate); a timeout
+  KEEPS the session (never a false reap under load).
+- In-flight-sync-op marker as the block-vs-sleep discriminator — works
+  cross-process so the lifeline won't restart a busy-but-alive server.
+- Signal-only `DegradedTmuxGuard` (bounded-accumulation ring, load-gated, never
+  kills tmux) surfacing a persistently degraded tmux server via `GET /guards`.
+- All three behind dev-gated flags (`monitoring.tmuxResilience.*`,
+  `monitoring.degradedTmuxGuard`): live on development agents, dark on the fleet.
+
+## Evidence
+
+Reproduced and guarded by a 3-tier suite (307 tests): InFlightSyncOpMarker
+round-trip + TTL self-heal; SleepWakeDetector block-vs-sleep on both sides of the
+boundary; SessionManager async tri-state (timeout ⇒ indeterminate KEEP, the
+no-false-reap regression guard); wake-handler + ServerSupervisor amplifier guards;
+DegradedTmuxGuard bounded-accumulation burst-invariant (10,000 samples, ring
+length never exceeds windowSize); an integration slow-stub (a ~15s tmux stub
+proves `/health` and `/sessions` never hang and zero sessions are reaped); and an
+e2e dev-gate lifecycle (the three flags resolve live-on-dev / dark-on-fleet, the
+guard reports alive not missing). `tsc`, `lint-sync-subprocess-chokepoint`,
+`lint-guard-manifest`, and the `no-silent-fallbacks` ratchet are all green.

package/upgrades/side-effects/false-excuse-deferral-stop-guard.md

@@ -0,0 +1,75 @@
+# Side-Effects Review — False-Excuse Deferral Stop-Guard
+
+**Slug:** `false-excuse-deferral-stop-guard` · **Tier:** 1 (small, low-risk hook behavior addition;
+operator-requested direct fix, no spec). Touches `PostUpdateMigrator.getStopGateRouterHook()` (a
+hook template), so the tier signal flags `belowFloor` (a PostUpdateMigrator touch raises the floor) —
+accepted: the change is a self-contained substring guard, fully reversible, no external surface.
+
+## Summary of the change
+
+`stop-gate-router` hook gains `falseExcuseDeferralGuard`, mirroring the existing
+`statedContinuationGuard`: a mode-independent IIFE that, on a Stop, blocks ONCE when the final
+assistant message contains BOTH (a) a named piece of remaining work AND (b) a self-protective
+deferral rationalization (session-length / time-of-day / made-mistakes / don't-rush /
+tracked-so-it-won't-slip / next-session), re-feeding a "this excuse is false — proceed" directive.
+Same change in the deployed copy is overwritten from this source on the next update (Migration
+Parity — always-overwrite built-in hooks), so it reaches every agent.
+
+## 1. Over-block / false positive
+
+The AND-of-both-signals requirement is the false-positive control: a self-protective phrase alone
+(e.g. "it's late, but everything is done") does NOT block — `knownWork` must also match. Tested:
+a genuine completion and a time-reference-with-no-pending-work both pass through (no block). Cost of
+any residual false positive is bounded to ONE extra turn (the agent re-affirms and re-stops), because
+the guard fires once.
+
+## 2. Under-block
+
+If the agent stops with an excuse but uses wording outside the phrase lists, it slips through (a
+false negative) — acceptable: the guard is a high-precision catch for the documented recurring
+phrasings, not a complete classifier. The lists cover the operator-cited forms and the agent's actual
+observed messages. Tunable by extending the arrays.
+
+## 4. Signal vs authority
+
+The guard is a one-shot re-feed (block decision with a reminder), exactly like the
+stated-continuation guard. It never silences or rewrites a message and takes no destructive action.
+The `stop_hook_active` loop guard guarantees the agent is never trapped — a legitimate stop (real
+external blocker / work complete / a user-only decision) re-stops cleanly on the next attempt.
+
+## 5. Interactions
+
+Sits directly after `statedContinuationGuard` in the same hook, before the server round-trip — so it
+works even when the server-side stop-gate is in shadow/off mode (which is exactly when these stalls
+slip through). No new dependency, no network call, no state. Pure substring matching. Cannot recurse
+or grow unbounded.
+
+## 6. External surfaces
+
+None. No route, no egress, no spend (no LLM call — deterministic substring matching in the hook
+subprocess).
+
+## 7. Multi-machine posture
+
+Machine-local: the hook runs per-session on whichever machine hosts the session. No replicated state.
+Each agent on each machine gets the guard via the standard hook update.
+
+## 8. Rollback cost
+
+Trivial: delete the `falseExcuseDeferralGuard` IIFE from `getStopGateRouterHook()`; the next update
+overwrites the deployed copy back to guard-free. No data, no migration to unwind.
+
+## Evidence pointers
+
+- `tests/unit/stop-gate-false-excuse-deferral.test.ts` (6): renders valid JS containing the guard;
+  blocks the real-world excuse-stop (named work + self-protective excuse); blocks a
+  session-length/next-session deferral; does NOT block a genuine completion; does NOT block a time
+  reference with no deferred work (false-positive control); does NOT re-block under `stop_hook_active`.
+- `tests/unit/stop-gate-stated-continuation.test.ts` + `tests/unit/generated-hooks-parse.test.ts`
+  stay green (the template literal still renders valid JS). Full `tsc` clean.
+
+## Conclusion
+
+Delivers the operator-requested structural catch for the recurring false-excuse early-stop, as an
+instar feature that ships to every agent. High-precision, loop-safe, reversible, no external surface.
+Ship.

package/upgrades/side-effects/tmux-event-loop-resilience.md

@@ -0,0 +1,86 @@
+# Side-Effects Review — tmux Event-Loop Resilience (Increment 1)
+
+**Slug:** `tmux-event-loop-resilience` · **Tier:** 2 (converged + approved spec
+`docs/specs/tmux-event-loop-resilience.md`). Touches `PostUpdateMigrator` (a
+config-strip migration), so the tier signal flags `belowFloor` — accepted: the
+PostUpdateMigrator touch is a dev-gate cleanup strip (removes a stale persisted
+`enabled:false` so the dev-gate resolves live-on-dev), fully reversible, no
+external surface.
+
+## Summary of the change
+
+The server's frequent synchronous `tmux` calls blocked the single event loop; a
+slow shared tmux server froze the loop ~15s, dropping the dashboard websocket and
+getting misread as host sleep. Increment 1: (A) an async, timeout-bounded tmux
+hot path with cache-served dashboard/session reads; (B) an in-flight-sync-op
+marker as the block-vs-sleep discriminator (in-process for SleepWakeDetector,
+cross-process via a mirror file for ServerSupervisor); (C) a signal-only
+`DegradedTmuxGuard` that surfaces a persistently slow tmux without ever killing
+it. All three are dev-gated: LIVE on a development agent, DARK on the fleet.
+
+## 1. Over-block / false signal
+
+`DegradedTmuxGuard` is signal-only — it raises ONE deduped attention item and
+never kills, restarts, or reaps anything, so a false degradation signal costs at
+most one surfaced notice. It is load-gated (suppressed when per-core load is high
+— that is expected slowness, not a tmux fault) and requires N-cycle corroboration
+(a single hiccup never raises an episode), with episode dedup + age-escalation.
+The in-flight marker can only SUPPRESS a wake / DEFER a restart when affirmatively
+present, in-flight, and non-stale — a false positive there fails toward the SAFE
+direction (keep the session alive / let a genuinely-dead server restart).
+
+## 2. Under-block / missed signal
+
+A real freeze the marker doesn't cover is still caught: SleepWakeDetector's
+existing cpuBlockBusyRatio path (CPU-spin blocks) is unchanged, and the marker
+adds the ~0-CPU I/O-wait case #1240 couldn't see. The marker self-heals via a
+2×timeout TTL so a leaked depth can never permanently blind sleep detection. The
+supervisor defer is hard-capped (`starvationRestartThreshold`) so a stuck
+in-flight marker can never wedge the restart path forever — it gives up loudly.
+
+## 3. Blast radius / fail-open
+
+Every failure path fails toward SAFE and is non-silent (no-silent-fallbacks
+ratchet enforced): a tmux call that times out maps to INDETERMINATE, never to
+"absent" — `isSessionAliveAsync` returns `'indeterminate'` (never `false`) on a
+timeout, so a slow tmux can NEVER cause a live session to be reaped (the
+line-2352 regression the tri-state guards). The cross-process marker reader is
+fail-OPEN: an absent/unparseable mirror ⇒ null ⇒ the supervisor proceeds to
+restart a genuinely dead server. The mirror writer is best-effort: an unwritable
+mirror never breaks a tmux call. `DegradedTmuxGuard` uses a fixed-capacity
+modulo-write ring (Bounded Accumulation — 10,000-sample burst-invariant test
+proves the ring length never exceeds windowSize), so it cannot grow memory under
+a flood.
+
+## 4. Signal vs authority
+
+The only blocking authority introduced is the supervisor restart-DEFER, and it is
+bounded (hard cap + TTL + fail-open) and additive to the existing
+`deferRestartForCpuStarvation` (the `||` keeps the CPU side-effect — verified by
+test). The wake-handler amplifier guard short-circuits a marker-covered wake and
+bounds the cascade's tmux re-validation (async 9000+SIGKILL); it never silences a
+genuine recovery. `DegradedTmuxGuard` holds zero authority — it observes and
+surfaces only.
+
+## 5. Interactions
+
+Reuses the existing `'event-loop-block'` WakeSuppressionReason + StallEvent (no
+new telemetry shape). The async hot-path twins coexist with the legacy sync
+methods behind `tmuxAsyncEnabled` (off ⇒ byte-identical legacy behavior — the e2e
+observable-equivalence test). `getCachedRunningSessions()` is the cache the
+request routes (GET /status et al.) now read; the non-request consumers
+(JobScheduler, WebSocketManager, AutoUpdater) intentionally keep the live read
+(documented residual, Increment 2 territory). Per-agent socket isolation is
+explicitly OUT of Increment 1 (Increment 2).
+
+## 6. Rollback
+
+Each of the three flags is independently disable-able in `.instar/config.json`
+(`monitoring.tmuxResilience.asyncHotPath.enabled`,
+`monitoring.tmuxResilience.inFlightMarker.enabled`,
+`monitoring.degradedTmuxGuard.enabled`) — set to `false` to revert that layer to
+the pre-change behavior. The async hot path falls back to the sync path; the
+marker branch goes inert; the guard stops observing. The migration strips only a
+default-shaped persisted `false`, never an explicit operator `true`/`false`.
+Config defaults OMIT `enabled` so the dev-gate (resolveDevAgentGate) governs:
+fleet = dark, dev = live.