instar 1.3.211 → 1.3.213

package/upgrades/1.3.212.md

@@ -0,0 +1,65 @@
+# Upgrade Guide — vNEXT
+
+<!-- assembled-by: assemble-next-md -->
+<!-- bump: patch -->
+
+## What Changed
+
+A codex conversational session can wedge: the server is healthy and messages are
+delivered, but the session sits paused with an injected message stuck at the
+prompt, never draining into a turn. `StuckInputSentinel` already detects this
+(marker-based), but its recovery topped out at keypresses — which, live, weren't
+enough; the session needed a full server restart + queue replay, performed by
+hand. This adds an **escalating self-recovery** so a codex session heals itself
+with no external nudge.
+
+The hard part is a process boundary: the detector (`StuckInputSentinel`) runs in
+the server process, but the restart authority (`ServerSupervisor` + queue replay)
+runs in the lifeline process. So the fix is split cleanly:
+
+- **`SessionRecoveryChannel`** — a cross-process request/ack channel. The server
+  emits recovery requests (sole writer of the request file); the lifeline writes
+  acks (sole writer of the ack file) — single-writer-per-file, atomic, so the two
+  processes never race.
+- **Sentinel escalation** — after the keypress ladder exhausts but the marker is
+  still stuck, the sentinel requests a tier-C recovery, polls the ack, verifies,
+  and bounds the wait (no restart loop).
+- **`SessionRecoveryConsumer`** (lifeline) — reads tier-C requests and performs
+  `performGracefulRestart` + `replayQueue`, dry-run-first, guarded by a **durable**
+  cooldown (a server restart wipes the sentinel's in-memory bound, so the loop
+  guard has to survive the restart).
+
+Ships **dark** behind `monitoring.codexWedgeRecovery` (default off; `dryRun:true`
+first). With no config it is byte-for-byte the legacy behavior.
+
+## What to Tell Your User
+
+When one of my codex-driven sessions gets stuck — paused with a message that never
+turns into a reply — I can now work myself loose without anyone stepping in:
+first by nudging the prompt, and if that is not enough, by cleanly restarting my
+own server and replaying the queued message. It is turned off by default and ships
+in a watch-only dry-run mode first, with a built-in cooldown so it can never get
+into a restart loop. You decide when to turn it on.
+
+## Summary of New Capabilities
+
+- A codex session that wedges (input delivered, not draining) can self-recover by
+  escalating from keypresses to a server restart + queue replay — no external agent.
+- The escalation is bounded and has a durable cross-restart cooldown, so a wedge
+  that a restart can't fix can never cause a restart loop.
+- Ships dark + dry-run on the Graduated-Feature-Rollout track
+  (`monitoring.codexWedgeRecovery`); absence of config = off, so no migration is
+  needed and existing agents are unaffected until explicitly enabled.
+
+## Evidence
+
+- 78 tests across 3 tiers: unit (`SessionRecoveryChannel` 15, `StuckInputSentinel`
+  escalation 7, `SessionRecoveryConsumer` 8, plus the existing stuck-input suites)
+  + integration (`tests/integration/codex-wedge-recovery.test.ts` 3 — the full
+  sentinel→channel→consumer→ack→sentinel loop through the real channel, live +
+  dry-run + dark-disabled paths).
+- `tsc --noEmit` clean; `pnpm build` clean.
+- Dark-by-default verified: with no config, the sentinel never escalates and the
+  lifeline never starts the consumer (asserted in unit + integration).
+- Spec: `docs/specs/CODEX-SESSION-WEDGE-SELF-RECOVERY.md` (grounded gap-analysis +
+  cross-process design + the durable-cooldown finding + the increment plan).

package/upgrades/1.3.213.md

@@ -0,0 +1,50 @@
+# Upgrade Guide — vNEXT
+
+<!-- assembled-by: assemble-next-md -->
+<!-- bump: patch -->
+
+## What Changed
+
+A scheduled codex job (`codex exec --json`) can wedge: the process stays alive but
+makes no progress — 0% CPU, no output. One such job sat wedged ~12h undetected. The
+StaleSessionBackstop (which raises a "stale but unkillable" attention item, never
+kills) missed it because its forward-progress check used process **existence**:
+`hasActiveProcesses` walks descendant processes and returns true if any non-baseline
+child is alive — which a wedged-but-alive job is. So the backstop read it as
+"active" and never escalated. (Conversational claude sessions are saved by transcript
+growth, so only jobs — whose codex transcript doesn't resolve — were exposed.)
+
+The fix gives the backstop a real CPU signal: `SessionManager.descendantCpuSeconds`
+reads each non-baseline descendant's accumulated CPU time, and the backstop compares
+the delta across its two snapshots — real CPU *used* in the interval, not mere
+existence. This is scoped to **job sessions** (which have no legitimate-idle state):
+a job that uses no CPU between snapshots reads as no-progress and escalates.
+Conversational sessions keep the existing existence-based check, so a legitimately-idle
+session with a background process is *not* falsely flagged.
+
+## What to Tell Your User
+
+If one of my scheduled background jobs ever freezes — alive but doing nothing — I'll
+now actually notice it and flag it for attention, instead of mistaking "the process
+is still running" for "it's making progress." Before, a job could sit stuck for hours
+unseen; now a job that stops using any CPU gets surfaced. This only adds an
+attention notice — it never kills anything — and it doesn't change how I treat your
+normal interactive sessions.
+
+## Summary of New Capabilities
+
+- The stale-session backstop now detects a wedged-but-alive **job** (0% CPU, no
+  output) and raises its existing attention notice, where it previously read process
+  existence as "progress" and stayed silent.
+- New `SessionManager.descendantCpuSeconds` (accumulated CPU-seconds of a session's
+  real children) + `parseProcTimeToSeconds` — a CPU-delta progress signal.
+- Scoped to job sessions only; conversational sessions are unchanged (no new
+  false-positives on idle-with-a-background-process). Never kills — attention-only.
+
+## Evidence
+
+- 19 tests: `stale-session-backstop` (13 — incl. wedged-job→escalate, busy-job→no,
+  conversational-idle-with-bg→no-regression) + `parseProcTimeToSeconds` (6, covering
+  `MM:SS` / `MM:SS.ss` / `HH:MM:SS` / `DD-HH:MM:SS`). `tsc --noEmit` + `pnpm build` clean.
+- The escalation path is unchanged (still attention-only, never a kill); only the
+  forward-progress test gains a CPU-delta requirement for job sessions.

package/upgrades/side-effects/codex-wedge-self-recovery.md

@@ -0,0 +1,162 @@
+# Side-Effects Review — Codex session-wedge self-recovery
+
+**Version / slug:** `codex-wedge-self-recovery`
+**Date:** `2026-06-03`
+**Author:** `Echo`
+**Second-pass reviewer:** `Echo (self) — Tier-2, single-pass under standing preapproval; ships dark`
+
+## Summary of the change
+
+Adds an escalating self-recovery for wedged codex sessions (input delivered but
+not draining). The detector (`StuckInputSentinel`, server process) escalates past
+the keypress ladder by requesting a tier-C recovery — a server restart + queue
+replay — which is executed by a new lifeline-process consumer
+(`SessionRecoveryConsumer`), because `ServerSupervisor` lives in the lifeline. The
+two processes communicate through a new on-disk `SessionRecoveryChannel`
+(single-writer-per-file request + ack + cooldown). Files: `SessionRecoveryChannel.ts`
+(new), `SessionRecoveryConsumer.ts` (new), `StuckInputSentinel.ts` (escalation state
+machine + recovered-request cleanup), `types.ts` (config shape), `server.ts` +
+`TelegramLifeline.ts` (wiring). Highest-blast-radius decision point: **whether to
+restart the agent server.** Ships dark (`monitoring.codexWedgeRecovery`, default
+off + dryRun).
+
+## Decision-point inventory
+
+- `StuckInputSentinel exhausted-record path` — modify — after the keypress ladder,
+  escalate (request tier C) instead of stopping, when enabled.
+- `SessionRecoveryConsumer.handle` — add — decides whether to restart the server
+  for a given request (cooldown + dedup + dry-run gates).
+- `monitoring.codexWedgeRecovery` config gate — add — master enable + dryRun +
+  cooldown + timeout. Default off.
+- No message block/allow decision point is touched.
+
+---
+
+## 1. Over-block
+
+No message block/allow surface — over-block not applicable. The closest analogue
+is "would it restart a server it shouldn't?" Guards against that: it only fires
+when (a) the config is enabled, (b) a codex marker is stuck past the keypress
+ladder (≥ minTicks + maxAttempts keypresses all failed), and (c) the durable
+cooldown is clear. In dry-run (the shipping default) it never restarts at all.
+
+---
+
+## 2. Under-block
+
+No message block/allow surface. Failure-to-recover modes it still misses: a wedge
+whose pane signature the marker-based detector doesn't recognize won't escalate
+(out of scope — detection is unchanged); a wedge that a server restart genuinely
+can't fix will restart once, hit the cooldown, ack failed, and stop (by design —
+bounded, not infinite). Tier B (a gentler server-side message re-inject before the
+restart) is a documented planned refinement, not in this cut.
+
+---
+
+## 3. Level-of-abstraction fit
+
+Correct layering, and it was the central design question. Detection is a low-level
+signal in the server; the restart is a high-level authority in the lifeline. They
+are split across the real process boundary via the channel rather than forcing the
+sentinel to reach across it. Reuses existing primitives (`fireStuckInputRecovery`,
+`ServerSupervisor.performGracefulRestart`, `replayQueue`) instead of reimplementing
+them.
+
+---
+
+## 4. Signal vs authority compliance
+
+**Required reference:** [docs/signal-vs-authority.md](../../docs/signal-vs-authority.md)
+
+- [x] No — this change has no message block/allow surface.
+
+This is the textbook Signal-vs-Authority split: the sentinel emits a SIGNAL (a
+recovery request); the lifeline holds the AUTHORITY to act on it (or decline, via
+cooldown/dry-run). The sentinel never restarts anything; the consumer is the only
+actor, and it is gated.
+
+---
+
+## 5. Interactions
+
+- **Shadowing:** The escalation hangs off the END of the existing keypress ladder —
+  it only runs after `attempts >= maxAttempts`, so it never shadows the keypress
+  recovery (that still runs first, unchanged). The lifeline consumer runs on its own
+  15s interval alongside the existing replay + restart-signal intervals; it acts
+  only on tier-C requests, so it can't interfere with version-skew restart signals
+  (different file, different tier).
+- **Double-fire:** The consumer dedups on `(sessionId, attemptId)` (terminal ack →
+  skip) and records the restart durably BEFORE acting, so a crash mid-restart still
+  counts against the cooldown. The sentinel won't emit a second request while one is
+  in flight (idempotent by attemptId).
+- **Races:** Single-writer-per-file (server owns request + cooldown-trigger via the
+  sentinel's clears; lifeline owns ack + cooldown record) — no cross-process write
+  contention. Atomic writes via SafeFsExecutor.
+- **Feedback loops:** THE risk for this feature — a tier-C restart wipes the
+  sentinel's in-memory escalation bound, so a still-stuck session could re-detect →
+  re-restart forever. Closed by the DURABLE cooldown the lifeline checks before every
+  restart (the sentinel's in-memory bound is no longer load-bearing across restarts).
+  Unit-tested explicitly (cooldown blocks the second restart; allows it after the
+  window).
+
+---
+
+## 6. External surfaces
+
+- **Other agents / users:** none — per-agent state files + per-agent processes.
+- **External systems:** none (Telegram/Slack/GitHub untouched). The server restart
+  is the agent restarting itself — the same action the auto-updater + `/lifeline
+  restart` already perform; it briefly drops the local API/dashboard while the
+  server bounces (seconds), then queue replay re-delivers. This only happens when
+  the feature is enabled AND not in dry-run.
+- **Persistent state:** three new small JSON files under `state/`
+  (`session-recovery-requested/acked/cooldown.json`). Inert when the feature is off.
+- **Timing:** the consumer ticks every 15s only when enabled.
+
+---
+
+## 7. Rollback cost
+
+Pure code + new files; the feature is dark by default. Back-out is `git revert` +
+ship the next patch — nothing to undo on existing agents because nothing runs
+until `monitoring.codexWedgeRecovery.enabled` is set (which no agent has). The new
+state files are only written when enabled and are inert otherwise. No data
+migration, no user-visible regression. Disabling live is a config flip (set
+`enabled:false` or `dryRun:true`).
+
+---
+
+## Conclusion
+
+The review's central concern — the restart-loop feedback path — was found DURING
+the build (a tier-C restart wipes the in-memory bound) and closed with a durable
+cooldown, which is the load-bearing safety mechanism and is unit-tested. The
+feature ships dark + dry-run on the Graduated-Feature-Rollout track, with the only
+high-blast-radius action (server restart) gated four ways (config enabled, not
+dry-run, past the keypress ladder, cooldown clear) and bounded. Clear to ship as a
+dark feature for review; enabling (dry-run → live) is a deliberate, reversible
+config decision for the operator.
+
+---
+
+## Second-pass review (if required)
+
+**Reviewer:** Echo (self) — Tier-2, ships dark; recommend a dry-run soak before live.
+**Independent read of the artifact: concur**
+
+The Signal-vs-Authority split is clean, the restart-loop guard is durable and
+tested, dark-by-default is verified in both unit and integration tiers, and the
+rollback cost is a revert. One flagged follow-up (not blocking): add tier B (a
+server-side message re-inject) as a gentler step before the restart, to reduce how
+often tier C fires once this soaks.
+
+---
+
+## Evidence pointers
+
+- `tests/unit/SessionRecoveryChannel.test.ts` (15), `tests/unit/StuckInputSentinel-escalation.test.ts` (7),
+  `tests/unit/SessionRecoveryConsumer.test.ts` (8), `tests/integration/codex-wedge-recovery.test.ts` (3),
+  plus existing stuck-input suites — 78 total green.
+- `docs/specs/CODEX-SESSION-WEDGE-SELF-RECOVERY.md` — design, cross-process
+  constraint, durable-cooldown finding, increment plan.
+- `tsc --noEmit` + `pnpm build` clean.

package/upgrades/side-effects/codex-wedged-job-cpu-detection.md

@@ -0,0 +1,118 @@
+# Side-Effects Review — Codex wedged-job CPU-stall detection
+
+**Version / slug:** `codex-wedged-job-cpu-detection`
+**Date:** `2026-06-03`
+**Author:** `Echo`
+**Second-pass reviewer:** `Echo (self) — Tier-1; the regression analysis below is the load-bearing part`
+
+## Summary of the change
+
+StaleSessionBackstop's `hasForwardProgress` gains a CPU-delta progress signal for
+**job sessions**, replacing the existence-based `mainProcessActive` check for them.
+A wedged-but-alive codex job (0% CPU) now reads as no-progress and triggers the
+existing attention escalation (never a kill). Files: `SessionManager.ts`
+(`descendantCpuSeconds` + `parseProcTimeToSeconds`), `StaleSessionBackstop.ts`
+(ProgressSnapshot fields `descendantCpuSeconds`/`isJobSession`, the job-scoped
+`hasForwardProgress` branch, `cpuFloorSeconds` option), `server.ts` (snapshot
+populates the two fields). The only decision point: *is this session making forward
+progress* — and the change narrows what counts as progress for jobs.
+
+## Decision-point inventory
+
+- `StaleSessionBackstop.hasForwardProgress` (ii) — modify — for job sessions, require
+  cpu-seconds delta > floor instead of process existence.
+- No message block/allow surface. No kill/terminate surface (the backstop has none —
+  asserted by an existing structural test).
+
+## 1. Over-block
+
+No block/allow surface. The analogue ("does it raise a false attention notice?") is
+THE risk and is handled by scoping: the stricter cpu-delta test applies ONLY to job
+sessions (`!!session.jobSlug`). A conversational session that's legitimately idle
+with a background process keeps the existence-based check → not flagged. Verified by
+the `conversational-idle-with-bg → no` test.
+
+## 2. Under-block
+
+A job that uses *some* CPU between two 120s snapshots (above `cpuFloorSeconds=1`) is
+treated as progressing — so a job that's busy-looping (high CPU, no real output)
+would not be flagged by this signal. That's acceptable: this targets the 0%-CPU
+freeze; a CPU-burning runaway is a different detector's concern (and would show on
+load). I/O-bound jobs at 0% CPU with no output would be flagged — but a job has no
+legitimate long idle-wait state (it runs to completion), so flagging it for attention
+(not killing) is correct.
+
+## 3. Level-of-abstraction fit
+
+Correct layer. This is a cheap, deterministic process-metric read feeding an existing
+attention-raising backstop. It reuses `hasActiveProcesses`'s exact descendant
+tree-walk + baseline filter, adding only the CPU-time accumulation. No LLM needed —
+the signal is objective (CPU seconds), and the escalation it feeds is attention-only.
+
+## 4. Signal vs authority compliance
+
+**Required reference:** [docs/signal-vs-authority.md](../../docs/signal-vs-authority.md)
+
+- [x] No — this change has no message block/allow surface.
+
+It refines a SIGNAL (forward-progress) consumed by an existing smart-ish gate
+(the backstop's episode state machine), which only ever raises an attention item and
+never kills. No new authority.
+
+## 5. Interactions
+
+- **Shadowing:** The change is inside `hasForwardProgress`, one branch (ii). Branches
+  (i) transcript-growth and (iii) idle-token-change are unchanged and still short-
+  circuit to "progress" first, so a job that IS producing output/transcript is still
+  exempt before the CPU test is consulted.
+- **Double-fire:** None — same single escalation path, same per-episode dedup
+  (`episodeActive`), so a persistently-wedged job still raises exactly one item.
+- **Races:** `descendantCpuSeconds` is a synchronous `ps` read per snapshot, same
+  cadence/shape as the existing `hasActiveProcesses` call; no shared mutable state.
+- **Feedback loops:** None — the output is an attention notice, which does not feed
+  back into the session.
+
+## 6. External surfaces
+
+- **Persistent state:** none new. ProgressSnapshot lives in memory per tick.
+- **External systems:** none (the attention item uses the existing poster).
+- **Other agents/users:** none — per-agent backstop.
+- **`ps` cost:** one extra `ps -eo pid,ppid,time,command` per session per 120s tick —
+  negligible, same class as the existing `hasActiveProcesses` `ps`.
+- **Cross-platform:** `parseProcTimeToSeconds` handles the `ps -o time` formats
+  (`MM:SS`, `MM:SS.ss`, `HH:MM:SS`, `DD-HH:MM:SS`); a bad parse reads as 0 (no growth),
+  which is safe (the delta just shows no progress; attention, never kill).
+
+## 7. Rollback cost
+
+Pure code change; no persistent state, no migration. Back-out is `git revert` + ship
+the next patch — the backstop reverts to existence-based progress (the prior, looser
+behavior). No user-visible regression during rollback (it only changes whether a
+*job* gets a stale-attention notice). Worst case if the floor is mis-tuned: a busy
+job flagged spuriously (attention only) or a wedged job missed — both are
+attention-level, never destructive.
+
+## Conclusion
+
+The review's central risk — falsely flagging legitimately-idle sessions — is closed
+by scoping the cpu-delta requirement to job sessions (no legitimate-idle state) and
+leaving conversational sessions on the existing existence check. That scoping is the
+load-bearing design decision and is directly tested. The escalation remains
+attention-only (never a kill), and the change reuses the existing process-walk. Clear
+to ship.
+
+## Second-pass review (if required)
+
+**Reviewer:** Echo (self) — Tier-1.
+**Independent read of the artifact: concur**
+
+The job-scoping is the right way to avoid the idle-with-bg false-positive; the three
+behavior tests pin both the fix and the no-regression case; rollback is a revert.
+No concerns.
+
+## Evidence pointers
+
+- `tests/unit/stale-session-backstop.test.ts` (13: incl. the 3 new behavior cases),
+  `tests/unit/parseProcTimeToSeconds.test.ts` (6). `tsc` + `pnpm build` clean.
+- Root-cause + design trail: `.instar/autonomous/13435.local.md` item 2 (the iterative
+  diagnosis that caught the idle-with-bg / extended-think regression traps).