instar 1.3.642 → 1.3.644

package/upgrades/1.3.644.md

@@ -0,0 +1,45 @@
+# Upgrade Guide — vNEXT
+
+<!-- assembled-by: assemble-next-md -->
+<!-- bump: patch -->
+
+## What Changed
+
+The macOS keychain credential read (`OAuthRefresher.defaultCredentialStore.read`) was a synchronous
+`execFileSync('security', …)` with no timeout, run on the server's single event loop. The
+credential-audit loop reads all of an agent's claude account slots sequentially through it, so when
+several co-resident agents contend on one `securityd`, each read stalled seconds and the loop froze
+4–13s every ~30–65s — dropping the dashboard websocket (the "Disconnected" flapping) and false-firing
+the SleepWakeDetector (a ~0-CPU I/O-wait block the CPU check can't see). This is the SECOND
+event-loop blocker behind the dashboard flapping; the tmux Event-Loop Resilience fix (v1.3.643)
+removed the first (the tmux calls), this removes the keychain one.
+
+The read now has an async, off-loop variant (`readAsync`, promisified `execFile`, 3s timeout) used on
+the audit hot path (`CredentialIdentityOracle.resolveSlotTenant` awaits it, so each slot read yields
+the loop), plus a `timeout: 3000` on the remaining sync read/write as a bound for any non-hot-path
+caller. The sibling `CredentialProvider.ts` already set a keychain timeout; OAuthRefresher was missed.
+
+## What to Tell Your User
+
+If your dashboard kept showing "Disconnected" even after the tmux fix, this is the rest of the cause:
+a slow shared macOS keychain was freezing the server loop. After this update the dashboard stays
+connected under multi-agent load, and the agent stops occasionally misreporting itself as having gone
+to sleep.
+
+## Summary of New Capabilities
+
+- Async, timeout-bounded keychain credential read (`CredentialStore.readAsync`, optional + backward
+  compatible); the credential-audit hot path reads off the event loop.
+- A 3s timeout on the remaining synchronous keychain read/write — an unbounded `securityd` stall can
+  no longer wedge the loop (a timeout maps to needs-reauth, retried next cycle).
+- Removes the second event-loop blocker behind the dashboard "disconnected" flapping + the
+  SleepWakeDetector false-wakes.
+
+## Evidence
+
+Root cause diagnosed by a live `/usr/bin/sample` of the running server during a freeze (1567/1567
+main-thread samples in `SyncProcessRunner::Spawn → kevent` with `com.apple.security` loaded) +
+correlation probes catching freezes ~64s apart. Fix covered by a new async-read test suite (parity
+with the sync read, fallback-to-sync for stores without `readAsync`, and a deferred-promise test
+proving `resolveSlotTenant` awaits the async path); 83/83 tests across the credential suites green,
+tsc clean, no-silent-fallbacks ratchet + sync-subprocess chokepoint lint green.

package/upgrades/eli16/credential-keychain-async-read.md

@@ -0,0 +1,44 @@
+# Credential keychain read — async + timeout-bounded — ELI16
+
+## What this is
+
+Your agent's server has a single main thread that handles everything — answering the dashboard,
+replying to messages, running every background check. If any one operation on that thread stops to
+wait for something slow, *everything* waits. That's a "freeze."
+
+This fixes a freeze. To know which account a credential belongs to, the server reads it from the
+macOS keychain by running the `security` command — and it did that the BLOCKING way (`execFileSync`)
+with no time limit. Normally a keychain read is instant. But you run several agents on one Mac, and
+they all go through one macOS keychain service (`securityd`); when they pile up, each read can take
+several seconds. The server checks all 5 of your Claude accounts one after another, so the freezes
+added up to **4–13 seconds, roughly every 30–60 seconds**.
+
+That's what was actually still breaking the dashboard. The earlier tmux fix (v1.3.643) correctly
+removed the *tmux* version of this same problem, but there were TWO blocking calls, and this is the
+second one. During each freeze the dashboard's live connection drops (you saw "Disconnected"), and
+the freeze even looked to the agent like the laptop had gone to sleep, so it kept false-alarming a
+"wake."
+
+## What already exists
+
+The agent already has the right pattern in a sibling file (`CredentialProvider.ts`) — it reads the
+keychain with a 10-second time limit. This read path just never got that treatment. And the
+credential-audit loop that triggers the reads is already `async`, so switching it to a non-blocking
+read is clean.
+
+## What's new
+
+- The keychain read now has an **async, off-the-main-thread version** (`readAsync`) that the
+  audit loop uses — so while one account's keychain read is in flight, the main thread is free to
+  answer the dashboard, messages, and everything else. The freeze is gone, regardless of how slow
+  `securityd` gets or how many accounts there are.
+- Both the old synchronous read and write now also carry a **3-second time limit**, so even a
+  caller that still reads synchronously can't wedge forever (it just falls back to "needs re-auth"
+  and retries next cycle — exactly what it did before for a missing entry).
+- It's backward-compatible: the new async method is optional, so nothing else has to change.
+
+## What you need to decide
+
+Nothing. It's a self-contained, low-risk fix on one read path, fully covered by tests. Once it ships
+and your agent updates, the dashboard should stay connected under load instead of flapping, and the
+spurious "wake" alarms stop.

package/upgrades/side-effects/credential-keychain-async-read.md

@@ -0,0 +1,60 @@
+# Side-Effects Review — Credential keychain read: async + timeout-bounded
+
+**Slug:** `credential-keychain-async-read` · **Tier:** 1 (focused bug fix, no spec; rigorous
+root-cause diagnosis via a live `/usr/bin/sample` of the running server). Parent principle:
+**Structure beats Willpower** — the same "never block the event loop" guarantee the tmux
+Event-Loop Resilience fix made structural, applied to the second blocking call site it didn't cover.
+
+## Summary of the change
+
+The macOS keychain credential read in `OAuthRefresher.ts` (`defaultCredentialStore.read`) was a
+SYNCHRONOUS `execFileSync('security', …)` with NO timeout, on the event loop. The credential-audit
+hot path — `CredentialLocationLedger.auditIdentities()` → loops sequentially over all 5 claude
+account slots → `await CredentialIdentityOracle.resolveSlotTenant()` → `readClaudeOauth()` → that
+sync `security` spawn — froze the whole event loop 4–13s every ~30–65s under multi-agent `securityd`
+contention, dropping the dashboard websocket (user-visible flapping) and false-firing the
+SleepWakeDetector (~0-CPU I/O-wait). This adds an async `readAsync` (promisified `execFile`,
+3s-timeout) used on the hot path (`resolveSlotTenant` awaits it → each slot read yields the loop),
+plus a `timeout: 3000` on the remaining sync `read`/`write` for any non-hot-path caller. The sibling
+`monitoring/CredentialProvider.ts` already set `timeout:10000`; OAuthRefresher was simply missed.
+
+## 1. Behavioral equivalence / correctness
+
+`readAsync` mirrors the sync read's args + null-on-error semantics exactly; `readClaudeOauthAsync`
+reuses the identical parse and falls back to the sync `read` for any store without `readAsync`
+(the interface method is OPTIONAL, so all existing `CredentialStore` mocks compile unchanged). The
+only consumer switched to the async path is `resolveSlotTenant` (already an `async` method awaited by
+the audit loop) — verified nothing after the read assumes synchrony (only `oauth?.accessToken` is
+used). 83/83 tests across the credential suites green, including a deferred-promise test proving
+`resolveSlotTenant` does not resolve until the async read resolves (it genuinely awaits the async
+path, not the sync read).
+
+## 2. Failure modes / fail-safe
+
+Every error path returns `null` (unreadable → caller falls to needs-reauth, retried next cycle) —
+identical to the prior sync behavior. A 3s timeout now bounds a wedged `securityd` instead of an
+unbounded block: a timeout maps to `null` (needs-reauth) exactly like a missing entry. The async
+`execFile` buffers stdout/stderr (the `stdio` option is dropped on the promisified overload — stderr
+is captured-then-ignored, matching the sync read's `stdio:['ignore','pipe','ignore']`).
+
+## 3. Blast radius
+
+Two files of behavior (`OAuthRefresher.ts`, `CredentialIdentityOracle.ts`) + a one-line comment in
+`QuotaPoller.ts`. `QuotaPoller.defaultTokenResolver` stays sync by design (a single periodic read,
+now timeout-bounded — making it async would ripple through `SubscriptionAccount` token resolution
+for no benefit). No write-path semantics change (only the timeout bound is added). No credential
+VALUE ever leaves the funnel; no new external surface.
+
+## 4. Interactions
+
+Complements the tmux Event-Loop Resilience fix (v1.3.643): that fix took the SYNC TMUX calls off the
+loop; this takes the SYNC KEYCHAIN call off the loop. Together they remove the two periodic
+event-loop blockers that caused the dashboard "disconnected" flapping. The async read also removes
+the SleepWakeDetector false-wakes (no ~0-CPU block → no misread as sleep), so the spurious
+wake-recovery cascade this triggered stops.
+
+## 5. Rollback
+
+Revert the 2 source files. The change is additive (a new optional interface method + an async
+function) plus a one-line switch in `resolveSlotTenant`; the sync `read` remains the fallback, so a
+partial revert (keeping only the `timeout: 3000` bound) is also safe.

package/upgrades/side-effects/tmux-event-loop-resilience.md

@@ -0,0 +1,86 @@
+# Side-Effects Review — tmux Event-Loop Resilience (Increment 1)
+
+**Slug:** `tmux-event-loop-resilience` · **Tier:** 2 (converged + approved spec
+`docs/specs/tmux-event-loop-resilience.md`). Touches `PostUpdateMigrator` (a
+config-strip migration), so the tier signal flags `belowFloor` — accepted: the
+PostUpdateMigrator touch is a dev-gate cleanup strip (removes a stale persisted
+`enabled:false` so the dev-gate resolves live-on-dev), fully reversible, no
+external surface.
+
+## Summary of the change
+
+The server's frequent synchronous `tmux` calls blocked the single event loop; a
+slow shared tmux server froze the loop ~15s, dropping the dashboard websocket and
+getting misread as host sleep. Increment 1: (A) an async, timeout-bounded tmux
+hot path with cache-served dashboard/session reads; (B) an in-flight-sync-op
+marker as the block-vs-sleep discriminator (in-process for SleepWakeDetector,
+cross-process via a mirror file for ServerSupervisor); (C) a signal-only
+`DegradedTmuxGuard` that surfaces a persistently slow tmux without ever killing
+it. All three are dev-gated: LIVE on a development agent, DARK on the fleet.
+
+## 1. Over-block / false signal
+
+`DegradedTmuxGuard` is signal-only — it raises ONE deduped attention item and
+never kills, restarts, or reaps anything, so a false degradation signal costs at
+most one surfaced notice. It is load-gated (suppressed when per-core load is high
+— that is expected slowness, not a tmux fault) and requires N-cycle corroboration
+(a single hiccup never raises an episode), with episode dedup + age-escalation.
+The in-flight marker can only SUPPRESS a wake / DEFER a restart when affirmatively
+present, in-flight, and non-stale — a false positive there fails toward the SAFE
+direction (keep the session alive / let a genuinely-dead server restart).
+
+## 2. Under-block / missed signal
+
+A real freeze the marker doesn't cover is still caught: SleepWakeDetector's
+existing cpuBlockBusyRatio path (CPU-spin blocks) is unchanged, and the marker
+adds the ~0-CPU I/O-wait case #1240 couldn't see. The marker self-heals via a
+2×timeout TTL so a leaked depth can never permanently blind sleep detection. The
+supervisor defer is hard-capped (`starvationRestartThreshold`) so a stuck
+in-flight marker can never wedge the restart path forever — it gives up loudly.
+
+## 3. Blast radius / fail-open
+
+Every failure path fails toward SAFE and is non-silent (no-silent-fallbacks
+ratchet enforced): a tmux call that times out maps to INDETERMINATE, never to
+"absent" — `isSessionAliveAsync` returns `'indeterminate'` (never `false`) on a
+timeout, so a slow tmux can NEVER cause a live session to be reaped (the
+line-2352 regression the tri-state guards). The cross-process marker reader is
+fail-OPEN: an absent/unparseable mirror ⇒ null ⇒ the supervisor proceeds to
+restart a genuinely dead server. The mirror writer is best-effort: an unwritable
+mirror never breaks a tmux call. `DegradedTmuxGuard` uses a fixed-capacity
+modulo-write ring (Bounded Accumulation — 10,000-sample burst-invariant test
+proves the ring length never exceeds windowSize), so it cannot grow memory under
+a flood.
+
+## 4. Signal vs authority
+
+The only blocking authority introduced is the supervisor restart-DEFER, and it is
+bounded (hard cap + TTL + fail-open) and additive to the existing
+`deferRestartForCpuStarvation` (the `||` keeps the CPU side-effect — verified by
+test). The wake-handler amplifier guard short-circuits a marker-covered wake and
+bounds the cascade's tmux re-validation (async 9000+SIGKILL); it never silences a
+genuine recovery. `DegradedTmuxGuard` holds zero authority — it observes and
+surfaces only.
+
+## 5. Interactions
+
+Reuses the existing `'event-loop-block'` WakeSuppressionReason + StallEvent (no
+new telemetry shape). The async hot-path twins coexist with the legacy sync
+methods behind `tmuxAsyncEnabled` (off ⇒ byte-identical legacy behavior — the e2e
+observable-equivalence test). `getCachedRunningSessions()` is the cache the
+request routes (GET /status et al.) now read; the non-request consumers
+(JobScheduler, WebSocketManager, AutoUpdater) intentionally keep the live read
+(documented residual, Increment 2 territory). Per-agent socket isolation is
+explicitly OUT of Increment 1 (Increment 2).
+
+## 6. Rollback
+
+Each of the three flags is independently disable-able in `.instar/config.json`
+(`monitoring.tmuxResilience.asyncHotPath.enabled`,
+`monitoring.tmuxResilience.inFlightMarker.enabled`,
+`monitoring.degradedTmuxGuard.enabled`) — set to `false` to revert that layer to
+the pre-change behavior. The async hot path falls back to the sync path; the
+marker branch goes inert; the guard stops observing. The migration strips only a
+default-shaped persisted `false`, never an explicit operator `true`/`false`.
+Config defaults OMIT `enabled` so the dev-gate (resolveDevAgentGate) governs:
+fleet = dark, dev = live.