instar 1.3.682 → 1.3.683

package/upgrades/process-footprint-monitor.eli16.md

@@ -0,0 +1,47 @@
+# Process-footprint monitor — ELI16 overview
+
+## The problem in plain words
+
+On 2026-06-26 the machine running this agent crashed hard — a kernel panic that forced
+a reboot. The trigger was resource exhaustion: too many processes piled up on one
+machine (several full agent stacks plus their heavy helper "MCP" servers — a whole
+Chromium for the browser tool, an Electron app for another — most of them sitting idle).
+The number of processes climbed slowly over time until the operating system hit an
+internal limit and gave up.
+
+The painful part: nobody SAW it coming. The agent already tracks CPU and memory, but it
+never tracked the simplest, most relevant number for this kind of crash — *how many
+processes are running on this machine right now, and is that number climbing?* That
+measurement was missing, so the buildup was invisible until it was too late.
+
+## What this change does
+
+This adds a small, observe-only monitor that fills exactly that gap. On an interval it
+counts the agent-relevant processes on the machine and sorts them into buckets:
+
+- **agent CLIs** — the per-session reasoning processes (claude / codex / gemini),
+- **MCP servers** — the heavy, mostly-idle helper servers (Playwright's Chromium, etc.),
+  matched by the same precise signatures the cleanup sweep already uses,
+- **other node** — the agent's servers, lifelines, and wrappers.
+
+It keeps a rolling window of those readings so a TREND is visible — is the count rising,
+stable, or falling? You can read it any time:
+
+`GET /resources/footprint` → `{ enabled, latest: { total, byKind, rssBytes }, trend,
+overThreshold, samples }`
+
+## What it deliberately does NOT do
+
+It is **observe-only**. It never kills, throttles, or blocks anything — reclaiming
+processes is the job of the existing reapers. This monitor only MEASURES, so it can't
+cause harm. There is an optional heads-up that can fire when the count crosses a
+threshold, but it's off by default ("measure first") — the goal of this first step is
+just to make the climb visible.
+
+It ships **dark** (off on the fleet, on for development agents) so it's exercised here
+before any wider rollout, and every reading path fails safe: if the process scan fails,
+the monitor keeps its last reading rather than crashing. The route returns a clear
+"unavailable" (503) when the monitor is turned off.
+
+The point: the next time the process footprint starts climbing toward danger, it will be
+a number you can watch — not an invisible buildup that ends in a crash.

package/upgrades/side-effects/process-footprint-monitor.md

@@ -0,0 +1,69 @@
+# Side-Effects Review — Process-footprint monitor
+
+**Version / slug:** `process-footprint-monitor`
+**Date:** `2026-06-27`
+**Author:** `echo`
+**Tier:** 1 — a NEW observe-only monitor + a read-only route. Ships DARK (developmentAgent
+gate). No authority, no gate, no deletion, no mutation; it only READS process metadata
+and exposes a status. The threshold heads-up is opt-in and its sink is unwired in this
+increment (measure-first).
+
+## Summary of the change
+
+Adds `ProcessFootprintMonitor` — the per-machine process-COUNT measurement missing before
+the 2026-06-26 resource-exhaustion kernel panic (the ResourceLedger samples CPU%/RSS but
+not process count). On an interval it counts agent-relevant processes (agent CLIs / MCP
+servers via the shared `MCP_PROCESS_SIGNATURES` / other node), keeps a bounded rolling
+window for a trend, and exposes `GET /resources/footprint`. Files:
+`src/monitoring/ProcessFootprintMonitor.ts` (new — pure classifier + sampler + a
+`ps`-backed scanner funneled through `withSyncOp`), `src/server/AgentServer.ts` (construct
++ start, dev-gated; null when disabled), `src/server/routes.ts` (the read-only route +
+ctx type), `src/core/types.ts` (config), `src/scaffold/templates.ts` +
+`src/core/PostUpdateMigrator.ts` (awareness). Tests: 11 unit + 3 integration + 4 e2e.
+
+## 1. Over-block / 2. Under-block
+
+**No block/allow surface — not applicable.** The monitor never gates, kills, or throttles.
+The closest failure is a wrong reading: a throwing `ps` scan fails safe (keeps the last
+sample, never crashes `sample()`); classification is best-effort (a miscount only changes
+a number, never an action). Both sides of the classifier and the alert latch are unit-tested.
+
+## 3. Level-of-abstraction fit
+
+Correct layer. It sits beside the ResourceLedger as read-only observability, constructed in
+`AgentServer` like the other monitors and exposed via a `/resources/*` route. The pure core
+(classifier, sample builder, trend, alert latch) is fully injectable/fake-testable; only the
+production `ps` scanner touches the host.
+
+## 4. Signal vs authority compliance
+
+Pure SIGNAL. The status is consumed by a human/dashboard, not by any gate. The optional
+threshold heads-up only ever ADDS one de-duplicated attention item (per episode, with
+hysteresis) — it carries zero blocking authority and is OFF by default.
+
+## 5. Interactions
+
+- **Sampling cost:** one `ps` scan per multi-minute interval, funneled through `withSyncOp`
+  (the in-flight marker sees the bounded blocking spawn). Dark by default → zero fleet cost.
+- **Alert latch:** one heads-up per threshold-crossing episode; re-arms only after the count
+  drops below 90% of the threshold (hysteresis) — no flapping. Tested both sides.
+- **Disabled = null:** when the dev-gate/config resolves disabled, the monitor is not
+  constructed → the route 503s (matches the ResourceLedger contract; e2e-tested).
+
+## 6. External surfaces
+
+- **New route** `GET /resources/footprint` (Bearer-gated, read-only; POST → 404). No new
+  external call beyond the local `ps` scan. No new credential.
+- Config: `monitoring.processFootprintMonitor` (enabled/sampleIntervalMs/windowSamples/
+  alertThreshold/alertEnabled). `enabled` undefined rides the developmentAgent gate. No
+  config MIGRATION needed (absence resolves via the gate; `?? defaults` cover the rest).
+
+## 7. Rollback
+
+Pure additive code + one config block + docs. Reverting the commit removes the monitor and
+the route entirely. Dark default means no agent runs it until explicitly enabled.
+
+## Known follow-up (tracked, not in this increment)
+
+The threshold heads-up's `emitAttention` sink is left unwired (the alert is opt-in and OFF
+by default — measure-first). Wiring it to the aggregated attention surface is increment 2.