instar 1.3.229 → 1.3.231

package/src/scaffold/templates.ts

@@ -650,6 +650,11 @@ This routes feedback to the Instar maintainers automatically. Valid types: \`bug
 - Returns \`{ totals, features: [{ feature, calls, tokensIn, tokensOut, fired, noop, fireRate, p50LatencyMs, p95LatencyMs, ... }] }\` — one row per system (e.g. MessagingToneGate, CoherenceReviewer). Filter with \`?feature=<name>\`.
 - **When to use**: when asked "which checks cost the most / fire the least?", "is this gate worth it?", or before tuning a sentinel/gate — read the numbers instead of guessing. (Spec: \`docs/specs/llm-feature-metrics-spec.md\`.)
 
+**Resource Usage (rate-limit events)** — Durable per-agent record of rate-limit events, so they survive restarts instead of vanishing. Every circuit-breaker trip (the account got throttled) and session-sentinel detection is written down. Read-only observability — it never gates. (Phase A; CPU/memory tracking is a follow-up.)
+- Check: \`curl -H "Authorization: Bearer $AUTH" "http://localhost:${port}/resources/rate-limits?sinceHours=24"\`
+- Returns \`{ windowHours, summary: { circuitOpenCount, circuitRecoverCount, sentinelCount, tripsPerHour, ... }, byKind, events }\`. \`circuitOpenCount\` is the headline (breaker trips); \`sentinelCount\` is session-scoped detections, counted separately.
+- **When to use**: when asked "how many times were we throttled today?", "is the rate-limit pressure getting worse?", or before/while tuning anything that reacts to rate limits — read the durable count instead of guessing. (Spec: \`docs/specs/per-agent-resource-ledger.md\`.)
+
 **Session Clock** — How long have you been running, and how much time is left? For any active time-boxed (autonomous) session, this returns the computed elapsed + remaining so you never have to guess or compute it yourself. Read-only observability.
 - Check: \`curl -H "Authorization: Bearer $AUTH" "http://localhost:${port}/session/clock"\` (optional \`?topic=<N>\` to bind to one session)
 - Returns \`{ now, nowIso, sessions: [{ label, kind, startedAt, endsAt, elapsedSeconds, remainingSeconds, elapsedHuman, remainingHuman, percentElapsed, status }] }\`; \`{ sessions: [] }\` when nothing is time-boxed. Per-machine (the record is local).
@@ -755,6 +760,8 @@ Instar contributors can run \`instar dev:preflight\` before opening PRs to run l
 
 Run \`instar dev:ci-failures <pr>\` to print a red PR's exact failing tests (file:line + assertion) via the GitHub check-run annotations API — handy when \`gh run view --log\` returns nothing.
 
+Run \`instar dev:profile-node [pid]\` to CPU-profile a hot RUNNING node process and print its hottest JS functions (function + file:line + self-time %). It uses SIGUSR1 + node's inspector + a CDP CPU profile, so it sees the JS frames macOS \`sample\` can't symbolicate — the way to pin which function a busy agent server is burning CPU in. No pid → it profiles the hottest node process.
+
 **Critical rule**: If this CLAUDE.md says a feature is "for standalone agents" or "when configured" or uses any qualifier — do NOT conclude you lack the feature. Check \`/capabilities\` instead. Documentation describes features in general; the API tells you what's actually running for YOU right now. When they conflict, the API wins.
 
 ### Registry First, Explore Second

package/upgrades/1.3.230.md

@@ -0,0 +1,66 @@
+# Upgrade Guide — vNEXT
+
+<!-- assembled-by: assemble-next-md -->
+<!-- bump: minor -->
+
+## What Changed
+
+Adds `instar dev:profile-node [pid]` — a contributor/agent dev command that
+CPU-profiles a **running** node process and prints its hottest JS functions
+(function + `file:line` + self-time %).
+
+**Why this exists (the technique that cracked a real bug).** macOS `sample` — and
+lsof snapshots, empty `gh run --log`, port-grep — cannot symbolicate a node
+process's **JS** frames; they show only native/V8 frames, so a busy agent server's
+actual hot *function* stays invisible. The way to see it is the node process's own
+introspection: `SIGUSR1` opens node's inspector on the running process, then a CDP
+CPU profile over the inspector websocket reports the exact JS call frames. This is
+literally how the systemic agent-server hot-loop was pinned — `StateManager.listSessions`
+burning ~30% of CPU in `readFileUtf8`, invisible to every other tool. This command
+turns that hard-won procedure into one reusable command.
+
+**What it does:** resolves the pid (or auto-finds the hottest `node` process) →
+`SIGUSR1` → probes `127.0.0.1:9229-9235` for the inspector target → captures a CPU
+profile (default 5s, `--duration`) → prints the top frames by self-time (`--top`).
+The biggest non-idle frame is the hot path. Read-only aside from the SIGUSR1, which
+opens the inspector on localhost for the process's lifetime (it closes on the next
+restart) — the command says so in its output.
+
+Pairs with `dev:ci-failures` and `dev:preflight` as the contributor dev-loop
+toolkit (friction → tooling).
+
+## What to Tell Your User
+
+Nothing required — it is a developer/agent tool for working on Instar. When an
+agent server is mysteriously burning CPU, the instar dev:profile-node command
+prints the exact JS function responsible, which native profilers cannot show.
+
+## Summary of New Capabilities
+
+- `instar dev:profile-node [pid] [--duration <sec>] [--top <n>]` — CPU-profile a
+  running node process (SIGUSR1 + inspector + CDP) and print its hottest JS frames.
+  No pid → profiles the hottest node process.
+
+## Evidence
+
+- Built directly from the technique that pinned the real hot-loop: after `sample`,
+  lsof, grep, and log-diffing all failed, SIGUSR1 + a CDP CPU profile showed 30.8%
+  `readFileUtf8` + 7% `listSessions` (StateManager) — the JS frame nothing else
+  could see — leading to the cache fix.
+- Live end-to-end smoke: profiling a spawned CPU-busy node correctly reported the
+  hot frame (`listOnTimeout`, where the busy loop ran).
+- Tests: `tests/unit/devProfileNode.test.ts` — pure `aggregateHotFrames` (ranking,
+  file:line normalization, anonymous-frame fallback, empty profile, topN) +
+  `findInspectorTarget` (first-hit / none) + `runDevProfileNode` orchestration with
+  injected deps (happy path, no-pid→hottest, no-node→exit1, signal-fail→exit1,
+  no-inspector→exit1, capture-throw→exit1).
+
+## Also in this change
+
+Fixes a real false-negative in `scripts/docs-coverage.mjs`: a command's coverage
+was keyed on its camelCase **file** name (`devProfileNode`), so the needle
+`instar devProfileNode` never matched the shipped colon-command
+`instar dev:profile-node` — every `dev:*` command (ci-failures, preflight,
+profile-node) read as undocumented. The matcher now also tries a kebab form and
+treats `:`/`-` as interchangeable, so colon-commands count. (The new command
+surfaced this — friction → structural fix.)

package/upgrades/1.3.231.md

@@ -0,0 +1,39 @@
+# Upgrade Guide — vNEXT
+
+<!-- assembled-by: assemble-next-md -->
+<!-- bump: patch -->
+
+## What Changed
+
+**Rate-limit events are now recorded durably (per-agent ResourceLedger, Phase A).**
+Until now, every time the account got throttled — a circuit-breaker trip, or a
+session hitting Anthropic's server-side rate limit — was counted only in
+process-local memory and lost on restart, so "how many times were we throttled
+today?" had no answer. This adds a small read-only SQLite ledger
+(`server-data/resource-ledger.db`, same pattern as the TokenLedger) that persists
+each rate-limit event, fed by a minimal new trip/recover observer on the circuit
+breaker plus the existing RateLimitSentinel.
+
+New read-only route `GET /resources/rate-limits?sinceHours=N` returns the durable
+count + rate (breaker trips as the headline; session-sentinel detections counted
+separately) and recent events. Default-on (event-driven, negligible cost);
+`monitoring.resourceLedger.enabled: false` opts out. It never gates or changes any
+behavior — the breaker observer is wrapped so a listener error can never affect
+the breaker. CPU/memory tracking and a unified `/resources` snapshot are Phase B.
+
+Spec: `docs/specs/per-agent-resource-ledger.md`.
+
+## What to Tell Your User
+
+Nothing to configure. Your agent now keeps a durable record of how often it's
+being rate-limited, so that history survives restarts instead of resetting to
+zero. You can ask it how many times it was throttled today, or whether the
+pressure is getting worse, and it can answer from real recorded numbers.
+
+## Summary of New Capabilities
+
+- New read-only endpoint GET /resources/rate-limits (durable rate-limit-event
+  count, rate, and recent events).
+- Circuit-breaker trips and session rate-limit detections are now persisted
+  across restarts instead of being lost.
+- New config flag monitoring.resourceLedger.enabled (default on).

package/upgrades/dev-profile-node-tool.eli16.md

@@ -0,0 +1,42 @@
+# A stethoscope for a node process that's running hot
+
+## The one-sentence version
+When one of our programs is burning the CPU for no obvious reason, the usual tools can tell you *that* it's busy but not *what line of code* is doing it. This new command — `instar dev:profile-node` — presses a stethoscope to the running program and prints the exact function that's hogging the CPU.
+
+## The problem it solves
+Picture a car idling in the driveway but somehow guzzling gas. You pop the hood with your usual gauge — it says "engine's working hard" but won't tell you *which part*. That's macOS's built-in profiler on a Node.js program: it sees the engine block (the low-level machinery) but can't read the actual code, so the real culprit stays hidden.
+
+This bit us for real: three of our agent servers were each burning ~50–60% CPU, bouncing them never helped, and four different tools all came back "busy, but I can't tell you why." 
+
+## The trick that worked (now a one-liner)
+Node programs have a hidden superpower: send them a specific nudge (the SIGUSR1 signal) and they open a little diagnostic port on themselves. Connect to that port, record a few seconds of "what function is running right now," and it tells you — in plain code terms — exactly where the time goes.
+
+That's how we finally caught it: **30% of the CPU was in `readFileUtf8`, inside `listSessions`** — a function re-reading every session file off the disk, over and over. Invisible to everything else; obvious once we looked through the right lens.
+
+`instar dev:profile-node` bakes that whole trick into one command:
+- Point it at a process (or let it auto-find the hottest one).
+- It nudges the process, records ~5 seconds, and prints the hottest functions with their percentages.
+- The biggest non-idle line is your culprit.
+
+## What it looks like
+```
+Hottest JS frames (self-time) for pid 12673:
+   87.6%  listOnTimeout  node:internal/timers:546
+    9.9%  (program)
+    2.5%  (garbage collector)
+```
+(That's from a test process running a busy loop inside a timer — and it correctly fingered the timer callback.)
+
+## The safety notes
+1. The only thing it changes is opening that diagnostic port — and only on **localhost** (your own machine), only until the process next restarts. The command tells you it did this.
+2. It **samples** the process — it never kills it, restarts it, or edits anything.
+3. It's a developer tool, run on demand — nothing in the running system uses it.
+
+## Why it matters
+The hard part of fixing a mystery slowdown isn't the fix — it's *finding* the cause. This turns a clever, hard-won debugging trick into a reusable one-liner, so the next time any node process runs hot, the answer is one command away instead of an afternoon of dead ends. (Same spirit as our `dev:ci-failures` tool: when a workaround saves the day, make it permanent.)
+
+---
+
+**Rendered (verified HTTP 200):** https://echo.dawn-tunnel.dev/view/e5433071-bf6f-4cf0-b97a-5e16d9f7b160?sig=2c015aaeedc96ebb20d57b326f44d88eaa93dd2bcba0b4e8347a33d23f4d2b4b
+
+*See also: the full success-story + meta-insights writeup for this run.*

package/upgrades/side-effects/dev-profile-node-tool.md

@@ -0,0 +1,85 @@
+# Side-Effects Review — `instar dev:profile-node`
+
+**Version / slug:** `dev-profile-node-tool`
+**Date:** `2026-06-03`
+**Author:** `echo`
+**Second-pass reviewer:** `not required (dev-only CLI; read-only sampling; no server/runtime surface)`
+
+## Summary of the change
+
+A new dev CLI command that CPU-profiles a running node process via SIGUSR1 + the
+node inspector + a CDP CPU profile, and prints the hottest JS frames. New file
+`src/commands/devProfileNode.ts` + a cli.ts registration + a manifest entry +
+awareness lines. No server, route, config, or runtime-behavior change.
+
+## Decision-point inventory
+
+1. **pid resolution** — explicit `[pid]` digits win; otherwise auto-pick the
+   hottest `node` process (`ps -Aceo pcpu,pid,comm`). No node found → exit 1.
+2. **Inspector discovery** — probe ports 9229–9235 (node's default + increments
+   when the default is taken); first target with a `webSocketDebuggerUrl` wins;
+   none → exit 1 with a clear message.
+3. **Profile aggregation** — `aggregateHotFrames` sums per-node `hitCount` by call
+   frame, normalizes the path to `dist/…`/`src/…`, +1 to the 0-based line, and
+   returns top-N by self-time. Idle/native frames are kept on purpose (seeing
+   "48% idle, 30% readFileUtf8" is the signal).
+4. **Exit codes** — 0 on a successful print (even "no samples"); 1 only on an
+   operational failure (no node, signal failed, no inspector, capture threw).
+
+## 1. The one real side effect — SIGUSR1 opens the inspector
+
+SIGUSR1 tells node to start its inspector on `127.0.0.1` (localhost-only) for the
+life of the process. Implications, all bounded:
+- **Localhost-only** — the inspector binds 127.0.0.1, not a public interface; it is
+  not remotely reachable.
+- **Transient** — it closes when the process restarts; it does not persist.
+- **Disclosed** — the command prints that it left the inspector open on the pid.
+- **No state mutation** — profiling samples the process; it does not change its
+  behavior, memory, or files.
+
+This is the same standard procedure node documents for live debugging; it is the
+*only* way to symbolicate JS frames on a running process. The benefit (pinning a
+hot loop that no other tool can see) outweighs a transient localhost debug port on
+a dev box. Operators who object can simply not run the command.
+
+## 2. Over/under-block
+
+Not applicable — it gates nothing and changes no behavior. A wrong pid or a
+non-node target fails cleanly with exit 1; it never acts on the target beyond
+SIGUSR1 + reading the profile.
+
+## 3. Blast radius
+
+Zero runtime surface. The command runs only when a contributor/agent invokes it.
+No server code path imports it; it is a leaf CLI command behind a dynamic import
+(like `dev:ci-failures`). The `ws` dependency is already present (`^8.19.0`).
+
+## 4. Migration parity
+
+CLAUDE.md template (`generateClaudeMd`) + `site/.../cli.md` + the builtin-manifest
+(`cli:dev-profile-node`) are updated in lockstep so the awareness + discoverability
+checks stay green. No config/hook/skill migration — it is a pure CLI addition
+(same shape as `dev:ci-failures`, which needed none).
+
+## 5. Testability
+
+`aggregateHotFrames` and `findInspectorTarget` are pure/dep-injected; the
+SIGUSR1 / inspector-http / ws-CDP boundary is behind `ProfileNodeDeps` so the
+orchestration is unit-tested without a real process. A live end-to-end smoke
+(profiling a spawned busy node) confirmed the real path.
+
+## 6. What it does NOT do
+
+- Does not kill, restart, or modify the target (only SIGUSR1 + sample).
+- Does not run as part of the server or any job — invoked on demand only.
+- Does not expose a remote/public debug port (inspector is localhost).
+
+## 7. Collateral fix — docs-coverage matcher
+
+Adding the command surfaced a false-negative in `scripts/docs-coverage.mjs`: it
+matched a command by its camelCase file name, so colon-commands (`dev:*`) never
+matched their documented form and read as undocumented (the floor was sitting
+right at the limit, so the new command tipped it). The matcher now also tries a
+kebab form and normalizes `:`↔`-`. Audit-only script (the weekly docs-coverage
+report); no runtime/behavior effect. It re-classifies the three `dev:*` commands
+as documented, which is correct (they ARE in cli.md + the CLAUDE.md template).

package/upgrades/side-effects/resource-ledger-phase-a.md

@@ -0,0 +1,101 @@
+# Side-Effects Review — Per-agent ResourceLedger, Phase A (durable rate-limit events)
+
+**Version / slug:** `resource-ledger-phase-a`
+**Date:** `2026-06-03`
+**Author:** `Echo (instar-dev agent)`
+**Second-pass reviewer:** `not required`
+
+## Summary of the change
+
+Phase A of `docs/specs/per-agent-resource-ledger.md`: a durable, read-only store
+for rate-limit events, so breaker trips + session-sentinel detections survive
+restarts (today they're process-local and vanish). New:
+- `src/monitoring/ResourceLedger.ts` — SQLite store (`rate_limit_events`),
+  TokenLedger pattern (NativeModuleHealer/WAL/registerSqliteHandle), swallow-on-write.
+- `src/monitoring/ResourceLedgerPoller.ts` — event-driven; subscribes to the
+  breaker observer (+ optional RateLimitSentinel) and writes one row per emission.
+- `src/core/LlmCircuitBreaker.ts` — added a minimal `onTrip`/`onRecover` observer
+  (emit in `onRateLimited` and on open→closed `onResolved`).
+- `src/server/AgentServer.ts` — construct ledger (own try/catch) + start poller
+  (breaker = `getLlmCircuitBreaker()`); inject into route ctx; stop on shutdown.
+- `src/server/routes.ts` — `GET /resources/rate-limits` (503 on null ledger).
+- `src/config/ConfigDefaults.ts` — `monitoring.resourceLedger.enabled` (default on).
+- `src/scaffold/templates.ts` — CLAUDE.md Capabilities entry (Agent Awareness).
+
+## Decision-point inventory
+- `LlmCircuitBreaker.onRateLimited / onResolved` — **modify (additive emit only)** —
+  fire observer callbacks; no change to gating/state logic.
+- `ResourceLedger` writes, route, poller, config default, template — **add**.
+
+## 1. Over-block
+No block/allow surface — not applicable (read-only observability).
+
+## 2. Under-block
+No block/allow surface. (Phase A captures breaker trips + sentinel detections; it
+does not claim to capture every provider 429 — the breaker trip is the canonical
+account-level signal, sentinel detections are counted separately by `source`.)
+
+## 3. Level-of-abstraction fit
+Correct. The ledger is a recorder, not an authority; it FEEDS the read-only
+`/resources` surface and never gates. The breaker observer is a pure side-channel
+on a component that DOES hold gating authority — see §4.
+
+> CI-fix addendum (full-suite caught two regressions the local subset missed,
+> both no-behavior-change): (a) the new `/resources` route prefix was unclassified
+> in `CapabilityIndex` — added a read-only `resourceLedger` capability entry
+> (enabled only when `ctx.resourceLedger` exists); (b) `ResourceLedger.ts`
+> registers a SQLite handle so it was added to the `SqliteRegistry-wiring` test's
+> `LONG_LIVED_STORES` allowlist. No new surface; purely discoverability/test wiring.
+
+## 4. Signal vs authority compliance
+**Required reference:** docs/signal-vs-authority.md
+- [x] No — this change has no block/allow surface of its own.
+
+The one sensitive point: `LlmCircuitBreaker` holds real blocking authority (it
+gates ALL LLM work). I added an *observer* to it, NOT logic that affects gating.
+Safety properties, all tested (`tests/unit/llm-circuit-breaker-observers.test.ts`):
+- `emitTrip`/`emitRecover` iterate listeners inside a per-listener `try/catch`, so
+  a throwing observer **cannot** propagate into the breaker — verified by a test
+  that registers a throwing listener and asserts `onRateLimited` doesn't throw and
+  `tripCount` is still correct.
+- The emit happens AFTER all state mutation (last line of `onRateLimited`; end of
+  `onResolved`), so an observer can never see/affect an intermediate state.
+- The 30 existing breaker tests still pass unchanged — no behavioral drift.
+
+## 5. Interactions
+- **Shadowing/double-fire:** none. The observer is additive; the breaker's
+  acquire/onRateLimited/onResolved behavior is byte-identical to before.
+- **Races:** the poller's per-source `seq` + the ledger's `(source,ts,seq)` id +
+  `INSERT OR IGNORE` make writes idempotent; a replay can't double-count.
+- **Lifecycle:** ledger constructed in its OWN try/catch in AgentServer (cascade
+  isolation — a failure can't 503 TokenLedger/FeatureMetricsLedger); poller
+  `stop()` unsubscribes the breaker observer + the ledger closes on shutdown.
+- **Feedback loops:** none — observe-only.
+
+## 6. External surfaces
+- New route `GET /resources/rate-limits` (additive; Bearer-auth; read-only).
+- Persistent state: a new `server-data/resource-ledger.db` per agent. No schema
+  change to existing DBs. Naturally bounded (~hundreds of events/day) — no
+  retention needed for `rate_limit_events` (the high-frequency `resource_samples`
+  table that WILL need retention is Phase B, not in this PR).
+- Config: `monitoring.resourceLedger.enabled` default true (ships ON — event-driven,
+  negligible cost; `false` leaves the ledger null and the route 503s).
+
+## 7. Rollback cost
+Pure additive code + a new DB file. Revert and ship a patch; the orphaned
+`resource-ledger.db` is harmless (nothing else reads it). No migration of existing
+state, no agent-state repair, no user-visible regression.
+
+## Conclusion
+Complete, working Phase A with all three test tiers green (unit store+emitter+poller
+incl. a real-breaker→real-ledger wiring test; integration route 200/503; e2e
+"feature is alive" 200 on the real AgentServer init path). The only authority-
+adjacent edit (the breaker observer) is provably unable to affect gating. CPU/mem
+sampling + retention + the unified `/resources` snapshot + dashboard are Phase B
+(specced, not in this PR). Clear to ship.
+
+## Evidence pointers
+- `tsc --noEmit` clean; vitest green: `ResourceLedger.test.ts` (6),
+  `ResourceLedgerPoller.test.ts` (4), `llm-circuit-breaker-observers.test.ts` (4),
+  `llm-circuit-breaker-wait.test.ts` (30, unchanged),
+  `resources-rate-limits-routes.test.ts` (2), `resources-rate-limits-lifecycle.test.ts` (3).