@yemi33/minions 0.1.2451 → 0.1.2452

package/docs/README.md

@@ -44,6 +44,7 @@ Architecture, design proposals, and lifecycle references for people working on t
 - [cooldown-merge-semantics.md](cooldown-merge-semantics.md) — Scoping deliverable defining merge semantics for `saveCooldowns` (longer-of TTL merge, key-level upserts, gitignored on-disk format).
 - [copilot-cli-schema.md](copilot-cli-schema.md) — Behavior and schema reference for the GitHub Copilot CLI adapter (capability flags, stdin vs `-p`, model discovery, effort levels).
 - [cross-repo-plans.md](cross-repo-plans.md) — Cross-repo plans: a single plan whose work items ship into two or more configured projects — per-item `project` field, per-project work-item fan-out, and one verify work item per touched repo.
+- [dashboard-reliability-and-responsiveness.md](dashboard-reliability-and-responsiveness.md) — Reliability/perf deep dive on what starves the single dashboard event loop: the `GET /api/work-items` full-enrichment stall (measured ~3.3 s p50 at 2,300 items; production 7–17 s), the ETag/`MAX(events.id)` re-run trigger, the health-probe/CC-SSE false-disconnect chain, a threading/process-boundary decision matrix, a P0/P1/P2 roadmap, and explicit worker-thread non-recommendations. Distinguishes merged safeguards from the still-in-flight fix (PR #1198).
 - [dead-code-audit-retractions.md](dead-code-audit-retractions.md) — Retracted dead-code-audit findings (false positives) that future audits MUST read before re-citing.
 - [default-branch-ci.md](default-branch-ci.md) — The post-merge default-branch gate: why a `pull_request` gate cannot catch merge-order defects (two green PRs that break only in combination), why the post-merge workflow *calls* the PR gate rather than copying it or inverting the direction (the ruleset requires the `Unit Tests` context by exact string), the cost controls, and the evaluated-but-rejected merge-queue alternative.
 - [deprecated-process.md](deprecated-process.md) — Schema for `docs/deprecated.json` and the weekly `cleanup-deprecated` audit walk that retires entries past their removal signal.

package/docs/dashboard-reliability-and-responsiveness.md

@@ -0,0 +1,281 @@
+# Dashboard Reliability & Responsiveness
+
+Deep dive into what keeps (and what starves) the Minions dashboard/API event loop
+under realistic engine + UI load. Verified against `engine/core/queries.js`,
+`dashboard.js`, and `engine/observability/diagnostics-memory.js` on `work/W-msci5nby0044fd1f`
+(branched from `main`), with reproducible measurements from an isolated benchmark
+(see [Appendix A](#appendix-a--reproduction)).
+
+**Audience:** engine/dashboard contributors filing follow-up performance work. This is
+a decision document — each finding names the module boundary that changes, the fix
+class, and an acceptance metric. It is **not** an implementation.
+
+---
+
+## 1. Executive summary
+
+The dashboard is a single Node process (`dashboard.js`, default port 7331) sharing one
+event loop across: the 4 s SPA poll (`/api/status` + per-page list endpoints), the
+250 ms-cadence `/api/health` restart probe, Command Center / doc-chat SSE streams, and
+all mutating API calls. Any synchronous region longer than a few hundred milliseconds on
+that loop degrades *every* concurrent consumer at once.
+
+**The dominant offender is `GET /api/work-items`.** Its list builder runs
+`queries.getWorkItems()` with **full enrichment** (`enrich: true`), whose
+`_artifacts`/`_notes` step does per-item bucketed scans over the knowledge-base snapshot,
+the notes archive, per-agent output dirs, and the inbox — `O(items × files)` synchronous
+work. On a history of ~2,300 work items with a comparably sized notes archive, a single
+cold/cache-busted enrichment blocks the event loop for **~3.3 s p50 / ~3.9 s p95**
+(measured; [Appendix A](#appendix-a--reproduction)); production incidents at ~2,281 items
+reported **7–17 s** (larger KB, colder OS file cache, slower disk).
+
+**User-visible failure chain:**
+
+```
+engine tick writes state (events.id bumps every tick)
+  → /api/work-items ETag busts → list builder runs full enrichment every ~4s poll
+  → synchronous O(items×files) scan blocks the single event loop for seconds
+  → /api/health 250ms probe misses  → supervisor sees a "dead" dashboard
+  → CC SSE 15s heartbeat fires late → [cc-stall] logged, browser reconnects
+  → browser fetch() times out       → "TypeError: Failed to fetch" false-disconnect UI
+```
+
+**Status of the fix.** The root-cause fix — serving the list from the *lean*
+(`enrich: false`) path — is **in flight, not merged**: PR **#1198** ("Make work-items
+list event-loop safe") rewrites the route builder to `getWorkItems(enrich:false)` +
+pagination + a cheap map-based PR link-up, leaving detail-only fields on
+`GET /api/work-items/<id>`. On `main` today the route still calls the enriching
+`getWorkItems().map(slimWorkItemForList)` (`dashboard.js`, `/api/work-items` GET route),
+so the starvation path is **still present until #1198 lands**. An earlier merged fix
+(PR **#456**) reduced the same enrichment's complexity with per-agent bucketing but did
+not take the route off the enrich path. See [§7](#7-already-fixed-vs-still-outstanding).
+
+---
+
+## 2. Current architecture (responsiveness-relevant slice)
+
+| Surface | Where | Freshness / isolation today |
+|---|---|---|
+| `/api/status` | `handleStatus` → `refreshStatusAsync` (`dashboard.js`) | Single-flight `_statusRebuildPromise`; ETag on `_statusCacheVersion`; one cooperative `await _yieldEventLoop()` between fast/slow halves; invalidation-race guard. **Well isolated.** |
+| `/api/work-items` (list) | `/api/work-items` GET builder → `queries.getWorkItems()` | `serveFreshJson`, ETag = `tag-variant-mtime-eventVersion`. `eventVersion` = `SELECT MAX(id) FROM events` (`_getCurrentEventVersion`), so on a busy engine the ETag busts **every poll** and the builder re-runs. Builder is **synchronous** with no yield. |
+| `/api/work-items/<id>` (detail) | `handleWorkItemsById` | Full single-record read; backs the detail/edit modal. |
+| `/api/health` | `handleHealth` | Minimal inline agent array from config + dispatch only (two cached reads, no dir enumeration) — deliberately kept off `getAgents()`/`/api/status`. **Well isolated.** |
+| CC / doc-chat SSE | `ccCallStreaming`, doc-chat handlers | 15 s heartbeat (`CC_STREAM_HEARTBEAT_MS`); `[cc-stall]` logged when it fires > 3 s late (`CC_STREAM_STALL_THRESHOLD_MS`); per-tab queue shed at 4 MB / 30 s backpressure (`SSE_MAX_QUEUE_BYTES`, `SSE_STUCK_KILL_MS`). Heartbeat is a timer → **starved by any long sync region**. |
+| Memory / event-loop-lag observability | `engine/observability/diagnostics-memory.js` | Engine samples every `memoryBaselineEveryTicks` (≈60 s). Dashboard samples via `startPeriodicSampling({ intervalMs: 60000 })`. Event-loop-lag histogram exists but is **sampled only once per 60 s** on the dashboard side. |
+
+### The read-side aggregation layer
+
+`queries.getWorkItems(config, { enrich })` (`engine/core/queries.js`) is the hinge:
+
+- **`enrich: false` (lean):** SQL rows (`work-items-store.readAllWorkItems()`) + dispatch
+  cross-reference + pending-gate annotation + the deterministic sort. **No filesystem
+  scans.** 1 s TTL cache (`_workItemsLeanCache`).
+- **`enrich: true` (default):** everything above **plus** PR cross-reference and the
+  per-item `_artifacts`/`_notes`/`_model` detail-modal fields. This is where the
+  `O(items × files)` cost lives. Its own header comment documents the freeze and the
+  per-agent bucketing that PR #456 added; a `_buildNotesByWiMap` + mtime-gated archive
+  cache (`_buildArchiveNoteRefs`, 5 min TTL) further amortize the archive read — **but any
+  archive add/remove busts that cache**, and consolidation moves inbox → archive
+  routinely, so the "warm" state is not durable under a live engine.
+
+Both share a 1 s TTL so intra-request duplication (`getWorkItems` is called 5–7× per
+`/api/status` rebuild) collapses to one scan; the TTL does not help *across* poll cycles
+once `eventVersion` busts the route ETag.
+
+---
+
+## 3. Measured bottlenecks (evidence table)
+
+Confidence: **H** = directly measured or unambiguous in code; **M** = strong code
+evidence + incident correlation; **L** = plausible, needs instrumentation to confirm.
+
+| # | Finding | Severity | Conf. | Reproduction | Affected surfaces |
+|---|---|---|---|---|---|
+| E1 | `/api/work-items` list runs full `enrich:true` enrichment; a cold/cache-busted call blocks the loop ~3.3 s p50 / ~3.9 s p95 at 2,300 items (production 7–17 s) | **P0** | H | [Appendix A](#appendix-a--reproduction) | health probe, CC SSE, every concurrent fetch |
+| E2 | Route ETag includes `MAX(events.id)`; a busy engine bumps it every tick, so the enrich builder re-runs on essentially every 4 s poll instead of 304-ing | **P0** | H | `serveFreshJson` ETag = `…-mtime-eventVersion`; `_getCurrentEventVersion` | list latency, loop fairness |
+| E3 | The enrich builder is one uninterrupted synchronous region — no `await _yieldEventLoop()` mid-scan (unlike `/api/status`) | **P0** | H | `getWorkItems` enrich block is straight-line sync | loop fairness |
+| E4 | Archive notes-by-WI cache is busted by any archive add/remove; consolidation writes archive notes under a live engine, so the multi-second cold scan recurs | **P1** | H | Bench with per-iteration archive write: p50 rebuild 3.3 s vs 63 ms when cache holds | list latency spikes |
+| E5 | KB snapshot bucketing scales with KB size (production KB ≈ 4,600 indexed + up to ~10k entries); benchmark understates it (cold snapshot returns `[]`) | **P1** | M | `getKnowledgeBaseEntriesSnapshot()` async warm; `_kbByAgent` bucketing in `getWorkItems` | list latency at scale |
+| E6 | Dashboard event-loop-lag is sampled once per 60 s; a 3–17 s spike is usually invisible to `/api/diagnostics/memory` | **P1** | H | `DIAGNOSTICS_MEMORY_SAMPLE_INTERVAL_MS = 60000` | observability blind spot |
+| E7 | No per-route duration / event-loop-delay instrumentation on the dashboard; `[cc-stall]` is the only starvation signal and it is CC-specific | **P1** | H | grep: no `monitorEventLoopDelay` in `dashboard.js` request path | triage blind spot |
+| E8 | `_getCurrentEventVersion()` runs a synchronous `prepare().get()` against SQLite on every `serveFreshJson` request (cheap, but on the hot loop) | **P2** | M | `_getCurrentEventVersion` | micro-overhead |
+| E9 | Supervisor treats a starved-but-alive dashboard as dead → false restart, which drops in-flight SSE and compounds the perceived outage | **P1** | M | `/api/health` polled every 250 ms for 15 s post-restart (`handleHealth` header) | restart churn |
+
+---
+
+## 4. Threading / process-boundary decision matrix
+
+The primary question is not "which threads help?" but "which synchronous work should not
+exist on the hot loop at all?". Prefer, in order: **remove → cache/index → chunk/yield →
+worker thread → child process.**
+
+| Work | Keep on loop | Chunk/yield | Cache/index/query | Worker thread | Child process | Chosen | Why |
+|---|:--:|:--:|:--:|:--:|:--:|---|---|
+| Work-items **list** payload | | | ✅ | | | **Serve lean (E1/E2)** | Remove the fs scan from the list path entirely (PR #1198). No concurrency needed — the data is already in SQL. |
+| Per-item `_artifacts`/`_notes` enrichment | | ✅ | ✅ | | | **Detail-only + index** | Only the detail modal needs it; compute per-id on `/<id>`, or precompute a WI→artifacts index in SQL. |
+| `/api/status` rebuild | | ✅ | ✅ | | | **Already done** | Single-flight + mid-rebuild yield already isolate it. |
+| KB / notes text search for linkage | | | ✅ | ⚠️ | | **Index in SQL/FTS5** | FTS5 already backs memory retrieval; reuse it instead of `String.includes` scans. A worker still pays transfer cost for a scan that an index removes. |
+| Health probe | ✅ | | | | | **Already minimal** | Must never touch heavy reads. |
+| CC / doc-chat LLM turns | | | | | ✅ | **Already isolated** | Spawned children; only their SSE writer touches the loop. |
+| Skills / tools / MCP discovery | | | ✅ | | ✅ | **Already child-process** | `/api/tools` discovers in a child; keep it off `/api/status`. |
+
+**Guiding rule:** if an index or a cache removes the cost, a worker thread is the *wrong*
+tool — it adds serialization + transfer overhead to hide work that should not run. Worker
+threads earn their keep only for genuinely CPU-bound, non-SQL, non-fs compute that cannot
+be indexed away (none identified today).
+
+---
+
+## 5. Prioritized roadmap
+
+### P0 — take the list off the enrich path (removes the starvation)
+
+- **P0-1 — Land PR #1198 (list serves `enrich:false`).**
+  - **Modules:** `dashboard.js` `/api/work-items` GET builder; `queries.attachWorkItemPrLinks` (new, `engine/core/queries.js`); `dashboard/js/render-work-items.js` (detail hydration).
+  - **Success metric:** with an engine actively writing state, `/api/health` p99 stays < 500 ms and no `[cc-stall]` fires while `/api/work-items` is polled at 4 s over a 2,300-item history; list rebuild p95 < 50 ms (from ~3.9 s).
+  - **Rollback:** the change is builder-local and byte-compatible on the no-paging path; revert the single route builder to restore prior behavior.
+- **P0-2 — Regression guard.** Add/keep a source-and-timing test asserting the list route
+  never calls `getWorkItems(enrich:true)` and that a synthetic 2k-item rebuild stays under
+  a budget (the `test/unit/dashboard-work-items-list-event-loop-safe.test.js` added by
+  #1198 is the anchor).
+  - **Success metric:** test fails if any future edit reintroduces enrich-on-list.
+
+### P1 — establish performance budgets & observability *before* adding concurrency
+
+- **P1-1 — Dashboard request-path event-loop-delay instrumentation.**
+  - **Modules:** `dashboard.js` request wrapper; reuse `engine/observability/diagnostics-memory.js` histogram.
+  - **Deliverable:** a bounded per-route duration + a continuously-enabled
+    `monitorEventLoopDelay` histogram surfaced on `/api/diagnostics/memory` (not just a
+    60 s point sample), plus a `[slow-route]` warn log over a configurable threshold.
+  - **Success metric:** a synthetic 1 s block is visible in `/api/diagnostics/memory`
+    within one poll interval; today it is usually invisible (E6/E7).
+- **P1-2 — Precompute the WI→artifacts/notes index in SQL.**
+  - **Modules:** a numbered migration + store helper; `queries.getWorkItems` enrich block
+    reads the index instead of scanning KB snapshot + archive + agent dirs.
+  - **Success metric:** detail enrichment for one item is O(1) index lookups, no directory
+    walk; the archive-cache-bust cliff (E4) disappears.
+- **P1-3 — Supervisor false-positive hardening.** Confirm the restart probe distinguishes
+  "dashboard process alive, status cache slow" (already exposed via
+  `handleHealth`'s `dashboardStartedAt`) from "new process". Ensure a single slow
+  `/api/status` cannot trip a restart when `/api/health` is answering.
+  - **Success metric:** no supervisor restart is triggered while `/api/health` p99
+    < 500 ms, even if `/api/status` is momentarily stale.
+
+### P2 — micro-overheads & scale hygiene
+
+- **P2-1 — Cache `_getCurrentEventVersion()` for a sub-second TTL** so the per-request
+  SQLite `MAX(id)` read (E8) is not repeated across concurrent polls in the same tick.
+- **P2-2 — Pagination default-on for the SPA list poll.** The `_parsePageParams` /
+  `_paginateList` foundation and `engine/api-contracts/paging.js` bounds already exist;
+  have the classic 4 s loop request a bounded first page instead of the full array.
+- **P2-3 — Database-growth soak.** Extend `test/perf/soak.test.js` with a work-items /
+  archive growth axis so latency regressions at 5k–10k items are caught pre-merge.
+
+---
+
+## 6. Proposed follow-up work items (independently shippable)
+
+Each is small enough to file directly as a WI. All are documentation-derived; none bundle
+a speculative production refactor.
+
+1. **Merge/rebase PR #1198** and confirm P0-1 acceptance metric on a live-ish dataset
+   (regression test + manual `/api/health` latency check). *(depends: nothing)*
+2. **Dashboard event-loop-delay + per-route timing** (P1-1). Ships a diagnostic endpoint
+   surface + `[slow-route]` log; observability-only, no behavior change.
+   *(depends: nothing; unblocks measuring everything below)*
+3. **SQL WI→artifacts/notes index** (P1-2). Migration + store helper + `getWorkItems`
+   enrich read-path swap; keep the fs-scan fallback for one release behind the index.
+   *(depends: #2 for before/after numbers)*
+4. **Event-version read cache** (P2-1). One-line-scope TTL around `_getCurrentEventVersion`.
+   *(depends: nothing)*
+5. **SPA list poll → paginated** (P2-2). Frontend + route wiring only.
+   *(depends: #1 so the page path is already lean)*
+6. **Work-items growth soak axis** (P2-3). Test-only.
+   *(depends: #2 for the budget threshold)*
+
+Every item above ships with (a) a regression test or benchmark, and (b) an observability
+requirement satisfied by #2 (a starvation must be *visible* before and *absent* after).
+
+---
+
+## 7. Already-fixed vs still-outstanding
+
+**Merged / present on `main` (do not re-file):**
+
+- `/api/status` event-loop isolation: single-flight `_statusRebuildPromise`, ETag on
+  `_statusCacheVersion`, mid-rebuild `await _yieldEventLoop()`, invalidation-race guard.
+- `getWorkItems` 1 s TTL caches (enriched + lean, held separately) and per-agent bucketing
+  of the KB/archive scans (PR #456) — reduced the constant factor.
+- `slimWorkItemForList`: hard-caps `description` to 2,048 chars and replaces
+  `acceptanceCriteria`/`references` arrays with `*Count` integers, so a single oversized
+  transcript can no longer balloon the list payload (bounds bytes, not compute).
+- `/api/health` minimal inline agents (no directory enumeration) — keeps the restart probe
+  cheap.
+- CC SSE backpressure shedding (`SSE_MAX_QUEUE_BYTES` / `SSE_STUCK_KILL_MS`) and
+  `[cc-stall]` / `[cc-sse-backpressure]` logging.
+- List pagination *foundation* (`_parsePageParams` / `_paginateList`, bounds from
+  `engine/api-contracts/paging.js`) — present but not yet the default for the SPA poll.
+
+**Still outstanding:**
+
+- **P0:** the list route still enriches on `main` (fixed only once PR #1198 merges) — E1/E2/E3.
+- **P1:** archive-cache-bust cliff (E4), KB-scale enrichment cost (E5), coarse dashboard
+  lag sampling (E6), no per-route timing (E7), supervisor false-positive hardening (E9).
+- **P2:** per-request `MAX(id)` read (E8), pagination-by-default, growth soak.
+
+---
+
+## 8. Non-recommendations & traps
+
+- **Do not move `getWorkItems` enrichment onto a worker thread.** The cost is fs + SQL
+  scanning, not CPU math. `node:sqlite` statement objects are **not transferable** across
+  worker boundaries, and structured-cloning a 2k-item enriched list back to the main
+  thread would re-introduce a serialization stall. Remove the scan (index it) instead.
+- **Do not wrap cheap operations in workers.** Worker spawn + message serialization dwarfs
+  a sub-millisecond SQL read; the lean list path is already ~11–20 ms.
+- **Do not "fix" starvation by raising health/probe timeouts.** That hides the loop block
+  from the supervisor while browsers still see `Failed to fetch`. Extending
+  `restartHealthTimeoutMs` treats the symptom; removing the sync region treats the cause.
+- **Do not add a second always-on cache without an invalidation source.** The archive
+  cache already shows how a cache that a live engine busts is worse than no cache (a
+  surprise multi-second cliff). Prefer an index keyed by the same events the ETag reads.
+- **Do not push `LIMIT/OFFSET` into the per-scope stores for the list** without accounting
+  for the post-enrichment cross-scope sort — the current design slices the deterministic
+  cached array at the endpoint precisely so `total` matches `/api/status` counts and pages
+  never overlap (see the pagination design note in `dashboard.js`).
+- **Do not conflate `/api/status` with the list endpoints.** `/api/status` is already
+  isolated; the regression risk is re-coupling a heavy list build back into the status
+  rebuild.
+
+---
+
+## Appendix A — Reproduction
+
+Method: an isolated `MINIONS_TEST_DIR` (via the test harness `createTestMinionsDir()`)
+seeded with **2,300 work items**, **2,300 archive notes**, and 5 agents × 40 output logs,
+exercising the **real** `engine/core/queries.getWorkItems`. Event-loop blocking equals the
+synchronous wall-time of the call (the builder has no `await`), so a cold call of duration
+_D_ blocks the loop for _D_. The archive notes-by-WI cache was busted each iteration (one
+archive write) to model consolidation writing under a live engine.
+
+Machine/runtime: Node v24.14.0, win32, warm SSD. Numbers are order-of-magnitude, not a
+guaranteed SLA — production disks and a populated KB snapshot make the enrich path slower,
+consistent with the 7–17 s incident reports.
+
+```
+dataset: 2300 work items, 2300 archive notes, 5 agents
+enrich:true  COLD     = ~3.2–3.8 s      (2300 items)   ← single event-loop block
+enrich:false COLD     = ~15–20 ms       (2300 items)
+enrich:true  rebuild  p50=3264ms p95=3942ms  (archive cache busted each poll)
+enrich:false rebuild  p50=11ms   p95=13ms
+```
+
+Ratio: the lean path is **~250–300× cheaper** and, crucially, does **zero** filesystem
+scanning, so it cannot be pushed back onto the multi-second cliff by a KB/archive write.
+
+The benchmark script is intentionally **not committed** — it is a throwaway diagnostic. To
+reproduce, adapt the harness in `test/unit/queries-work-items-cache.test.js`
+(`createTestMinionsDir` + `seedWorkItems`), seed a large archive under
+`notes/archive/`, and time `queries.getWorkItems(null, { enrich })` both ways. The
+regression anchor that *is* committed (once PR #1198 lands) is
+`test/unit/dashboard-work-items-list-event-loop-safe.test.js`.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@yemi33/minions",
-  "version": "0.1.2451",
+  "version": "0.1.2452",
   "description": "Multi-agent AI dev team that runs from ~/.minions/ — five autonomous agents share a single engine, dashboard, and knowledge base",
   "bin": {
     "minions": "bin/minions.js"