@yemi33/minions 0.1.2178 → 0.1.2180

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (39) hide show
  1. package/README.md +7 -5
  2. package/bin/minions.js +39 -17
  3. package/dashboard/js/command-parser.js +1 -1
  4. package/dashboard/js/memory-panel.js +324 -0
  5. package/dashboard/js/qa.js +2 -2
  6. package/dashboard/js/refresh.js +19 -1
  7. package/dashboard/js/render-other.js +143 -2
  8. package/dashboard/js/render-prs.js +2 -1
  9. package/dashboard/js/render-schedules.js +1 -1
  10. package/dashboard/js/render-watches.js +1 -1
  11. package/dashboard/js/render-work-items.js +18 -1
  12. package/dashboard/js/settings.js +23 -0
  13. package/dashboard/pages/engine-memory-panel.html +56 -0
  14. package/dashboard/pages/engine.html +1 -0
  15. package/dashboard/pages/tools.html +8 -0
  16. package/dashboard/slim/js/link-pr.js +5 -5
  17. package/dashboard/slim/js/modals-tiles.js +44 -3
  18. package/dashboard/slim/js/projects.js +8 -6
  19. package/dashboard/slim/styles.css +20 -0
  20. package/dashboard-build.js +17 -2
  21. package/dashboard.js +693 -19
  22. package/docs/branch-derivation.md +13 -1
  23. package/docs/diagnostics-memory.md +446 -0
  24. package/docs/harness-propagation.md +273 -0
  25. package/docs/human-vs-automated.md +1 -1
  26. package/docs/runtime-adapters.md +5 -0
  27. package/engine/cli.js +24 -5
  28. package/engine/diagnostics-memory.js +190 -0
  29. package/engine/lifecycle.js +111 -1
  30. package/engine/preflight.js +265 -0
  31. package/engine/queries.js +331 -19
  32. package/engine/runtimes/claude.js +36 -0
  33. package/engine/runtimes/codex.js +19 -0
  34. package/engine/runtimes/copilot.js +27 -36
  35. package/engine/shared.js +390 -15
  36. package/engine/spawn-agent.js +178 -12
  37. package/engine/watchdog.js +6 -0
  38. package/engine.js +277 -4
  39. package/package.json +2 -2
@@ -44,6 +44,18 @@ When a ref is detected, `copyWorkItemPrFields` stamps
44
44
  `targetPr` / `pr_id` / `prNumber`, `item.branch` is unset, and
45
45
  `discoverFromWorkItems` reuses the PR's source branch.
46
46
 
47
+ **Type gate (W-mqbaby2a000pa8ee).** The loose description/title scan
48
+ (step 4 above) only stamps on `type: "fix"` WIs. Non-fix WIs
49
+ (implement / explore / test / review / …) only get stamped from
50
+ structured fields (steps 1-3). Without this gate, an implement WI
51
+ whose description merely mentions an existing PR in prose
52
+ ("Class bug surfaced today on pull request 130") would silently
53
+ get `targetPr` / `pr_id` / `prNumber` stamped, route through the
54
+ PR-fix dispatch path, fail the PR-branch lookup, and stick in
55
+ `_pendingReason: null` forever. Structured intent
56
+ (`targetPr` / `prUrl` / `references[].url` / etc.) still stamps on
57
+ every type — only the loose regex scan is gated.
58
+
47
59
  ## Structured-vs-loose split (W-mq18ec6h000p7b87)
48
60
 
49
61
  The PR-ref extractor has **two** variants — pick the right one for the
@@ -52,7 +64,7 @@ call site:
52
64
  | Helper | What it walks | Used by | Why |
53
65
  |--------|---------------|---------|-----|
54
66
  | `shared.extractStructuredWorkItemPrRef(item)` | Structured fields + `references[*].url` + `meta.pr_followup.parent_pr_url`. **No** description / title scan. | `engine.js#getStructuredWorkItemPrRef` → `pr_not_found` dispatch gate. | Gating blocks dispatch and MUST require explicit operator intent. A description like "see PR #3015 for context" must NOT trip the gate. |
55
- | `shared.extractWorkItemPrRef(item)` | Structured walk + last-resort description / title scan. | `engine.js#getWorkItemPrRef` (branch derivation, prompt PR context, `resolveWorkItemPrRecord`); `dashboard.js#getWorkItemPrRef` (POST `/api/work-items` create-time `targetPr` stamping). | Callers downgrade gracefully when no PR record matches; stamp path preserves the operator UX of pasting a PR URL into description prose and getting `targetPr` auto-stamped. |
67
+ | `shared.extractWorkItemPrRef(item)` | Structured walk + last-resort description / title scan. | `engine.js#getWorkItemPrRef` (branch derivation, prompt PR context, `resolveWorkItemPrRecord`); `dashboard.js#getWorkItemPrRef` (POST `/api/work-items` create-time `targetPr` stamping, **type-gated on `fix` — W-mqbaby2a000pa8ee**). | Callers downgrade gracefully when no PR record matches; stamp path preserves the operator UX of pasting a PR URL into description prose and getting `targetPr` auto-stamped on `type: "fix"` WIs. |
56
68
 
57
69
  **Rule of thumb: gate uses structured-only; stamp uses loose.**
58
70
  Stamping is best-effort and reversible; gating blocks dispatch and should
@@ -0,0 +1,446 @@
1
+ # Diagnostics: memory, GC, event-loop, heap snapshots, CPU/heap profiles
2
+
3
+ Operator runbook for the in-process memory + perf observability surface shipped
4
+ under PRD `minions-opg-2026-06-12.json` (plan items `P-a1b2c3d4` →
5
+ `P-e5f6a7b8`). Covers what the API exposes, how to read the engine log
6
+ emissions, how to capture and analyse heap snapshots, how to drive a
7
+ `--cpu-prof` / `--heap-prof` session, and how to retune the
8
+ `test/perf/soak.test.js` regression gate.
9
+
10
+ If you just want to *see* the live numbers, open the dashboard's `/engine`
11
+ page — the Memory panel renders the same data this doc describes.
12
+
13
+ ## 1. `GET /api/diagnostics/memory` and `GET /api/diagnostics/memory/history`
14
+
15
+ Both routes are registered in the dashboard route registry
16
+ ([`dashboard.js:13079-13080`](../dashboard.js)) and exposed at
17
+ `GET /api/routes`. They are read-only, do no shelling-out, and complete in
18
+ well under 10 ms warm.
19
+
20
+ ### `GET /api/diagnostics/memory`
21
+
22
+ Returns the latest in-process **dashboard** sample plus the latest **engine**
23
+ sample (read fresh from `engine/diagnostics-memory.json` via
24
+ `safeJsonObj`) plus a staleness flag. Engine staleness is `true` when the
25
+ sidecar is missing/unparseable OR when its `capturedAt` is more than
26
+ 5 minutes old (`DIAGNOSTICS_MEMORY_STALE_MS`,
27
+ [`dashboard.js:117`](../dashboard.js)).
28
+
29
+ ```bash
30
+ curl -s http://localhost:7331/api/diagnostics/memory | jq
31
+ ```
32
+
33
+ ```json
34
+ {
35
+ "dashboard": {
36
+ "rss": 184250368,
37
+ "heapUsed": 92847232,
38
+ "heapTotal": 134217728,
39
+ "external": 4214528,
40
+ "arrayBuffers": 532480,
41
+ "heapSizeLimit": 4341563392,
42
+ "eventLoopLagP50": 1.42,
43
+ "eventLoopLagP99": 18.36,
44
+ "eventLoopLagMax": 64.20,
45
+ "lastGcPauseMs": 4.83,
46
+ "lastGcKind": "minor",
47
+ "gcPausesTotalMs": 312.47,
48
+ "gcCount": 87,
49
+ "uptime": 612.4,
50
+ "pid": 18324,
51
+ "label": "dashboard",
52
+ "capturedAt": 1749758480123
53
+ },
54
+ "engine": {
55
+ "rss": 152784896,
56
+ "heapUsed": 78643200,
57
+ "heapTotal": 117440512,
58
+ "external": 2228224,
59
+ "arrayBuffers": 131072,
60
+ "heapSizeLimit": 4341563392,
61
+ "eventLoopLagP50": 0.94,
62
+ "eventLoopLagP99": 12.18,
63
+ "eventLoopLagMax": 41.62,
64
+ "lastGcPauseMs": 2.11,
65
+ "lastGcKind": "minor",
66
+ "gcPausesTotalMs": 198.34,
67
+ "gcCount": 64,
68
+ "uptime": 603.7,
69
+ "pid": 18301,
70
+ "label": "engine",
71
+ "capturedAt": 1749758460112
72
+ },
73
+ "engineStale": false
74
+ }
75
+ ```
76
+
77
+ Field reference (every value originates from
78
+ [`engine/diagnostics-memory.js`](../engine/diagnostics-memory.js)
79
+ `sampleSelf()`):
80
+
81
+ | Field | Source | Units |
82
+ |---|---|---|
83
+ | `rss` | `process.memoryUsage().rss` | bytes |
84
+ | `heapUsed` | `process.memoryUsage().heapUsed` | bytes |
85
+ | `heapTotal` | `process.memoryUsage().heapTotal` | bytes |
86
+ | `external` | `process.memoryUsage().external` | bytes |
87
+ | `arrayBuffers` | `process.memoryUsage().arrayBuffers` | bytes |
88
+ | `heapSizeLimit` | `v8.getHeapStatistics().heap_size_limit` | bytes |
89
+ | `eventLoopLagP50` / `P99` / `Max` | `perf_hooks.monitorEventLoopDelay({ resolution: 20 })` | ms (fractional) |
90
+ | `lastGcPauseMs` / `lastGcKind` | `PerformanceObserver({ entryTypes: ['gc'] })` last entry | ms / `'minor'\|'major'\|'incremental'\|'weakcb'` |
91
+ | `gcPausesTotalMs` / `gcCount` | accumulated from same observer | ms / int |
92
+ | `uptime` | `process.uptime()` | seconds |
93
+ | `pid` | `process.pid` | int |
94
+ | `label` | constant per process | `'engine'` or `'dashboard'` |
95
+ | `capturedAt` | `Date.now()` at sample time | epoch ms |
96
+
97
+ When the engine has not yet written the sidecar (cold dashboard, engine
98
+ down, or `memoryBaselineEveryTicks <= 0`), `engine` is `null` and
99
+ `engineStale` is `true`.
100
+
101
+ ### `GET /api/diagnostics/memory/history`
102
+
103
+ In-memory ring buffer of samples. Two backing rings:
104
+
105
+ - `process=dashboard` — the dashboard's own collector (populated by
106
+ `diagnosticsMemory.startPeriodicSampling` on dashboard boot,
107
+ `DIAGNOSTICS_MEMORY_SAMPLE_INTERVAL_MS = 60 000` ms,
108
+ [`dashboard.js:118`](../dashboard.js)). Capacity is
109
+ `RING_BUFFER_CAP = 1440` samples (≈ 24 h at 60 s cadence,
110
+ [`engine/diagnostics-memory.js:28`](../engine/diagnostics-memory.js)).
111
+ - `process=engine` — the dashboard's **polled accumulation** of the
112
+ engine's sidecar. Engine.js only persists the latest sample to
113
+ `engine/diagnostics-memory.json`, so engine-side history is rebuilt by
114
+ the dashboard poller (dedup by `capturedAt`). Capacity is
115
+ `DIAGNOSTICS_MEMORY_ENGINE_RING_CAP = 1440` samples
116
+ ([`dashboard.js:119`](../dashboard.js)).
117
+
118
+ ```bash
119
+ # All samples
120
+ curl -s "http://localhost:7331/api/diagnostics/memory/history?process=engine" | jq '.count'
121
+
122
+ # Newest 60 samples (≈ last hour at 60s cadence)
123
+ curl -s "http://localhost:7331/api/diagnostics/memory/history?process=dashboard&limit=60" | jq
124
+ ```
125
+
126
+ ```json
127
+ {
128
+ "process": "engine",
129
+ "count": 60,
130
+ "samples": [
131
+ { "rss": 152784896, "heapUsed": 78643200, "...": "...", "capturedAt": 1749754860111 },
132
+ { "rss": 153038848, "heapUsed": 79233024, "...": "...", "capturedAt": 1749754920119 }
133
+ ]
134
+ }
135
+ ```
136
+
137
+ `samples` is ordered oldest-first. `limit` clamps to the **newest** N.
138
+ Both rings are in-process — restarting the dashboard zeroes them. There
139
+ is no on-disk replay for history (by design — the soak gate
140
+ `test/perf/soak.test.js` covers the regression case the disk history
141
+ would otherwise serve).
142
+
143
+ A 400 is returned when `process` is missing or not in
144
+ `{engine, dashboard}`.
145
+
146
+ ## 2. `MEMORY_BASELINE engine ...` log lines
147
+
148
+ Every `ENGINE_DEFAULTS.memoryBaselineEveryTicks` ticks (default `6`,
149
+ defined at
150
+ [`engine/shared.js:2540`](../engine/shared.js)), the engine emits one
151
+ structured log line and writes the latest sample to the sidecar. At the
152
+ default `tickInterval: 10` the cadence is ≈ 60 s.
153
+
154
+ ```
155
+ [2026-06-12T20:01:00.115Z] MEMORY_BASELINE engine rss=152784896 heapUsed=78643200 eventLoopLagP99=12.18ms gcPauses=64/198.34ms tickN=360
156
+ ```
157
+
158
+ Anatomy ([`engine.js:9539-9542`](../engine.js)):
159
+
160
+ | Token | Meaning |
161
+ |---|---|
162
+ | `rss` | Resident set size in bytes — total memory the OS has mapped for the process. |
163
+ | `heapUsed` | V8 heap actually in use in bytes (after the most recent allocation; not GC-stabilised). |
164
+ | `eventLoopLagP99` | 99th-percentile event-loop lag in ms across the histogram's lifetime. **The most useful single number** — when it stays above ~50 ms you have a synchronous hot path. |
165
+ | `gcPauses` | `<count>/<totalMs>` since process start. Rapidly-growing `count` with low `totalMs` is healthy (lots of cheap minor GCs); slow-growing `count` with large `totalMs` per increment is concerning (long major GCs). |
166
+ | `tickN` | Engine tick number when the sample was taken. Useful for correlating against the same tick's other log lines. |
167
+
168
+ ### Cadence + opt-out
169
+
170
+ `ENGINE_DEFAULTS.memoryBaselineEveryTicks` is the only knob:
171
+
172
+ - Set to `6` (default) → emit every 6 ticks ≈ 60 s at 10 s tick interval.
173
+ - Set to `1` → emit every tick (≈ 10 s). Useful during a leak hunt; turns
174
+ the log into ~6× more noise.
175
+ - Set to `0` or any non-positive integer → **both** the log emission and
176
+ the sidecar write are disabled cleanly. The dashboard's
177
+ `/api/diagnostics/memory` will surface `engineStale: true` and
178
+ `engine: null` after the sidecar ages past
179
+ `DIAGNOSTICS_MEMORY_STALE_MS` (5 minutes).
180
+
181
+ Override via `config.json`:
182
+
183
+ ```json
184
+ {
185
+ "engine": {
186
+ "memoryBaselineEveryTicks": 1
187
+ }
188
+ }
189
+ ```
190
+
191
+ Restart the engine after editing — `ENGINE_DEFAULTS` is resolved once at
192
+ process start.
193
+
194
+ The sidecar file (`engine/diagnostics-memory.json`) is a passive
195
+ single-object cache, gitignored, and intentionally **exempt from the
196
+ SQL-first state rule** — same exemption as `engine/dashboard-port.json`.
197
+ Do not promote it to SQLite.
198
+
199
+ ## 3. `POST /api/diagnostics/heap-snapshot`
200
+
201
+ Captures `v8.writeHeapSnapshot()` for **both** the dashboard and engine
202
+ processes. This is the heavyweight diagnostic — read this section in
203
+ full before invoking.
204
+
205
+ ### ⚠ Stall warning
206
+
207
+ `v8.writeHeapSnapshot` is synchronous and stalls the calling process for
208
+ several seconds (≈ 2–6 s typical, longer on a hot engine with 100 MB+
209
+ heap). It also writes a 50–200 MB file per process. During the capture:
210
+
211
+ - The dashboard does not serve any other HTTP requests.
212
+ - The engine does not service ticks (no new dispatches, no PR polling,
213
+ no timeout checks). Existing child agents continue running, but their
214
+ output is buffered until the engine resumes.
215
+
216
+ Do not invoke during a live incident unless you've already accepted the
217
+ ≈ 10–15 s engine pause as worth the diagnostic.
218
+
219
+ ### Invocation
220
+
221
+ The endpoint is guarded by a literal confirm token
222
+ (`HEAP_SNAPSHOT_CONFIRM_TOKEN`,
223
+ [`dashboard.js:185`](../dashboard.js)) so accidental POSTs return 400
224
+ with the warning text:
225
+
226
+ ```bash
227
+ curl -X POST -s \
228
+ "http://localhost:7331/api/diagnostics/heap-snapshot?confirm=YES_I_UNDERSTAND_THIS_STALLS_THE_ENGINE" \
229
+ | jq
230
+ ```
231
+
232
+ ```json
233
+ {
234
+ "dashboardSnapshot": "D:\\squad-opg\\engine\\diagnostics\\heap-dashboard-2026-06-12T20-15-30-123Z.heapsnapshot",
235
+ "engineSnapshot": "D:\\squad-opg\\engine\\diagnostics\\heap-engine-2026-06-12T20-15-30-123Z.heapsnapshot",
236
+ "engineTimedOut": false,
237
+ "timeoutMs": 30000
238
+ }
239
+ ```
240
+
241
+ - Without the `?confirm=` token: `400` with hint.
242
+ - Within `HEAP_SNAPSHOT_RATE_LIMIT_MS` (60 000 ms) of the previous
243
+ successful capture: `429` with `Retry-After` and `retryAfterSeconds` in
244
+ the body. The window stamps on completion (not request start), so a
245
+ back-to-back POST won't double-stall.
246
+ - If the engine doesn't write its snapshot within 30 s
247
+ (`HEAP_SNAPSHOT_ENGINE_TIMEOUT_MS`): `engineSnapshot: null` and
248
+ `engineTimedOut: true`. The dashboard snapshot still landed. Common
249
+ causes: engine is paused/stopping, or a tick is in a long phase
250
+ (`pollPrStatus` against a slow remote).
251
+
252
+ ### Where files land
253
+
254
+ `engine/diagnostics/heap-<process>-<iso>.heapsnapshot`, where the
255
+ timestamp comes from the dashboard handler (engine reuses the same ISO
256
+ from the sentinel at
257
+ `engine/diagnostics/heap-snapshot-request.json` so the pair shares a
258
+ predictable filename suffix). ISO colons + the millis dot are replaced
259
+ with `-` for NTFS compatibility.
260
+
261
+ ### Loading in Chrome DevTools
262
+
263
+ 1. Open Chrome (or Edge) → DevTools → **Memory** tab.
264
+ 2. Pick the **Heap snapshot** profiling type.
265
+ 3. Right-click in the snapshot list → **Load…** → select the
266
+ `.heapsnapshot` file.
267
+ 4. Switch the dropdown from *Summary* to *Comparison* and load a second
268
+ snapshot to diff retained sizes across captures.
269
+
270
+ For a leak hunt, capture **three** snapshots ≥ 2 h apart while the
271
+ process is under representative load, then diff snapshot 1 → 2 → 3.
272
+ Look for object classes whose `# Delta` and `Size Delta` are both
273
+ positive and monotonically growing. The Phase 2 explore item
274
+ (`P-f6a7b8c9`) documents this workflow end-to-end and lists the
275
+ canonical retention paths to inspect inside the Minions engine
276
+ (`activeProcesses` Map, CC session map, `_buildStatusCache`, etc.).
277
+
278
+ ### Retention policy
279
+
280
+ `HEAP_SNAPSHOT_RETAIN_COUNT = 5` per process
281
+ ([`dashboard.js:189`](../dashboard.js)). On every successful capture,
282
+ `_heapSnapshotPrune` keeps the 5 newest `heap-dashboard-*.heapsnapshot`
283
+ and the 5 newest `heap-engine-*.heapsnapshot` by mtime, deleting the
284
+ rest. Best-effort: a prune failure is swallowed and retried on the next
285
+ capture, so transient Windows file locks don't break the endpoint.
286
+
287
+ If you need to preserve a snapshot, **copy it out** of
288
+ `engine/diagnostics/` before invoking the endpoint a sixth time.
289
+
290
+ ## 4. CPU + heap allocation profile via `NODE_OPTIONS`
291
+
292
+ The heap-snapshot endpoint gives you retained sizes at a single moment.
293
+ `--cpu-prof` and `--heap-prof` give you the orthogonal signal: where CPU
294
+ time and allocations are *spent* during a window of execution.
295
+
296
+ Both are Node built-ins; no extra deps. Restart the engine with both
297
+ flags pointed at `engine/diagnostics`:
298
+
299
+ ```powershell
300
+ # PowerShell
301
+ $env:NODE_OPTIONS = '--cpu-prof --cpu-prof-dir=engine/diagnostics --heap-prof --heap-prof-dir=engine/diagnostics'
302
+ minions restart
303
+ ```
304
+
305
+ ```bash
306
+ # bash
307
+ NODE_OPTIONS='--cpu-prof --cpu-prof-dir=engine/diagnostics --heap-prof --heap-prof-dir=engine/diagnostics' minions restart
308
+ ```
309
+
310
+ Drive 5–10 minutes of representative load:
311
+
312
+ - Manually dispatch ≥ 5 short fix WIs (mocked agents are fine).
313
+ - Poll `/api/status` every 4 s in a tight loop:
314
+ `while ($true) { curl -s http://localhost:7331/api/status > $null; Start-Sleep 4 }`.
315
+ - Trigger ≥ 3 PR poller refreshes (or wait through ≥ 3 polling cycles).
316
+
317
+ Then **stop the engine cleanly** (`minions stop`) — the `.cpuprofile`
318
+ and `.heapprofile` files are only flushed on graceful exit.
319
+
320
+ The two file types land in `engine/diagnostics/`:
321
+
322
+ - `CPU.<date>.<pid>.<tid>.<seq>.cpuprofile` — V8 CPU sampler output.
323
+ - `Heap.<date>.<pid>.<tid>.<seq>.heapprofile` — V8 sampling heap
324
+ profiler output (allocations, *not* live retention; complements the
325
+ heap-snapshot endpoint).
326
+
327
+ ### Loading in Chrome DevTools
328
+
329
+ 1. Open DevTools → **Performance** tab → ⋯ menu → **Load profile…** for
330
+ `.cpuprofile`. Look at the flame chart for any synchronous frame
331
+ > 50 ms (these violate the `/api/status` isolation invariant — the
332
+ tick should never block the dashboard event loop that long).
333
+ 2. Open DevTools → **Memory** tab → pick **Allocation sampling** →
334
+ right-click → **Load…** for `.heapprofile`. Sort by *Self size* to
335
+ identify the top allocation sites by byte count.
336
+
337
+ The Phase 3 explore item (`P-a7b8c9d0`) calls out the specific hot
338
+ paths to inspect (route-registry rebuild in `dashboard.js`,
339
+ `dashboard-build.js` HTML assembly, per-tick allocation hotspots in
340
+ `engine.js#tick`, regex / JSON.parse frames).
341
+
342
+ Remember to **clear** `NODE_OPTIONS` before the next normal start —
343
+ `--cpu-prof` writes a new file on every process exit and the directory
344
+ will fill up.
345
+
346
+ ## 5. `test/perf/soak.test.js` threshold tuning
347
+
348
+ The soak test drives 30 simulated engine ticks against an isolated
349
+ `MINIONS_TEST_DIR` and asserts that `heapUsed` neither grows unboundedly
350
+ across the soak nor exceeds an absolute ceiling. Source:
351
+ [`test/perf/soak.test.js`](../test/perf/soak.test.js).
352
+
353
+ ```bash
354
+ node test/perf/soak.test.js # RSS fallback, looser bound
355
+ node --expose-gc test/perf/soak.test.js # heap measurement, tight bound
356
+ ```
357
+
358
+ ### The two thresholds
359
+
360
+ | Constant | Default | What it gates |
361
+ |---|---|---|
362
+ | `HEAP_DELTA_MAX` | 8 MB | `heapUsed(30t) - heapUsed(5t)` — the delta between the post-warmup baseline (5 ticks) and the end of the soak (30 ticks). Catches compounding leaks. |
363
+ | `HEAP_ABSOLUTE_MAX` | 200 MB | `heapUsed(30t)` after the final forced GC. Catches "engine's resting footprint regressed", not just "per-tick leak". |
364
+ | `HEAP_DELTA_MAX_NO_GC` | 32 MB (= 4× `HEAP_DELTA_MAX`) | Fallback delta bound when `--expose-gc` is absent. Measured against RSS (includes uncollected garbage), so 4× slack. |
365
+
366
+ The absolute-ceiling test is informational (warning only) when GC is not
367
+ forced — without `--expose-gc`, `heapUsed` can include arbitrary
368
+ uncollected garbage that doesn't reflect a real leak.
369
+
370
+ ### When to retune
371
+
372
+ **Lower the bounds** (tighten the gate) when you've shipped a real
373
+ reduction in per-tick allocations and want to lock it in:
374
+
375
+ 1. Run the soak ≥ 5 times locally with `--expose-gc`.
376
+ 2. Take the **max** observed delta across the runs.
377
+ 3. Set the new `HEAP_DELTA_MAX` to `ceil(maxObservedDelta * 1.5)` MB,
378
+ rounded to the nearest MB.
379
+ 4. Same recipe for `HEAP_ABSOLUTE_MAX` against the max post-soak
380
+ `heapUsed`.
381
+ 5. Land the constant change with the PR that earned the headroom — link
382
+ the 5 sample runs in the PR description.
383
+
384
+ **Raise the bounds** (loosen the gate) only with evidence that the
385
+ existing limit is too tight for the *legitimate* per-tick cost (not to
386
+ mask a regression):
387
+
388
+ 1. The CI gate must be flaking *across multiple unrelated PRs* — a
389
+ single PR failing the gate is a signal to fix the PR, not the test.
390
+ 2. Capture ≥ 10 runs on the CI host (or a comparable Windows
391
+ self-hosted runner) with `--expose-gc` enabled. Record max delta and
392
+ max post-soak heapUsed.
393
+ 3. If the variance is the issue (max - min > 50% of the current bound),
394
+ **add samples or extend warmup** before bumping. The header comment
395
+ on the test file is explicit about this: "If they flake on a noisy
396
+ CI host, prefer adding samples / longer warmup over raising the
397
+ bound — the gate's value is detecting compounding growth."
398
+ 4. If max + 2σ truly exceeds the current bound, bump to
399
+ `ceil((max + 2σ) * 1.25)` MB. Link the 10 runs in the PR
400
+ description. Add a comment line above the constant referencing the
401
+ PR that justified the bump.
402
+
403
+ ### Acceptable variance
404
+
405
+ On a quiet developer workstation with `--expose-gc`:
406
+
407
+ - `HEAP_DELTA_MAX` should clear with 1–3 MB observed delta. > 5 MB
408
+ means there is something to investigate, even if it still passes.
409
+ - `HEAP_ABSOLUTE_MAX` should clear with 60–110 MB observed post-soak
410
+ heapUsed. Approaching 150 MB means the resting footprint has grown
411
+ recently.
412
+
413
+ On a noisy CI host without `--expose-gc`, RSS variance can be 4–8 MB
414
+ between runs — this is exactly why the no-GC fallback uses 4× slack.
415
+
416
+ ## 6. Phase 2 and Phase 3 findings notes
417
+
418
+ The diagnostics surface this doc describes exists to make two follow-up
419
+ explores (and any future ones) *cheap to run*. Findings live in
420
+ `knowledge/architecture/` once the KB sweep promotes them; until then,
421
+ they sit in `notes/inbox/` with predictable slugs:
422
+
423
+ | Phase | Plan item | Inbox slug (pre-sweep) | Promoted path (post-sweep) |
424
+ |---|---|---|---|
425
+ | 2 — heap dominators + leak candidates | `P-f6a7b8c9` | `notes/inbox/ripley-memory-audit-phase-2-heap-dominators*.md` | `knowledge/architecture/2026-06-12-ripley-memory-audit-phase-2-heap-dominators.md` |
426
+ | 3 — CPU + allocation hot frames | `P-a7b8c9d0` | `notes/inbox/ripley-memory-audit-phase-3-cpu-and-allocations*.md` | `knowledge/architecture/2026-06-12-ripley-memory-audit-phase-3-cpu-and-allocations.md` |
427
+
428
+ Use [`engine/kb-sweep.js`](../engine/kb-sweep.js) (see
429
+ [`docs/kb-sweep.md`](kb-sweep.md)) or the dashboard's "Sweep" button to
430
+ trigger promotion. The cheap-win PR items downstream of the explores
431
+ (`P-b8c9d0e1`, `P-c9d0e1f2`, `P-d0e1f2a3`) consume the prioritized lists
432
+ from these notes — read them before opening one of those PRs to avoid
433
+ re-deriving the same findings.
434
+
435
+ ## Related modules and tests
436
+
437
+ - Module: [`engine/diagnostics-memory.js`](../engine/diagnostics-memory.js) — sampler + ring buffer + GC observer.
438
+ - Engine wiring: [`engine.js`](../engine.js) `emitMemoryBaseline` + `processHeapSnapshotRequest`.
439
+ - Dashboard wiring: [`dashboard.js`](../dashboard.js) `handleDiagnosticsMemory`, `handleDiagnosticsMemoryHistory`, `handleDiagnosticsHeapSnapshot`.
440
+ - Frontend: [`dashboard/pages/engine-memory-panel.html`](../dashboard/pages/engine-memory-panel.html) + [`dashboard/js/memory-panel.js`](../dashboard/js/memory-panel.js).
441
+ - Tests:
442
+ - [`test/unit/diagnostics-memory-api.test.js`](../test/unit/diagnostics-memory-api.test.js)
443
+ - [`test/unit/heap-snapshot-api.test.js`](../test/unit/heap-snapshot-api.test.js)
444
+ - [`test/unit/memory-panel.test.js`](../test/unit/memory-panel.test.js)
445
+ - [`test/perf/memory-baseline-tick-overhead.test.js`](../test/perf/memory-baseline-tick-overhead.test.js)
446
+ - [`test/perf/soak.test.js`](../test/perf/soak.test.js)