@yemi33/minions 0.1.2178 → 0.1.2180
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +7 -5
- package/bin/minions.js +39 -17
- package/dashboard/js/command-parser.js +1 -1
- package/dashboard/js/memory-panel.js +324 -0
- package/dashboard/js/qa.js +2 -2
- package/dashboard/js/refresh.js +19 -1
- package/dashboard/js/render-other.js +143 -2
- package/dashboard/js/render-prs.js +2 -1
- package/dashboard/js/render-schedules.js +1 -1
- package/dashboard/js/render-watches.js +1 -1
- package/dashboard/js/render-work-items.js +18 -1
- package/dashboard/js/settings.js +23 -0
- package/dashboard/pages/engine-memory-panel.html +56 -0
- package/dashboard/pages/engine.html +1 -0
- package/dashboard/pages/tools.html +8 -0
- package/dashboard/slim/js/link-pr.js +5 -5
- package/dashboard/slim/js/modals-tiles.js +44 -3
- package/dashboard/slim/js/projects.js +8 -6
- package/dashboard/slim/styles.css +20 -0
- package/dashboard-build.js +17 -2
- package/dashboard.js +693 -19
- package/docs/branch-derivation.md +13 -1
- package/docs/diagnostics-memory.md +446 -0
- package/docs/harness-propagation.md +273 -0
- package/docs/human-vs-automated.md +1 -1
- package/docs/runtime-adapters.md +5 -0
- package/engine/cli.js +24 -5
- package/engine/diagnostics-memory.js +190 -0
- package/engine/lifecycle.js +111 -1
- package/engine/preflight.js +265 -0
- package/engine/queries.js +331 -19
- package/engine/runtimes/claude.js +36 -0
- package/engine/runtimes/codex.js +19 -0
- package/engine/runtimes/copilot.js +27 -36
- package/engine/shared.js +390 -15
- package/engine/spawn-agent.js +178 -12
- package/engine/watchdog.js +6 -0
- package/engine.js +277 -4
- package/package.json +2 -2
|
@@ -44,6 +44,18 @@ When a ref is detected, `copyWorkItemPrFields` stamps
|
|
|
44
44
|
`targetPr` / `pr_id` / `prNumber`, `item.branch` is unset, and
|
|
45
45
|
`discoverFromWorkItems` reuses the PR's source branch.
|
|
46
46
|
|
|
47
|
+
**Type gate (W-mqbaby2a000pa8ee).** The loose description/title scan
|
|
48
|
+
(step 4 above) only stamps on `type: "fix"` WIs. Non-fix WIs
|
|
49
|
+
(implement / explore / test / review / …) only get stamped from
|
|
50
|
+
structured fields (steps 1-3). Without this gate, an implement WI
|
|
51
|
+
whose description merely mentions an existing PR in prose
|
|
52
|
+
("Class bug surfaced today on pull request 130") would silently
|
|
53
|
+
get `targetPr` / `pr_id` / `prNumber` stamped, route through the
|
|
54
|
+
PR-fix dispatch path, fail the PR-branch lookup, and stick in
|
|
55
|
+
`_pendingReason: null` forever. Structured intent
|
|
56
|
+
(`targetPr` / `prUrl` / `references[].url` / etc.) still stamps on
|
|
57
|
+
every type — only the loose regex scan is gated.
|
|
58
|
+
|
|
47
59
|
## Structured-vs-loose split (W-mq18ec6h000p7b87)
|
|
48
60
|
|
|
49
61
|
The PR-ref extractor has **two** variants — pick the right one for the
|
|
@@ -52,7 +64,7 @@ call site:
|
|
|
52
64
|
| Helper | What it walks | Used by | Why |
|
|
53
65
|
|--------|---------------|---------|-----|
|
|
54
66
|
| `shared.extractStructuredWorkItemPrRef(item)` | Structured fields + `references[*].url` + `meta.pr_followup.parent_pr_url`. **No** description / title scan. | `engine.js#getStructuredWorkItemPrRef` → `pr_not_found` dispatch gate. | Gating blocks dispatch and MUST require explicit operator intent. A description like "see PR #3015 for context" must NOT trip the gate. |
|
|
55
|
-
| `shared.extractWorkItemPrRef(item)` | Structured walk + last-resort description / title scan. | `engine.js#getWorkItemPrRef` (branch derivation, prompt PR context, `resolveWorkItemPrRecord`); `dashboard.js#getWorkItemPrRef` (POST `/api/work-items` create-time `targetPr` stamping). | Callers downgrade gracefully when no PR record matches; stamp path preserves the operator UX of pasting a PR URL into description prose and getting `targetPr` auto-stamped. |
|
|
67
|
+
| `shared.extractWorkItemPrRef(item)` | Structured walk + last-resort description / title scan. | `engine.js#getWorkItemPrRef` (branch derivation, prompt PR context, `resolveWorkItemPrRecord`); `dashboard.js#getWorkItemPrRef` (POST `/api/work-items` create-time `targetPr` stamping, **type-gated on `fix` — W-mqbaby2a000pa8ee**). | Callers downgrade gracefully when no PR record matches; stamp path preserves the operator UX of pasting a PR URL into description prose and getting `targetPr` auto-stamped on `type: "fix"` WIs. |
|
|
56
68
|
|
|
57
69
|
**Rule of thumb: gate uses structured-only; stamp uses loose.**
|
|
58
70
|
Stamping is best-effort and reversible; gating blocks dispatch and should
|
|
@@ -0,0 +1,446 @@
|
|
|
1
|
+
# Diagnostics: memory, GC, event-loop, heap snapshots, CPU/heap profiles
|
|
2
|
+
|
|
3
|
+
Operator runbook for the in-process memory + perf observability surface shipped
|
|
4
|
+
under PRD `minions-opg-2026-06-12.json` (plan items `P-a1b2c3d4` →
|
|
5
|
+
`P-e5f6a7b8`). Covers what the API exposes, how to read the engine log
|
|
6
|
+
emissions, how to capture and analyse heap snapshots, how to drive a
|
|
7
|
+
`--cpu-prof` / `--heap-prof` session, and how to retune the
|
|
8
|
+
`test/perf/soak.test.js` regression gate.
|
|
9
|
+
|
|
10
|
+
If you just want to *see* the live numbers, open the dashboard's `/engine`
|
|
11
|
+
page — the Memory panel renders the same data this doc describes.
|
|
12
|
+
|
|
13
|
+
## 1. `GET /api/diagnostics/memory` and `GET /api/diagnostics/memory/history`
|
|
14
|
+
|
|
15
|
+
Both routes are registered in the dashboard route registry
|
|
16
|
+
([`dashboard.js:13079-13080`](../dashboard.js)) and exposed at
|
|
17
|
+
`GET /api/routes`. They are read-only, do no shelling-out, and complete in
|
|
18
|
+
well under 10 ms warm.
|
|
19
|
+
|
|
20
|
+
### `GET /api/diagnostics/memory`
|
|
21
|
+
|
|
22
|
+
Returns the latest in-process **dashboard** sample plus the latest **engine**
|
|
23
|
+
sample (read fresh from `engine/diagnostics-memory.json` via
|
|
24
|
+
`safeJsonObj`) plus a staleness flag. Engine staleness is `true` when the
|
|
25
|
+
sidecar is missing/unparseable OR when its `capturedAt` is more than
|
|
26
|
+
5 minutes old (`DIAGNOSTICS_MEMORY_STALE_MS`,
|
|
27
|
+
[`dashboard.js:117`](../dashboard.js)).
|
|
28
|
+
|
|
29
|
+
```bash
|
|
30
|
+
curl -s http://localhost:7331/api/diagnostics/memory | jq
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
```json
|
|
34
|
+
{
|
|
35
|
+
"dashboard": {
|
|
36
|
+
"rss": 184250368,
|
|
37
|
+
"heapUsed": 92847232,
|
|
38
|
+
"heapTotal": 134217728,
|
|
39
|
+
"external": 4214528,
|
|
40
|
+
"arrayBuffers": 532480,
|
|
41
|
+
"heapSizeLimit": 4341563392,
|
|
42
|
+
"eventLoopLagP50": 1.42,
|
|
43
|
+
"eventLoopLagP99": 18.36,
|
|
44
|
+
"eventLoopLagMax": 64.20,
|
|
45
|
+
"lastGcPauseMs": 4.83,
|
|
46
|
+
"lastGcKind": "minor",
|
|
47
|
+
"gcPausesTotalMs": 312.47,
|
|
48
|
+
"gcCount": 87,
|
|
49
|
+
"uptime": 612.4,
|
|
50
|
+
"pid": 18324,
|
|
51
|
+
"label": "dashboard",
|
|
52
|
+
"capturedAt": 1749758480123
|
|
53
|
+
},
|
|
54
|
+
"engine": {
|
|
55
|
+
"rss": 152784896,
|
|
56
|
+
"heapUsed": 78643200,
|
|
57
|
+
"heapTotal": 117440512,
|
|
58
|
+
"external": 2228224,
|
|
59
|
+
"arrayBuffers": 131072,
|
|
60
|
+
"heapSizeLimit": 4341563392,
|
|
61
|
+
"eventLoopLagP50": 0.94,
|
|
62
|
+
"eventLoopLagP99": 12.18,
|
|
63
|
+
"eventLoopLagMax": 41.62,
|
|
64
|
+
"lastGcPauseMs": 2.11,
|
|
65
|
+
"lastGcKind": "minor",
|
|
66
|
+
"gcPausesTotalMs": 198.34,
|
|
67
|
+
"gcCount": 64,
|
|
68
|
+
"uptime": 603.7,
|
|
69
|
+
"pid": 18301,
|
|
70
|
+
"label": "engine",
|
|
71
|
+
"capturedAt": 1749758460112
|
|
72
|
+
},
|
|
73
|
+
"engineStale": false
|
|
74
|
+
}
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
Field reference (every value originates from
|
|
78
|
+
[`engine/diagnostics-memory.js`](../engine/diagnostics-memory.js)
|
|
79
|
+
`sampleSelf()`):
|
|
80
|
+
|
|
81
|
+
| Field | Source | Units |
|
|
82
|
+
|---|---|---|
|
|
83
|
+
| `rss` | `process.memoryUsage().rss` | bytes |
|
|
84
|
+
| `heapUsed` | `process.memoryUsage().heapUsed` | bytes |
|
|
85
|
+
| `heapTotal` | `process.memoryUsage().heapTotal` | bytes |
|
|
86
|
+
| `external` | `process.memoryUsage().external` | bytes |
|
|
87
|
+
| `arrayBuffers` | `process.memoryUsage().arrayBuffers` | bytes |
|
|
88
|
+
| `heapSizeLimit` | `v8.getHeapStatistics().heap_size_limit` | bytes |
|
|
89
|
+
| `eventLoopLagP50` / `P99` / `Max` | `perf_hooks.monitorEventLoopDelay({ resolution: 20 })` | ms (fractional) |
|
|
90
|
+
| `lastGcPauseMs` / `lastGcKind` | `PerformanceObserver({ entryTypes: ['gc'] })` last entry | ms / `'minor'\|'major'\|'incremental'\|'weakcb'` |
|
|
91
|
+
| `gcPausesTotalMs` / `gcCount` | accumulated from same observer | ms / int |
|
|
92
|
+
| `uptime` | `process.uptime()` | seconds |
|
|
93
|
+
| `pid` | `process.pid` | int |
|
|
94
|
+
| `label` | constant per process | `'engine'` or `'dashboard'` |
|
|
95
|
+
| `capturedAt` | `Date.now()` at sample time | epoch ms |
|
|
96
|
+
|
|
97
|
+
When the engine has not yet written the sidecar (cold dashboard, engine
|
|
98
|
+
down, or `memoryBaselineEveryTicks <= 0`), `engine` is `null` and
|
|
99
|
+
`engineStale` is `true`.
|
|
100
|
+
|
|
101
|
+
### `GET /api/diagnostics/memory/history`
|
|
102
|
+
|
|
103
|
+
In-memory ring buffer of samples. Two backing rings:
|
|
104
|
+
|
|
105
|
+
- `process=dashboard` — the dashboard's own collector (populated by
|
|
106
|
+
`diagnosticsMemory.startPeriodicSampling` on dashboard boot,
|
|
107
|
+
`DIAGNOSTICS_MEMORY_SAMPLE_INTERVAL_MS = 60 000` ms,
|
|
108
|
+
[`dashboard.js:118`](../dashboard.js)). Capacity is
|
|
109
|
+
`RING_BUFFER_CAP = 1440` samples (≈ 24 h at 60 s cadence,
|
|
110
|
+
[`engine/diagnostics-memory.js:28`](../engine/diagnostics-memory.js)).
|
|
111
|
+
- `process=engine` — the dashboard's **polled accumulation** of the
|
|
112
|
+
engine's sidecar. Engine.js only persists the latest sample to
|
|
113
|
+
`engine/diagnostics-memory.json`, so engine-side history is rebuilt by
|
|
114
|
+
the dashboard poller (dedup by `capturedAt`). Capacity is
|
|
115
|
+
`DIAGNOSTICS_MEMORY_ENGINE_RING_CAP = 1440` samples
|
|
116
|
+
([`dashboard.js:119`](../dashboard.js)).
|
|
117
|
+
|
|
118
|
+
```bash
|
|
119
|
+
# All samples
|
|
120
|
+
curl -s "http://localhost:7331/api/diagnostics/memory/history?process=engine" | jq '.count'
|
|
121
|
+
|
|
122
|
+
# Newest 60 samples (≈ last hour at 60s cadence)
|
|
123
|
+
curl -s "http://localhost:7331/api/diagnostics/memory/history?process=dashboard&limit=60" | jq
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
```json
|
|
127
|
+
{
|
|
128
|
+
"process": "engine",
|
|
129
|
+
"count": 60,
|
|
130
|
+
"samples": [
|
|
131
|
+
{ "rss": 152784896, "heapUsed": 78643200, "...": "...", "capturedAt": 1749754860111 },
|
|
132
|
+
{ "rss": 153038848, "heapUsed": 79233024, "...": "...", "capturedAt": 1749754920119 }
|
|
133
|
+
]
|
|
134
|
+
}
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
`samples` is ordered oldest-first. `limit` clamps to the **newest** N.
|
|
138
|
+
Both rings are in-process — restarting the dashboard zeroes them. There
|
|
139
|
+
is no on-disk replay for history (by design — the soak gate
|
|
140
|
+
`test/perf/soak.test.js` covers the regression case the disk history
|
|
141
|
+
would otherwise serve).
|
|
142
|
+
|
|
143
|
+
A 400 is returned when `process` is missing or not in
|
|
144
|
+
`{engine, dashboard}`.
|
|
145
|
+
|
|
146
|
+
## 2. `MEMORY_BASELINE engine ...` log lines
|
|
147
|
+
|
|
148
|
+
Every `ENGINE_DEFAULTS.memoryBaselineEveryTicks` ticks (default `6`,
|
|
149
|
+
defined at
|
|
150
|
+
[`engine/shared.js:2540`](../engine/shared.js)), the engine emits one
|
|
151
|
+
structured log line and writes the latest sample to the sidecar. At the
|
|
152
|
+
default `tickInterval: 10` the cadence is ≈ 60 s.
|
|
153
|
+
|
|
154
|
+
```
|
|
155
|
+
[2026-06-12T20:01:00.115Z] MEMORY_BASELINE engine rss=152784896 heapUsed=78643200 eventLoopLagP99=12.18ms gcPauses=64/198.34ms tickN=360
|
|
156
|
+
```
|
|
157
|
+
|
|
158
|
+
Anatomy ([`engine.js:9539-9542`](../engine.js)):
|
|
159
|
+
|
|
160
|
+
| Token | Meaning |
|
|
161
|
+
|---|---|
|
|
162
|
+
| `rss` | Resident set size in bytes — total memory the OS has mapped for the process. |
|
|
163
|
+
| `heapUsed` | V8 heap actually in use in bytes (after the most recent allocation; not GC-stabilised). |
|
|
164
|
+
| `eventLoopLagP99` | 99th-percentile event-loop lag in ms across the histogram's lifetime. **The most useful single number** — when it stays above ~50 ms you have a synchronous hot path. |
|
|
165
|
+
| `gcPauses` | `<count>/<totalMs>` since process start. Rapidly-growing `count` with low `totalMs` is healthy (lots of cheap minor GCs); slow-growing `count` with large `totalMs` per increment is concerning (long major GCs). |
|
|
166
|
+
| `tickN` | Engine tick number when the sample was taken. Useful for correlating against the same tick's other log lines. |
|
|
167
|
+
|
|
168
|
+
### Cadence + opt-out
|
|
169
|
+
|
|
170
|
+
`ENGINE_DEFAULTS.memoryBaselineEveryTicks` is the only knob:
|
|
171
|
+
|
|
172
|
+
- Set to `6` (default) → emit every 6 ticks ≈ 60 s at 10 s tick interval.
|
|
173
|
+
- Set to `1` → emit every tick (≈ 10 s). Useful during a leak hunt; turns
|
|
174
|
+
the log into ~6× more noise.
|
|
175
|
+
- Set to `0` or any non-positive integer → **both** the log emission and
|
|
176
|
+
the sidecar write are disabled cleanly. The dashboard's
|
|
177
|
+
`/api/diagnostics/memory` will surface `engineStale: true` and
|
|
178
|
+
`engine: null` after the sidecar ages past
|
|
179
|
+
`DIAGNOSTICS_MEMORY_STALE_MS` (5 minutes).
|
|
180
|
+
|
|
181
|
+
Override via `config.json`:
|
|
182
|
+
|
|
183
|
+
```json
|
|
184
|
+
{
|
|
185
|
+
"engine": {
|
|
186
|
+
"memoryBaselineEveryTicks": 1
|
|
187
|
+
}
|
|
188
|
+
}
|
|
189
|
+
```
|
|
190
|
+
|
|
191
|
+
Restart the engine after editing — `ENGINE_DEFAULTS` is resolved once at
|
|
192
|
+
process start.
|
|
193
|
+
|
|
194
|
+
The sidecar file (`engine/diagnostics-memory.json`) is a passive
|
|
195
|
+
single-object cache, gitignored, and intentionally **exempt from the
|
|
196
|
+
SQL-first state rule** — same exemption as `engine/dashboard-port.json`.
|
|
197
|
+
Do not promote it to SQLite.
|
|
198
|
+
|
|
199
|
+
## 3. `POST /api/diagnostics/heap-snapshot`
|
|
200
|
+
|
|
201
|
+
Captures `v8.writeHeapSnapshot()` for **both** the dashboard and engine
|
|
202
|
+
processes. This is the heavyweight diagnostic — read this section in
|
|
203
|
+
full before invoking.
|
|
204
|
+
|
|
205
|
+
### ⚠ Stall warning
|
|
206
|
+
|
|
207
|
+
`v8.writeHeapSnapshot` is synchronous and stalls the calling process for
|
|
208
|
+
several seconds (≈ 2–6 s typical, longer on a hot engine with 100 MB+
|
|
209
|
+
heap). It also writes a 50–200 MB file per process. During the capture:
|
|
210
|
+
|
|
211
|
+
- The dashboard does not serve any other HTTP requests.
|
|
212
|
+
- The engine does not service ticks (no new dispatches, no PR polling,
|
|
213
|
+
no timeout checks). Existing child agents continue running, but their
|
|
214
|
+
output is buffered until the engine resumes.
|
|
215
|
+
|
|
216
|
+
Do not invoke during a live incident unless you've already accepted the
|
|
217
|
+
≈ 10–15 s engine pause as worth the diagnostic.
|
|
218
|
+
|
|
219
|
+
### Invocation
|
|
220
|
+
|
|
221
|
+
The endpoint is guarded by a literal confirm token
|
|
222
|
+
(`HEAP_SNAPSHOT_CONFIRM_TOKEN`,
|
|
223
|
+
[`dashboard.js:185`](../dashboard.js)) so accidental POSTs return 400
|
|
224
|
+
with the warning text:
|
|
225
|
+
|
|
226
|
+
```bash
|
|
227
|
+
curl -X POST -s \
|
|
228
|
+
"http://localhost:7331/api/diagnostics/heap-snapshot?confirm=YES_I_UNDERSTAND_THIS_STALLS_THE_ENGINE" \
|
|
229
|
+
| jq
|
|
230
|
+
```
|
|
231
|
+
|
|
232
|
+
```json
|
|
233
|
+
{
|
|
234
|
+
"dashboardSnapshot": "D:\\squad-opg\\engine\\diagnostics\\heap-dashboard-2026-06-12T20-15-30-123Z.heapsnapshot",
|
|
235
|
+
"engineSnapshot": "D:\\squad-opg\\engine\\diagnostics\\heap-engine-2026-06-12T20-15-30-123Z.heapsnapshot",
|
|
236
|
+
"engineTimedOut": false,
|
|
237
|
+
"timeoutMs": 30000
|
|
238
|
+
}
|
|
239
|
+
```
|
|
240
|
+
|
|
241
|
+
- Without the `?confirm=` token: `400` with hint.
|
|
242
|
+
- Within `HEAP_SNAPSHOT_RATE_LIMIT_MS` (60 000 ms) of the previous
|
|
243
|
+
successful capture: `429` with `Retry-After` and `retryAfterSeconds` in
|
|
244
|
+
the body. The window stamps on completion (not request start), so a
|
|
245
|
+
back-to-back POST won't double-stall.
|
|
246
|
+
- If the engine doesn't write its snapshot within 30 s
|
|
247
|
+
(`HEAP_SNAPSHOT_ENGINE_TIMEOUT_MS`): `engineSnapshot: null` and
|
|
248
|
+
`engineTimedOut: true`. The dashboard snapshot still landed. Common
|
|
249
|
+
causes: engine is paused/stopping, or a tick is in a long phase
|
|
250
|
+
(`pollPrStatus` against a slow remote).
|
|
251
|
+
|
|
252
|
+
### Where files land
|
|
253
|
+
|
|
254
|
+
`engine/diagnostics/heap-<process>-<iso>.heapsnapshot`, where the
|
|
255
|
+
timestamp comes from the dashboard handler (engine reuses the same ISO
|
|
256
|
+
from the sentinel at
|
|
257
|
+
`engine/diagnostics/heap-snapshot-request.json` so the pair shares a
|
|
258
|
+
predictable filename suffix). ISO colons + the millis dot are replaced
|
|
259
|
+
with `-` for NTFS compatibility.
|
|
260
|
+
|
|
261
|
+
### Loading in Chrome DevTools
|
|
262
|
+
|
|
263
|
+
1. Open Chrome (or Edge) → DevTools → **Memory** tab.
|
|
264
|
+
2. Pick the **Heap snapshot** profiling type.
|
|
265
|
+
3. Right-click in the snapshot list → **Load…** → select the
|
|
266
|
+
`.heapsnapshot` file.
|
|
267
|
+
4. Switch the dropdown from *Summary* to *Comparison* and load a second
|
|
268
|
+
snapshot to diff retained sizes across captures.
|
|
269
|
+
|
|
270
|
+
For a leak hunt, capture **three** snapshots ≥ 2 h apart while the
|
|
271
|
+
process is under representative load, then diff snapshot 1 → 2 → 3.
|
|
272
|
+
Look for object classes whose `# Delta` and `Size Delta` are both
|
|
273
|
+
positive and monotonically growing. The Phase 2 explore item
|
|
274
|
+
(`P-f6a7b8c9`) documents this workflow end-to-end and lists the
|
|
275
|
+
canonical retention paths to inspect inside the Minions engine
|
|
276
|
+
(`activeProcesses` Map, CC session map, `_buildStatusCache`, etc.).
|
|
277
|
+
|
|
278
|
+
### Retention policy
|
|
279
|
+
|
|
280
|
+
`HEAP_SNAPSHOT_RETAIN_COUNT = 5` per process
|
|
281
|
+
([`dashboard.js:189`](../dashboard.js)). On every successful capture,
|
|
282
|
+
`_heapSnapshotPrune` keeps the 5 newest `heap-dashboard-*.heapsnapshot`
|
|
283
|
+
and the 5 newest `heap-engine-*.heapsnapshot` by mtime, deleting the
|
|
284
|
+
rest. Best-effort: a prune failure is swallowed and retried on the next
|
|
285
|
+
capture, so transient Windows file locks don't break the endpoint.
|
|
286
|
+
|
|
287
|
+
If you need to preserve a snapshot, **copy it out** of
|
|
288
|
+
`engine/diagnostics/` before invoking the endpoint a sixth time.
|
|
289
|
+
|
|
290
|
+
## 4. CPU + heap allocation profile via `NODE_OPTIONS`
|
|
291
|
+
|
|
292
|
+
The heap-snapshot endpoint gives you retained sizes at a single moment.
|
|
293
|
+
`--cpu-prof` and `--heap-prof` give you the orthogonal signal: where CPU
|
|
294
|
+
time and allocations are *spent* during a window of execution.
|
|
295
|
+
|
|
296
|
+
Both are Node built-ins; no extra deps. Restart the engine with both
|
|
297
|
+
flags pointed at `engine/diagnostics`:
|
|
298
|
+
|
|
299
|
+
```powershell
|
|
300
|
+
# PowerShell
|
|
301
|
+
$env:NODE_OPTIONS = '--cpu-prof --cpu-prof-dir=engine/diagnostics --heap-prof --heap-prof-dir=engine/diagnostics'
|
|
302
|
+
minions restart
|
|
303
|
+
```
|
|
304
|
+
|
|
305
|
+
```bash
|
|
306
|
+
# bash
|
|
307
|
+
NODE_OPTIONS='--cpu-prof --cpu-prof-dir=engine/diagnostics --heap-prof --heap-prof-dir=engine/diagnostics' minions restart
|
|
308
|
+
```
|
|
309
|
+
|
|
310
|
+
Drive 5–10 minutes of representative load:
|
|
311
|
+
|
|
312
|
+
- Manually dispatch ≥ 5 short fix WIs (mocked agents are fine).
|
|
313
|
+
- Poll `/api/status` every 4 s in a tight loop:
|
|
314
|
+
`while ($true) { curl -s http://localhost:7331/api/status > $null; Start-Sleep 4 }`.
|
|
315
|
+
- Trigger ≥ 3 PR poller refreshes (or wait through ≥ 3 polling cycles).
|
|
316
|
+
|
|
317
|
+
Then **stop the engine cleanly** (`minions stop`) — the `.cpuprofile`
|
|
318
|
+
and `.heapprofile` files are only flushed on graceful exit.
|
|
319
|
+
|
|
320
|
+
The two file types land in `engine/diagnostics/`:
|
|
321
|
+
|
|
322
|
+
- `CPU.<date>.<pid>.<tid>.<seq>.cpuprofile` — V8 CPU sampler output.
|
|
323
|
+
- `Heap.<date>.<pid>.<tid>.<seq>.heapprofile` — V8 sampling heap
|
|
324
|
+
profiler output (allocations, *not* live retention; complements the
|
|
325
|
+
heap-snapshot endpoint).
|
|
326
|
+
|
|
327
|
+
### Loading in Chrome DevTools
|
|
328
|
+
|
|
329
|
+
1. Open DevTools → **Performance** tab → ⋯ menu → **Load profile…** for
|
|
330
|
+
`.cpuprofile`. Look at the flame chart for any synchronous frame
|
|
331
|
+
> 50 ms (these violate the `/api/status` isolation invariant — the
|
|
332
|
+
tick should never block the dashboard event loop that long).
|
|
333
|
+
2. Open DevTools → **Memory** tab → pick **Allocation sampling** →
|
|
334
|
+
right-click → **Load…** for `.heapprofile`. Sort by *Self size* to
|
|
335
|
+
identify the top allocation sites by byte count.
|
|
336
|
+
|
|
337
|
+
The Phase 3 explore item (`P-a7b8c9d0`) calls out the specific hot
|
|
338
|
+
paths to inspect (route-registry rebuild in `dashboard.js`,
|
|
339
|
+
`dashboard-build.js` HTML assembly, per-tick allocation hotspots in
|
|
340
|
+
`engine.js#tick`, regex / JSON.parse frames).
|
|
341
|
+
|
|
342
|
+
Remember to **clear** `NODE_OPTIONS` before the next normal start —
|
|
343
|
+
`--cpu-prof` writes a new file on every process exit and the directory
|
|
344
|
+
will fill up.
|
|
345
|
+
|
|
346
|
+
## 5. `test/perf/soak.test.js` threshold tuning
|
|
347
|
+
|
|
348
|
+
The soak test drives 30 simulated engine ticks against an isolated
|
|
349
|
+
`MINIONS_TEST_DIR` and asserts that `heapUsed` neither grows unboundedly
|
|
350
|
+
across the soak nor exceeds an absolute ceiling. Source:
|
|
351
|
+
[`test/perf/soak.test.js`](../test/perf/soak.test.js).
|
|
352
|
+
|
|
353
|
+
```bash
|
|
354
|
+
node test/perf/soak.test.js # RSS fallback, looser bound
|
|
355
|
+
node --expose-gc test/perf/soak.test.js # heap measurement, tight bound
|
|
356
|
+
```
|
|
357
|
+
|
|
358
|
+
### The two thresholds
|
|
359
|
+
|
|
360
|
+
| Constant | Default | What it gates |
|
|
361
|
+
|---|---|---|
|
|
362
|
+
| `HEAP_DELTA_MAX` | 8 MB | `heapUsed(30t) - heapUsed(5t)` — the delta between the post-warmup baseline (5 ticks) and the end of the soak (30 ticks). Catches compounding leaks. |
|
|
363
|
+
| `HEAP_ABSOLUTE_MAX` | 200 MB | `heapUsed(30t)` after the final forced GC. Catches "engine's resting footprint regressed", not just "per-tick leak". |
|
|
364
|
+
| `HEAP_DELTA_MAX_NO_GC` | 32 MB (= 4× `HEAP_DELTA_MAX`) | Fallback delta bound when `--expose-gc` is absent. Measured against RSS (includes uncollected garbage), so 4× slack. |
|
|
365
|
+
|
|
366
|
+
The absolute-ceiling test is informational (warning only) when GC is not
|
|
367
|
+
forced — without `--expose-gc`, `heapUsed` can include arbitrary
|
|
368
|
+
uncollected garbage that doesn't reflect a real leak.
|
|
369
|
+
|
|
370
|
+
### When to retune
|
|
371
|
+
|
|
372
|
+
**Lower the bounds** (tighten the gate) when you've shipped a real
|
|
373
|
+
reduction in per-tick allocations and want to lock it in:
|
|
374
|
+
|
|
375
|
+
1. Run the soak ≥ 5 times locally with `--expose-gc`.
|
|
376
|
+
2. Take the **max** observed delta across the runs.
|
|
377
|
+
3. Set the new `HEAP_DELTA_MAX` to `ceil(maxObservedDelta * 1.5)` MB,
|
|
378
|
+
rounded to the nearest MB.
|
|
379
|
+
4. Same recipe for `HEAP_ABSOLUTE_MAX` against the max post-soak
|
|
380
|
+
`heapUsed`.
|
|
381
|
+
5. Land the constant change with the PR that earned the headroom — link
|
|
382
|
+
the 5 sample runs in the PR description.
|
|
383
|
+
|
|
384
|
+
**Raise the bounds** (loosen the gate) only with evidence that the
|
|
385
|
+
existing limit is too tight for the *legitimate* per-tick cost (not to
|
|
386
|
+
mask a regression):
|
|
387
|
+
|
|
388
|
+
1. The CI gate must be flaking *across multiple unrelated PRs* — a
|
|
389
|
+
single PR failing the gate is a signal to fix the PR, not the test.
|
|
390
|
+
2. Capture ≥ 10 runs on the CI host (or a comparable Windows
|
|
391
|
+
self-hosted runner) with `--expose-gc` enabled. Record max delta and
|
|
392
|
+
max post-soak heapUsed.
|
|
393
|
+
3. If the variance is the issue (max - min > 50% of the current bound),
|
|
394
|
+
**add samples or extend warmup** before bumping. The header comment
|
|
395
|
+
on the test file is explicit about this: "If they flake on a noisy
|
|
396
|
+
CI host, prefer adding samples / longer warmup over raising the
|
|
397
|
+
bound — the gate's value is detecting compounding growth."
|
|
398
|
+
4. If max + 2σ truly exceeds the current bound, bump to
|
|
399
|
+
`ceil((max + 2σ) * 1.25)` MB. Link the 10 runs in the PR
|
|
400
|
+
description. Add a comment line above the constant referencing the
|
|
401
|
+
PR that justified the bump.
|
|
402
|
+
|
|
403
|
+
### Acceptable variance
|
|
404
|
+
|
|
405
|
+
On a quiet developer workstation with `--expose-gc`:
|
|
406
|
+
|
|
407
|
+
- `HEAP_DELTA_MAX` should clear with 1–3 MB observed delta. > 5 MB
|
|
408
|
+
means there is something to investigate, even if it still passes.
|
|
409
|
+
- `HEAP_ABSOLUTE_MAX` should clear with 60–110 MB observed post-soak
|
|
410
|
+
heapUsed. Approaching 150 MB means the resting footprint has grown
|
|
411
|
+
recently.
|
|
412
|
+
|
|
413
|
+
On a noisy CI host without `--expose-gc`, RSS variance can be 4–8 MB
|
|
414
|
+
between runs — this is exactly why the no-GC fallback uses 4× slack.
|
|
415
|
+
|
|
416
|
+
## 6. Phase 2 and Phase 3 findings notes
|
|
417
|
+
|
|
418
|
+
The diagnostics surface this doc describes exists to make two follow-up
|
|
419
|
+
explores (and any future ones) *cheap to run*. Findings live in
|
|
420
|
+
`knowledge/architecture/` once the KB sweep promotes them; until then,
|
|
421
|
+
they sit in `notes/inbox/` with predictable slugs:
|
|
422
|
+
|
|
423
|
+
| Phase | Plan item | Inbox slug (pre-sweep) | Promoted path (post-sweep) |
|
|
424
|
+
|---|---|---|---|
|
|
425
|
+
| 2 — heap dominators + leak candidates | `P-f6a7b8c9` | `notes/inbox/ripley-memory-audit-phase-2-heap-dominators*.md` | `knowledge/architecture/2026-06-12-ripley-memory-audit-phase-2-heap-dominators.md` |
|
|
426
|
+
| 3 — CPU + allocation hot frames | `P-a7b8c9d0` | `notes/inbox/ripley-memory-audit-phase-3-cpu-and-allocations*.md` | `knowledge/architecture/2026-06-12-ripley-memory-audit-phase-3-cpu-and-allocations.md` |
|
|
427
|
+
|
|
428
|
+
Use [`engine/kb-sweep.js`](../engine/kb-sweep.js) (see
|
|
429
|
+
[`docs/kb-sweep.md`](kb-sweep.md)) or the dashboard's "Sweep" button to
|
|
430
|
+
trigger promotion. The cheap-win PR items downstream of the explores
|
|
431
|
+
(`P-b8c9d0e1`, `P-c9d0e1f2`, `P-d0e1f2a3`) consume the prioritized lists
|
|
432
|
+
from these notes — read them before opening one of those PRs to avoid
|
|
433
|
+
re-deriving the same findings.
|
|
434
|
+
|
|
435
|
+
## Related modules and tests
|
|
436
|
+
|
|
437
|
+
- Module: [`engine/diagnostics-memory.js`](../engine/diagnostics-memory.js) — sampler + ring buffer + GC observer.
|
|
438
|
+
- Engine wiring: [`engine.js`](../engine.js) `emitMemoryBaseline` + `processHeapSnapshotRequest`.
|
|
439
|
+
- Dashboard wiring: [`dashboard.js`](../dashboard.js) `handleDiagnosticsMemory`, `handleDiagnosticsMemoryHistory`, `handleDiagnosticsHeapSnapshot`.
|
|
440
|
+
- Frontend: [`dashboard/pages/engine-memory-panel.html`](../dashboard/pages/engine-memory-panel.html) + [`dashboard/js/memory-panel.js`](../dashboard/js/memory-panel.js).
|
|
441
|
+
- Tests:
|
|
442
|
+
- [`test/unit/diagnostics-memory-api.test.js`](../test/unit/diagnostics-memory-api.test.js)
|
|
443
|
+
- [`test/unit/heap-snapshot-api.test.js`](../test/unit/heap-snapshot-api.test.js)
|
|
444
|
+
- [`test/unit/memory-panel.test.js`](../test/unit/memory-panel.test.js)
|
|
445
|
+
- [`test/perf/memory-baseline-tick-overhead.test.js`](../test/perf/memory-baseline-tick-overhead.test.js)
|
|
446
|
+
- [`test/perf/soak.test.js`](../test/perf/soak.test.js)
|