@claude-flow/cli 3.25.5 → 3.25.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/helpers/helpers.manifest.json +2 -2
- package/dist/src/commands/doctor.js +74 -29
- package/dist/src/init/executor.js +16 -16
- package/dist/src/init/mcp-generator.js +11 -6
- package/package.json +5 -2
- package/plugins/ruflo-metaharness/.claude-flow/daemon-state.json +178 -0
- package/plugins/ruflo-metaharness/.claude-flow/daemon.pid +1 -0
- package/plugins/ruflo-metaharness/.claude-flow/logs/daemon.log +43 -0
- package/plugins/ruflo-metaharness/.claude-flow/logs/headless/audit_1783604774864_ozbujc_prompt.log +19 -0
- package/plugins/ruflo-metaharness/.claude-flow/logs/headless/audit_1783604774864_ozbujc_result.log +108 -0
- package/plugins/ruflo-metaharness/.claude-flow/logs/headless/audit_1783605513587_ulvmpb_prompt.log +19 -0
- package/plugins/ruflo-metaharness/.claude-flow/logs/headless/audit_1783605513587_ulvmpb_result.log +209 -0
- package/plugins/ruflo-metaharness/.claude-flow/logs/headless/audit_1783606368867_ahysui_prompt.log +19 -0
- package/plugins/ruflo-metaharness/.claude-flow/logs/headless/audit_1783606368867_ahysui_result.log +192 -0
- package/plugins/ruflo-metaharness/.claude-flow/logs/headless/optimize_1783604894861_v6n3ut_prompt.log +14 -0
- package/plugins/ruflo-metaharness/.claude-flow/logs/headless/optimize_1783604894861_v6n3ut_result.log +66 -0
- package/plugins/ruflo-metaharness/.claude-flow/logs/headless/optimize_1783605934532_9h8ikb_prompt.log +14 -0
- package/plugins/ruflo-metaharness/.claude-flow/logs/headless/optimize_1783605934532_9h8ikb_result.log +68 -0
- package/plugins/ruflo-metaharness/.claude-flow/logs/headless/testgaps_1783605134860_9jssz9_prompt.log +14 -0
- package/plugins/ruflo-metaharness/.claude-flow/logs/headless/testgaps_1783605134860_9jssz9_result.log +69 -0
- package/plugins/ruflo-metaharness/.claude-flow/logs/headless/testgaps_1783606516743_zftbaa_prompt.log +14 -0
- package/plugins/ruflo-metaharness/.claude-flow/metrics/backup.json +7 -0
- package/plugins/ruflo-metaharness/.claude-flow/metrics/codebase-map.json +11 -0
- package/plugins/ruflo-metaharness/.claude-flow/metrics/consolidation.json +16 -0
- package/plugins/ruflo-metaharness/.claude-flow/metrics/harness-loop.json +83 -0
- package/plugins/ruflo-metaharness/.claude-flow/metrics/performance.json +67 -0
- package/plugins/ruflo-metaharness/.claude-flow/metrics/security-audit.json +191 -0
- package/plugins/ruflo-metaharness/.claude-flow/metrics/test-gaps.json +68 -0
- package/plugins/ruflo-metaharness/.claude-flow/neural/stats.json +3 -3
- package/plugins/ruflo-metaharness/scripts/smoke.sh +18 -5
|
@@ -0,0 +1,68 @@
|
|
|
1
|
+
[2026-07-09T14:11:21.736Z] RESULT
|
|
2
|
+
============================================================
|
|
3
|
+
{
|
|
4
|
+
"success": true,
|
|
5
|
+
"output": "## Performance Analysis: `plugins/ruflo-metaharness`\n\nThis directory has no React and no database/ORM layer — it's ~30 standalone Node.js `.mjs` CLI scripts that shell out to `npx ruflo`/`@claude-flow/cli`, read/write JSON, and call memory/AgentDB APIs. So I mapped your five categories onto their actual Node-CLI analogues rather than inventing web-app issues. Two categories (React re-renders, classic SQL N+1) genuinely don't apply here — noted explicitly below instead of stretched to fit.\n\n### 1. N+1-shaped pattern (subprocess-per-item, the real analogue of N+1 queries)\n\n**`audit-list.mjs:56-65, 102-117`** — lists audit keys with one spawn, then does a **second spawn per key** to hydrate each record:\n```js\nfunction memRetrieve(key) {\n const r = spawnSync('npx', [CLI_PKG, 'memory', 'retrieve', '--namespace', NS, '--key', key], ...);\n}\n...\nfor (const key of slice) {\n const rec = memRetrieve(key); // one npx spawn PER record, sequential\n}\n```\n**`drift-from-history.mjs:154, 234-236`** compounds this: its default path (no `--baseline-key`/`--baseline-file`) calls `audit-list` with `--limit 50`, hydrating **50 full records**, sorts them, then uses only `sorted[0]` — 49 wasted subprocess spawns to extract one field. The file's own comments put per-subprocess cost at ~2-5s, so this is minutes of wasted time on the documented default UX.\n\n**Fix:** add a `--keys-only` mode to `audit-list.mjs` that returns from `memList()` without calling `memRetrieve` at all, and have `drift-from-history.mjs`'s default path use that (it only needs the newest key). For the general case, parallelize with `Promise.all` over an async spawn instead of serial `spawnSync`.\n\n### 2. Redundant subprocess spawns / missing caching\n\n**`_darwin.mjs:80,125`** never got migrated to the cached-install pattern the rest of the family uses:\n```js\nconst r = spawnSync('npx', ['-y', '-p', DARWIN_PIN, 'metaharness-darwin', ...argv], { ... });\n```\n`_harness.mjs`'s own header documents *why* this is expensive — `npx -y -p` forces registry/tarball resolution on every call — and `_redblue.mjs:79-111` already shows the fix (`ensureCachedInstall` → cache once → `spawnSync('node', [cliPath, ...])` thereafter). `_darwin.mjs` is used by `evolve.mjs:340`, `bench.mjs:64`, `security-bench.mjs:121`, so every Darwin-backed call pays this tax. It also lacks `_harness.mjs`'s `RESOLVED` memoization (`_harness.mjs:68-97`), so even within one process, repeated calls re-resolve from scratch.\n\n**Fix:** port `_darwin.mjs` to `_invoke.mjs`'s existing `findLocalPackageDir` + `ensureCachedInstall` helpers (drop-in, no new plumbing needed) and add a module-level `RESOLVED` cache.\n\n**Unpinned `@latest` dist-tag** — `audit-list.mjs:24`, `audit-trend.mjs:41`, `oia-audit.mjs:38`, `similarity.mjs:33`, `drift-from-history.mjs:47` all do:\n```js\nconst CLI_PKG = process.env.CLI_CORE === '1' ? '@claude-flow/cli-core@alpha' : '@claude-flow/cli@latest';\n```\n`@latest` forces an npm-registry metadata check on every invocation — exactly the anti-pattern `_harness.mjs` says it already fixed for the `metaharness` bin, but these five scripts never adopted it (`test-with-openrouter.mjs:109,118` hardcodes `metaharness@latest` too).\n\n**`mint.mjs:68`** spawns a whole extra `node` process just to detect degraded state via `runMetaharness(['--version'], ...)`, when `_harness.mjs:327-330` already exports an in-process `metaharnessResolution()` with identical semantics — swap it in and drop the second fork+exec.\n\n### 3. Blocking I/O in loops\n\n**`evolve.mjs:205-209`** — up to 100 serial `readFileSync` calls in the `--diagnose` path:\n```js\nfor (const f of readdirSync(runsDir).filter(f => f.endsWith('.json')).slice(0, 100)) {\n records.push({ id: ..., rec: JSON.parse(readFileSync(join(runsDir, f), 'utf-8')) });\n}\n```\n**Fix:**\n```js\nconst files = readdirSync(runsDir).filter(f => f.endsWith('.json')).slice(0, 100);\nconst records = (await Promise.all(files.map(async f => {\n try { return { id: f.replace(/\\.json$/, ''), rec: JSON.parse(await readFile(join(runsDir, f), 'utf-8')) }; }\n catch { return null; }\n}))).filter(Boolean);\n```\n\n### 4. Memory leaks — doesn't apply\n\nThese are one-shot CLI processes that `process.exit()` at the end of `main()`. Checked and confirmed clean: every `setTimeout` (`_harness.mjs:149-152`, `_darwin.mjs:144-151`) has a matching `clearTimeout`, `AbortSignal` listeners use `{ once: true }`, and no module-level array/Map accumulates across calls. No finding here — not manufacturing one.\n\n### 5. Redundant computation — mostly doesn't apply\n\nSimilarity/drift scoring (`_similarity.mjs`) is invoked exactly once per process, never re-run in a loop over a collection. The only real redundancy is the subprocess-spawn waste already covered in #1/#2 — not duplicate math.\n\n---\n\n**Summary table**\n\n| Issue | Location | Fix |\n|---|---|---|\n| N+1 subprocess spawns for 1 field | `drift-from-history.mjs:154` + `audit-list.mjs:102-117` | Add `--keys-only` list mode |\n| `npx -y -p` on every Darwin call | `_darwin.mjs:80,125` | Migrate to `ensureCachedInstall` + memoize |\n| Unpinned `@latest` CLI_PKG | 5 scripts (audit-list/trend, oia-audit, similarity, drift-from-history) | Pin + route through `_invoke.mjs` cache |\n| Extra spawn for degraded probe | `mint.mjs:68` | Use in-process `metaharnessResolution()` |\n| Serial `readFileSync` ×100 | `evolve.mjs:205-209` | `fs/promises` + `Promise.all` |\n\nWant me to implement any of these fixes? The `_darwin.mjs` migration and the `audit-list` `--keys-only` mode are the highest-value ones given ADR-150's cold-start budget constraints.\n",
|
|
6
|
+
"parsedOutput": {
|
|
7
|
+
"sections": [
|
|
8
|
+
{
|
|
9
|
+
"title": "Performance Analysis: `plugins/ruflo-metaharness`",
|
|
10
|
+
"content": "\nThis directory has no React and no database/ORM layer — it's ~30 standalone Node.js `.mjs` CLI scripts that shell out to `npx ruflo`/`@claude-flow/cli`, read/write JSON, and call memory/AgentDB APIs. So I mapped your five categories onto their actual Node-CLI analogues rather than inventing web-app issues. Two categories (React re-renders, classic SQL N+1) genuinely don't apply here — noted explicitly below instead of stretched to fit.\n\n",
|
|
11
|
+
"level": 2
|
|
12
|
+
},
|
|
13
|
+
{
|
|
14
|
+
"title": "1. N+1-shaped pattern (subprocess-per-item, the real analogue of N+1 queries)",
|
|
15
|
+
"content": "\n**`audit-list.mjs:56-65, 102-117`** — lists audit keys with one spawn, then does a **second spawn per key** to hydrate each record:\n```js\nfunction memRetrieve(key) {\n const r = spawnSync('npx', [CLI_PKG, 'memory', 'retrieve', '--namespace', NS, '--key', key], ...);\n}\n...\nfor (const key of slice) {\n const rec = memRetrieve(key); // one npx spawn PER record, sequential\n}\n```\n**`drift-from-history.mjs:154, 234-236`** compounds this: its default path (no `--baseline-key`/`--baseline-file`) calls `audit-list` with `--limit 50`, hydrating **50 full records**, sorts them, then uses only `sorted[0]` — 49 wasted subprocess spawns to extract one field. The file's own comments put per-subprocess cost at ~2-5s, so this is minutes of wasted time on the documented default UX.\n\n**Fix:** add a `--keys-only` mode to `audit-list.mjs` that returns from `memList()` without calling `memRetrieve` at all, and have `drift-from-history.mjs`'s default path use that (it only needs the newest key). For the general case, parallelize with `Promise.all` over an async spawn instead of serial `spawnSync`.\n\n",
|
|
16
|
+
"level": 3
|
|
17
|
+
},
|
|
18
|
+
{
|
|
19
|
+
"title": "2. Redundant subprocess spawns / missing caching",
|
|
20
|
+
"content": "\n**`_darwin.mjs:80,125`** never got migrated to the cached-install pattern the rest of the family uses:\n```js\nconst r = spawnSync('npx', ['-y', '-p', DARWIN_PIN, 'metaharness-darwin', ...argv], { ... });\n```\n`_harness.mjs`'s own header documents *why* this is expensive — `npx -y -p` forces registry/tarball resolution on every call — and `_redblue.mjs:79-111` already shows the fix (`ensureCachedInstall` → cache once → `spawnSync('node', [cliPath, ...])` thereafter). `_darwin.mjs` is used by `evolve.mjs:340`, `bench.mjs:64`, `security-bench.mjs:121`, so every Darwin-backed call pays this tax. It also lacks `_harness.mjs`'s `RESOLVED` memoization (`_harness.mjs:68-97`), so even within one process, repeated calls re-resolve from scratch.\n\n**Fix:** port `_darwin.mjs` to `_invoke.mjs`'s existing `findLocalPackageDir` + `ensureCachedInstall` helpers (drop-in, no new plumbing needed) and add a module-level `RESOLVED` cache.\n\n**Unpinned `@latest` dist-tag** — `audit-list.mjs:24`, `audit-trend.mjs:41`, `oia-audit.mjs:38`, `similarity.mjs:33`, `drift-from-history.mjs:47` all do:\n```js\nconst CLI_PKG = process.env.CLI_CORE === '1' ? '@claude-flow/cli-core@alpha' : '@claude-flow/cli@latest';\n```\n`@latest` forces an npm-registry metadata check on every invocation — exactly the anti-pattern `_harness.mjs` says it already fixed for the `metaharness` bin, but these five scripts never adopted it (`test-with-openrouter.mjs:109,118` hardcodes `metaharness@latest` too).\n\n**`mint.mjs:68`** spawns a whole extra `node` process just to detect degraded state via `runMetaharness(['--version'], ...)`, when `_harness.mjs:327-330` already exports an in-process `metaharnessResolution()` with identical semantics — swap it in and drop the second fork+exec.\n\n",
|
|
21
|
+
"level": 3
|
|
22
|
+
},
|
|
23
|
+
{
|
|
24
|
+
"title": "3. Blocking I/O in loops",
|
|
25
|
+
"content": "\n**`evolve.mjs:205-209`** — up to 100 serial `readFileSync` calls in the `--diagnose` path:\n```js\nfor (const f of readdirSync(runsDir).filter(f => f.endsWith('.json')).slice(0, 100)) {\n records.push({ id: ..., rec: JSON.parse(readFileSync(join(runsDir, f), 'utf-8')) });\n}\n```\n**Fix:**\n```js\nconst files = readdirSync(runsDir).filter(f => f.endsWith('.json')).slice(0, 100);\nconst records = (await Promise.all(files.map(async f => {\n try { return { id: f.replace(/\\.json$/, ''), rec: JSON.parse(await readFile(join(runsDir, f), 'utf-8')) }; }\n catch { return null; }\n}))).filter(Boolean);\n```\n\n",
|
|
26
|
+
"level": 3
|
|
27
|
+
},
|
|
28
|
+
{
|
|
29
|
+
"title": "4. Memory leaks — doesn't apply",
|
|
30
|
+
"content": "\nThese are one-shot CLI processes that `process.exit()` at the end of `main()`. Checked and confirmed clean: every `setTimeout` (`_harness.mjs:149-152`, `_darwin.mjs:144-151`) has a matching `clearTimeout`, `AbortSignal` listeners use `{ once: true }`, and no module-level array/Map accumulates across calls. No finding here — not manufacturing one.\n\n",
|
|
31
|
+
"level": 3
|
|
32
|
+
},
|
|
33
|
+
{
|
|
34
|
+
"title": "5. Redundant computation — mostly doesn't apply",
|
|
35
|
+
"content": "Similarity/drift scoring (`_similarity.mjs`) is invoked exactly once per process, never re-run in a loop over a collection. The only real redundancy is the subprocess-spawn waste already covered in #1/#2 — not duplicate math.\n\n---\n\n**Summary table**\n\n| Issue | Location | Fix |\n|---|---|---|\n| N+1 subprocess spawns for 1 field | `drift-from-history.mjs:154` + `audit-list.mjs:102-117` | Add `--keys-only` list mode |\n| `npx -y -p` on every Darwin call | `_darwin.mjs:80,125` | Migrate to `ensureCachedInstall` + memoize |\n| Unpinned `@latest` CLI_PKG | 5 scripts (audit-list/trend, oia-audit, similarity, drift-from-history) | Pin + route through `_invoke.mjs` cache |\n| Extra spawn for degraded probe | `mint.mjs:68` | Use in-process `metaharnessResolution()` |\n| Serial `readFileSync` ×100 | `evolve.mjs:205-209` | `fs/promises` + `Promise.all` |\n\nWant me to implement any of these fixes? The `_darwin.mjs` migration and the `audit-list` `--keys-only` mode are the highest-value ones given ADR-150's cold-start budget constraints.",
|
|
36
|
+
"level": 3
|
|
37
|
+
}
|
|
38
|
+
],
|
|
39
|
+
"codeBlocks": [
|
|
40
|
+
{
|
|
41
|
+
"language": "js",
|
|
42
|
+
"code": "function memRetrieve(key) {\n const r = spawnSync('npx', [CLI_PKG, 'memory', 'retrieve', '--namespace', NS, '--key', key], ...);\n}\n...\nfor (const key of slice) {\n const rec = memRetrieve(key); // one npx spawn PER record, sequential\n}"
|
|
43
|
+
},
|
|
44
|
+
{
|
|
45
|
+
"language": "js",
|
|
46
|
+
"code": "const r = spawnSync('npx', ['-y', '-p', DARWIN_PIN, 'metaharness-darwin', ...argv], { ... });"
|
|
47
|
+
},
|
|
48
|
+
{
|
|
49
|
+
"language": "js",
|
|
50
|
+
"code": "const CLI_PKG = process.env.CLI_CORE === '1' ? '@claude-flow/cli-core@alpha' : '@claude-flow/cli@latest';"
|
|
51
|
+
},
|
|
52
|
+
{
|
|
53
|
+
"language": "js",
|
|
54
|
+
"code": "for (const f of readdirSync(runsDir).filter(f => f.endsWith('.json')).slice(0, 100)) {\n records.push({ id: ..., rec: JSON.parse(readFileSync(join(runsDir, f), 'utf-8')) });\n}"
|
|
55
|
+
},
|
|
56
|
+
{
|
|
57
|
+
"language": "js",
|
|
58
|
+
"code": "const files = readdirSync(runsDir).filter(f => f.endsWith('.json')).slice(0, 100);\nconst records = (await Promise.all(files.map(async f => {\n try { return { id: f.replace(/\\.json$/, ''), rec: JSON.parse(await readFile(join(runsDir, f), 'utf-8')) }; }\n catch { return null; }\n}))).filter(Boolean);"
|
|
59
|
+
}
|
|
60
|
+
]
|
|
61
|
+
},
|
|
62
|
+
"durationMs": 347204,
|
|
63
|
+
"model": "sonnet",
|
|
64
|
+
"sandboxMode": "permissive",
|
|
65
|
+
"workerType": "optimize",
|
|
66
|
+
"timestamp": "2026-07-09T14:11:21.736Z",
|
|
67
|
+
"executionId": "optimize_1783605934532_9h8ikb"
|
|
68
|
+
}
|
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
[2026-07-09T13:52:14.861Z] PROMPT
|
|
2
|
+
============================================================
|
|
3
|
+
Analyze test coverage and identify gaps:
|
|
4
|
+
- Find untested functions and classes
|
|
5
|
+
- Identify edge cases not covered
|
|
6
|
+
- Suggest new test scenarios
|
|
7
|
+
- Check for missing error handling tests
|
|
8
|
+
- Identify integration test gaps
|
|
9
|
+
|
|
10
|
+
For each gap, provide a test skeleton.
|
|
11
|
+
|
|
12
|
+
## Instructions
|
|
13
|
+
|
|
14
|
+
Analyze the codebase and provide your response following the format specified in the task.
|
|
@@ -0,0 +1,69 @@
|
|
|
1
|
+
[2026-07-09T13:55:16.742Z] RESULT
|
|
2
|
+
============================================================
|
|
3
|
+
{
|
|
4
|
+
"success": true,
|
|
5
|
+
"output": "## Test Coverage Analysis — `plugins/ruflo-metaharness`\n\nThis plugin uses a hand-rolled test harness (`test-*.mjs` scripts with a custom `assert()` helper, no Jest/Vitest), plus `scripts/smoke.sh` (bash, mostly structural/CLI-registration grep checks) run in CI via `no-metaharness-smoke.yml` / `metaharness-ci.yml`.\n\n**Existing coverage:**\n| Test file | Covers |\n|---|---|\n| `test-similarity.mjs` | `_similarity.mjs` (all 5 fns) + `_harness.mjs` severity primitives |\n| `test-graceful-degradation.mjs` | CLI-level exit-0/degraded contract for `score/genome/mcp-scan/threat-model/oia-audit/audit-list/audit-trend/mint` only |\n| `test-mcp-tools.mjs` | MCP tool handler contract (shape, no-throw) against compiled dist |\n| `test-parallel-pipeline.mjs` | `router-parallel-analyze.mjs` as a black-box subprocess against a synthesized JSONL fixture |\n| `test-pipeline-roundtrip.mjs` | `oia-audit` → `audit-trend` e2e chain |\n| `test-with-openrouter.mjs` | Real e2e against OpenRouter (costs money, needs GCP secret) |\n| `smoke.sh` | Structural grep + CLI subcommand/`--help` registration for everything |\n\n## 1. Untested functions/classes\n\n**`_invoke.mjs` — zero direct unit tests.** This is the highest-value gap: it's the extracted shared plumbing (per its own header, \"converged security/perf/arch review\") used by ~15 scripts, made of small pure/near-pure functions, yet no test imports it directly — only indirectly through subprocess-level e2e tests that need npm/network.\n- `classifyDegraded(stderr, exitCode, reasonPrefix)` — untested\n- `injectJson(args, wantJson)` — untested\n- `parseTrailingJson(stdout)` — untested (this one **replaced a live bug** per the comment — the exact kind of function that regresses silently without a unit test)\n- `satisfiesTildeRange(version, pinVersion)` — untested\n- `findLocalPackageDir(pkg, pinVersion)` — untested (env-seam `RUFLO_METAHARNESS_SKIP_LOCAL` exists but nothing exercises it)\n- `cacheBaseDir()` — untested (env-seam `RUFLO_METAHARNESS_CACHE_BASE` exists but nothing exercises it)\n- `makeDegradedEmitter(pkg, pinVersion)` — untested\n\n**`_darwin.mjs` / `_redblue.mjs`** — only exercised transitively (network-dependent, via smoke.sh registration checks), never unit-tested directly:\n- `_darwin.mjs`: `runDarwin`, `runDarwinAsync` (streaming/`onProgress`/abort-signal paths), `importGepa`\n- `_redblue.mjs`: `runRedblue`, `emitRedblueDegradedJsonAndExit`\n\n**Pure parsing/logic functions, module-private (not exported), with no direct test:**\n- `security-bench.mjs: parseSecurityBenchMarkdown()` — 3 independent regex extractions (overall verdict, gate lines, baseline table)\n- `redblue.mjs: buildUpstreamArgs()` — per-subcommand argv construction + validation (`attack` family check, `report` requires `--in`)\n- `evolve.mjs: looksLikeGepaTranscript()`, `extractGepaTranscripts()`, `summarizeTraces()`, `safetyChecks()`\n- `audit-list.mjs: parseDurationMs()` — duration-spec parser (`7d`, `24h`, `1w`, `1m`)\n- `gepa.mjs: readJsonFile()`, `loadGenomeOrExit()`\n- `router-parallel-analyze.mjs: median()`, `pctile()`, `mean()` — pure stats, only exercised indirectly through a full subprocess run in `test-parallel-pipeline.mjs`\n\n**Note:** all of the above are unexported module-level functions in CLI scripts that run `main()` on import — that's *why* nobody unit-tests them directly today. Testing them requires either (a) exporting them (cheap, mirrors what `_similarity.mjs` already does) or (b) testing through stdout/exit-code capture of the whole script. Skeletons below assume (a) since it's the established pattern in this codebase.\n\n## 2. Edge cases not covered\n\n- `parseTrailingJson`: multiple `{...}` blocks where an earlier block is unparseable JSON but a *later* one nests braces the lazy regex fragments (the exact bug class the comment says this fixed) — no fixture actually exercises nested-object recovery via the greedy fallback.\n- `satisfiesTildeRange`: version with a `-alpha.N` suffix, `pinVersion` without a leading `~`, garbage input.\n- `classifyDegraded`: `exitCode === 0` with degraded-looking stderr (should NOT degrade — success takes precedence over noisy stderr); `stderr` undefined vs empty string.\n- `findLocalPackageDir`: version present but pre-1.0 (`0.x.y`) tilde matching; symlinked node_modules; two ancestors both having a stale version (should pick a satisfying one over the first found).\n- `parseDurationMs`: unit `m` is ambiguous (30-day \"month\" vs minutes) — worth a regression test pinning the intended semantics; malformed specs (`7`, `d7`, `-7d`).\n- `parseSecurityBenchMarkdown`: `overallMatch` is `null` (upstream changed emoji/wording) — does the caller handle a `null` gate list gracefully?\n- `buildUpstreamArgs('attack')`: missing `--count` (`ARGS.count === null`) should omit the flag, not emit `--count null`.\n\n## 3. Missing error handling tests\n\n- `ensureCachedInstall`: `mkdirSync` throw path (`cache-dir-create-failed`) and `npm install` non-zero exit (`install-failed`) — both branches untested.\n- `importOptionalLibrary`: the `recoverable` regex gate — an *unrelated* `TypeError` thrown from within an installed module must NOT be swallowed (re-thrown), only `MODULE_NOT_FOUND`-shaped errors should fall through to cached-install. No test proves non-recoverable errors propagate.\n- `evolve.mjs safetyChecks()`: none of `--generations`/`--children`/`--concurrency` out-of-range, invalid `--sandbox`/`--selection`/`--mutator`, or nonexistent `--repo` path are unit-tested (only reachable via full CLI spawn today).\n- `security-bench.mjs safetyChecks()`: `--population`/`--cycles` bounds untested directly.\n- `redblue.mjs`: `report` without `--in`, `attack` with invalid family — both call `process.exit(2)`, untested.\n\n## 4. Integration test gaps\n\n- **`evolve` / `gepa` / `learn` / `redblue` / `bench` / `security-bench` are absent from `test-graceful-degradation.mjs`.** The file's own header explicitly scopes to 8 skills and says nothing about these six. Given ADR-150's architectural constraint (\"ruflo remains operational if every MetaHarness package is removed\") is supposed to apply project-wide, this is a real gate gap, not just a nice-to-have — a regression in `_darwin.mjs`'s or `_redblue.mjs`'s degraded path for these specific subcommands would ship silently.\n- `drift-from-history.mjs` composing 3 primitives (`audit-list` → `similarity`/`audit-trend`) has no dedicated round-trip test analogous to `test-pipeline-roundtrip.mjs` — only structural grep in `smoke.sh`.\n- No test exercises `RUFLO_METAHARNESS_CACHE_BASE` / `RUFLO_METAHARNESS_SKIP_LOCAL` env seams that `_invoke.mjs` explicitly documents as \"used by test-graceful-degradation.mjs\" — grep confirms they aren't actually referenced there.\n\n## Test skeletons\n\n**New file: `scripts/test-invoke.mjs`** (mirrors `test-similarity.mjs`'s style; requires exporting nothing new — everything below is already exported)\n\n```js\n#!/usr/bin/env node\n// test-invoke.mjs — unit tests for the shared _invoke.mjs plumbing\n// (classifyDegraded, injectJson, parseTrailingJson, satisfiesTildeRange,\n// findLocalPackageDir, cacheBaseDir). No network, no subprocess spawns.\n\nimport { mkdtempSync, mkdirSync, writeFileSync, rmSync } from 'node:fs';\nimport { tmpdir } from 'node:os';\nimport { join } from 'node:path';\nimport {\n classifyDegraded, injectJson, parseTrailingJson,\n satisfiesTildeRange, findLocalPackageDir, cacheBaseDir,\n} from './_invoke.mjs';\n\nlet passed = 0, failed = 0;\nconst failures = [];\nfunction assert(cond, label) {\n if (cond) { console.log(` ✓ ${label}`); passed++; }\n else { console.log(` ✗ ${label}`); failures.push(label); failed++; }\n}\n\nconsole.log('# classifyDegraded');\nassert(classifyDegraded('', null, 'x').degraded === true, 'null exitCode -> timeout');\nassert(classifyDegraded('', null, 'x').reason === 'x-timeout', 'timeout reason prefixed');\nassert(classifyDegraded('npm ERR! 404', 1, 'x').reason === 'x-not-available', 'npm ERR matches DEGRADED_RX');\nassert(classifyDegraded('some noisy stderr', 0, 'x').degraded === false, 'exit 0 with noisy stderr is NOT degraded');\nassert(classifyDegraded(undefined, 1, 'x').degraded === false, 'undefined stderr does not throw / does not false-positive');\n\nconsole.log('\\n# injectJson');\nassert(injectJson(['run'], true).includes('--json'), 'appends --json when wanted');\nassert(!injectJson(['run'], false).includes('--json'), 'omits --json when not wanted');\nassert(injectJson(['run', '--json'], true).filter((a) => a === '--json').length === 1, 'no duplicate --json');\n\nconsole.log('\\n# parseTrailingJson');\nassert(parseTrailingJson('progress\\n{\"a\":1}\\n{\"b\":2}').b === 2, 'picks LAST block, not first');\nassert(parseTrailingJson('{\"a\": {\"nested\": {\"deep\": 1}}}').a.nested.deep === 1, 'greedy fallback recovers nested braces');\nassert(parseTrailingJson('no json here') === null, 'no braces -> null, never throws');\nassert(parseTrailingJson('{\"a\":1} garbage {not json}').a === 1, 'unparseable trailing block falls back to earlier valid one');\n\nconsole.log('\\n# satisfiesTildeRange');\nassert(satisfiesTildeRange('0.8.3', '~0.8.0') === true, 'patch >= pin satisfies');\nassert(satisfiesTildeRange('0.8.0', '~0.8.0') === true, 'exact patch satisfies');\nassert(satisfiesTildeRange('0.9.0', '~0.8.0') === false, 'minor bump does not satisfy tilde range');\nassert(satisfiesTildeRange('not-a-version', '~0.8.0') === false, 'garbage version -> false, not throw');\nassert(satisfiesTildeRange('0.8.3', 'garbage-pin') === false, 'garbage pin -> false, not throw');\n\nconsole.log('\\n# cacheBaseDir + RUFLO_METAHARNESS_CACHE_BASE seam');\n{\n const prev = process.env.RUFLO_METAHARNESS_CACHE_BASE;\n delete process.env.RUFLO_METAHARNESS_CACHE_BASE;\n assert(cacheBaseDir().endsWith('.ruflo'), 'defaults to ~/.ruflo');\n process.env.RUFLO_METAHARNESS_CACHE_BASE = '/tmp/custom-cache';\n assert(cacheBaseDir() === '/tmp/custom-cache', 'env override respected');\n if (prev === undefined) delete process.env.RUFLO_METAHARNESS_CACHE_BASE;\n else process.env.RUFLO_METAHARNESS_CACHE_BASE = prev;\n}\n\nconsole.log('\\n# findLocalPackageDir + RUFLO_METAHARNESS_SKIP_LOCAL seam');\n{\n const fixture = mkdtempSync(join(tmpdir(), 'invoke-fixture-'));\n const pkgDir = join(fixture, 'node_modules', '@metaharness', 'darwin');\n mkdirSync(pkgDir, { recursive: true });\n writeFileSync(join(pkgDir, 'package.json'), JSON.stringify({ version: '0.8.3' }));\n const prevCwd = process.cwd();\n process.chdir(fixture);\n try {\n assert(findLocalPackageDir('@metaharness/darwin', '~0.8.0') === pkgDir, 'finds satisfying local install by walking cwd');\n assert(findLocalPackageDir('@metaharness/darwin', '~0.9.0') === null, 'rejects local install that does not satisfy pin');\n process.env.RUFLO_METAHARNESS_SKIP_LOCAL = '1';\n assert(findLocalPackageDir('@metaharness/darwin', '~0.8.0') === null, 'SKIP_LOCAL seam disables resolution entirely');\n } finally {\n delete process.env.RUFLO_METAHARNESS_SKIP_LOCAL;\n process.chdir(prevCwd);\n rmSync(fixture, { recursive: true, force: true });\n }\n}\n\nconsole.log(`\\n${passed} passed, ${failed} failed`);\nif (failed > 0) { console.log('Failures:', failures.join(', ')); process.exit(1); }\n```\n\n**Extend `test-graceful-degradation.mjs`** — add the 6 missing skills to close the ADR-150 gate gap:\n\n```js\n// Add alongside the existing 8-skill list:\nconst MISSING_SKILLS = ['evolve', 'gepa', 'learn', 'redblue', 'bench', 'security-bench'];\nfor (const skill of MISSING_SKILLS) {\n // same drill as the existing loop: spawn with npm_config_registry pointed\n // at an unresolvable host, assert exitCode === 0 and stdout contains\n // \"degraded\": true — these six currently escape the drill entirely.\n}\n```\n\n**New file: `scripts/test-cli-parsers.mjs`** (requires adding `export` to the named functions in `security-bench.mjs`, `redblue.mjs`, `audit-list.mjs`, `evolve.mjs`, `gepa.mjs` — currently module-private, which is *why* they're untestable in isolation today):\n\n```js\n#!/usr/bin/env node\n// test-cli-parsers.mjs — unit tests for pure parsing/validation logic\n// embedded in CLI scripts. Requires exporting: parseSecurityBenchMarkdown\n// (security-bench.mjs), buildUpstreamArgs (redblue.mjs), parseDurationMs\n// (audit-list.mjs), looksLikeGepaTranscript/extractGepaTranscripts/\n// summarizeTraces (evolve.mjs).\n\nimport { parseSecurityBenchMarkdown } from './security-bench.mjs';\nimport { parseDurationMs } from './audit-list.mjs';\nimport { looksLikeGepaTranscript, extractGepaTranscripts, summarizeTraces } from './evolve.mjs';\n\nlet passed = 0, failed = 0;\nconst failures = [];\nfunction assert(cond, label) {\n if (cond) { console.log(` ✓ ${label}`); passed++; }\n else { console.log(` ✗ ${label}`); failures.push(label); failed++; }\n}\n\nconsole.log('# parseSecurityBenchMarkdown');\nconst md = `\n**Overall: ✅ PASS**\n- ✅ **TPR improvement ≥ 25% vs fixed harness** — +150%\n- ❌ **FPR ≤ 5%** — 8%\n| B1 baseline | 0.42 | 0.60 | 0.10 | 0.90 | 0.95 | 2 | $0.10 |\n`;\nconst parsed = parseSecurityBenchMarkdown(md);\nassert(parsed.overall.ok === true, 'overall PASS parsed');\nassert(parsed.gates.length === 2, 'both gate lines parsed');\nassert(parsed.gates[1].ok === false, 'FAIL gate icon parsed correctly');\nassert(parsed.baselines[0].fitness === 0.42, 'baseline table row parsed');\nassert(parseSecurityBenchMarkdown('no matches here').overall === null, 'missing Overall line -> null, not throw');\nassert(parseSecurityBenchMarkdown('no matches here').gates.length === 0, 'no gate lines -> empty array, not throw');\n\nconsole.log('\\n# parseDurationMs');\nassert(parseDurationMs('7d') === 7 * 86400_000, '7d parses');\nassert(parseDurationMs('24h') === 24 * 3600_000, '24h parses');\nassert(parseDurationMs('1w') === 7 * 86400_000, '1w parses');\nassert(parseDurationMs('garbage') === null, 'malformed spec -> null');\nassert(parseDurationMs('-7d') === null, 'negative spec -> null (regex requires digits only)');\nassert(parseDurationMs('7') === null, 'missing unit -> null');\n\nconsole.log('\\n# evolve.mjs gepa-transcript helpers');\nassert(looksLikeGepaTranscript([{ actionRaw: 'x' }]) === true, 'actionRaw-shaped entries recognized');\nassert(looksLikeGepaTranscript([{ obs: 'y' }]) === true, 'obs-shaped entries recognized');\nassert(looksLikeGepaTranscript([]) === false, 'empty array is not a transcript');\nassert(looksLikeGepaTranscript([{ foo: 1 }]) === false, 'unrelated shape rejected');\nassert(looksLikeGepaTranscript(null) === false, 'null input -> false, not throw');\n\nconst record = { transcript: [{ actionRaw: 'a' }], traces: [{ taskId: 't1', exitCode: 1, timedOut: true, blockedActions: ['x'] }] };\nassert(extractGepaTranscripts(record).length === 1, 'top-level transcript extracted');\nconst summary = summarizeTraces(record);\nassert(summary.tasks === 1 && summary.failed === 1 && summary.timedOut === 1 && summary.blockedActions === 1, 'summarizeTraces tallies correctly');\nassert(summarizeTraces({}).tasks === 0, 'missing traces array -> zeroed summary, not throw');\n\nconsole.log(`\\n${passed} passed, ${failed} failed`);\nif (failed > 0) { console.log('Failures:', failures.join(', ')); process.exit(1); }\n```\n\n**New file: `scripts/test-router-parallel-math.mjs`** (requires exporting `median`/`pctile`/`mean` from `router-parallel-analyze.mjs`):\n\n```js\n#!/usr/bin/env node\nimport { median, pctile, mean } from './router-parallel-analyze.mjs';\n\nlet passed = 0, failed = 0;\nfunction assert(cond, label) { if (cond) passed++; else { console.log(`✗ ${label}`); failed++; } }\n\nassert(median([1, 2, 3]) === 2, 'odd-length median');\nassert(median([1, 2, 3, 4]) === 2.5, 'even-length median averages middle two');\nassert(median([]) === undefined || Number.isNaN(median([])), 'empty array does not throw');\nassert(pctile([1, 2, 3, 4, 5], 0.5) === median([1, 2, 3, 4, 5]), 'p50 matches median');\nassert(mean([2, 4, 6]) === 4, 'mean computes correctly');\nassert(Number.isNaN(mean([])) || mean([]) === 0, 'empty array mean defined (pin the actual convention)');\n\nconsole.log(`${passed} passed, ${failed} failed`);\nif (failed > 0) process.exit(1);\n```\n\n## Priority\n\n1. `scripts/test-invoke.mjs` — closes the biggest real gap (shared plumbing, zero direct tests, already caused one documented regression).\n2. Extend `test-graceful-degradation.mjs` to the 6 missing skills — closes an actual ADR-150 CI gate hole.\n3. `test-cli-parsers.mjs` + `test-router-parallel-math.mjs` — cheap, pure-function tests once the target functions are exported (small, low-risk diffs to each source file).\n",
|
|
6
|
+
"parsedOutput": {
|
|
7
|
+
"sections": [
|
|
8
|
+
{
|
|
9
|
+
"title": "Test Coverage Analysis — `plugins/ruflo-metaharness`",
|
|
10
|
+
"content": "\nThis plugin uses a hand-rolled test harness (`test-*.mjs` scripts with a custom `assert()` helper, no Jest/Vitest), plus `scripts/smoke.sh` (bash, mostly structural/CLI-registration grep checks) run in CI via `no-metaharness-smoke.yml` / `metaharness-ci.yml`.\n\n**Existing coverage:**\n| Test file | Covers |\n|---|---|\n| `test-similarity.mjs` | `_similarity.mjs` (all 5 fns) + `_harness.mjs` severity primitives |\n| `test-graceful-degradation.mjs` | CLI-level exit-0/degraded contract for `score/genome/mcp-scan/threat-model/oia-audit/audit-list/audit-trend/mint` only |\n| `test-mcp-tools.mjs` | MCP tool handler contract (shape, no-throw) against compiled dist |\n| `test-parallel-pipeline.mjs` | `router-parallel-analyze.mjs` as a black-box subprocess against a synthesized JSONL fixture |\n| `test-pipeline-roundtrip.mjs` | `oia-audit` → `audit-trend` e2e chain |\n| `test-with-openrouter.mjs` | Real e2e against OpenRouter (costs money, needs GCP secret) |\n| `smoke.sh` | Structural grep + CLI subcommand/`--help` registration for everything |\n\n",
|
|
11
|
+
"level": 2
|
|
12
|
+
},
|
|
13
|
+
{
|
|
14
|
+
"title": "1. Untested functions/classes",
|
|
15
|
+
"content": "\n**`_invoke.mjs` — zero direct unit tests.** This is the highest-value gap: it's the extracted shared plumbing (per its own header, \"converged security/perf/arch review\") used by ~15 scripts, made of small pure/near-pure functions, yet no test imports it directly — only indirectly through subprocess-level e2e tests that need npm/network.\n- `classifyDegraded(stderr, exitCode, reasonPrefix)` — untested\n- `injectJson(args, wantJson)` — untested\n- `parseTrailingJson(stdout)` — untested (this one **replaced a live bug** per the comment — the exact kind of function that regresses silently without a unit test)\n- `satisfiesTildeRange(version, pinVersion)` — untested\n- `findLocalPackageDir(pkg, pinVersion)` — untested (env-seam `RUFLO_METAHARNESS_SKIP_LOCAL` exists but nothing exercises it)\n- `cacheBaseDir()` — untested (env-seam `RUFLO_METAHARNESS_CACHE_BASE` exists but nothing exercises it)\n- `makeDegradedEmitter(pkg, pinVersion)` — untested\n\n**`_darwin.mjs` / `_redblue.mjs`** — only exercised transitively (network-dependent, via smoke.sh registration checks), never unit-tested directly:\n- `_darwin.mjs`: `runDarwin`, `runDarwinAsync` (streaming/`onProgress`/abort-signal paths), `importGepa`\n- `_redblue.mjs`: `runRedblue`, `emitRedblueDegradedJsonAndExit`\n\n**Pure parsing/logic functions, module-private (not exported), with no direct test:**\n- `security-bench.mjs: parseSecurityBenchMarkdown()` — 3 independent regex extractions (overall verdict, gate lines, baseline table)\n- `redblue.mjs: buildUpstreamArgs()` — per-subcommand argv construction + validation (`attack` family check, `report` requires `--in`)\n- `evolve.mjs: looksLikeGepaTranscript()`, `extractGepaTranscripts()`, `summarizeTraces()`, `safetyChecks()`\n- `audit-list.mjs: parseDurationMs()` — duration-spec parser (`7d`, `24h`, `1w`, `1m`)\n- `gepa.mjs: readJsonFile()`, `loadGenomeOrExit()`\n- `router-parallel-analyze.mjs: median()`, `pctile()`, `mean()` — pure stats, only exercised indirectly through a full subprocess run in `test-parallel-pipeline.mjs`\n\n**Note:** all of the above are unexported module-level functions in CLI scripts that run `main()` on import — that's *why* nobody unit-tests them directly today. Testing them requires either (a) exporting them (cheap, mirrors what `_similarity.mjs` already does) or (b) testing through stdout/exit-code capture of the whole script. Skeletons below assume (a) since it's the established pattern in this codebase.\n\n",
|
|
16
|
+
"level": 2
|
|
17
|
+
},
|
|
18
|
+
{
|
|
19
|
+
"title": "2. Edge cases not covered",
|
|
20
|
+
"content": "\n- `parseTrailingJson`: multiple `{...}` blocks where an earlier block is unparseable JSON but a *later* one nests braces the lazy regex fragments (the exact bug class the comment says this fixed) — no fixture actually exercises nested-object recovery via the greedy fallback.\n- `satisfiesTildeRange`: version with a `-alpha.N` suffix, `pinVersion` without a leading `~`, garbage input.\n- `classifyDegraded`: `exitCode === 0` with degraded-looking stderr (should NOT degrade — success takes precedence over noisy stderr); `stderr` undefined vs empty string.\n- `findLocalPackageDir`: version present but pre-1.0 (`0.x.y`) tilde matching; symlinked node_modules; two ancestors both having a stale version (should pick a satisfying one over the first found).\n- `parseDurationMs`: unit `m` is ambiguous (30-day \"month\" vs minutes) — worth a regression test pinning the intended semantics; malformed specs (`7`, `d7`, `-7d`).\n- `parseSecurityBenchMarkdown`: `overallMatch` is `null` (upstream changed emoji/wording) — does the caller handle a `null` gate list gracefully?\n- `buildUpstreamArgs('attack')`: missing `--count` (`ARGS.count === null`) should omit the flag, not emit `--count null`.\n\n",
|
|
21
|
+
"level": 2
|
|
22
|
+
},
|
|
23
|
+
{
|
|
24
|
+
"title": "3. Missing error handling tests",
|
|
25
|
+
"content": "\n- `ensureCachedInstall`: `mkdirSync` throw path (`cache-dir-create-failed`) and `npm install` non-zero exit (`install-failed`) — both branches untested.\n- `importOptionalLibrary`: the `recoverable` regex gate — an *unrelated* `TypeError` thrown from within an installed module must NOT be swallowed (re-thrown), only `MODULE_NOT_FOUND`-shaped errors should fall through to cached-install. No test proves non-recoverable errors propagate.\n- `evolve.mjs safetyChecks()`: none of `--generations`/`--children`/`--concurrency` out-of-range, invalid `--sandbox`/`--selection`/`--mutator`, or nonexistent `--repo` path are unit-tested (only reachable via full CLI spawn today).\n- `security-bench.mjs safetyChecks()`: `--population`/`--cycles` bounds untested directly.\n- `redblue.mjs`: `report` without `--in`, `attack` with invalid family — both call `process.exit(2)`, untested.\n\n",
|
|
26
|
+
"level": 2
|
|
27
|
+
},
|
|
28
|
+
{
|
|
29
|
+
"title": "4. Integration test gaps",
|
|
30
|
+
"content": "\n- **`evolve` / `gepa` / `learn` / `redblue` / `bench` / `security-bench` are absent from `test-graceful-degradation.mjs`.** The file's own header explicitly scopes to 8 skills and says nothing about these six. Given ADR-150's architectural constraint (\"ruflo remains operational if every MetaHarness package is removed\") is supposed to apply project-wide, this is a real gate gap, not just a nice-to-have — a regression in `_darwin.mjs`'s or `_redblue.mjs`'s degraded path for these specific subcommands would ship silently.\n- `drift-from-history.mjs` composing 3 primitives (`audit-list` → `similarity`/`audit-trend`) has no dedicated round-trip test analogous to `test-pipeline-roundtrip.mjs` — only structural grep in `smoke.sh`.\n- No test exercises `RUFLO_METAHARNESS_CACHE_BASE` / `RUFLO_METAHARNESS_SKIP_LOCAL` env seams that `_invoke.mjs` explicitly documents as \"used by test-graceful-degradation.mjs\" — grep confirms they aren't actually referenced there.\n\n",
|
|
31
|
+
"level": 2
|
|
32
|
+
},
|
|
33
|
+
{
|
|
34
|
+
"title": "Test skeletons",
|
|
35
|
+
"content": "\n**New file: `scripts/test-invoke.mjs`** (mirrors `test-similarity.mjs`'s style; requires exporting nothing new — everything below is already exported)\n\n```js\n#!/usr/bin/env node\n// test-invoke.mjs — unit tests for the shared _invoke.mjs plumbing\n// (classifyDegraded, injectJson, parseTrailingJson, satisfiesTildeRange,\n// findLocalPackageDir, cacheBaseDir). No network, no subprocess spawns.\n\nimport { mkdtempSync, mkdirSync, writeFileSync, rmSync } from 'node:fs';\nimport { tmpdir } from 'node:os';\nimport { join } from 'node:path';\nimport {\n classifyDegraded, injectJson, parseTrailingJson,\n satisfiesTildeRange, findLocalPackageDir, cacheBaseDir,\n} from './_invoke.mjs';\n\nlet passed = 0, failed = 0;\nconst failures = [];\nfunction assert(cond, label) {\n if (cond) { console.log(` ✓ ${label}`); passed++; }\n else { console.log(` ✗ ${label}`); failures.push(label); failed++; }\n}\n\nconsole.log('# classifyDegraded');\nassert(classifyDegraded('', null, 'x').degraded === true, 'null exitCode -> timeout');\nassert(classifyDegraded('', null, 'x').reason === 'x-timeout', 'timeout reason prefixed');\nassert(classifyDegraded('npm ERR! 404', 1, 'x').reason === 'x-not-available', 'npm ERR matches DEGRADED_RX');\nassert(classifyDegraded('some noisy stderr', 0, 'x').degraded === false, 'exit 0 with noisy stderr is NOT degraded');\nassert(classifyDegraded(undefined, 1, 'x').degraded === false, 'undefined stderr does not throw / does not false-positive');\n\nconsole.log('\\n# injectJson');\nassert(injectJson(['run'], true).includes('--json'), 'appends --json when wanted');\nassert(!injectJson(['run'], false).includes('--json'), 'omits --json when not wanted');\nassert(injectJson(['run', '--json'], true).filter((a) => a === '--json').length === 1, 'no duplicate --json');\n\nconsole.log('\\n# parseTrailingJson');\nassert(parseTrailingJson('progress\\n{\"a\":1}\\n{\"b\":2}').b === 2, 'picks LAST block, not first');\nassert(parseTrailingJson('{\"a\": {\"nested\": {\"deep\": 1}}}').a.nested.deep === 1, 'greedy fallback recovers nested braces');\nassert(parseTrailingJson('no json here') === null, 'no braces -> null, never throws');\nassert(parseTrailingJson('{\"a\":1} garbage {not json}').a === 1, 'unparseable trailing block falls back to earlier valid one');\n\nconsole.log('\\n# satisfiesTildeRange');\nassert(satisfiesTildeRange('0.8.3', '~0.8.0') === true, 'patch >= pin satisfies');\nassert(satisfiesTildeRange('0.8.0', '~0.8.0') === true, 'exact patch satisfies');\nassert(satisfiesTildeRange('0.9.0', '~0.8.0') === false, 'minor bump does not satisfy tilde range');\nassert(satisfiesTildeRange('not-a-version', '~0.8.0') === false, 'garbage version -> false, not throw');\nassert(satisfiesTildeRange('0.8.3', 'garbage-pin') === false, 'garbage pin -> false, not throw');\n\nconsole.log('\\n# cacheBaseDir + RUFLO_METAHARNESS_CACHE_BASE seam');\n{\n const prev = process.env.RUFLO_METAHARNESS_CACHE_BASE;\n delete process.env.RUFLO_METAHARNESS_CACHE_BASE;\n assert(cacheBaseDir().endsWith('.ruflo'), 'defaults to ~/.ruflo');\n process.env.RUFLO_METAHARNESS_CACHE_BASE = '/tmp/custom-cache';\n assert(cacheBaseDir() === '/tmp/custom-cache', 'env override respected');\n if (prev === undefined) delete process.env.RUFLO_METAHARNESS_CACHE_BASE;\n else process.env.RUFLO_METAHARNESS_CACHE_BASE = prev;\n}\n\nconsole.log('\\n# findLocalPackageDir + RUFLO_METAHARNESS_SKIP_LOCAL seam');\n{\n const fixture = mkdtempSync(join(tmpdir(), 'invoke-fixture-'));\n const pkgDir = join(fixture, 'node_modules', '@metaharness', 'darwin');\n mkdirSync(pkgDir, { recursive: true });\n writeFileSync(join(pkgDir, 'package.json'), JSON.stringify({ version: '0.8.3' }));\n const prevCwd = process.cwd();\n process.chdir(fixture);\n try {\n assert(findLocalPackageDir('@metaharness/darwin', '~0.8.0') === pkgDir, 'finds satisfying local install by walking cwd');\n assert(findLocalPackageDir('@metaharness/darwin', '~0.9.0') === null, 'rejects local install that does not satisfy pin');\n process.env.RUFLO_METAHARNESS_SKIP_LOCAL = '1';\n assert(findLocalPackageDir('@metaharness/darwin', '~0.8.0') === null, 'SKIP_LOCAL seam disables resolution entirely');\n } finally {\n delete process.env.RUFLO_METAHARNESS_SKIP_LOCAL;\n process.chdir(prevCwd);\n rmSync(fixture, { recursive: true, force: true });\n }\n}\n\nconsole.log(`\\n${passed} passed, ${failed} failed`);\nif (failed > 0) { console.log('Failures:', failures.join(', ')); process.exit(1); }\n```\n\n**Extend `test-graceful-degradation.mjs`** — add the 6 missing skills to close the ADR-150 gate gap:\n\n```js\n// Add alongside the existing 8-skill list:\nconst MISSING_SKILLS = ['evolve', 'gepa', 'learn', 'redblue', 'bench', 'security-bench'];\nfor (const skill of MISSING_SKILLS) {\n // same drill as the existing loop: spawn with npm_config_registry pointed\n // at an unresolvable host, assert exitCode === 0 and stdout contains\n // \"degraded\": true — these six currently escape the drill entirely.\n}\n```\n\n**New file: `scripts/test-cli-parsers.mjs`** (requires adding `export` to the named functions in `security-bench.mjs`, `redblue.mjs`, `audit-list.mjs`, `evolve.mjs`, `gepa.mjs` — currently module-private, which is *why* they're untestable in isolation today):\n\n```js\n#!/usr/bin/env node\n// test-cli-parsers.mjs — unit tests for pure parsing/validation logic\n// embedded in CLI scripts. Requires exporting: parseSecurityBenchMarkdown\n// (security-bench.mjs), buildUpstreamArgs (redblue.mjs), parseDurationMs\n// (audit-list.mjs), looksLikeGepaTranscript/extractGepaTranscripts/\n// summarizeTraces (evolve.mjs).\n\nimport { parseSecurityBenchMarkdown } from './security-bench.mjs';\nimport { parseDurationMs } from './audit-list.mjs';\nimport { looksLikeGepaTranscript, extractGepaTranscripts, summarizeTraces } from './evolve.mjs';\n\nlet passed = 0, failed = 0;\nconst failures = [];\nfunction assert(cond, label) {\n if (cond) { console.log(` ✓ ${label}`); passed++; }\n else { console.log(` ✗ ${label}`); failures.push(label); failed++; }\n}\n\nconsole.log('# parseSecurityBenchMarkdown');\nconst md = `\n**Overall: ✅ PASS**\n- ✅ **TPR improvement ≥ 25% vs fixed harness** — +150%\n- ❌ **FPR ≤ 5%** — 8%\n| B1 baseline | 0.42 | 0.60 | 0.10 | 0.90 | 0.95 | 2 | $0.10 |\n`;\nconst parsed = parseSecurityBenchMarkdown(md);\nassert(parsed.overall.ok === true, 'overall PASS parsed');\nassert(parsed.gates.length === 2, 'both gate lines parsed');\nassert(parsed.gates[1].ok === false, 'FAIL gate icon parsed correctly');\nassert(parsed.baselines[0].fitness === 0.42, 'baseline table row parsed');\nassert(parseSecurityBenchMarkdown('no matches here').overall === null, 'missing Overall line -> null, not throw');\nassert(parseSecurityBenchMarkdown('no matches here').gates.length === 0, 'no gate lines -> empty array, not throw');\n\nconsole.log('\\n# parseDurationMs');\nassert(parseDurationMs('7d') === 7 * 86400_000, '7d parses');\nassert(parseDurationMs('24h') === 24 * 3600_000, '24h parses');\nassert(parseDurationMs('1w') === 7 * 86400_000, '1w parses');\nassert(parseDurationMs('garbage') === null, 'malformed spec -> null');\nassert(parseDurationMs('-7d') === null, 'negative spec -> null (regex requires digits only)');\nassert(parseDurationMs('7') === null, 'missing unit -> null');\n\nconsole.log('\\n# evolve.mjs gepa-transcript helpers');\nassert(looksLikeGepaTranscript([{ actionRaw: 'x' }]) === true, 'actionRaw-shaped entries recognized');\nassert(looksLikeGepaTranscript([{ obs: 'y' }]) === true, 'obs-shaped entries recognized');\nassert(looksLikeGepaTranscript([]) === false, 'empty array is not a transcript');\nassert(looksLikeGepaTranscript([{ foo: 1 }]) === false, 'unrelated shape rejected');\nassert(looksLikeGepaTranscript(null) === false, 'null input -> false, not throw');\n\nconst record = { transcript: [{ actionRaw: 'a' }], traces: [{ taskId: 't1', exitCode: 1, timedOut: true, blockedActions: ['x'] }] };\nassert(extractGepaTranscripts(record).length === 1, 'top-level transcript extracted');\nconst summary = summarizeTraces(record);\nassert(summary.tasks === 1 && summary.failed === 1 && summary.timedOut === 1 && summary.blockedActions === 1, 'summarizeTraces tallies correctly');\nassert(summarizeTraces({}).tasks === 0, 'missing traces array -> zeroed summary, not throw');\n\nconsole.log(`\\n${passed} passed, ${failed} failed`);\nif (failed > 0) { console.log('Failures:', failures.join(', ')); process.exit(1); }\n```\n\n**New file: `scripts/test-router-parallel-math.mjs`** (requires exporting `median`/`pctile`/`mean` from `router-parallel-analyze.mjs`):\n\n```js\n#!/usr/bin/env node\nimport { median, pctile, mean } from './router-parallel-analyze.mjs';\n\nlet passed = 0, failed = 0;\nfunction assert(cond, label) { if (cond) passed++; else { console.log(`✗ ${label}`); failed++; } }\n\nassert(median([1, 2, 3]) === 2, 'odd-length median');\nassert(median([1, 2, 3, 4]) === 2.5, 'even-length median averages middle two');\nassert(median([]) === undefined || Number.isNaN(median([])), 'empty array does not throw');\nassert(pctile([1, 2, 3, 4, 5], 0.5) === median([1, 2, 3, 4, 5]), 'p50 matches median');\nassert(mean([2, 4, 6]) === 4, 'mean computes correctly');\nassert(Number.isNaN(mean([])) || mean([]) === 0, 'empty array mean defined (pin the actual convention)');\n\nconsole.log(`${passed} passed, ${failed} failed`);\nif (failed > 0) process.exit(1);\n```\n\n",
|
|
36
|
+
"level": 2
|
|
37
|
+
},
|
|
38
|
+
{
|
|
39
|
+
"title": "Priority",
|
|
40
|
+
"content": "1. `scripts/test-invoke.mjs` — closes the biggest real gap (shared plumbing, zero direct tests, already caused one documented regression).\n2. Extend `test-graceful-degradation.mjs` to the 6 missing skills — closes an actual ADR-150 CI gate hole.\n3. `test-cli-parsers.mjs` + `test-router-parallel-math.mjs` — cheap, pure-function tests once the target functions are exported (small, low-risk diffs to each source file).",
|
|
41
|
+
"level": 2
|
|
42
|
+
}
|
|
43
|
+
],
|
|
44
|
+
"codeBlocks": [
|
|
45
|
+
{
|
|
46
|
+
"language": "js",
|
|
47
|
+
"code": "#!/usr/bin/env node\n// test-invoke.mjs — unit tests for the shared _invoke.mjs plumbing\n// (classifyDegraded, injectJson, parseTrailingJson, satisfiesTildeRange,\n// findLocalPackageDir, cacheBaseDir). No network, no subprocess spawns.\n\nimport { mkdtempSync, mkdirSync, writeFileSync, rmSync } from 'node:fs';\nimport { tmpdir } from 'node:os';\nimport { join } from 'node:path';\nimport {\n classifyDegraded, injectJson, parseTrailingJson,\n satisfiesTildeRange, findLocalPackageDir, cacheBaseDir,\n} from './_invoke.mjs';\n\nlet passed = 0, failed = 0;\nconst failures = [];\nfunction assert(cond, label) {\n if (cond) { console.log(` ✓ ${label}`); passed++; }\n else { console.log(` ✗ ${label}`); failures.push(label); failed++; }\n}\n\nconsole.log('# classifyDegraded');\nassert(classifyDegraded('', null, 'x').degraded === true, 'null exitCode -> timeout');\nassert(classifyDegraded('', null, 'x').reason === 'x-timeout', 'timeout reason prefixed');\nassert(classifyDegraded('npm ERR! 404', 1, 'x').reason === 'x-not-available', 'npm ERR matches DEGRADED_RX');\nassert(classifyDegraded('some noisy stderr', 0, 'x').degraded === false, 'exit 0 with noisy stderr is NOT degraded');\nassert(classifyDegraded(undefined, 1, 'x').degraded === false, 'undefined stderr does not throw / does not false-positive');\n\nconsole.log('\\n# injectJson');\nassert(injectJson(['run'], true).includes('--json'), 'appends --json when wanted');\nassert(!injectJson(['run'], false).includes('--json'), 'omits --json when not wanted');\nassert(injectJson(['run', '--json'], true).filter((a) => a === '--json').length === 1, 'no duplicate --json');\n\nconsole.log('\\n# parseTrailingJson');\nassert(parseTrailingJson('progress\\n{\"a\":1}\\n{\"b\":2}').b === 2, 'picks LAST block, not first');\nassert(parseTrailingJson('{\"a\": {\"nested\": {\"deep\": 1}}}').a.nested.deep === 1, 'greedy fallback recovers nested braces');\nassert(parseTrailingJson('no json here') === null, 'no braces -> null, never throws');\nassert(parseTrailingJson('{\"a\":1} garbage {not json}').a === 1, 'unparseable trailing block falls back to earlier valid one');\n\nconsole.log('\\n# satisfiesTildeRange');\nassert(satisfiesTildeRange('0.8.3', '~0.8.0') === true, 'patch >= pin satisfies');\nassert(satisfiesTildeRange('0.8.0', '~0.8.0') === true, 'exact patch satisfies');\nassert(satisfiesTildeRange('0.9.0', '~0.8.0') === false, 'minor bump does not satisfy tilde range');\nassert(satisfiesTildeRange('not-a-version', '~0.8.0') === false, 'garbage version -> false, not throw');\nassert(satisfiesTildeRange('0.8.3', 'garbage-pin') === false, 'garbage pin -> false, not throw');\n\nconsole.log('\\n# cacheBaseDir + RUFLO_METAHARNESS_CACHE_BASE seam');\n{\n const prev = process.env.RUFLO_METAHARNESS_CACHE_BASE;\n delete process.env.RUFLO_METAHARNESS_CACHE_BASE;\n assert(cacheBaseDir().endsWith('.ruflo'), 'defaults to ~/.ruflo');\n process.env.RUFLO_METAHARNESS_CACHE_BASE = '/tmp/custom-cache';\n assert(cacheBaseDir() === '/tmp/custom-cache', 'env override respected');\n if (prev === undefined) delete process.env.RUFLO_METAHARNESS_CACHE_BASE;\n else process.env.RUFLO_METAHARNESS_CACHE_BASE = prev;\n}\n\nconsole.log('\\n# findLocalPackageDir + RUFLO_METAHARNESS_SKIP_LOCAL seam');\n{\n const fixture = mkdtempSync(join(tmpdir(), 'invoke-fixture-'));\n const pkgDir = join(fixture, 'node_modules', '@metaharness', 'darwin');\n mkdirSync(pkgDir, { recursive: true });\n writeFileSync(join(pkgDir, 'package.json'), JSON.stringify({ version: '0.8.3' }));\n const prevCwd = process.cwd();\n process.chdir(fixture);\n try {\n assert(findLocalPackageDir('@metaharness/darwin', '~0.8.0') === pkgDir, 'finds satisfying local install by walking cwd');\n assert(findLocalPackageDir('@metaharness/darwin', '~0.9.0') === null, 'rejects local install that does not satisfy pin');\n process.env.RUFLO_METAHARNESS_SKIP_LOCAL = '1';\n assert(findLocalPackageDir('@metaharness/darwin', '~0.8.0') === null, 'SKIP_LOCAL seam disables resolution entirely');\n } finally {\n delete process.env.RUFLO_METAHARNESS_SKIP_LOCAL;\n process.chdir(prevCwd);\n rmSync(fixture, { recursive: true, force: true });\n }\n}\n\nconsole.log(`\\n${passed} passed, ${failed} failed`);\nif (failed > 0) { console.log('Failures:', failures.join(', ')); process.exit(1); }"
|
|
48
|
+
},
|
|
49
|
+
{
|
|
50
|
+
"language": "js",
|
|
51
|
+
"code": "// Add alongside the existing 8-skill list:\nconst MISSING_SKILLS = ['evolve', 'gepa', 'learn', 'redblue', 'bench', 'security-bench'];\nfor (const skill of MISSING_SKILLS) {\n // same drill as the existing loop: spawn with npm_config_registry pointed\n // at an unresolvable host, assert exitCode === 0 and stdout contains\n // \"degraded\": true — these six currently escape the drill entirely.\n}"
|
|
52
|
+
},
|
|
53
|
+
{
|
|
54
|
+
"language": "js",
|
|
55
|
+
"code": "#!/usr/bin/env node\n// test-cli-parsers.mjs — unit tests for pure parsing/validation logic\n// embedded in CLI scripts. Requires exporting: parseSecurityBenchMarkdown\n// (security-bench.mjs), buildUpstreamArgs (redblue.mjs), parseDurationMs\n// (audit-list.mjs), looksLikeGepaTranscript/extractGepaTranscripts/\n// summarizeTraces (evolve.mjs).\n\nimport { parseSecurityBenchMarkdown } from './security-bench.mjs';\nimport { parseDurationMs } from './audit-list.mjs';\nimport { looksLikeGepaTranscript, extractGepaTranscripts, summarizeTraces } from './evolve.mjs';\n\nlet passed = 0, failed = 0;\nconst failures = [];\nfunction assert(cond, label) {\n if (cond) { console.log(` ✓ ${label}`); passed++; }\n else { console.log(` ✗ ${label}`); failures.push(label); failed++; }\n}\n\nconsole.log('# parseSecurityBenchMarkdown');\nconst md = `\n**Overall: ✅ PASS**\n- ✅ **TPR improvement ≥ 25% vs fixed harness** — +150%\n- ❌ **FPR ≤ 5%** — 8%\n| B1 baseline | 0.42 | 0.60 | 0.10 | 0.90 | 0.95 | 2 | $0.10 |\n`;\nconst parsed = parseSecurityBenchMarkdown(md);\nassert(parsed.overall.ok === true, 'overall PASS parsed');\nassert(parsed.gates.length === 2, 'both gate lines parsed');\nassert(parsed.gates[1].ok === false, 'FAIL gate icon parsed correctly');\nassert(parsed.baselines[0].fitness === 0.42, 'baseline table row parsed');\nassert(parseSecurityBenchMarkdown('no matches here').overall === null, 'missing Overall line -> null, not throw');\nassert(parseSecurityBenchMarkdown('no matches here').gates.length === 0, 'no gate lines -> empty array, not throw');\n\nconsole.log('\\n# parseDurationMs');\nassert(parseDurationMs('7d') === 7 * 86400_000, '7d parses');\nassert(parseDurationMs('24h') === 24 * 3600_000, '24h parses');\nassert(parseDurationMs('1w') === 7 * 86400_000, '1w parses');\nassert(parseDurationMs('garbage') === null, 'malformed spec -> null');\nassert(parseDurationMs('-7d') === null, 'negative spec -> null (regex requires digits only)');\nassert(parseDurationMs('7') === null, 'missing unit -> null');\n\nconsole.log('\\n# evolve.mjs gepa-transcript helpers');\nassert(looksLikeGepaTranscript([{ actionRaw: 'x' }]) === true, 'actionRaw-shaped entries recognized');\nassert(looksLikeGepaTranscript([{ obs: 'y' }]) === true, 'obs-shaped entries recognized');\nassert(looksLikeGepaTranscript([]) === false, 'empty array is not a transcript');\nassert(looksLikeGepaTranscript([{ foo: 1 }]) === false, 'unrelated shape rejected');\nassert(looksLikeGepaTranscript(null) === false, 'null input -> false, not throw');\n\nconst record = { transcript: [{ actionRaw: 'a' }], traces: [{ taskId: 't1', exitCode: 1, timedOut: true, blockedActions: ['x'] }] };\nassert(extractGepaTranscripts(record).length === 1, 'top-level transcript extracted');\nconst summary = summarizeTraces(record);\nassert(summary.tasks === 1 && summary.failed === 1 && summary.timedOut === 1 && summary.blockedActions === 1, 'summarizeTraces tallies correctly');\nassert(summarizeTraces({}).tasks === 0, 'missing traces array -> zeroed summary, not throw');\n\nconsole.log(`\\n${passed} passed, ${failed} failed`);\nif (failed > 0) { console.log('Failures:', failures.join(', ')); process.exit(1); }"
|
|
56
|
+
},
|
|
57
|
+
{
|
|
58
|
+
"language": "js",
|
|
59
|
+
"code": "#!/usr/bin/env node\nimport { median, pctile, mean } from './router-parallel-analyze.mjs';\n\nlet passed = 0, failed = 0;\nfunction assert(cond, label) { if (cond) passed++; else { console.log(`✗ ${label}`); failed++; } }\n\nassert(median([1, 2, 3]) === 2, 'odd-length median');\nassert(median([1, 2, 3, 4]) === 2.5, 'even-length median averages middle two');\nassert(median([]) === undefined || Number.isNaN(median([])), 'empty array does not throw');\nassert(pctile([1, 2, 3, 4, 5], 0.5) === median([1, 2, 3, 4, 5]), 'p50 matches median');\nassert(mean([2, 4, 6]) === 4, 'mean computes correctly');\nassert(Number.isNaN(mean([])) || mean([]) === 0, 'empty array mean defined (pin the actual convention)');\n\nconsole.log(`${passed} passed, ${failed} failed`);\nif (failed > 0) process.exit(1);"
|
|
60
|
+
}
|
|
61
|
+
]
|
|
62
|
+
},
|
|
63
|
+
"durationMs": 181882,
|
|
64
|
+
"model": "sonnet",
|
|
65
|
+
"sandboxMode": "permissive",
|
|
66
|
+
"workerType": "testgaps",
|
|
67
|
+
"timestamp": "2026-07-09T13:55:16.742Z",
|
|
68
|
+
"executionId": "testgaps_1783605134860_9jssz9"
|
|
69
|
+
}
|
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
[2026-07-09T14:15:16.745Z] PROMPT
|
|
2
|
+
============================================================
|
|
3
|
+
Analyze test coverage and identify gaps:
|
|
4
|
+
- Find untested functions and classes
|
|
5
|
+
- Identify edge cases not covered
|
|
6
|
+
- Suggest new test scenarios
|
|
7
|
+
- Check for missing error handling tests
|
|
8
|
+
- Identify integration test gaps
|
|
9
|
+
|
|
10
|
+
For each gap, provide a test skeleton.
|
|
11
|
+
|
|
12
|
+
## Instructions
|
|
13
|
+
|
|
14
|
+
Analyze the codebase and provide your response following the format specified in the task.
|
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
{
|
|
2
|
+
"timestamp": "2026-07-09T14:14:14.857Z",
|
|
3
|
+
"projectRoot": "/Users/cohen/Projects/ruflo/plugins/ruflo-metaharness",
|
|
4
|
+
"structure": {
|
|
5
|
+
"hasPackageJson": false,
|
|
6
|
+
"hasTsConfig": false,
|
|
7
|
+
"hasClaudeConfig": false,
|
|
8
|
+
"hasClaudeFlow": true
|
|
9
|
+
},
|
|
10
|
+
"scannedAt": 1783606454857
|
|
11
|
+
}
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
{
|
|
2
|
+
"timestamp": "2026-07-09T13:50:14.860Z",
|
|
3
|
+
"distillationEnabled": true,
|
|
4
|
+
"patternsConsolidated": 0,
|
|
5
|
+
"memoryCleaned": 0,
|
|
6
|
+
"duplicatesRemoved": 0,
|
|
7
|
+
"episodes": 0,
|
|
8
|
+
"patternEmbeddings": 0,
|
|
9
|
+
"causalEdges": 0,
|
|
10
|
+
"promoted": 0,
|
|
11
|
+
"byProvenance": {},
|
|
12
|
+
"namespaces": [],
|
|
13
|
+
"dryRun": false,
|
|
14
|
+
"corrupt": false,
|
|
15
|
+
"skipped": "no-db"
|
|
16
|
+
}
|
|
@@ -0,0 +1,83 @@
|
|
|
1
|
+
{
|
|
2
|
+
"timestamp": "2026-07-09T13:56:14.863Z",
|
|
3
|
+
"flywheel": {
|
|
4
|
+
"ran": false,
|
|
5
|
+
"reason": "opt-in required (RUFLO_HARNESS_LOOP=1)",
|
|
6
|
+
"generation": 0
|
|
7
|
+
},
|
|
8
|
+
"lineage": {
|
|
9
|
+
"generations": 0,
|
|
10
|
+
"attempts": 0,
|
|
11
|
+
"lineage": {
|
|
12
|
+
"generations": 0,
|
|
13
|
+
"candidatesEvaluated": 0,
|
|
14
|
+
"promotions": 0,
|
|
15
|
+
"rejections": 0,
|
|
16
|
+
"cumulativeHeldOutImprovement": 0,
|
|
17
|
+
"rootHash": null,
|
|
18
|
+
"branches": [],
|
|
19
|
+
"lineageIntact": false,
|
|
20
|
+
"allReplayable": true,
|
|
21
|
+
"nodes": [],
|
|
22
|
+
"problems": [
|
|
23
|
+
"expected exactly one immutable root, found 0"
|
|
24
|
+
]
|
|
25
|
+
},
|
|
26
|
+
"plateau": {
|
|
27
|
+
"status": "insufficient-data",
|
|
28
|
+
"window": 5,
|
|
29
|
+
"medianImprovement": 0,
|
|
30
|
+
"promotionRate": 0,
|
|
31
|
+
"varianceShrinking": false,
|
|
32
|
+
"candidateVariance": 0,
|
|
33
|
+
"rationale": "need 5 generations, have 0"
|
|
34
|
+
},
|
|
35
|
+
"mutation": [],
|
|
36
|
+
"axisEffectiveness": [
|
|
37
|
+
{
|
|
38
|
+
"axis": "alpha",
|
|
39
|
+
"promotions": 0,
|
|
40
|
+
"meanDelta": 0
|
|
41
|
+
},
|
|
42
|
+
{
|
|
43
|
+
"axis": "subjectWeight",
|
|
44
|
+
"promotions": 0,
|
|
45
|
+
"meanDelta": 0
|
|
46
|
+
},
|
|
47
|
+
{
|
|
48
|
+
"axis": "mmrLambda",
|
|
49
|
+
"promotions": 0,
|
|
50
|
+
"meanDelta": 0
|
|
51
|
+
},
|
|
52
|
+
{
|
|
53
|
+
"axis": "bodyWeight",
|
|
54
|
+
"promotions": 0,
|
|
55
|
+
"meanDelta": 0
|
|
56
|
+
},
|
|
57
|
+
{
|
|
58
|
+
"axis": "typePenaltyFactor",
|
|
59
|
+
"promotions": 0,
|
|
60
|
+
"meanDelta": 0
|
|
61
|
+
}
|
|
62
|
+
],
|
|
63
|
+
"cumulativeBenchmarkDelta": 0,
|
|
64
|
+
"cumulativeHumanRelevanceDelta": 0,
|
|
65
|
+
"humanEvalHash": null,
|
|
66
|
+
"served": {
|
|
67
|
+
"championHash": null,
|
|
68
|
+
"config": null,
|
|
69
|
+
"servedAt": null,
|
|
70
|
+
"fromGeneration": null
|
|
71
|
+
},
|
|
72
|
+
"champion": {
|
|
73
|
+
"config": {
|
|
74
|
+
"alpha": 0.5,
|
|
75
|
+
"subjectWeight": 2,
|
|
76
|
+
"mmrLambda": 0.7,
|
|
77
|
+
"bodyWeight": 1,
|
|
78
|
+
"typePenaltyFactor": 1
|
|
79
|
+
},
|
|
80
|
+
"hash": null
|
|
81
|
+
}
|
|
82
|
+
}
|
|
83
|
+
}
|
|
@@ -0,0 +1,67 @@
|
|
|
1
|
+
{
|
|
2
|
+
"timestamp": "2026-07-09T14:11:21.736Z",
|
|
3
|
+
"mode": "headless",
|
|
4
|
+
"workerType": "optimize",
|
|
5
|
+
"model": "sonnet",
|
|
6
|
+
"durationMs": 347204,
|
|
7
|
+
"executionId": "optimize_1783605934532_9h8ikb",
|
|
8
|
+
"success": true,
|
|
9
|
+
"findings": {
|
|
10
|
+
"sections": [
|
|
11
|
+
{
|
|
12
|
+
"title": "Performance Analysis: `plugins/ruflo-metaharness`",
|
|
13
|
+
"content": "\nThis directory has no React and no database/ORM layer — it's ~30 standalone Node.js `.mjs` CLI scripts that shell out to `npx ruflo`/`@claude-flow/cli`, read/write JSON, and call memory/AgentDB APIs. So I mapped your five categories onto their actual Node-CLI analogues rather than inventing web-app issues. Two categories (React re-renders, classic SQL N+1) genuinely don't apply here — noted explicitly below instead of stretched to fit.\n\n",
|
|
14
|
+
"level": 2
|
|
15
|
+
},
|
|
16
|
+
{
|
|
17
|
+
"title": "1. N+1-shaped pattern (subprocess-per-item, the real analogue of N+1 queries)",
|
|
18
|
+
"content": "\n**`audit-list.mjs:56-65, 102-117`** — lists audit keys with one spawn, then does a **second spawn per key** to hydrate each record:\n```js\nfunction memRetrieve(key) {\n const r = spawnSync('npx', [CLI_PKG, 'memory', 'retrieve', '--namespace', NS, '--key', key], ...);\n}\n...\nfor (const key of slice) {\n const rec = memRetrieve(key); // one npx spawn PER record, sequential\n}\n```\n**`drift-from-history.mjs:154, 234-236`** compounds this: its default path (no `--baseline-key`/`--baseline-file`) calls `audit-list` with `--limit 50`, hydrating **50 full records**, sorts them, then uses only `sorted[0]` — 49 wasted subprocess spawns to extract one field. The file's own comments put per-subprocess cost at ~2-5s, so this is minutes of wasted time on the documented default UX.\n\n**Fix:** add a `--keys-only` mode to `audit-list.mjs` that returns from `memList()` without calling `memRetrieve` at all, and have `drift-from-history.mjs`'s default path use that (it only needs the newest key). For the general case, parallelize with `Promise.all` over an async spawn instead of serial `spawnSync`.\n\n",
|
|
19
|
+
"level": 3
|
|
20
|
+
},
|
|
21
|
+
{
|
|
22
|
+
"title": "2. Redundant subprocess spawns / missing caching",
|
|
23
|
+
"content": "\n**`_darwin.mjs:80,125`** never got migrated to the cached-install pattern the rest of the family uses:\n```js\nconst r = spawnSync('npx', ['-y', '-p', DARWIN_PIN, 'metaharness-darwin', ...argv], { ... });\n```\n`_harness.mjs`'s own header documents *why* this is expensive — `npx -y -p` forces registry/tarball resolution on every call — and `_redblue.mjs:79-111` already shows the fix (`ensureCachedInstall` → cache once → `spawnSync('node', [cliPath, ...])` thereafter). `_darwin.mjs` is used by `evolve.mjs:340`, `bench.mjs:64`, `security-bench.mjs:121`, so every Darwin-backed call pays this tax. It also lacks `_harness.mjs`'s `RESOLVED` memoization (`_harness.mjs:68-97`), so even within one process, repeated calls re-resolve from scratch.\n\n**Fix:** port `_darwin.mjs` to `_invoke.mjs`'s existing `findLocalPackageDir` + `ensureCachedInstall` helpers (drop-in, no new plumbing needed) and add a module-level `RESOLVED` cache.\n\n**Unpinned `@latest` dist-tag** — `audit-list.mjs:24`, `audit-trend.mjs:41`, `oia-audit.mjs:38`, `similarity.mjs:33`, `drift-from-history.mjs:47` all do:\n```js\nconst CLI_PKG = process.env.CLI_CORE === '1' ? '@claude-flow/cli-core@alpha' : '@claude-flow/cli@latest';\n```\n`@latest` forces an npm-registry metadata check on every invocation — exactly the anti-pattern `_harness.mjs` says it already fixed for the `metaharness` bin, but these five scripts never adopted it (`test-with-openrouter.mjs:109,118` hardcodes `metaharness@latest` too).\n\n**`mint.mjs:68`** spawns a whole extra `node` process just to detect degraded state via `runMetaharness(['--version'], ...)`, when `_harness.mjs:327-330` already exports an in-process `metaharnessResolution()` with identical semantics — swap it in and drop the second fork+exec.\n\n",
|
|
24
|
+
"level": 3
|
|
25
|
+
},
|
|
26
|
+
{
|
|
27
|
+
"title": "3. Blocking I/O in loops",
|
|
28
|
+
"content": "\n**`evolve.mjs:205-209`** — up to 100 serial `readFileSync` calls in the `--diagnose` path:\n```js\nfor (const f of readdirSync(runsDir).filter(f => f.endsWith('.json')).slice(0, 100)) {\n records.push({ id: ..., rec: JSON.parse(readFileSync(join(runsDir, f), 'utf-8')) });\n}\n```\n**Fix:**\n```js\nconst files = readdirSync(runsDir).filter(f => f.endsWith('.json')).slice(0, 100);\nconst records = (await Promise.all(files.map(async f => {\n try { return { id: f.replace(/\\.json$/, ''), rec: JSON.parse(await readFile(join(runsDir, f), 'utf-8')) }; }\n catch { return null; }\n}))).filter(Boolean);\n```\n\n",
|
|
29
|
+
"level": 3
|
|
30
|
+
},
|
|
31
|
+
{
|
|
32
|
+
"title": "4. Memory leaks — doesn't apply",
|
|
33
|
+
"content": "\nThese are one-shot CLI processes that `process.exit()` at the end of `main()`. Checked and confirmed clean: every `setTimeout` (`_harness.mjs:149-152`, `_darwin.mjs:144-151`) has a matching `clearTimeout`, `AbortSignal` listeners use `{ once: true }`, and no module-level array/Map accumulates across calls. No finding here — not manufacturing one.\n\n",
|
|
34
|
+
"level": 3
|
|
35
|
+
},
|
|
36
|
+
{
|
|
37
|
+
"title": "5. Redundant computation — mostly doesn't apply",
|
|
38
|
+
"content": "Similarity/drift scoring (`_similarity.mjs`) is invoked exactly once per process, never re-run in a loop over a collection. The only real redundancy is the subprocess-spawn waste already covered in #1/#2 — not duplicate math.\n\n---\n\n**Summary table**\n\n| Issue | Location | Fix |\n|---|---|---|\n| N+1 subprocess spawns for 1 field | `drift-from-history.mjs:154` + `audit-list.mjs:102-117` | Add `--keys-only` list mode |\n| `npx -y -p` on every Darwin call | `_darwin.mjs:80,125` | Migrate to `ensureCachedInstall` + memoize |\n| Unpinned `@latest` CLI_PKG | 5 scripts (audit-list/trend, oia-audit, similarity, drift-from-history) | Pin + route through `_invoke.mjs` cache |\n| Extra spawn for degraded probe | `mint.mjs:68` | Use in-process `metaharnessResolution()` |\n| Serial `readFileSync` ×100 | `evolve.mjs:205-209` | `fs/promises` + `Promise.all` |\n\nWant me to implement any of these fixes? The `_darwin.mjs` migration and the `audit-list` `--keys-only` mode are the highest-value ones given ADR-150's cold-start budget constraints.",
|
|
39
|
+
"level": 3
|
|
40
|
+
}
|
|
41
|
+
],
|
|
42
|
+
"codeBlocks": [
|
|
43
|
+
{
|
|
44
|
+
"language": "js",
|
|
45
|
+
"code": "function memRetrieve(key) {\n const r = spawnSync('npx', [CLI_PKG, 'memory', 'retrieve', '--namespace', NS, '--key', key], ...);\n}\n...\nfor (const key of slice) {\n const rec = memRetrieve(key); // one npx spawn PER record, sequential\n}"
|
|
46
|
+
},
|
|
47
|
+
{
|
|
48
|
+
"language": "js",
|
|
49
|
+
"code": "const r = spawnSync('npx', ['-y', '-p', DARWIN_PIN, 'metaharness-darwin', ...argv], { ... });"
|
|
50
|
+
},
|
|
51
|
+
{
|
|
52
|
+
"language": "js",
|
|
53
|
+
"code": "const CLI_PKG = process.env.CLI_CORE === '1' ? '@claude-flow/cli-core@alpha' : '@claude-flow/cli@latest';"
|
|
54
|
+
},
|
|
55
|
+
{
|
|
56
|
+
"language": "js",
|
|
57
|
+
"code": "for (const f of readdirSync(runsDir).filter(f => f.endsWith('.json')).slice(0, 100)) {\n records.push({ id: ..., rec: JSON.parse(readFileSync(join(runsDir, f), 'utf-8')) });\n}"
|
|
58
|
+
},
|
|
59
|
+
{
|
|
60
|
+
"language": "js",
|
|
61
|
+
"code": "const files = readdirSync(runsDir).filter(f => f.endsWith('.json')).slice(0, 100);\nconst records = (await Promise.all(files.map(async f => {\n try { return { id: f.replace(/\\.json$/, ''), rec: JSON.parse(await readFile(join(runsDir, f), 'utf-8')) }; }\n catch { return null; }\n}))).filter(Boolean);"
|
|
62
|
+
}
|
|
63
|
+
]
|
|
64
|
+
},
|
|
65
|
+
"rawOutputPreview": "## Performance Analysis: `plugins/ruflo-metaharness`\n\nThis directory has no React and no database/ORM layer — it's ~30 standalone Node.js `.mjs` CLI scripts that shell out to `npx ruflo`/`@claude-flow/cli`, read/write JSON, and call memory/AgentDB APIs. So I mapped your five categories onto their actual Node-CLI analogues rather than inventing web-app issues. Two categories (React re-renders, classic SQL N+1) genuinely don't apply here — noted explicitly below instead of stretched to fit.\n\n### 1. N+1-shaped pattern (subprocess-per-item, the real analogue of N+1 queries)\n\n**`audit-list.mjs:56-65, 102-117`** — lists audit keys with one spawn, then does a **second spawn per key** to hydrate each record:\n```js\nfunction memRetrieve(key) {\n const r = spawnSync('npx', [CLI_PKG, 'memory', 'retrieve', '--namespace', NS, '--key', key], ...);\n}\n...\nfor (const key of slice) {\n const rec = memRetrieve(key); // one npx spawn PER record, sequential\n}\n```\n**`drift-from-history.mjs:154, 234-236`** compounds this: its default path (no `--baseline-key`/`--baseline-file`) calls `audit-list` with `--limit 50`, hydrating **50 full records**, sorts them, then uses only `sorted[0]` — 49 wasted subprocess spawns to extract one field. The file's own comments put per-subprocess cost at ~2-5s, so this is minutes of wasted time on the documented default UX.\n\n**Fix:** add a `--keys-only` mode to `audit-list.mjs` that returns from `memList()` without calling `memRetrieve` at all, and have `drift-from-history.mjs`'s default path use that (it only needs the newest key). For the general case, parallelize with `Promise.all` over an async spawn instead of serial `spawnSync`.\n\n### 2. Redundant subprocess spawns / missing caching\n\n**`_darwin.mjs:80,125`** never got migrated to the cached-install pattern the rest of the family uses:\n```js\nconst r = spawnSync('npx', ['-y', '-p', DARWIN_PIN, 'metaharness-darwin', ...argv], { ... });\n```\n`_harness.mjs`'s own header documents *why* this is expensive — `n",
|
|
66
|
+
"rawOutputLength": 5580
|
|
67
|
+
}
|