control-arm 1.0.0 → 1.2.0

package/CHANGELOG.md

@@ -0,0 +1,44 @@
+# Changelog
+
+All notable changes to this project are documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this
+project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [1.2.0] - 2026-10-06
+
+> **Note on the missing 1.1.0.** `package.json` was bumped to `1.1.0` on 2026-09-25 but that
+> version was never tagged in git, never published to npm, and never released on GitHub —
+> npm's `latest` stayed on `1.0.0` until now. The `1.1.0` work (vitest and jest runners, the
+> findings-contract v1 envelope, the child-process reaper, and the fixes that shipped with
+> them) is folded into this release.
+
+### Fixed
+
+- The module-identity proof silently did nothing on Node < 20.6: `import.meta.resolve` is
+  undefined there, and the per-specifier error was swallowed as "genuinely missing", so every
+  verdict was "proven" without resolving a single import. The probe now reports when it
+  cannot resolve, and the verdict is withheld instead of trusted.
+- The findings-contract `version` field was hardcoded to `1.1.0`; it is now read from
+  `package.json` so it cannot drift.
+
+### Changed
+
+- Minimum Node version raised to `20.6.0` (was `18`).
+
+### Added
+
+- `scripts/precision.mjs`: measure the precision (and recall, when the sample is labelled)
+  of the `BLIND` verdict against a hand-labelled corpus.
+- Unit coverage for the vitest/jest report parser (`parseReport`) against captured runner
+  output.
+
+### Removed
+
+- Dead code: the tautological `nodeTest.matches` predicate, the unused `RUNNERS` export,
+  `isFileLevelFailure`, and a leftover destructure in `commitInfo`.
+
+## [1.0.0] - 2026-09-25
+
+- Initial public release: `ca doctor` / `ca verify` / `ca audit` over `node:test`, published
+  to npm and the GitHub Marketplace (`mekanhan/control-arm@v1`).

package/DESIGN.md

@@ -109,11 +109,19 @@ weighting: a codebase the author did not write, did not pick for a flattering re
 could not tune against. 190 commits matched the subject filter; 99 ship both a test and a
 source change; 40 were drawn from those.
 
-**One usability trap found while running it.** `audit` defaults to a six-month window. On
-a mature repo that silently shrinks the sample — the first dayjs run judged **5 commits
-instead of 40** and printed a confident, meaningless 100%. The window *is* echoed in the
-header line, so it is visible rather than hidden, but a `--n 40` that quietly returns 5
-deserves a louder signal. Pass `--since` explicitly on any repo older than six months.
+**One usability trap found while running it.** `audit` defaults to a **twelve-month**
+window, and dayjs ships few `fix:` commits in a year that also touch a test. The first
+dayjs run judged **5 commits instead of 40** and printed a confident, meaningless 100%.
+Nothing was broken: the pool was simply smaller than the request.
+
+What exposed it was that two different `--n` values and two different seeds gave
+**byte-identical output**. A rate that does not move when you change the sample size is
+not a rate.
+
+`audit` now says so when the draw comes up short, naming whichever cause applies — the
+window, or the test-plus-source pre-filter — and never blaming a `--since` the caller
+passed themselves (`src/sample-warning.mjs`, `SAMPLE-001..006`). Pass `--since` explicitly
+on any repo with more than a year of history.
 
 `INCONCLUSIVE` is 29% and 40% respectively. That is the honest denominator, not a rounding
 error: a fix that adds an export its test imports cannot be replayed against the parent,
@@ -122,22 +130,30 @@ either way.
 
 ## How often is this tool wrong?
 
-Ask any measuring instrument this. Here is the answer for this one, on the 300-commit run.
-
-The raw headline was **13 `BLIND` commits**. After running arm C on each and checking them
-by hand, **5 were real**:
-
-| | |
-|---|---|
-| 13 | raw `BLIND` |
-| −3 | `↻ REPAIRED SINCE` — the gap was closed after that commit |
-| −2 | `? cannot tell` — the test file no longer exists at HEAD |
-| −3 | category errors: two `fix(test):` (the bug WAS the test) and one build failure no unit test can catch |
-| **5** | genuinely open, **2.3% of answerable commits** |
-
-**A 62% false-positive rate on the raw number.** Arm C and the corpus filter exist because
-of it. Publish the raw count and you hand someone thirteen tickets, eight of which waste
-their afternoon.
+**[README.md → "How often is the TOOL right?"](README.md#does-it-work)** holds the audit:
+13 raw `BLIND` on the 300-commit run, 5 genuinely open after arm C, a 62% false-positive
+rate on the raw number.
+
+It lives there and not here because it is the first thing a skeptic should see, and because
+a table kept in two places drifts in one of them. What belongs here is the design
+consequence:
+
+**Arm C is not a nicety, it is the reason the raw signal is publishable at all.** Eight of
+those thirteen were not defects in anybody's tests — three gaps had been closed by a later
+commit, two named test files no longer exist, two were `fix(test):` commits where the bug
+*was* the test, and one was a build failure no unit test could catch. None of that is
+visible from the two arms alone; all of it needs a third look at HEAD.
+
+That is also why the corpus filter drops `fix(test):` by default, and why
+`--include-test-fixes` exists for anyone who wants them back.
+
+**Measuring precision properly.** "5 for 5, too small to quote" is a count, not a rate.
+To turn it into a rate — precision, and recall when the whole sample is labelled — label
+each judged commit `gap`/`not-gap` and run `scripts/precision.mjs`. It reads `ca audit
+--out` plus the labels, and prints the confusion matrix for both the raw `BLIND` signal and
+the post-arm-C "still open" signal. Until that labelled corpus exists, the defensible claim
+is the one the README already makes: the raw signal is over-sensitive, the filters remove
+the eight known false positives, and the residual precision is unmeasured.
 
 ## Every bug found in this tool so far
 

package/README.md

@@ -40,7 +40,7 @@ because it runs the new test against the old code.
 npm i -g control-arm     # or run it from a clone: node bin/ca.mjs
 ```
 
-Node 18+. No other dependencies.
+Node 20.6+. No other dependencies.
 
 ## Use it
 
@@ -87,9 +87,82 @@ It comments on the PR with a line per test. It does **not** fail the build by de
 PR can legitimately ship only regression guards, and a gate that fires on those gets
 switched off within a week.
 
+**On a feature PR it refuses to claim anything.** This matters, because the check is
+meaningless there and would otherwise read as proof. On a feature the base is code where
+the thing does not exist yet, so essentially *any* test touching it fails —
+`expected 0 to be greater than 0` is a real assertion failure that says nothing about
+whether the test is well aimed. So the comment changes its own headline:
+
+> ### `control-arm` — new code — these tests cannot be judged this way
+>
+> **3 of 4 tests fail without this change** — but this is a feature, so the base does not
+> have it at all.
+>
+> That is expected, and it is weak evidence: on new code almost any test that reads the new
+> thing fails on the base, whether or not it is well aimed. A test asserting `1 === 1` in
+> the same file would NOT show up here; one that merely reads the new field would.
+
+The classification is `fix`/`feat` from the conventional-commit prefix, plus a supporting
+hint — a commit that only *added* source lines and deleted none looks like new code
+regardless of its prefix. Both were measured on a real corpus before being trusted: the
+prefix is used consistently (1,213 `fix` / 988 `feat`), while additions-only fires on 9 of
+28 features but also on 1 of 35 fixes — **specific, not sensitive**, so it qualifies the
+claim and never decides the verdict on its own. A `CAUGHT` on a feature is still `CAUGHT`;
+it just does not get to say it would have caught a bug, because there was no bug.
+
+## All flags
+
+```bash
+ca doctor  --repo .                       # can this repo be measured?
+ca verify  <sha> --repo . --runs 3 --timeout 120000 --work .ca-work
+ca verify  <sha> --against origin/main --pr-comment --fail-on-blind
+ca audit   --n 100 --since '6 years' --grep '^fix' --seed 1 --out r.csv --html r.html
+ca issues  --apply --include-test-fixes
+```
+
+| flag | |
+|---|---|
+| `--repo <path>` | the repository to measure (default: cwd) |
+| `--against <ref>` | compare against a base ref — turns `verify` into a PR check |
+| `--runs <n>` | repeat each case N times to expose flakes |
+| `--timeout <ms>` | per-run timeout |
+| `--work <path>` | where the throwaway worktrees go |
+| `--keep` | leave them behind for inspection |
+| `--n <n>` · `--since <when>` · `--grep <re>` · `--seed <n>` | which commits `audit` samples |
+| `--include-test-fixes` | do not skip `fix(test):` commits, where the bug WAS the test |
+| `--out <file>` · `--html <file>` | write the audit as CSV or HTML |
+| `--pr-comment` | print the PR comment markdown and nothing else |
+| `--apply` | `issues` actually files them, rather than printing |
+| **`--json`** | **findings-contract v1 envelope on stdout** (see below) |
+| `--json <path>` | the older behaviour: write the audit summary to a file |
+| **`--fail-on-blind`** | **exit 1 when a commit is BLIND** |
+
+### The findings contract
+
+Bare `--json` emits one [findings-contract v1](https://github.com/mekanhan/findings-contract)
+object on stdout and nothing else — progress goes to stderr, so it pipes. It works on
+`doctor`, `verify` and `audit`.
+
+The mapping is not one-to-one, and the interesting part is what is **not** a finding:
+
+| | |
+|---|---|
+| commit `CAUGHT` | no finding. Nothing is wrong |
+| commit `BLIND` | a finding, severity `warn` |
+| commit `INCONCLUSIVE` | **not** a finding — it goes to `skipped`, with its reason |
+| case `NON-DISCRIMINATING` | never a finding. A fact about a guard, not a fault |
+| case `FLAKY` | a finding. Runs disagreed, which is a real defect |
+
+**Exit codes:** `0` ran cleanly · `1` found blockers · `2` the tool itself failed.
+
+`ca verify` used to exit `1` on `BLIND`. It no longer does unless you pass
+`--fail-on-blind`, because `BLIND` is a `warn` — a test that cannot catch a bug breaks
+nothing today — and putting a warning into an exit code makes every caller guess. The
+GitHub Action is unaffected: it reads the verdict from stdout.
+
 ## What you get
 
-> ### `control-arm` — this branch is proven
+> ### `control-arm` — 1 test here fails without this change
 >
 > **1 of 5 tests here genuinely catch this change.** Run against the code as it was before
 > this PR, they fail; with the change, they pass.
@@ -121,41 +194,149 @@ measurement that cannot tell zero from failure is not a measurement.
 
 ## Which of your tests it looks at
 
-It does not care what *kind* of test it is, only whether the runner can execute the file.
+**Unit and integration tests** — anything that runs in-process, without a deployed
+application. That is the whole scope.
 
-| kind | covered | why |
-|---|---|---|
-| unit tests | yes | the easy case |
-| backend / server logic | yes | same runner, same rewind |
-| database-backed tests | yes | needs a live DB, else they report `SKIPPED` |
-| component tests (React / RN) | yes | via vitest and jest |
-| browser e2e (Playwright) | **no** | no Playwright runner yet |
-| performance / load | **no** | they measure speed, not correctness |
-| manual QA | **no** | nothing to execute |
+**Every row carries what proves it.** A row without evidence beside it is an aspiration, and
+that is where this table was wrong before: vitest and jest were listed as covered because the
+code has adapters for them, not because either had ever been run.
+
+| runner | proven by |
+|---|---|
+| `node:test` | fixtures `01`–`12`, built and run by `npm test` on every push |
+| vitest | observed on a real commit, 2026-09-25 — **no fixture yet** (#21) |
+| jest | observed on a real commit, 2026-09-25 — **no fixture yet** (#21) |
+
+A database-backed test is judged like any other, and needs its database present. The README
+used to claim it reports `SKIPPED` without one; that has never actually been observed, so the
+claim is withdrawn until it is (#21).
+
+**"fixtures"** means anyone can re-run it: `npm test` builds throwaway repos where the right
+answer is known by construction, and the suite fails if this tool cannot tell
+`02-blind-direction` from `01-caught-value`. **"observed on a real commit"** means it worked
+once, on a repository you cannot see — weaker, and marked weaker.
+
+### Out of scope, permanently
+
+Browser e2e (Playwright), device e2e (WDIO/Appium) and load tests (k6).
+
+Arm B has to run your test against the **old code**, and for an e2e test the old code is a
+*running application* — built and served at the parent commit, with its database and its
+services. That is minutes per commit at best, and frequently the parent will not boot at all.
+Load tests measure speed rather than correctness, so the question this tool asks does not
+apply to them.
+
+Point it at one of those and it will name the runner and decline, rather than attempting it
+with the wrong one. That is a courtesy, not a roadmap — none of these is planned.
 
 ## Does it work?
 
-Three repositories, random draws, seeds recorded so the samples are reproducible.
+**Two different questions live here, and mixing them flatters the tool.** How good are the
+tests it measured, and how often is the tool itself right? Only the second is about
+`control-arm`.
+
+### How often is the TOOL right?
+
+This is the number to judge it on. On the 300-commit run it raised **13 `BLIND` verdicts**
+— "not one test in this commit would have caught the bug". Each was then re-run through
+arm C and checked by hand:
+
+| | |
+|---|---|
+| 13 | raw `BLIND` |
+| −3 | `↻ REPAIRED SINCE` — the gap was real, and closed after that commit |
+| −2 | `? cannot tell` — the test file no longer exists at HEAD |
+| −3 | category errors: two `fix(test):` (the bug WAS the test) and one build failure no unit test can catch |
+| **5** | genuinely open — **2.3% of answerable commits** |
+
+**A 62% false-positive rate on the raw number.** Publish it and you hand someone thirteen
+tickets, eight of which waste their afternoon. Arm C and the corpus filter exist entirely
+because of that, and they run by default.
+
+All five survivors held up under hand inspection. **That is 5 for 5 out of 13 candidates —
+far too small a sample to quote as a precision rate**, and it is stated as a count for that
+reason. The honest summary is: the raw signal is badly over-sensitive, the filters remove
+eight of eight known false positives, and what precision remains after them has not been
+measured on a sample large enough to have a rate.
+
+`BLIND` is the only verdict that accuses anybody, so every ambiguity resolves away from it.
+
+### How good were the TESTS it measured?
+
+These say nothing about the tool's accuracy — they are a property of the repositories.
+Counts, not just percentages, because a percentage over 25 commits invites arithmetic
+nobody should have to do themselves.
 
 | | a private monorepo | nodejs/undici | iamkun/dayjs |
 |---|---|---|---|
 | fix commits sampled | 300 | 25 | 40 |
 | answerable | 214 | 15 | 25 |
-| **CAUGHT** | **93.9%** | **80.0%** | **92.0%** |
-| BLIND | 6.1% | 20.0% | 8.0% |
+| **CAUGHT** | **201 — 93.9%** | **12 — 80.0%** | **23 — 92.0%** |
+| BLIND | 13 — 6.1% | **3** — 20.0% | **2** — 8.0% |
+| INCONCLUSIVE | 86 | 10 | 9 |
 | runtime | 7.1 s/commit | 24.9 s/commit | 1.4 s/commit |
 
-**The dayjs column is the one to weigh.** It is a codebase the author did not write, did
-not choose for a flattering result, and could not tune the tool against — 190 fix commits
-matched the filter, 99 ship both a test and a source change, 40 drawn at random with the
-seed recorded. It lands within a point of the private monorepo it was built on.
+**The small columns are small.** dayjs's 8% is **two commits** and undici's 20% is **three**.
+Those are proof-of-concept numbers; do not read a difference between 80% and 92% as a
+difference between the repositories.
+
+**The dayjs column is the one to weigh** anyway: a codebase the author did not write, did
+not choose for a flattering result, and could not tune against — 190 fix commits matched
+the filter, 99 ship both a test and a source change, 40 drawn at random with the seed
+recorded. It lands within a point of the private monorepo it was built on.
+
+### Why is so much unanswerable?
+
+29% and 40% is a lot to exclude, so here is where it goes. `INCONCLUSIVE` is never folded
+into the other columns — a measurement that cannot tell zero from failure is not a
+measurement — and there are exactly four ways to earn it:
+
+| cause | what happened |
+|---|---|
+| **the test will not load on the parent** | the dominant one. The fix *added* an export, a module, a fixture; the test imports it; on the parent that import throws before a single assertion runs. `SyntaxError: does not provide an export named …` is not a test failing, it is a test never starting |
+| **the test is not green on the fix either** | no before/after to compare. Usually an environment gap — a database, a browser, a missing `.env` |
+| **module identity unprovable** | the run could not be shown to have loaded the worktree's code rather than the real repo's. Refused rather than guessed |
+| **a skipped case is present** | the skipped one might have been the discriminating one, so the commit cannot be called `BLIND` |
+
+The breakdown *between* these four has not been counted per repository, so no split is
+quoted here.
+
+## Working on this repo
+
+```bash
+git config core.hooksPath .githooks   # once per clone; worktrees inherit it
+```
+
+`pre-push` refuses three things CI can only tell you about after the fact: a direct push to
+`main` (everything lands through a PR), a branch **named** like a default that is not this
+repo's default, and a push from a base that has already moved.
+
+The middle one is not hypothetical. A local `master` once sat ten commits behind `main`
+while `main` moved on through four PRs; pushing it created a parallel remote branch, and the
+Node 20/22/24 matrix came back green against the wrong base. The only tell was a line of
+push output — `* [new branch] master -> master` — on a repo that is anything but new.
+
+If a push prints none of that hook's output, it is not armed. Check `core.hooksPath` before
+trusting anything it did not say.
+
+## Prior art
+
+The fail-before / pass-after check is not new, and it is worth saying who got there first:
+
+- **[Defects4J](https://github.com/rjust/defects4j)** records, for each reproducible bug,
+  the **trigger tests** that fail on the buggy version and pass on the fixed one.
+- **[SWE-bench](https://www.swebench.com/)** builds every task around **`FAIL_TO_PASS`**
+  tests, with `PASS_TO_PASS` as the regression guard — the same two arms.
 
-Of the 13 raw `BLIND` verdicts in that 300-commit run, **5 were genuinely open** — a 62%
-false-positive rate on the raw number, which is why the filters exist.
+Both use the property to **construct benchmarks**: they start from a known bug and keep the
+tests that prove it. `control-arm` runs the same check in the other direction — on ordinary
+commits, in CI, where the answer is not known in advance and a `BLIND` result is news
+rather than a data-cleaning step.
 
-**[DESIGN.md](DESIGN.md) has the rest**: how the rewind works, the three traps that
-separate an instrument from a random number generator, the full accuracy audit, and every
-bug found in this tool so far — none of which were found by its own test suite.
+The consequence of that difference is most of this repository: a benchmark can discard
+anything ambiguous, because it only needs *some* clean examples. A CI check cannot discard
+the commit in front of it, so it has to be able to say `INCONCLUSIVE` out loud, and it has
+to be right when it says `BLIND` about somebody's work.
 
 ## Scope, stated plainly
 

package/bin/ca.mjs

@@ -16,11 +16,28 @@ import { renderVerify, renderAudit, MARK } from '../src/report.mjs';
 import { renderHtml, issueBody } from '../src/html-report.mjs';
 import { prComment } from '../src/markdown-report.mjs';
 import { analyseCase, extractCase } from '../src/assertions.mjs';
+import { COMMANDS, COMMAND_NAMES } from '../src/cli-spec.mjs';
+import { sampleWarning, relativeWindowWarning } from '../src/sample-warning.mjs';
+import { verifyEnvelope, auditEnvelope, doctorEnvelope } from '../src/contract.mjs';
 
 const argv = process.argv.slice(2);
 const cmd = argv[0];
 const flag = (n, d = null) => { const i = argv.indexOf(`--${n}`); return i === -1 ? d : argv[i + 1]; };
 const has = n => argv.includes(`--${n}`);
+// `--json` predates the contract and took a PATH. Bare `--json` — no value, or the next
+// token is another flag — now means "contract envelope on stdout", which is what C-001
+// asks for. `--json <path>` is unchanged, so nothing that worked stops working.
+const bareJson = (() => {
+    const i = argv.indexOf('--json');
+    if (i === -1) return false;
+    const next = argv[i + 1];
+    return next === undefined || next.startsWith('-');
+})();
+/** C-001: one object on stdout, nothing else. Then C-007 decides the code. */
+const emit = (env, gate = false) => {
+    process.stdout.write(JSON.stringify(env, null, 2) + '\n');
+    process.exit(gate && env.summary.blocker ? 1 : 0);
+};
 const repo = path.resolve(flag('repo', process.cwd()));
 const workDir = path.resolve(flag('work', path.join(repo, '.ca-work')));
 
@@ -59,6 +76,8 @@ async function doctor() {
         checks.push({ ok: true, name: 'workspaces', detail: `${[].concat(pkg.workspaces).join(', ')} — workspace links will be re-pointed into the worktree (this is the trap that produces false BLIND)` });
     }
 
+    if (bareJson) emit(doctorEnvelope(checks, { repo }), true);
+
     const w = 22;
     console.log(`\n  ca doctor — ${repo}\n`);
     for (const c of checks) console.log(`  ${c.ok ? MARK.ok : MARK.no} ${c.name.padEnd(w)} ${c.detail}`);
@@ -74,10 +93,18 @@ async function verify() {
     const r = await verifyCommit({ repo, workDir, sha, against: flag('against'), runs, timeoutMs: Number(flag('timeout', 120_000)),
         onStep: s => process.stderr.write(`\r  … ${s}      `) });
     process.stderr.write('\r' + ' '.repeat(40) + '\r');
+    if (bareJson) { if (!has('keep')) await removeWorktrees(repo, workDir); emit(verifyEnvelope(r, { repo, sha })); }
     if (has('pr-comment')) console.log(prComment(r, { repoName: flag('repo', '.') }));
     else console.log(renderVerify(r));
     if (!has('keep')) await removeWorktrees(repo, workDir);
-    process.exit(r.verdict === BLIND ? 1 : 0);
+
+    // C-007. This used to exit 1 on BLIND unconditionally, which made a `warn` look like
+    // a blocker and put this tool's own opinion into an exit code every caller has to
+    // interpret. Gating is opt-in now, under the same name the Action already uses.
+    // BEHAVIOUR CHANGE: `ca verify` on a BLIND commit exits 0 unless --fail-on-blind.
+    // The Action is unaffected — it reads the verdict from stdout and already wraps the
+    // call in `|| true`.
+    process.exit(has('fail-on-blind') && r.verdict === BLIND ? 1 : 0);
 }
 
 async function audit() {
@@ -115,7 +142,17 @@ async function audit() {
     const pool = [...eligible];
     for (let i = pool.length - 1; i > 0; i--) { const j = Math.floor(rand() * (i + 1)); [pool[i], pool[j]] = [pool[j], pool[i]]; }
     const sample = pool.slice(0, n);
-    process.stderr.write(`  drawing ${sample.length} at random (seed ${seed})\n\n`);
+    process.stderr.write(`  drawing ${sample.length} at random (seed ${seed})\n`);
+
+    // A draw that came up short is the difference between a measurement and a number.
+    const moving = relativeWindowWarning(since, argv.includes('--seed'));
+    if (moving) process.stderr.write(`\n${moving}\n`);
+
+    const short = sampleWarning({
+        requested: n, matched: candidates.length, eligible: eligible.length,
+        drawn: sample.length, since, sinceWasExplicit: argv.includes('--since'),
+    });
+    process.stderr.write(short ? `\n${short}\n\n` : '\n');
 
     const results = [];
     const t0 = Date.now();
@@ -130,6 +167,8 @@ async function audit() {
     }
     process.stderr.write('\r' + ' '.repeat(120) + '\r');
 
+    if (bareJson) emit(auditEnvelope(results, { repo, since, n: sample.length, seed }));
+
     console.log(renderAudit(results, { since, n: sample.length, eligible: eligible.length, matched: candidates.length, seed, seconds: (Date.now() - t0) / 1000 }));
 
     const htmlOut = flag('html');
@@ -180,6 +219,7 @@ async function audit() {
             caught_pct: answerable ? Number(((results.filter(r => r.verdict === CAUGHT).length / answerable) * 100).toFixed(1)) : null,
             still_open: byStatus('open'),
             repaired_since: byStatus('repaired'),
+            likely_repaired_elsewhere: byStatus('repaired-elsewhere'),
             cannot_tell: results.filter(r => r.verdict === BLIND && (!r.stillOpen || r.stillOpen.status === 'unknown'))
                 .map(r => ({ sha: r.sha, short: r.short, subject: r.subject })),
         }, null, 2));
@@ -229,14 +269,21 @@ async function issues() {
     if (!has('keep')) await removeWorktrees(repo, workDir);
 }
 
-const table = { doctor, verify, audit, issues };
+// Built from the spec, not written out again here. `test/readme.test.mjs` checks the
+// docs against COMMANDS, so a command that exists only in this file would make that
+// check reject a command that genuinely works.
+const impl = { doctor, verify, audit, issues };
+const missing = COMMAND_NAMES.filter(n => !impl[n]);
+if (missing.length) throw new Error(`cli-spec names commands with no implementation: ${missing}`);
+const table = Object.fromEntries(COMMAND_NAMES.map(n => [n, impl[n]]));
+
 if (!table[cmd]) {
     console.log(`
   ca — does a test actually fail on the code it was written to catch?
 
-    ca doctor                                  can this repo be measured?
-    ca verify <commit> [--runs 3]              one commit, per-case verdicts
-    ca audit --n 100 [--since '6 months'] [--grep '^fix'] [--seed 1] [--out r.csv]
+    ca doctor                                  ${COMMANDS.doctor}
+    ca verify <commit> [--runs 3]              ${COMMANDS.verify}
+    ca audit --n 100 [--since '6 years'] [--grep '^fix'] [--seed 1] [--out r.csv]
 
   common:  --repo <path>   --timeout <ms>   --keep (leave worktrees for inspection)
 `);

package/package.json

@@ -1,17 +1,22 @@
 {
   "name": "control-arm",
-  "version": "1.0.0",
+  "version": "1.2.0",
   "description": "Does a test actually fail on the code it was written to catch?",
   "type": "module",
-  "bin": { "ca": "./bin/ca.mjs" },
+  "bin": {
+    "ca": "./bin/ca.mjs"
+  },
   "files": [
     "bin",
     "src",
     "README.md",
+    "CHANGELOG.md",
     "DESIGN.md",
     "LICENSE"
   ],
-  "engines": { "node": ">=18" },
+  "engines": {
+    "node": ">=20.6.0"
+  },
   "keywords": [
     "testing",
     "test-quality",
@@ -26,7 +31,9 @@
     "url": "git+https://github.com/mekanhan/control-arm.git"
   },
   "homepage": "https://github.com/mekanhan/control-arm#readme",
-  "bugs": { "url": "https://github.com/mekanhan/control-arm/issues" },
+  "bugs": {
+    "url": "https://github.com/mekanhan/control-arm/issues"
+  },
   "scripts": {
     "test": "node --test test/*.test.mjs",
     "fixtures": "node fixtures/build.mjs"

package/src/assertions.mjs

@@ -245,6 +245,39 @@ function titleMatches(lit, caseName) {
     try { return new RegExp('^' + pattern + '$').test(caseName); } catch { return false; }
 }
 
+/**
+ * The opening brace of the CASE BODY — skipping an options object if one is there.
+ *
+ * `node:test`, vitest and jest all accept a middle argument:
+ *
+ *     test('name', { skip: SKIP },      () => { …assertions… })
+ *     test('name', { timeout: 5000 }, async () => { … })
+ *
+ * Taking the first `{` after the title grabs `{ skip: SKIP }` and analyses THAT as the
+ * body. It holds no assertions, so every such case was reported as
+ * "the case body contains no assertion at all" — about tests that are full of them.
+ *
+ * FOUND IN THE WILD, on five tests holding eleven assertions between them. The VERDICT
+ * was not affected — that comes from red/green across the two arms — but the REASON was,
+ * and a wrong reason attached to a BLIND verdict sends someone to look at the wrong
+ * thing. That is the same failure this tool exists to prevent, one level up.
+ */
+function bodyBraceAfter(source, from) {
+    let i = from;
+    // Walk past `,` and whitespace, stepping over any balanced `{...}` met before the
+    // callback. An options object is the only thing that can legally appear there.
+    for (let guard = 0; guard < 4; guard++) {
+        while (i < source.length && /[\s,]/.test(source[i])) i++;
+        if (source[i] !== '{') break;
+        const end = matchBrace(source, i);
+        if (end === -1) return -1;
+        i = end + 1;
+    }
+    // Now at the callback — `() => {`, `async () => {`, `function () {`. Its body is the
+    // next brace, with no options object left to confuse it with.
+    return source.indexOf('{', i);
+}
+
 function extractExact(source, caseName) {
     CALL.lastIndex = 0;
     let m;
@@ -253,7 +286,10 @@ function extractExact(source, caseName) {
         const lit = readLiteral(source, qi);
         if (!lit) continue;
         if (!titleMatches(lit, caseName)) continue;
-        const open = source.indexOf('{', lit.end);
+        // +1 because readLiteral returns the index OF the closing quote, not after it.
+        // Starting on the quote made the options-object skip a no-op, which is how this
+        // fix silently did nothing on its first attempt.
+        const open = bodyBraceAfter(source, lit.end + 1);
         if (open === -1) continue;
         const close = matchBrace(source, open);
         // A failed brace scan on ONE site must not abandon the search — a later site may

package/src/children.mjs

@@ -0,0 +1,55 @@
+/**
+ * Reap the test processes when this process goes away.
+ *
+ * A machine running these audits was found carrying ten stray `node --test` processes,
+ * seven of them TWO DAYS old, each holding a worktree and file descriptors open. They sat
+ * at 0% CPU, which is why nothing noticed: the tool looked idle rather than leaky.
+ *
+ * The cause is NOT the timeout — that path already killed what it spawned. It is the audit
+ * being killed itself. Observed directly:
+ *
+ *     before   1242399  ppid 1242397   node --test hangs.test.mjs
+ *     (kill the parent)
+ *     after    1242399  ppid 2099      node --test hangs.test.mjs   <- survived
+ *
+ * An audit runs for the better part of an hour, so it gets interrupted often — and each
+ * interruption stranded whatever was mid-run. `detached: true` alone makes this WORSE,
+ * because a detached child is meant to outlive its parent. So the spawn stays detached (to
+ * get a killable process GROUP for runners that fork workers) and every live group is
+ * registered here, to be swept when this process ends however it ends.
+ */
+const live = new Set();
+
+/** Kill a process group, tolerating one that has already gone. */
+export function killGroup(pid) {
+    try { process.kill(-pid, 'SIGKILL'); } catch { /* already gone, or never grouped */ }
+}
+
+export function register(pid) { live.add(pid); }
+export function unregister(pid) { live.delete(pid); }
+export function liveCount() { return live.size; }
+
+export function reapAll() {
+    for (const pid of live) killGroup(pid);
+    live.clear();
+}
+
+let armed = false;
+/**
+ * Idempotent, and deliberately not installed at import time: importing a module should not
+ * change how the process handles signals. The runners call this the first time they spawn.
+ */
+export function armReaper() {
+    if (armed) return;
+    armed = true;
+    process.on('exit', reapAll);
+    for (const sig of ['SIGINT', 'SIGTERM', 'SIGHUP']) {
+        process.on(sig, () => {
+            reapAll();
+            // Re-raise with the handler removed, so the exit code still says "signalled"
+            // rather than pretending this was a clean exit.
+            process.removeAllListeners(sig);
+            process.kill(process.pid, sig);
+        });
+    }
+}