testguard-cli 0.10.4 → 0.11.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -7,6 +7,40 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
7
7
 
8
8
  ## [Unreleased]
9
9
 
10
+ ## [0.11.0] - 2026-09-21
11
+
12
+ Automated weekly release — everything merged since `v0.10.4`.
13
+
14
+ ### Changed
15
+
16
+ - feat(sweep): findings on a repository that has no claims yet (#151)
17
+ - chore(homebrew): sha256 for v0.10.4 (#149)
18
+
19
+
20
+ ### Added
21
+
22
+ - `testguard sweep [dir] --changed <ref>` — the cold start. Proposes faults for
23
+ the changed source files that carry no claim, probes a bounded selection, and
24
+ reports what a green suite did not notice. Never writes
25
+ `testguard.claims.json`; its evidence never replaces
26
+ `.testguard/evidence.json`; only `survived` and `nocover` exit 1, so a
27
+ proposal the tool could not anchor or compile is reported and never gates.
28
+ Offline, like every other verifying command.
29
+ - Selection follows Google's mutation service: at most 7 × |files| candidates,
30
+ spread round-robin across files, ordered on the measured productivity of the
31
+ fault class and shrunk toward a neutral prior so one observation never
32
+ dominates (`src/supply/select.mjs`).
33
+ - `persistence-payload-unasserted` — why a survivor on a write path was missed:
34
+ the defender mocks the persistence layer and the file's call assertions never
35
+ name an exact payload, so a field dropped from a write cannot fail them.
36
+ Descriptive, never predictive; attached only to a survivor on a write path
37
+ (`src/supply/persistence.mjs`).
38
+ - `urn:claimspec:v1:sweep` — the sweep document, with semantic rules that check
39
+ a cap never hides its own remainder.
40
+ - Five self-claims (8 faults) covering the cap arithmetic, the exit-code rule,
41
+ where the persistence signal may be attached, per-file cap fairness, and that
42
+ one assertion's arguments are read without reading the next one's.
43
+
10
44
  ## [0.10.4] - 2026-09-21
11
45
 
12
46
  Automated weekly release — everything merged since `v0.10.3`.
package/README.md CHANGED
@@ -127,9 +127,9 @@ raises coverage. TestGuard admits it because it fails when the claim is false.
127
127
  | npm | `npm i -D testguard-cli` then `npx testguard probe` |
128
128
  | pip | `pip install testguard-cli` then `testguard probe` (needs Node ≥ 20) |
129
129
  | Homebrew | `brew tap raccioly/tap && brew install testguard` |
130
- | GitHub Action | `uses: raccioly/testguard@v0.10.4` — see [`action.yml`](./action.yml) |
130
+ | GitHub Action | `uses: raccioly/testguard@v0.11.0` — see [`action.yml`](./action.yml) |
131
131
  | pre-commit | `repo: https://github.com/raccioly/testguard`, hooks `testguard-claims`, `testguard-probe` |
132
- | GitLab CI | `include: - remote: https://raw.githubusercontent.com/raccioly/testguard/v0.10.4/packaging/gitlab/testguard.gitlab-ci.yml` with `inputs:` — see [`packaging/gitlab/`](./packaging/gitlab/testguard.gitlab-ci.yml) |
132
+ | GitLab CI | `include: - remote: https://raw.githubusercontent.com/raccioly/testguard/v0.11.0/packaging/gitlab/testguard.gitlab-ci.yml` with `inputs:` — see [`packaging/gitlab/`](./packaging/gitlab/testguard.gitlab-ci.yml) |
133
133
 
134
134
  Projects that set `min-release-age` in `.npmrc` cannot see a version published
135
135
  less than that many days ago (`ENOVERSIONS`); install that one with
@@ -147,6 +147,7 @@ npx testguard-cli probe # try to falsify each claim; report
147
147
  npx testguard-cli admit test/auth.test.ts --claim AUTH-ADMIN # does this test satisfy the two-gate rule?
148
148
  npx testguard-cli baseline # freeze today's unproven findings; only new ones gate
149
149
  npx testguard-cli gate --changed origin/main # fail when a changed source file carries no claim
150
+ npx testguard-cli sweep --changed origin/main # no claims yet? propose faults for the changed files and report what nothing noticed
150
151
  npx testguard-cli brief --text # tell the agent where the suite is blind, before it writes
151
152
  npx testguard-cli replay --since v1.0..HEAD # would this suite have caught the bugs that escaped?
152
153
  npx testguard-cli mcp # serve the read-only loop over MCP, on stdio
@@ -400,6 +401,71 @@ on-demand helper, not a hook**: the session-start hook never touches the
400
401
  network, and the helper exits 0 with a message when the CLI or the artifact
401
402
  is missing. TestGuard itself still makes no network calls.
402
403
 
404
+ ### Starting from zero claims
405
+
406
+ `gate` names the changed files that carry no claim, and stops there —
407
+ correctly, because stating a claim is a human act. But a project adopting this
408
+ tool reads that list, has nothing to compare it against, and closes the tab.
409
+ The bottleneck was never verification; it is **oracle supply**.
410
+
411
+ `sweep` is the lowest rung of that supply. No claim, no concern, nothing
412
+ written by anybody:
413
+
414
+ ```bash
415
+ npx testguard-cli sweep --changed origin/main # propose, probe a bounded selection, report
416
+ npx testguard-cli sweep --changed HEAD --include-dirty --cap 20
417
+ ```
418
+
419
+ It takes the files `gate` just called uncovered, proposes faults with the same
420
+ mechanical producers `scaffold` uses, probes a bounded selection, and reports
421
+ what a green suite did not notice. A fault that survives needs no claim to be
422
+ alarming: something was deliberately broken and not one test failed.
423
+
424
+ ```
425
+ swept 13 of 14 unclaimed changed files against origin/main.
426
+ proposed 510 faults, probed 10 (cap 10), deferred 500.
427
+ The 500 deferred are not a verdict: raise --cap, or sweep a smaller change.
428
+
429
+ 9 findings — a deliberate break that no test noticed:
430
+
431
+ SURVIVED src/app/actions/register.ts:118 [write path]
432
+ [line 118] Field dropped: `password_hash` is no longer written.
433
+ src/tests/actions/register.test.ts mocks @/lib/prisma; 1 exact, 8 partial,
434
+ 31 argument-free call assertions in the file. A field dropped from the write
435
+ payload fails only against an assertion that names that field exactly.
436
+ Assert the whole object written by the call this fault changes, including
437
+ `password_hash` — or read the record back and assert on what was stored.
438
+ ```
439
+
440
+ Three rules make it safe to run on a repository that has never seen this tool:
441
+
442
+ - **It never writes `testguard.claims.json`**, and never will. A claim is a
443
+ sentence someone is willing to stand behind; a sentence nobody wrote is not
444
+ one. Its drafts land beside the evidence to keep or drop.
445
+ - **Its evidence never replaces `.testguard/evidence.json`.** A sweep probes
446
+ faults nobody stated, under TODO statements; folding that into the document
447
+ `status` and `baseline` read would corrupt the record of what the project
448
+ actually claims.
449
+ - **It does not fail on its own bad guess.** Only `survived` and `nocover`
450
+ exit 1. A proposal that would not compile, or an anchor that did not locate,
451
+ is reported and never gates — a tool that fails because its own guess was bad
452
+ is a tool people switch off.
453
+
454
+ The cap and the ordering are Google's: their mutation service surfaces at most
455
+ 7 × |files| mutants per change and orders candidates on the measured
456
+ productivity of their operator in similar context, which took their productive
457
+ rate from 15% to 89%
458
+ ([Petrović et al., *Practical Mutation Testing at Scale*, 2021](https://arxiv.org/abs/2102.11378)).
459
+ TestGuard's ordering prior is measured the same way — from probed faults on a
460
+ real AI-authored codebase — and shrunk toward a neutral value, so one
461
+ observation of a class surviving never dominates a sweep. The cap is spread
462
+ across files: a single file with many candidates cannot make the others look
463
+ clean.
464
+
465
+ A sweep is weaker evidence than a probe, and says so. Nobody stated that the
466
+ behaviour mattered. It is stronger than nothing, which is what a repository
467
+ with no claims has — and the survivors worth defending become the first claims.
468
+
403
469
  ### Every change needs a claim
404
470
 
405
471
  `probe` can only verify claims that exist. Every escaped defect in the field
@@ -481,12 +547,12 @@ detected base that does not resolve is a warning for `status` and `brief`
481
547
  # GitHub Actions
482
548
  - uses: actions/checkout@v4
483
549
  with: { fetch-depth: 0 }
484
- - uses: raccioly/testguard@v0.10.4
550
+ - uses: raccioly/testguard@v0.11.0
485
551
  with: { command: gate }
486
552
 
487
553
  # GitLab CI — the component-shaped template: gate + probe, brief as an artifact and, opted in, as a merge-request note
488
554
  include:
489
- - remote: 'https://raw.githubusercontent.com/raccioly/testguard/v0.10.4/packaging/gitlab/testguard.gitlab-ci.yml'
555
+ - remote: 'https://raw.githubusercontent.com/raccioly/testguard/v0.11.0/packaging/gitlab/testguard.gitlab-ci.yml'
490
556
  inputs: { dir: backend, post_note: true } # post_note needs TESTGUARD_GITLAB_TOKEN (api scope); one note, updated in place
491
557
  ```
492
558
 
@@ -824,7 +890,7 @@ verdicts.
824
890
  ## Status
825
891
 
826
892
  **v0.5.** Eleven commands (`status`, `init`, `claims`, `probe`, `admit`,
827
- `replay`, `baseline`, `brief`, `gate`, `scaffold`, `mcp`), vitest, jest,
893
+ `replay`, `baseline`, `brief`, `gate`, `scaffold`, `sweep`, `mcp`), vitest, jest,
828
894
  Playwright and Python (pytest / stdlib unittest) runners, hand-authored faults
829
895
  plus a mechanical scaffold for JavaScript and Python, an agent operating layer
830
896
  (`status`, `init`) and a change gate (`gate`). The contract spine — eight JSON
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "testguard-cli",
3
- "version": "0.10.4",
3
+ "version": "0.11.0",
4
4
  "description": "Breaks your code on purpose and reports every promise your tests did not notice breaking. Mutation testing bound to the claims a project makes — vitest, jest, Playwright, pytest and unittest.",
5
5
  "type": "module",
6
6
  "license": "MIT",
@@ -7,7 +7,7 @@ import { reproduces, decimals, MAX_PLACES } from './wilson.mjs';
7
7
 
8
8
  const schemaDir = join(dirname(fileURLToPath(import.meta.url)), '..', 'schemas');
9
9
 
10
- export const KINDS = Object.freeze(['claims', 'evidence', 'baseline', 'ignore', 'calibration', 'brief', 'status', 'gate', 'replay']);
10
+ export const KINDS = Object.freeze(['claims', 'evidence', 'baseline', 'ignore', 'calibration', 'brief', 'status', 'gate', 'replay', 'sweep']);
11
11
  /**
12
12
  * How a document says it tried to falsify its claims. Absent means
13
13
  * `fault-injection`, so every document written before the field existed is
@@ -305,6 +305,48 @@ const semantic = {
305
305
  return errors;
306
306
  },
307
307
 
308
+ sweep(doc) {
309
+ const errors = [];
310
+ const s = doc.selection;
311
+ // A cap that hides its remainder is a coverage claim nobody made, so the
312
+ // arithmetic that proves nothing was dropped is checked, not trusted.
313
+ if (s.proposed !== s.selected + s.deferred) errors.push({ path: '/selection/proposed', message: `proposed (${s.proposed}) must equal selected (${s.selected}) + deferred (${s.deferred})` });
314
+ if (s.selected > s.cap) errors.push({ path: '/selection/selected', message: `selected (${s.selected}) exceeds the cap (${s.cap})` });
315
+ if (doc.scope.swept > doc.scope.targets) errors.push({ path: '/scope/swept', message: `swept (${doc.scope.swept}) cannot exceed targets (${doc.scope.targets})` });
316
+
317
+ const byClass = Object.values(s.byClass);
318
+ const proposedByClass = byClass.reduce((a, c) => a + c.proposed, 0);
319
+ const selectedByClass = byClass.reduce((a, c) => a + c.selected, 0);
320
+ if (proposedByClass !== s.proposed) errors.push({ path: '/selection/byClass', message: `byClass proposed sums to ${proposedByClass}; selection.proposed is ${s.proposed}` });
321
+ if (selectedByClass !== s.selected) errors.push({ path: '/selection/byClass', message: `byClass selected sums to ${selectedByClass}; selection.selected is ${s.selected}` });
322
+ for (const [k, c] of Object.entries(s.byClass)) {
323
+ if (c.selected > c.proposed) errors.push({ path: `/selection/byClass/${k}`, message: `selected (${c.selected}) exceeds proposed (${c.proposed})` });
324
+ if (c.writePath > c.proposed) errors.push({ path: `/selection/byClass/${k}`, message: `writePath (${c.writePath}) exceeds proposed (${c.proposed})` });
325
+ }
326
+
327
+ const probed = Object.values(doc.counts).reduce((a, n) => a + n, 0);
328
+ if (probed > s.selected) errors.push({ path: '/counts', message: `${probed} verdicts recorded for ${s.selected} selected faults` });
329
+ // Findings are every record that was not killed. Stating that here keeps a
330
+ // document from quietly reporting fewer findings than it reached verdicts.
331
+ const notKilled = probed - (doc.counts.killed ?? 0);
332
+ if (doc.findings.length !== notKilled) errors.push({ path: '/findings', message: `${doc.findings.length} findings for ${notKilled} records that were not killed; findings are every non-killed record` });
333
+
334
+ const gating = doc.findings.filter((f) => f.verdict === 'survived' || f.verdict === 'nocover').length;
335
+ if (gating > 0 && doc.exitCode !== 1) errors.push({ path: '/exitCode', message: 'a survived or nocover finding must exit 1; a sweep never passes over a deliberate break nothing noticed' });
336
+ if (gating === 0 && doc.exitCode !== 0) errors.push({ path: '/exitCode', message: 'exit 1 requires a survived or nocover finding; a sweep does not fail on its own unanchorable proposal' });
337
+
338
+ doc.findings.forEach((f, i) => {
339
+ if (f.verdict === 'killed') errors.push({ path: `/findings/${i}/verdict`, message: 'a killed fault is not a finding' });
340
+ // The signal explains why a defender could not notice a changed payload.
341
+ // On any other verdict, or off the write path, it explains nothing and
342
+ // would be exactly the non-actionable noise that gets a check ignored.
343
+ if (f.signals && !(f.verdict === 'survived' && f.writePath)) {
344
+ errors.push({ path: `/findings/${i}/signals`, message: 'signals belong to a survived finding on the write path' });
345
+ }
346
+ });
347
+ return errors;
348
+ },
349
+
308
350
  brief(doc) {
309
351
  const errors = [];
310
352
  if (!doc.text.startsWith(doc.heading)) errors.push({ path: '/text', message: 'text must begin with heading' });
@@ -0,0 +1,127 @@
1
+ {
2
+ "$id": "urn:claimspec:v1:sweep",
3
+ "$schema": "https://json-schema.org/draft/2020-12/schema",
4
+ "title": "claimspec v1 — sweep",
5
+ "description": "What a sweep proposed and what survived it. A sweep probes faults NOBODY STATED: it takes the changed source files that carry no claim, proposes faults with the mechanical producers, and reports the ones a green suite did not notice. Its findings are therefore weaker evidence than a probe's — no human said the behaviour mattered — and stronger than nothing, which is what a repository with no claims has. The drafts it carries are proposals for a human to keep or drop; they are not claims, and nothing here belongs in a claims file until someone states the sentence it defends.",
6
+ "type": "object",
7
+ "required": ["schemaVersion", "tool", "generatedAt", "ref", "base", "head", "includeDirty", "scope", "selection", "counts", "findings", "exitCode"],
8
+ "additionalProperties": false,
9
+ "properties": {
10
+ "$schema": { "type": "string" },
11
+ "schemaVersion": { "const": 1 },
12
+ "tool": { "$ref": "urn:claimspec:v1:common#/$defs/tool" },
13
+ "generatedAt": { "$ref": "urn:claimspec:v1:common#/$defs/isoDateTime" },
14
+ "ref": { "type": "string", "description": "The reference the change was measured against." },
15
+ "base": { "$ref": "urn:claimspec:v1:common#/$defs/gitSha" },
16
+ "head": { "$ref": "urn:claimspec:v1:common#/$defs/gitSha" },
17
+ "includeDirty": { "type": "boolean" },
18
+ "scope": {
19
+ "type": "object",
20
+ "required": ["changed", "targets", "swept"],
21
+ "additionalProperties": false,
22
+ "properties": {
23
+ "changed": { "type": "integer", "minimum": 0, "description": "Files in the change, before any filter." },
24
+ "targets": { "type": "integer", "minimum": 0, "description": "Changed SOURCE files carrying no claim — what the gate calls uncovered. A sweep proposes against these and nothing else." },
25
+ "swept": { "type": "integer", "minimum": 0, "description": "Targets that yielded at least one proposal. Never more than `targets`: a file the producers cannot touch is skipped, not swept." },
26
+ "skipped": {
27
+ "type": "array",
28
+ "description": "Targets that produced nothing, and why. Recorded rather than dropped: a file that silently yielded no proposal reads as a file with no findings.",
29
+ "items": {
30
+ "type": "object",
31
+ "required": ["file", "reason"],
32
+ "additionalProperties": false,
33
+ "properties": {
34
+ "file": { "$ref": "urn:claimspec:v1:common#/$defs/repoPath" },
35
+ "reason": { "type": "string", "minLength": 1 }
36
+ }
37
+ }
38
+ }
39
+ }
40
+ },
41
+ "selection": {
42
+ "type": "object",
43
+ "description": "What was proposed and what was probed. A sweep is capped — past a few findings per file nobody reads them, and the runner is spent either way — so the count NOT probed is reported beside the count that was. A cap that hides its own remainder is a coverage claim nobody made.",
44
+ "required": ["proposed", "selected", "deferred", "cap", "byClass"],
45
+ "additionalProperties": false,
46
+ "properties": {
47
+ "proposed": { "type": "integer", "minimum": 0 },
48
+ "selected": { "type": "integer", "minimum": 0 },
49
+ "deferred": { "type": "integer", "minimum": 0, "description": "Proposed and not probed. Not a verdict of any kind." },
50
+ "cap": { "type": "integer", "minimum": 1 },
51
+ "byClass": {
52
+ "type": "object",
53
+ "description": "Per fault class: how many were proposed, how many probed, and how many sat on a write payload.",
54
+ "additionalProperties": {
55
+ "type": "object",
56
+ "required": ["proposed", "selected", "writePath"],
57
+ "additionalProperties": false,
58
+ "properties": {
59
+ "proposed": { "type": "integer", "minimum": 0 },
60
+ "selected": { "type": "integer", "minimum": 0 },
61
+ "writePath": { "type": "integer", "minimum": 0 }
62
+ }
63
+ }
64
+ }
65
+ }
66
+ },
67
+ "counts": {
68
+ "type": "object",
69
+ "description": "Every verdict the probe reached, including `killed`. A sweep whose faults were all caught is a real result and must be readable as one.",
70
+ "additionalProperties": { "type": "integer", "minimum": 0 },
71
+ "propertyNames": { "$ref": "urn:claimspec:v1:common#/$defs/verdict" }
72
+ },
73
+ "findings": {
74
+ "type": "array",
75
+ "description": "Every record that was not killed, most actionable first. `survived` and `nocover` are statements about the test suite; the rest are statements about the PROPOSAL and do not gate, because a sweep's faults are machine-made and a tool that fails on its own bad guess gets switched off.",
76
+ "items": {
77
+ "type": "object",
78
+ "required": ["file", "claimId", "faultId", "faultClass", "verdict", "description", "writePath", "hint"],
79
+ "additionalProperties": false,
80
+ "properties": {
81
+ "file": { "$ref": "urn:claimspec:v1:common#/$defs/repoPath" },
82
+ "line": { "type": "integer", "minimum": 1 },
83
+ "claimId": { "$ref": "urn:claimspec:v1:common#/$defs/identifier" },
84
+ "faultId": { "$ref": "urn:claimspec:v1:common#/$defs/identifier" },
85
+ "faultClass": { "$ref": "urn:claimspec:v1:common#/$defs/faultClass" },
86
+ "verdict": { "$ref": "urn:claimspec:v1:common#/$defs/verdict" },
87
+ "reason": { "type": "string", "minLength": 1 },
88
+ "description": { "type": "string", "minLength": 1 },
89
+ "writePath": { "type": "boolean", "description": "The fault sits on a database write or the payload of one. Every surviving fault in the measurement that prompted this field was such a line." },
90
+ "hint": { "type": "string", "minLength": 1, "description": "One line naming the mechanism, never only the verdict." },
91
+ "signals": {
92
+ "type": "array",
93
+ "minItems": 1,
94
+ "description": "Static, annotation-grade signals about the defenders of a SURVIVOR on the write path. They never change a verdict. `persistence-payload-unasserted`: the defender mocks the persistence layer and the file's call assertions never name an exact payload, so a field dropped from a write cannot fail them. Descriptive, never predictive: no threshold on the assertion mix separates the files whose payload faults survive from the files whose do not.",
95
+ "items": {
96
+ "type": "object",
97
+ "required": ["file", "signal", "reason"],
98
+ "additionalProperties": false,
99
+ "properties": {
100
+ "file": { "$ref": "urn:claimspec:v1:common#/$defs/repoPath" },
101
+ "signal": { "enum": ["persistence-payload-unasserted"] },
102
+ "reason": { "type": "string", "minLength": 1 },
103
+ "specifiers": { "type": "array", "items": { "type": "string", "minLength": 1 } },
104
+ "counts": {
105
+ "type": "object",
106
+ "required": ["exact", "partial", "argless"],
107
+ "additionalProperties": false,
108
+ "properties": {
109
+ "exact": { "type": "integer", "minimum": 0 },
110
+ "partial": { "type": "integer", "minimum": 0 },
111
+ "argless": { "type": "integer", "minimum": 0 }
112
+ }
113
+ }
114
+ }
115
+ }
116
+ }
117
+ }
118
+ }
119
+ },
120
+ "exitCode": { "enum": [0, 1] },
121
+ "drafts": {
122
+ "type": "array",
123
+ "description": "The draft claims documents the proposals came from, carried so a human can keep one without re-running the sweep. Drafts, never claims: every statement is a TODO nobody has written yet.",
124
+ "items": { "type": "object" }
125
+ }
126
+ }
127
+ }
package/src/cli.mjs CHANGED
@@ -18,6 +18,7 @@ import { gateCommand } from './commands/gate.mjs';
18
18
  import { replayCommand } from './commands/replay.mjs';
19
19
  import { mcpCommand } from './commands/mcp.mjs';
20
20
  import { admitCommand } from './commands/admit.mjs';
21
+ import { sweepCommand } from './commands/sweep.mjs';
21
22
 
22
23
  const VERSION = JSON.parse(readFileSync(join(fileURLToPath(import.meta.url), '..', '..', 'package.json'), 'utf8')).version;
23
24
  const ISSUES = 'https://github.com/raccioly/testguard/issues';
@@ -32,6 +33,7 @@ export const USAGE = `testguard ${VERSION} — proves a test suite defends the c
32
33
  testguard brief [dir] emit the blind-spot block for an agent's session-start context
33
34
  testguard gate [dir] fail when a changed source file carries no claim and no excusing ignore entry
34
35
  testguard scaffold <file> propose faults mechanically for one source file, as a draft claims document
36
+ testguard sweep [dir] propose faults for the changed source files that carry no claim, probe a bounded selection, and report what nothing noticed
35
37
  testguard mcp serve the read-only status/brief/claims/evidence tools over MCP on stdio, so the loop works in any agent harness
36
38
  testguard replay [dir] would this suite have caught the bugs that already escaped? Replays real fix commits and calibrates fault classes against them
37
39
  testguard admit <test> --claim <ID> the two-gate rule as one verb: is this test green on HEAD and does it fail on every fault of the claim?
@@ -74,6 +76,10 @@ probe
74
76
  --json the status document plus this run's result (records, newSinceBaseline, exitCode)
75
77
 
76
78
  scaffold --claim <ID> (put every proposal under this claim; copies it if it exists) --out <path> --json
79
+ sweep --changed <ref> (required; CI bases and a safe local default are detected) --cap <n> (default 7 x unclaimed files)
80
+ --include-dirty --exclude <glob> --confirm <n> --budget <ms> --max <n> --out <path> --json
81
+ the cold start: no claim and no concern needed. Everything it proposes is a DRAFT — it never writes testguard.claims.json,
82
+ and its evidence never replaces .testguard/evidence.json. exit 1 on a fault that survived, or a file no test imports
77
83
  shapes: if-guard → if (false) · single-line guard/mutation removed · return <check> → return true
78
84
  · security flag/window/cost literal weakened · verify/validate/check call removed
79
85
  · field dropped from a payload/allow-list/schema/merge · parameter-derived argument swapped for undefined/{}
@@ -117,7 +123,7 @@ brief --evidence <path> --baseline <path> --max <n> --text (print only;
117
123
  exit codes: 0 nothing new to prove · 1 unproven claims (or claim drift, or unclaimed changes) · 2 precondition failed · 3 usage
118
124
  `;
119
125
 
120
- const COMMANDS = { mcp: mcpCommand, replay: replayCommand, probe: probeCommand, claims: claimsCommand, baseline: baselineCommand, brief: briefCommand, scaffold: scaffoldCommand, status: statusCommand, init: initCommand, gate: gateCommand, admit: admitCommand };
126
+ const COMMANDS = { mcp: mcpCommand, replay: replayCommand, probe: probeCommand, claims: claimsCommand, baseline: baselineCommand, brief: briefCommand, scaffold: scaffoldCommand, status: statusCommand, init: initCommand, gate: gateCommand, admit: admitCommand, sweep: sweepCommand };
121
127
 
122
128
  export async function main(argv, io = { out: (s) => process.stdout.write(s + '\n'), err: (s) => process.stderr.write(s + '\n') }) {
123
129
  let parsed;
@@ -143,6 +149,7 @@ export async function main(argv, io = { out: (s) => process.stdout.write(s + '\n
143
149
  changed: { type: 'string' },
144
150
  since: { type: 'string' },
145
151
  exclude: { type: 'string', multiple: true },
152
+ cap: { type: 'string' },
146
153
  strict: { type: 'boolean', default: false },
147
154
  explain: { type: 'boolean', default: false },
148
155
  ignore: { type: 'string' },
@@ -0,0 +1,74 @@
1
+ import { join, resolve } from 'node:path';
2
+ import { sweep, renderSweep } from '../sweep/sweep.mjs';
3
+ import { resolveChangedRef } from '../gate/changed.mjs';
4
+ import { writeSpecDoc } from '../evidence/writer.mjs';
5
+
6
+ export const sweepPath = (projectDir) => join(projectDir, '.testguard', 'sweep.json');
7
+
8
+ /**
9
+ * `testguard sweep [dir] --changed <ref>`: propose faults for the changed
10
+ * source files that carry no claim, probe a bounded selection, and report what
11
+ * a green suite did not notice.
12
+ *
13
+ * Exit 0 nothing survived · 1 a survivor or an untested file · 2 cannot
14
+ * evaluate · 3 no reference.
15
+ *
16
+ * The sweep document goes to `.testguard/sweep.json` and NEVER to
17
+ * `.testguard/evidence.json`. The canonical evidence is the record of what the
18
+ * project claims; a sweep probes faults nobody stated, under TODO statements,
19
+ * and folding one into the other would corrupt what `status` and `baseline`
20
+ * read.
21
+ */
22
+ export async function sweepCommand({ projectDir, values, version }, io) {
23
+ const resolved = resolveChangedRef({ explicit: values.changed, projectDir });
24
+ if (!resolved) {
25
+ io.err('sweep needs a reference to measure the change against: --changed <ref> (e.g. origin/main), or set TESTGUARD_CHANGED_REF. CI bases and a safe local remote default or differently named upstream are detected automatically.');
26
+ return 3;
27
+ }
28
+ const confirmRuns = Number(values.confirm);
29
+ const budgetMs = Number(values.budget);
30
+ if (!Number.isInteger(confirmRuns) || confirmRuns < 1 || !Number.isInteger(budgetMs) || budgetMs < 1000) {
31
+ io.err('--confirm must be a positive integer and --budget at least 1000');
32
+ return 3;
33
+ }
34
+ let cap;
35
+ if (values.cap !== undefined) {
36
+ cap = Number(values.cap);
37
+ if (!Number.isInteger(cap) || cap < 1) {
38
+ io.err('--cap must be a positive integer');
39
+ return 3;
40
+ }
41
+ }
42
+ if (!resolved.required && !values.json && !values.quiet) io.err(`sweep: comparing against ${resolved.ref} (${resolved.from})`);
43
+
44
+ const doc = await sweep({
45
+ projectDir,
46
+ ref: resolved.ref,
47
+ includeDirty: values['include-dirty'],
48
+ exclude: values.exclude ?? [],
49
+ cap,
50
+ confirmRuns,
51
+ budgetMs,
52
+ runnerCommand: values['runner-cmd'],
53
+ runnerName: values.runner,
54
+ nodeModules: values['node-modules'] ? resolve(values['node-modules']) : process.env.TESTGUARD_NODE_MODULES,
55
+ toolVersion: version,
56
+ claimsPath: values.claims ? resolve(values.claims) : undefined,
57
+ ignorePath: values.ignore ? resolve(values.ignore) : undefined,
58
+ onStage: !values.quiet && !values.json && process.stderr.isTTY
59
+ ? ({ claimId, faultId, stage, i, n }) => process.stderr.write(`\r\x1b[K … ${claimId}/${faultId} ${stage} ${i}/${n}`)
60
+ : undefined,
61
+ onWarn: (m) => { if (!values.quiet) io.err(`warning: ${m}`); },
62
+ });
63
+ if (process.stderr.isTTY && !values.quiet && !values.json) process.stderr.write('\r\x1b[K');
64
+
65
+ const outPath = values.out ? resolve(values.out) : sweepPath(projectDir);
66
+ writeSpecDoc('sweep', outPath, doc);
67
+ if (values.json) {
68
+ io.out(JSON.stringify(doc, null, 2));
69
+ } else {
70
+ io.out(renderSweep(doc, { limit: Number(values.max) || 20 }));
71
+ if (!values.quiet) io.out(`\nsweep: ${outPath}`);
72
+ }
73
+ return doc.exitCode;
74
+ }
@@ -0,0 +1,173 @@
1
+ /**
2
+ * The save-button signal: why a fault on the persistence path survived.
3
+ *
4
+ * `mocks.mjs` answers "does this test replace the SUBJECT?", and a file that
5
+ * does cannot detect any fault in it. This is the other half, and it is the
6
+ * one that explains a green suite over a save that does not save. Such a test
7
+ * does NOT mock the subject — it mocks the persistence layer underneath it —
8
+ * so it is a legitimate defender and stays one. It can catch a deleted write.
9
+ * What it cannot catch is a wrong PAYLOAD, and only because of how it asserts.
10
+ *
11
+ * Measured on a real AI-authored Next.js application (2026-09-21): 10 of 10
12
+ * server-action tests mocked `@/lib/prisma`, and 12 of 12 surviving faults on
13
+ * the persistence path were a field dropped from a `data` / `where` object.
14
+ * The assertion that should have caught them was `toHaveBeenCalledWith(
15
+ * expect.objectContaining({...}))` naming every field except the one carrying
16
+ * the data — the pathology this project's README reports from the field.
17
+ *
18
+ * DESCRIPTIVE, NOT PREDICTIVE — and that is a correction the measurement
19
+ * forced. The first version of this file silenced itself when a test contained
20
+ * any exact payload assertion. On the same 26 probed faults that rule fired on
21
+ * 1 of 10 files while 4 had survivors, because a file asserts many calls and
22
+ * the one exact assertion is usually about something else:
23
+ * `identity-password-recovery.test.ts` has one exact assertion among forty and
24
+ * every one of its payload faults survived, while `contact.test.ts` has the
25
+ * same one exact assertion and none did. No file-level threshold separates
26
+ * those two, so this reports the SHAPE and refuses to score it.
27
+ *
28
+ * It is therefore attached to a verdict the probe already reached, never used
29
+ * to predict one or to select what gets probed. The probe says a fault
30
+ * survived; this says the structural reason and what to write instead.
31
+ *
32
+ * Line- and regex-oriented, like the scaffold producers: no AST, no type
33
+ * information, and an honest miss rather than a confident wrong claim. In
34
+ * particular there is no read-back detection here. A regex for "the test asked
35
+ * for the row again" matched every file that mocks `findUnique` and asserts
36
+ * anything, including files whose payload faults all survived, so it measured
37
+ * nothing and was removed rather than shipped as a silencer.
38
+ */
39
+ import { readFileSync } from 'node:fs';
40
+ import { join } from 'node:path';
41
+
42
+ /**
43
+ * Module specifiers that mean "the database". A project-local path counts when
44
+ * it names one of these — `@/lib/prisma`, `../db`, `~/server/drizzle` — which
45
+ * is how these layers are named in practice.
46
+ */
47
+ const PERSISTENCE_SPECIFIER = /(?:^|[/@~.])(?:prisma|db|database|knex|sequelize|mongoose|typeorm|drizzle|supabase|pg|mysql2?|sqlite3?|dynamodb|mongodb)(?:$|[/.])/i;
48
+
49
+ /** `vi.mock('x')` / `jest.mock('x')` — the same call shapes mocks.mjs reads. */
50
+ const MOCK_CALL = /\b(?:vi|jest)\.(?:mock|doMock|unstable_mockModule)\(\s*(['"])([^'"]+)\1/g;
51
+
52
+ /** Partial matchers: true of a subset, so a dropped field outside the subset passes. */
53
+ const PARTIAL_MATCHER = /expect\.(?:objectContaining|arrayContaining|any|anything|stringContaining|stringMatching)\s*\(/;
54
+
55
+ /** The persistence specifiers this source mocks, with line numbers. */
56
+ export function persistenceMocks(source) {
57
+ const out = [];
58
+ source.split('\n').forEach((text, i) => {
59
+ for (const m of text.matchAll(MOCK_CALL)) {
60
+ if (PERSISTENCE_SPECIFIER.test(m[2])) out.push({ specifier: m[2], line: i + 1 });
61
+ }
62
+ });
63
+ return out;
64
+ }
65
+
66
+ /**
67
+ * The text between an open parenthesis at `open` and its match, so one
68
+ * assertion's arguments are read without reading the next one's.
69
+ *
70
+ * A fixed-width window cannot do this: two assertions on adjacent lines put the
71
+ * second one's `expect.objectContaining` inside the first one's window, and
72
+ * every exact assertion in a dense test file is reported as partial. Measured —
73
+ * the first version of this scored a file with one exact and one partial
74
+ * assertion as two partials.
75
+ *
76
+ * Quotes are tracked because a parenthesis inside a string is not a
77
+ * parenthesis; anything unbalanced returns what it read, which errs toward
78
+ * reporting a partial matcher that is really there.
79
+ */
80
+ function balancedArgs(source, open) {
81
+ let depth = 0;
82
+ let quote = null;
83
+ for (let i = open; i < source.length; i++) {
84
+ const c = source[i];
85
+ if (quote) {
86
+ if (c === '\\') i += 1;
87
+ else if (c === quote) quote = null;
88
+ continue;
89
+ }
90
+ if (c === '"' || c === "'" || c === '`') quote = c;
91
+ else if (c === '(') depth += 1;
92
+ else if (c === ')') {
93
+ depth -= 1;
94
+ if (depth === 0) return source.slice(open + 1, i);
95
+ }
96
+ }
97
+ return source.slice(open + 1);
98
+ }
99
+
100
+ /**
101
+ * How this source asserts on its mocks: how many assertions name the exact
102
+ * arguments, how many name them only partially, and how many assert that a
103
+ * call happened without saying with what.
104
+ *
105
+ * Counting rather than deciding. The counts go into the reason string so a
106
+ * reader can disagree with the inference instead of only with a verdict.
107
+ */
108
+ export function payloadAssertions(source) {
109
+ let exact = 0;
110
+ let partial = 0;
111
+ let argless = 0;
112
+ for (const m of source.matchAll(/\.toHaveBeenCalledWith\s*\(/g)) {
113
+ const open = m.index + m[0].length - 1;
114
+ if (PARTIAL_MATCHER.test(balancedArgs(source, open))) partial += 1;
115
+ else exact += 1;
116
+ }
117
+ for (const _ of source.matchAll(/\.(?:toHaveBeenCalled|toHaveBeenCalledTimes|toHaveBeenCalledOnce)\s*\(/g)) argless += 1;
118
+ return { exact, partial, argless };
119
+ }
120
+
121
+ /**
122
+ * The shape of one defender's relationship to the persistence layer, or null
123
+ * when it does not mock one. Shaped like the records `mocks.mjs` produces so a
124
+ * caller can merge the two lists without reshaping either.
125
+ */
126
+ export function analyzePersistence(projectDir, testRel) {
127
+ let source;
128
+ try {
129
+ source = readFileSync(join(projectDir, testRel), 'utf8');
130
+ } catch {
131
+ return null;
132
+ }
133
+ const mocks = persistenceMocks(source);
134
+ if (mocks.length === 0) return null;
135
+ const counts = payloadAssertions(source);
136
+ const specifiers = [...new Set(mocks.map((m) => m.specifier))].sort();
137
+ const shape = [
138
+ counts.exact ? `${counts.exact} exact` : '',
139
+ counts.partial ? `${counts.partial} partial` : '',
140
+ counts.argless ? `${counts.argless} argument-free` : '',
141
+ ].filter(Boolean).join(', ') || 'no';
142
+ return {
143
+ file: testRel,
144
+ signal: 'persistence-payload-unasserted',
145
+ reason: `mocks ${specifiers.join(', ')}; ${shape} call assertion${counts.exact + counts.partial + counts.argless === 1 ? '' : 's'} in the file. A field dropped from the write payload fails only against an assertion that names that field exactly.`,
146
+ counts,
147
+ specifiers,
148
+ };
149
+ }
150
+
151
+ /**
152
+ * The signal for a claim's defenders. Only ever consulted for a SURVIVOR whose
153
+ * fault is on the persistence path: on a kill it explains nothing, and on any
154
+ * other line a loose mock assertion is not evidence of anything. Reporting it
155
+ * everywhere would be the non-actionable noise that gets a check switched off.
156
+ */
157
+ export function persistenceSignals(projectDir, defenders) {
158
+ return defenders.map((d) => analyzePersistence(projectDir, d)).filter(Boolean);
159
+ }
160
+
161
+ /**
162
+ * One line an agent can act on, in the register `hintFor()` uses.
163
+ *
164
+ * The field name is read from either property form. Shorthand is not an edge
165
+ * case here: every surviving payload fault measured on the register action was
166
+ * shorthand — `email,` `username,` `password_hash,` — so a pattern that only
167
+ * matched `name: value` would fail to name the field in exactly the cases that
168
+ * prompted this signal.
169
+ */
170
+ export function persistenceHint(signal, faultLine) {
171
+ const field = /^\s*([A-Za-z_$][\w$]*)\s*(?::|,\s*$)/.exec(faultLine ?? '')?.[1];
172
+ return `${signal.file} ${signal.reason} Assert the whole object written by the call this fault changes${field ? `, including \`${field}\`` : ''} — or read the record back and assert on what was stored.`;
173
+ }
@@ -0,0 +1,168 @@
1
+ /**
2
+ * Which drafted faults are worth probing, and in what order.
3
+ *
4
+ * `scaffold` proposes mechanically and proposes a lot: eight source files of a
5
+ * real application yielded 238 proposals. Probing all of them is the cost model
6
+ * the tool already refuses — `(baseline + faults x N) x defender-set cost`,
7
+ * per claim — so a sweep that probes everything is a sweep nobody runs twice.
8
+ *
9
+ * Google's mutation service solved the same problem by never surfacing more
10
+ * than 7 x |files| mutants per change and by ordering candidates on the
11
+ * measured productivity of their operator in similar context, which took their
12
+ * productive rate from 15% to 89% ("Practical Mutation Testing at Scale",
13
+ * Petrovic et al., 2021). This module is that idea at this tool's scale.
14
+ *
15
+ * TWO THINGS IT MUST NOT DO. It must not drop a candidate silently — a cap
16
+ * that hides work is a coverage claim nobody made, so everything not selected
17
+ * is returned as `deferred` and the caller is expected to say so. And it must
18
+ * not be nondeterministic: the same drafts in the same repository must select
19
+ * the same faults in the same order, or a sweep's output stops being evidence.
20
+ * Every tie therefore breaks on file, then line, then id.
21
+ *
22
+ * Pure: drafts in, a selection out. Nothing here reads a file or runs anything.
23
+ */
24
+
25
+ /**
26
+ * P(a fault of this class survives a green suite), measured — not guessed.
27
+ *
28
+ * From 56 probed faults across 12 modules of a real AI-authored Next.js
29
+ * application (2026-09-21). Survival is the right prior for ordering because a
30
+ * fault that dies everywhere teaches nobody anything: the whole output of a
31
+ * sweep is the faults that lived.
32
+ *
33
+ * `n` is carried so a reader can see which of these is one observation and
34
+ * which is twelve, and so a later feedback pass can weight them honestly
35
+ * instead of treating 100%-of-one as certainty.
36
+ */
37
+ export const MEASURED_PRODUCTIVITY = Object.freeze({
38
+ 'element-removed': { p: 1.0, n: 1 },
39
+ 'call-removed': { p: 0.67, n: 3 },
40
+ 'field-dropped': { p: 0.5, n: 12 },
41
+ 'statement-deleted': { p: 0.33, n: 12 },
42
+ 'literal-changed': { p: 0.33, n: 3 },
43
+ 'condition-forced': { p: 0.1, n: 10 },
44
+ 'argument-swapped': { p: 0.09, n: 11 },
45
+ 'return-altered': { p: 0.0, n: 4 },
46
+ });
47
+
48
+ /**
49
+ * A class nobody measured is worth more attention than the worst measured one
50
+ * and less than the best. Ranking it last would make every new producer
51
+ * invisible until someone hand-edited this table; ranking it first would let a
52
+ * new producer crowd out everything known to be productive.
53
+ */
54
+ const UNMEASURED = 0.4;
55
+
56
+ /**
57
+ * Shrink a small sample toward the unmeasured prior. One observation of
58
+ * `element-removed` is not evidence that it always survives, and ordering the
59
+ * whole sweep on it would be exactly the overconfidence this tool exists to
60
+ * refuse. m is the pseudo-count: at n = m the measurement and the prior weigh
61
+ * the same.
62
+ */
63
+ const M = 4;
64
+ export function priorFor(faultClass, table = MEASURED_PRODUCTIVITY) {
65
+ const hit = table[faultClass];
66
+ if (!hit) return UNMEASURED;
67
+ return (hit.p * hit.n + UNMEASURED * M) / (hit.n + M);
68
+ }
69
+
70
+ /**
71
+ * The write payload of a persisted record is where this project's own
72
+ * measurement found every survivor: 12 of 12 were a field dropped from a
73
+ * `data` / `where` / `update` object reaching a database call. A fault on such
74
+ * a line is worth more than its class alone says, because the assertion that
75
+ * would have to catch it is the one tests most often write loosely
76
+ * (`expect.objectContaining({...})` naming everything except the field that
77
+ * carried the data).
78
+ *
79
+ * Deliberately a line-shape test, not a type-aware one: the scaffold producers
80
+ * are line-oriented and this must not become the one part of the pipeline that
81
+ * needs an AST.
82
+ */
83
+ const PERSISTENCE_CALL = /\b(?:prisma|db|tx|trx|knex|sequelize|mongoose|supabase|drizzle)\b\s*[.[]|\.(?:create|createMany|update|updateMany|upsert|insert|save|delete|deleteMany|destroy)\s*\(/;
84
+ const PAYLOAD_KEY = /^\s*(?:data|where|update|create|select|set|values)\s*:/;
85
+
86
+ /** Is this fault on the persistence path — a write call, or a field of its payload? */
87
+ export function onWritePath(fault) {
88
+ const line = fault.find ?? '';
89
+ return PERSISTENCE_CALL.test(line) || PAYLOAD_KEY.test(line) || PERSISTENCE_CALL.test(fault.description ?? '');
90
+ }
91
+
92
+ /** The boost a write-path fault gets. Additive, not multiplicative: it must not be able to overtake a measured 1.0. */
93
+ const WRITE_PATH_BONUS = 0.25;
94
+
95
+ /** What one draft is worth probing, in [0, 1.25]. Exported so a test can pin the ordering rule rather than the order. */
96
+ export function scoreOf(fault, { table } = {}) {
97
+ return priorFor(fault.faultClass, table) + (onWritePath(fault) ? WRITE_PATH_BONUS : 0);
98
+ }
99
+
100
+ /**
101
+ * Google surfaces at most 7 x |files| mutants per change. The number is theirs
102
+ * and the reason is general: past it the reader stops reading, and a finding
103
+ * nobody reads is worse than no finding, because it also spends the runner.
104
+ */
105
+ export const capFor = (fileCount, perFile = 7) => Math.max(perFile, fileCount * perFile);
106
+
107
+ /**
108
+ * Order every candidate and take the cap.
109
+ *
110
+ * `candidates` are `{ claim, fault }` pairs — the claim is carried so the
111
+ * caller can rebuild a claims document from the selection without a second
112
+ * lookup. Returns the selection, everything deferred, and the per-class tally
113
+ * a report needs to explain itself.
114
+ */
115
+ export function selectFaults(candidates, { cap, table } = {}) {
116
+ const limit = Number.isInteger(cap) && cap > 0 ? cap : capFor(new Set(candidates.map((c) => c.fault.file)).size);
117
+ const ranked = candidates
118
+ .map((c) => ({ ...c, score: scoreOf(c.fault, { table }), writePath: onWritePath(c.fault) }))
119
+ // Deterministic to the last tie: score, then file, then line, then id.
120
+ // Two runs of the same sweep on the same tree are the same sweep.
121
+ .sort((a, b) =>
122
+ b.score - a.score
123
+ || (a.fault.file ?? '').localeCompare(b.fault.file ?? '')
124
+ || (a.fault.line ?? 0) - (b.fault.line ?? 0)
125
+ || `${a.claim.id}/${a.fault.id}`.localeCompare(`${b.claim.id}/${b.fault.id}`));
126
+
127
+ // ── Spread the cap across FILES, best-first within each. ──
128
+ // Taking the global top N looks right and is not: a file with many
129
+ // high-scoring lines takes the whole budget and every other changed file is
130
+ // reported as having no findings, which is indistinguishable from having
131
+ // none. Measured — the first sweep of a 14-file change spent all six slots
132
+ // on one page component and said nothing about the other twelve.
133
+ // The cap is already expressed per file (7 x |files|), so this is what that
134
+ // number was always supposed to mean.
135
+ const byFile = new Map();
136
+ for (const r of ranked) {
137
+ const key = r.fault.file ?? '';
138
+ if (!byFile.has(key)) byFile.set(key, []);
139
+ byFile.get(key).push(r);
140
+ }
141
+ // Files in the order of their best candidate, so a round that cannot serve
142
+ // everyone still serves the most promising files first. Ties on file name.
143
+ const queues = [...byFile.entries()]
144
+ .sort((a, b) => b[1][0].score - a[1][0].score || a[0].localeCompare(b[0]))
145
+ .map(([, q]) => q);
146
+
147
+ const selected = [];
148
+ for (let round = 0; selected.length < limit; round++) {
149
+ let served = false;
150
+ for (const q of queues) {
151
+ if (q.length <= round) continue;
152
+ selected.push(q[round]);
153
+ served = true;
154
+ if (selected.length >= limit) break;
155
+ }
156
+ if (!served) break;
157
+ }
158
+ const taken = new Set(selected);
159
+ const deferred = ranked.filter((r) => !taken.has(r));
160
+ const byClass = {};
161
+ for (const r of ranked) {
162
+ const e = (byClass[r.fault.faultClass] ??= { proposed: 0, selected: 0, writePath: 0 });
163
+ e.proposed += 1;
164
+ if (r.writePath) e.writePath += 1;
165
+ }
166
+ for (const r of selected) byClass[r.fault.faultClass].selected += 1;
167
+ return { selected, deferred, cap: limit, byClass };
168
+ }
@@ -0,0 +1,274 @@
1
+ /**
2
+ * The cold start: findings on a repository that has no claims yet.
3
+ *
4
+ * `gate --changed` already says which changed source files carry no claim, and
5
+ * stops there — correctly, because stating a claim is a human act. But a
6
+ * project adopting this tool reads that list, has nothing to compare it
7
+ * against, and closes the tab. Every escaped defect reported from the field so
8
+ * far was a claim gap, not a probe miss, so the bottleneck was never
9
+ * verification. It is oracle SUPPLY.
10
+ *
11
+ * Sweep is the lowest rung of that supply: no concern, no statement, nothing
12
+ * written by anybody. It takes the files the gate just called uncovered,
13
+ * proposes faults mechanically with the existing scaffold producers, probes a
14
+ * bounded selection of them, and reports what survived. A fault that survives
15
+ * needs no claim to be alarming: the code changed, something was deliberately
16
+ * broken, and not one test noticed.
17
+ *
18
+ * WHAT IT IS NOT. It does not write `testguard.claims.json`, and it never
19
+ * will — a claim is a sentence a human is willing to stand behind, and a
20
+ * sentence nobody wrote is not one. Its drafts land beside the evidence for a
21
+ * human to keep or drop, exactly as `scaffold` already works. It does not
22
+ * overwrite the canonical `.testguard/evidence.json` either: a sweep probes
23
+ * machine-proposed faults under TODO statements, and folding that into the
24
+ * document `status` and `baseline` read from would corrupt the record of what
25
+ * the project actually claims.
26
+ *
27
+ * The cap and the ordering come from Google's mutation service, which surfaces
28
+ * at most 7 x |files| mutants per change and orders candidates on the measured
29
+ * productivity of their operator, taking their productive rate from 15% to 89%
30
+ * ("Practical Mutation Testing at Scale", Petrovic et al., 2021). See
31
+ * `src/supply/select.mjs`.
32
+ */
33
+ import { existsSync } from 'node:fs';
34
+ import { loadClaims, defaultClaimsPath } from '../claims/load.mjs';
35
+ import { computeChangedGate } from '../gate/changed.mjs';
36
+ import { scaffoldFile } from '../scaffold/scaffold.mjs';
37
+ import { selectFaults, onWritePath, capFor } from '../supply/select.mjs';
38
+ import { persistenceSignals, persistenceHint } from '../supply/persistence.mjs';
39
+ import { probe } from '../probe/probe.mjs';
40
+ import { hintFor } from '../brief/brief.mjs';
41
+
42
+ /**
43
+ * Findings first, in the order a reader should act on them. The same order
44
+ * `brief` uses, minus the verdicts a sweep does not gate on.
45
+ */
46
+ const FINDING_ORDER = ['survived', 'nocover', 'timeout', 'flaky-defender', 'unverifiable', 'fault-invalid'];
47
+
48
+ /**
49
+ * Which verdicts are a finding ABOUT THE PROJECT, and so decide the exit code.
50
+ *
51
+ * `survived` and `nocover` are statements about the test suite: something was
52
+ * broken and nothing failed, or nothing imports the file at all. The rest are
53
+ * statements about the PROPOSAL — an anchor that did not locate, a replacement
54
+ * that does not compile — and a sweep's proposals are machine-made and thrown
55
+ * away. Gating on them would make the tool fail because its own guess was bad,
56
+ * which is the fastest way to get a check switched off. They are still
57
+ * reported; they just do not gate.
58
+ */
59
+ export const GATING = new Set(['survived', 'nocover']);
60
+
61
+ /**
62
+ * Turn one evidence record into a finding, or null when it is not one.
63
+ *
64
+ * Pure, and exported, for the reason `classify.mjs` gives about itself: the
65
+ * only thing that could otherwise falsify these rules is a full fixture sweep,
66
+ * which is a minutes-long acceptance test standing in for a decision you can
67
+ * state in four lines. Self-probing this file found exactly that gap — the
68
+ * schema was guarded and the engine that fills it was not.
69
+ *
70
+ * `signalsFor` is injected so the rule about WHEN a signal applies can be
71
+ * tested without a filesystem; the default in `sweep()` reads the defenders.
72
+ */
73
+ export function findingFrom(record, fault, signalsFor = () => []) {
74
+ if (record.verdict === 'killed') return null;
75
+ const writePath = fault ? onWritePath(fault) : false;
76
+ // The persistence signal explains a survivor on the write path and nothing
77
+ // else. On a kill it explains nothing, and on any other line a loose mock
78
+ // assertion is not evidence — reporting it everywhere is the
79
+ // non-actionable noise that gets a check switched off.
80
+ const signals = record.verdict === 'survived' && writePath ? signalsFor(record) : [];
81
+ return {
82
+ file: record.subject.file,
83
+ ...(fault?.line ? { line: fault.line } : {}),
84
+ claimId: record.claim.id,
85
+ faultId: record.subject.id,
86
+ faultClass: record.subject.faultClass,
87
+ verdict: record.verdict,
88
+ ...(record.detail?.reason ? { reason: record.detail.reason } : {}),
89
+ description: record.subject.description,
90
+ writePath,
91
+ hint: signals.length ? persistenceHint(signals[0], fault?.find) : hintFor(record),
92
+ ...(signals.length ? { signals } : {}),
93
+ };
94
+ }
95
+
96
+ /** Findings in the order a reader should act on them. Pure; deterministic to the last tie. */
97
+ export function sortFindings(findings) {
98
+ return [...findings].sort((a, b) =>
99
+ FINDING_ORDER.indexOf(a.verdict) - FINDING_ORDER.indexOf(b.verdict)
100
+ || Number(b.writePath) - Number(a.writePath)
101
+ || a.file.localeCompare(b.file)
102
+ || (a.line ?? 0) - (b.line ?? 0));
103
+ }
104
+
105
+ /** How many findings a sweep is willing to fail on. */
106
+ export const gatingCount = (findings) => findings.filter((f) => GATING.has(f.verdict)).length;
107
+
108
+ /** Rebuild claim documents from a flat selection, keeping each claim's own faults together. */
109
+ function claimsFromSelection(selected) {
110
+ const byClaim = new Map();
111
+ for (const { claim, fault } of selected) {
112
+ if (!byClaim.has(claim.id)) byClaim.set(claim.id, { ...claim, faults: [] });
113
+ byClaim.get(claim.id).faults.push(fault);
114
+ }
115
+ // Deterministic: the probe's own ordering, and the evidence it writes, must
116
+ // not depend on the order a Map happened to be filled in.
117
+ return [...byClaim.values()]
118
+ .map((c) => ({ ...c, faults: [...c.faults].sort((a, b) => a.id.localeCompare(b.id)) }))
119
+ .sort((a, b) => a.id.localeCompare(b.id));
120
+ }
121
+
122
+ /**
123
+ * Run a sweep. Returns the sweep document; the caller owns writing and exit
124
+ * codes, as every other engine module in this tool does.
125
+ */
126
+ export async function sweep({
127
+ projectDir,
128
+ ref,
129
+ includeDirty = false,
130
+ exclude = [],
131
+ cap,
132
+ confirmRuns = 3,
133
+ budgetMs = 120_000,
134
+ runnerCommand,
135
+ runnerName,
136
+ nodeModules,
137
+ toolVersion = '0.0.0',
138
+ generatedAt = new Date().toISOString(),
139
+ claimsPath,
140
+ ignorePath,
141
+ onStage,
142
+ onWarn = () => {},
143
+ }) {
144
+ const gate = computeChangedGate({ projectDir, ref, includeDirty, exclude, toolVersion, claimsPath, ignorePath });
145
+ // A test file with no claim is a different problem — it has nothing to
146
+ // falsify — and the gate already names it. Sweep proposes against source.
147
+ const targets = gate.uncovered.filter((u) => u.kind === 'source').map((u) => u.file);
148
+
149
+ const cPath = claimsPath ?? defaultClaimsPath(projectDir);
150
+ const existingClaims = existsSync(cPath) ? loadClaims(cPath) : { claims: [] };
151
+
152
+ const candidates = [];
153
+ const skipped = [];
154
+ const drafts = [];
155
+ for (const file of targets) {
156
+ let result;
157
+ try {
158
+ result = scaffoldFile({ projectDir, file, existingClaims, toolVersion });
159
+ } catch (e) {
160
+ // One unscaffoldable file must not take the sweep with it: the other
161
+ // files' findings are still true, and a reader needs to know which file
162
+ // produced nothing and why.
163
+ skipped.push({ file, reason: e.message.split('\n')[0].slice(0, 200) });
164
+ continue;
165
+ }
166
+ if (result.stats.proposals === 0) {
167
+ skipped.push({ file, reason: 'no line in this file matches a fault producer' });
168
+ continue;
169
+ }
170
+ drafts.push(result.doc);
171
+ for (const claim of result.doc.claims) for (const fault of claim.faults) candidates.push({ claim, fault });
172
+ }
173
+
174
+ const limit = cap ?? capFor(targets.length);
175
+ const selection = selectFaults(candidates, { cap: limit });
176
+ const claims = claimsFromSelection(selection.selected);
177
+
178
+ let evidence = { records: [] };
179
+ if (claims.length > 0) {
180
+ evidence = await probe({
181
+ projectDir,
182
+ claims: { schemaVersion: 1, claims },
183
+ confirmRuns,
184
+ budgetMs,
185
+ mode: 'worktree',
186
+ includeDirty,
187
+ // A proposed fault has no declared defenders to be missing from, so the
188
+ // question escalation answers — "did something UNDECLARED catch it?" —
189
+ // has no meaning here, and it is the most expensive thing the probe does.
190
+ escalate: false,
191
+ runnerCommand,
192
+ runnerName,
193
+ nodeModules,
194
+ toolVersion,
195
+ onStage,
196
+ onWarn,
197
+ });
198
+ }
199
+
200
+ // Index the selection so a record can find the fault it came from without a
201
+ // second scan; `line` and `find` live on the draft, not on the evidence.
202
+ const draftOf = new Map(selection.selected.map(({ claim, fault }) => [`${claim.id}/${fault.id}`, fault]));
203
+
204
+ const signalsFor = (r) => persistenceSignals(projectDir, r.defenders?.resolved ?? []);
205
+ const counts = {};
206
+ const findings = [];
207
+ for (const r of evidence.records) {
208
+ counts[r.verdict] = (counts[r.verdict] ?? 0) + 1;
209
+ const f = findingFrom(r, draftOf.get(`${r.claim.id}/${r.subject.id}`), signalsFor);
210
+ if (f) findings.push(f);
211
+ }
212
+ const ordered = sortFindings(findings);
213
+ const gating = gatingCount(ordered);
214
+ return {
215
+ schemaVersion: 1,
216
+ tool: { name: 'testguard', version: toolVersion },
217
+ generatedAt,
218
+ ref,
219
+ base: gate.base,
220
+ head: gate.head,
221
+ includeDirty,
222
+ scope: {
223
+ changed: gate.changed,
224
+ targets: targets.length,
225
+ swept: drafts.length,
226
+ ...(skipped.length ? { skipped } : {}),
227
+ },
228
+ selection: {
229
+ proposed: candidates.length,
230
+ selected: selection.selected.length,
231
+ deferred: selection.deferred.length,
232
+ cap: selection.cap,
233
+ byClass: selection.byClass,
234
+ },
235
+ counts,
236
+ findings: ordered,
237
+ exitCode: gating > 0 ? 1 : 0,
238
+ drafts,
239
+ };
240
+ }
241
+
242
+ /** The sweep as text, for someone who ran it in a terminal or reads it in a CI log. */
243
+ export function renderSweep(doc, { limit = 20 } = {}) {
244
+ const out = [];
245
+ const { scope, selection } = doc;
246
+ if (scope.targets === 0) {
247
+ out.push(`sweep: no changed source file is without a claim against ${doc.ref}. Nothing to propose.`);
248
+ return out.join('\n');
249
+ }
250
+ out.push(`swept ${scope.swept} of ${scope.targets} unclaimed changed file${scope.targets === 1 ? '' : 's'} against ${doc.ref}.`);
251
+ out.push(`proposed ${selection.proposed} fault${selection.proposed === 1 ? '' : 's'}, probed ${selection.selected} (cap ${selection.cap})${selection.deferred ? `, deferred ${selection.deferred}` : ''}.`);
252
+ if (selection.deferred) out.push(` The ${selection.deferred} deferred are not a verdict: raise --cap, or sweep a smaller change.`);
253
+ for (const s of scope.skipped ?? []) out.push(` skipped ${s.file}: ${s.reason}`);
254
+ out.push('');
255
+
256
+ const gating = doc.findings.filter((f) => GATING.has(f.verdict));
257
+ if (gating.length === 0) {
258
+ out.push(`No fault survived. ${doc.counts.killed ?? 0} of ${selection.selected} probed faults were caught.`);
259
+ } else {
260
+ out.push(`${gating.length} finding${gating.length === 1 ? '' : 's'} — a deliberate break that no test noticed:`);
261
+ out.push('');
262
+ for (const f of doc.findings.slice(0, limit)) {
263
+ const where = f.line ? `${f.file}:${f.line}` : f.file;
264
+ out.push(` ${f.verdict.toUpperCase().padEnd(13)} ${where}${f.writePath ? ' [write path]' : ''}`);
265
+ out.push(` ${f.description}`);
266
+ out.push(` ${f.hint}`);
267
+ out.push('');
268
+ }
269
+ if (doc.findings.length > limit) out.push(` … ${doc.findings.length - limit} more in the sweep document`);
270
+ }
271
+ out.push('These are PROPOSALS, not claims. Keep the ones worth defending: state the');
272
+ out.push('claim, copy its fault into testguard.claims.json, and probe it from then on.');
273
+ return out.join('\n');
274
+ }