testguard-cli 0.12.0 → 0.13.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -7,6 +7,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
7
7
 
8
8
  ## [Unreleased]
9
9
 
10
+ ## [0.13.0] - 2026-09-22
11
+
12
+ Automated weekly release — everything merged since `v0.12.0`.
13
+
14
+ ### Changed
15
+
16
+ - chore(homebrew): sha256 for v0.12.0 (#159)
17
+ - feat: persistence provability, and an ordering that learns from this project's own verdicts (#157)
18
+
19
+
10
20
  ## [0.12.0] - 2026-09-22
11
21
 
12
22
  Automated weekly release — everything merged since `v0.11.0`.
@@ -18,6 +28,53 @@ Automated weekly release — everything merged since `v0.11.0`.
18
28
  - chore(homebrew): sha256 for v0.11.0 (#153)
19
29
 
20
30
 
31
+ ### Added
32
+
33
+ - **Concerns** — one sentence a human writes that generates faults across a
34
+ whole surface, the rung between writing nothing and writing a claim per
35
+ behaviour. A concern says *where* to look and *which shapes of break* are
36
+ relevant: `"every write persists the data it was given"` targets the write
37
+ sites and applies the payload producers. `testguard concerns` lists them,
38
+ `sweep --concern <ID>` aims a sweep by one, and `--save-paths` is now sugar
39
+ for the built-in `SAVE-PERSISTS`.
40
+ - Two built-ins ship, and only two: an authorization concern would need a
41
+ heuristic for "which files check permissions", and a guess there is the
42
+ non-actionable noise that gets a check switched off. A project that knows
43
+ its own auth layer says so with a `glob` target in three lines.
44
+ - A project concern with the same id **replaces** a built-in, so a team can
45
+ retune `SAVE-PERSISTS` for an ORM this tool does not recognise without
46
+ forking anything. The replacement is reported, never silent.
47
+ - `urn:claimspec:v1:concerns`, with rules that refuse a glob target naming no
48
+ globs (it would match nothing and report clean) and an empty fault-class
49
+ list (a typo for "every class", every time).
50
+ - **No model is required, deliberately.** A concern's useful core is a named
51
+ scope plus a producer selection, which the mechanical producers already
52
+ satisfy — so every verifying command stays offline and NFR-01 is unchanged.
53
+ An LLM can later propose faults a line-oriented producer cannot see, as an
54
+ additional source for a concern that already works.
55
+
56
+ ### Added
57
+
58
+ - A save-path sweep now reports **what its defenders could prove**, before it
59
+ reports what they did: per file, whether every defender mocks the persistence
60
+ layer (call shape only), at least one does not (a real write is possible), or
61
+ there is no defender at all. When nothing is unmocked the report says so
62
+ outright — a clean probe there is not evidence of persistence, it is evidence
63
+ that nothing could have measured it. Measured on a real AI-authored
64
+ application: of 37 files that write to storage, 32 were defended only by
65
+ mocking tests.
66
+ - `scope.provability` on the sweep document, with a validator rule that every
67
+ target is classified exactly once.
68
+ - The candidate ordering now **learns from this project's own probed verdicts**
69
+ instead of only the shipped productivity prior. Every evidence record already
70
+ carries a `faultClass` and a `verdict`, so "how often does a fault of this
71
+ class survive *here*" is a tally rather than a new thing to capture. A sweep
72
+ persists its own evidence to `.testguard/sweep-evidence.json` — beside the
73
+ canonical one, never replacing it — so the next run learns from
74
+ machine-proposed faults, the same distribution the ranker orders. The sweep
75
+ document records `ordering.observed` and `ordering.sources`, because an
76
+ ordering nobody can trace is a number nobody should trust.
77
+
21
78
  ### Added
22
79
 
23
80
  - `sweep --save-paths` — sweep every file that writes to storage instead of the
package/README.md CHANGED
@@ -127,9 +127,9 @@ raises coverage. TestGuard admits it because it fails when the claim is false.
127
127
  | npm | `npm i -D testguard-cli` then `npx testguard probe` |
128
128
  | pip | `pip install testguard-cli` then `testguard probe` (needs Node ≥ 20) |
129
129
  | Homebrew | `brew tap raccioly/tap && brew install testguard` |
130
- | GitHub Action | `uses: raccioly/testguard@v0.12.0` — see [`action.yml`](./action.yml) |
130
+ | GitHub Action | `uses: raccioly/testguard@v0.13.0` — see [`action.yml`](./action.yml) |
131
131
  | pre-commit | `repo: https://github.com/raccioly/testguard`, hooks `testguard-claims`, `testguard-probe` |
132
- | GitLab CI | `include: - remote: https://raw.githubusercontent.com/raccioly/testguard/v0.12.0/packaging/gitlab/testguard.gitlab-ci.yml` with `inputs:` — see [`packaging/gitlab/`](./packaging/gitlab/testguard.gitlab-ci.yml) |
132
+ | GitLab CI | `include: - remote: https://raw.githubusercontent.com/raccioly/testguard/v0.13.0/packaging/gitlab/testguard.gitlab-ci.yml` with `inputs:` — see [`packaging/gitlab/`](./packaging/gitlab/testguard.gitlab-ci.yml) |
133
133
 
134
134
  Projects that set `min-release-age` in `.npmrc` cannot see a version published
135
135
  less than that many days ago (`ENOVERSIONS`); install that one with
@@ -451,6 +451,14 @@ Three rules make it safe to run on a repository that has never seen this tool:
451
451
  is reported and never gates — a tool that fails because its own guess was bad
452
452
  is a tool people switch off.
453
453
 
454
+ The ordering **learns from your own runs**. Every evidence record already
455
+ carries a fault class and a verdict, so "how often does a fault of this class
456
+ survive here" is a tally, not a new thing to collect — a sweep persists its
457
+ evidence and the next one reads it. A project with no evidence falls back to
458
+ the shipped prior and says so; one with its own records overrides it in
459
+ proportion to how many it has. The report names which, because an ordering
460
+ nobody can trace is a number nobody should trust.
461
+
454
462
  The cap and the ordering are Google's: their mutation service surfaces at most
455
463
  7 × |files| mutants per change and orders candidates on the measured
456
464
  productivity of their operator in similar context, which took their productive
@@ -466,6 +474,60 @@ A sweep is weaker evidence than a probe, and says so. Nobody stated that the
466
474
  behaviour mattered. It is stronger than nothing, which is what a repository
467
475
  with no claims has — and the survivors worth defending become the first claims.
468
476
 
477
+ #### Concerns: one sentence, two hundred screens
478
+
479
+ A claim names one promise precisely, which is why a repository with two hundred
480
+ screens never finishes writing them. A **concern** names a *kind* of promise and
481
+ says where to look for it:
482
+
483
+ ```bash
484
+ npx testguard-cli concerns # what this project can be swept by
485
+ npx testguard-cli sweep --concern ADMIN-GUARDS # aim a sweep at one
486
+ ```
487
+
488
+ ```json
489
+ {
490
+ "schemaVersion": 1,
491
+ "concerns": [
492
+ {
493
+ "id": "ADMIN-GUARDS",
494
+ "statement": "Every admin route refuses a caller without the admin role.",
495
+ "severity": "critical",
496
+ "targets": { "kind": "glob", "globs": ["src/app/api/admin/**"] },
497
+ "faultClasses": ["guard-removed", "condition-forced", "return-altered"]
498
+ }
499
+ ]
500
+ }
501
+ ```
502
+
503
+ Five lines, and the sweep reports:
504
+
505
+ ```
506
+ concern ADMIN-GUARDS matches 24 files. Swept 23.
507
+ 158 further proposals were outside this concern's fault classes.
508
+
509
+ SURVIVED src/app/api/admin/claims/[id]/route.ts
510
+ [line 41] Guard never triggers: `if (!membership || (membership.role !== 'owner'
511
+ && membership.role !== 'manager'))` becomes `if (false)`.
512
+ ```
513
+
514
+ The idea is Meta's: ACH has an engineer describe an area of concern in plain
515
+ text and gates every generated step by execution. What is taken here is the
516
+ **unit** — the concern as the thing a human writes — not their generator, and
517
+ **no model is required**. A concern's useful core is a named scope plus a
518
+ producer selection, which the mechanical producers already satisfy, so every
519
+ verifying command stays offline.
520
+
521
+ Two concerns ship built in, and only two. An authorization concern would need a
522
+ heuristic for *"which files check permissions"*, and a guess there is exactly
523
+ the noise that gets a check switched off — so `SAVE-PERSISTS` and
524
+ `CHANGED-CODE` are the defaults, and a project that knows its own auth layer
525
+ says so in three lines. Declaring a concern with a built-in's id replaces it,
526
+ and the replacement is reported rather than silent.
527
+
528
+ A concern is **not** a claim and never becomes one by itself. It says where to
529
+ look; the probe says what it found; a human states the sentence worth defending.
530
+
469
531
  #### Sweeping the save surface instead of the diff
470
532
 
471
533
  A diff cannot answer *"when the user clicks save, does it actually save?"* — it
@@ -494,10 +556,26 @@ denominator comes first and the document carries `writeSites` and
494
556
  AI-authored application: **12 of 26 probed persistence faults survived a fully
495
557
  green suite, and all 12 were a dropped payload field.**
496
558
 
497
- This is a *sample* of the surface, and says so. It does not yet prove a write
498
- reached storage — only that a test would notice if the payload changed.
499
- Read-back oracles are the next rung
500
- ([#154](https://github.com/raccioly/testguard/issues/154)).
559
+ It also states, before any verdict, **what the defenders could prove at all**:
560
+
561
+ ```
562
+ of those 37 files: 32 defended only by tests that mock the persistence layer,
563
+ 1 with an unmocked defender, 4 with no defender at all.
564
+ ```
565
+
566
+ That line is the read-back answer. A test that replaced the database can prove
567
+ the *call shape* and nothing beyond it — a `where` that matches no rows, a
568
+ rolled-back transaction and a rejected constraint all pass against a mock that
569
+ recorded the arguments and returned a plausible object. So rather than pretend
570
+ to measure persistence, TestGuard states the limit, the same way `nocover` says
571
+ "no test imports this" instead of guessing.
572
+
573
+ When every defender mocks the layer, the report says so outright: *nothing in
574
+ this suite can prove a write reached storage*. A clean probe there is not
575
+ evidence of persistence — it is evidence that nothing could have measured it.
576
+
577
+ `unmocked` is deliberately weaker than "proves persistence": a test that does
578
+ not replace the database may simply never reach it. Possible is the honest word.
501
579
 
502
580
  ### Every change needs a claim
503
581
 
@@ -580,12 +658,12 @@ detected base that does not resolve is a warning for `status` and `brief`
580
658
  # GitHub Actions
581
659
  - uses: actions/checkout@v4
582
660
  with: { fetch-depth: 0 }
583
- - uses: raccioly/testguard@v0.12.0
661
+ - uses: raccioly/testguard@v0.13.0
584
662
  with: { command: gate }
585
663
 
586
664
  # GitLab CI — the component-shaped template: gate + probe, brief as an artifact and, opted in, as a merge-request note
587
665
  include:
588
- - remote: 'https://raw.githubusercontent.com/raccioly/testguard/v0.12.0/packaging/gitlab/testguard.gitlab-ci.yml'
666
+ - remote: 'https://raw.githubusercontent.com/raccioly/testguard/v0.13.0/packaging/gitlab/testguard.gitlab-ci.yml'
589
667
  inputs: { dir: backend, post_note: true } # post_note needs TESTGUARD_GITLAB_TOKEN (api scope); one note, updated in place
590
668
  ```
591
669
 
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "testguard-cli",
3
- "version": "0.12.0",
3
+ "version": "0.13.0",
4
4
  "description": "Breaks your code on purpose and reports every promise your tests did not notice breaking. Mutation testing bound to the claims a project makes — vitest, jest, Playwright, pytest and unittest.",
5
5
  "type": "module",
6
6
  "license": "MIT",
@@ -7,7 +7,7 @@ import { reproduces, decimals, MAX_PLACES } from './wilson.mjs';
7
7
 
8
8
  const schemaDir = join(dirname(fileURLToPath(import.meta.url)), '..', 'schemas');
9
9
 
10
- export const KINDS = Object.freeze(['claims', 'evidence', 'baseline', 'ignore', 'calibration', 'brief', 'status', 'gate', 'replay', 'sweep']);
10
+ export const KINDS = Object.freeze(['claims', 'evidence', 'baseline', 'ignore', 'calibration', 'brief', 'status', 'gate', 'replay', 'sweep', 'concerns']);
11
11
  /**
12
12
  * How a document says it tried to falsify its claims. Absent means
13
13
  * `fault-injection`, so every document written before the field existed is
@@ -305,6 +305,29 @@ const semantic = {
305
305
  return errors;
306
306
  },
307
307
 
308
+ concerns(doc) {
309
+ const errors = [];
310
+ const seen = new Set();
311
+ doc.concerns.forEach((c, i) => {
312
+ if (seen.has(c.id)) errors.push({ path: `/concerns/${i}/id`, message: `duplicate concern id "${c.id}"` });
313
+ seen.add(c.id);
314
+ // A glob target with no globs matches nothing, which reads as "this
315
+ // concern found no problems" rather than "this concern was never aimed".
316
+ if (c.targets?.kind === 'glob' && !(c.targets.globs?.length > 0)) {
317
+ errors.push({ path: `/concerns/${i}/targets/globs`, message: `concern "${c.id}" targets a glob but names none; it would match nothing and report clean` });
318
+ }
319
+ if (c.targets?.kind !== 'glob' && c.targets?.globs) {
320
+ errors.push({ path: `/concerns/${i}/targets/globs`, message: `concern "${c.id}" names globs but does not target them` });
321
+ }
322
+ // An empty list is not "every class"; it is a concern that can never
323
+ // propose anything, which is a typo for `null` every time.
324
+ if (Array.isArray(c.faultClasses) && c.faultClasses.length === 0) {
325
+ errors.push({ path: `/concerns/${i}/faultClasses`, message: `concern "${c.id}" allows no fault class; omit the field for every class` });
326
+ }
327
+ });
328
+ return errors;
329
+ },
330
+
308
331
  sweep(doc) {
309
332
  const errors = [];
310
333
  const s = doc.selection;
@@ -318,10 +341,17 @@ const semantic = {
318
341
  // measured, and its absence on a save-paths sweep hides the denominator
319
342
  // entirely — which is the whole reason that mode exists.
320
343
  const saves = doc.scope.mode === 'save-paths';
321
- for (const k of ['writeSites', 'payloadFields']) {
344
+ for (const k of ['writeSites', 'payloadFields', 'provability']) {
322
345
  if (saves && doc.scope[k] === undefined) errors.push({ path: `/scope/${k}`, message: `a save-paths sweep must report ${k}: the surface is the denominator its findings are read against` });
323
346
  if (!saves && doc.scope[k] !== undefined) errors.push({ path: `/scope/${k}`, message: `${k} belongs to a save-paths sweep; a changed sweep did not measure the write surface` });
324
347
  }
348
+ if (saves && doc.scope.provability) {
349
+ const pv = doc.scope.provability;
350
+ const total = pv.mocked + pv.unmocked + pv.none;
351
+ // Every target is classified exactly once. A surface whose provability
352
+ // does not account for all of it has a bucket nobody looked in.
353
+ if (total !== doc.scope.targets) errors.push({ path: '/scope/provability', message: `provability accounts for ${total} files; targets is ${doc.scope.targets}` });
354
+ }
325
355
  if (saves && doc.scope.writeSites !== undefined && doc.scope.targets > doc.scope.writeSites) {
326
356
  errors.push({ path: '/scope/targets', message: `targets (${doc.scope.targets}) exceeds writeSites (${doc.scope.writeSites}); every target is a file with at least one write` });
327
357
  }
@@ -343,6 +373,12 @@ const semantic = {
343
373
  const notKilled = probed - (doc.counts.killed ?? 0);
344
374
  if (doc.findings.length !== notKilled) errors.push({ path: '/findings', message: `${doc.findings.length} findings for ${notKilled} records that were not killed; findings are every non-killed record` });
345
375
 
376
+ // An ordering that claims to have learned from something must name it.
377
+ if (doc.ordering) {
378
+ if (doc.ordering.observed > 0 && doc.ordering.sources.length === 0) errors.push({ path: '/ordering/sources', message: 'an ordering learned from evidence must name the documents it was learned from' });
379
+ if (doc.ordering.observed === 0 && doc.ordering.sources.length > 0) errors.push({ path: '/ordering/observed', message: 'sources are named but nothing was observed; the ordering used the shipped prior' });
380
+ }
381
+
346
382
  const gating = doc.findings.filter((f) => f.verdict === 'survived' || f.verdict === 'nocover').length;
347
383
  if (gating > 0 && doc.exitCode !== 1) errors.push({ path: '/exitCode', message: 'a survived or nocover finding must exit 1; a sweep never passes over a deliberate break nothing noticed' });
348
384
  if (gating === 0 && doc.exitCode !== 0) errors.push({ path: '/exitCode', message: 'exit 1 requires a survived or nocover finding; a sweep does not fail on its own unanchorable proposal' });
@@ -0,0 +1,41 @@
1
+ {
2
+ "$id": "urn:claimspec:v1:concerns",
3
+ "$schema": "https://json-schema.org/draft/2020-12/schema",
4
+ "title": "claimspec v1 — concerns",
5
+ "description": "A concern is one sentence a human writes that generates faults across a whole surface: it says WHERE to look and WHICH shapes of break are relevant. It is the rung between writing nothing and writing a claim per behaviour. A concern is never a claim and never becomes one by itself — it says where to look, the probe says what it found, and a human states the sentence worth defending. Nothing in this document is a promise about the code; only a claim is that.",
6
+ "type": "object",
7
+ "required": ["schemaVersion", "concerns"],
8
+ "additionalProperties": false,
9
+ "properties": {
10
+ "$schema": { "type": "string" },
11
+ "schemaVersion": { "const": 1 },
12
+ "concerns": {
13
+ "type": "array",
14
+ "items": {
15
+ "type": "object",
16
+ "required": ["id", "statement"],
17
+ "additionalProperties": false,
18
+ "properties": {
19
+ "id": { "$ref": "urn:claimspec:v1:common#/$defs/identifier", "description": "Declaring an id that matches a built-in REPLACES it, which is how a project retunes a default for an ORM or a layout this tool does not recognise. The replacement is reported, never silent." },
20
+ "statement": { "type": "string", "minLength": 1, "description": "The sentence, in plain language. It describes a KIND of promise, not one behaviour: \"every write persists the data it was given\", not \"the register action stores password_hash\". The second is a claim." },
21
+ "severity": { "$ref": "urn:claimspec:v1:common#/$defs/severity" },
22
+ "targets": {
23
+ "type": "object",
24
+ "required": ["kind"],
25
+ "additionalProperties": false,
26
+ "description": "Where this concern applies. `write-sites` asks the write enumerator. `glob` matches paths. `changed` defers to whatever diff the caller is already measuring — a concern does not re-derive what the gate computed.",
27
+ "properties": {
28
+ "kind": { "enum": ["write-sites", "glob", "changed"] },
29
+ "globs": { "type": "array", "items": { "type": "string", "minLength": 1 } }
30
+ }
31
+ },
32
+ "faultClasses": {
33
+ "type": ["array", "null"],
34
+ "items": { "$ref": "urn:claimspec:v1:common#/$defs/faultClass" },
35
+ "description": "Which producers are relevant. `null` or absent means every class, the honest default for a concern with no shape to narrow by. Narrowing is what makes one sentence useful across two hundred files: a persistence concern that also reported every altered return value would bury the payload findings it exists to surface."
36
+ }
37
+ }
38
+ }
39
+ }
40
+ }
41
+ }
@@ -62,6 +62,10 @@
62
62
  ],
63
63
  "description": "What the sweep was pointed at. `changed` proposes for the changed source files that carry no claim. `save-paths` proposes for every file that writes to storage, because a diff cannot answer how many save paths exist — and a clean report over an unstated denominator is a sample of unknown size, not a guarantee."
64
64
  },
65
+ "concern": {
66
+ "$ref": "urn:claimspec:v1:common#/$defs/identifier",
67
+ "description": "The concern this sweep was aimed by, when one was. A concern narrows both the targets and the fault classes, so the same surface swept under two concerns is two different questions."
68
+ },
65
69
  "changed": {
66
70
  "type": "integer",
67
71
  "minimum": 0,
@@ -107,6 +111,30 @@
107
111
  "type": "integer",
108
112
  "minimum": 0,
109
113
  "description": "save-paths only: payload keys sitting on their own line, which is what a field-dropped fault can anchor to."
114
+ },
115
+ "provability": {
116
+ "type": "object",
117
+ "required": [
118
+ "mocked",
119
+ "unmocked",
120
+ "none"
121
+ ],
122
+ "additionalProperties": false,
123
+ "description": "save-paths only: what the defenders of this surface are CAPABLE of proving, before anything is probed. `mocked` — every defender replaces the persistence layer, so it can prove the call shape and never that the row landed. `unmocked` — at least one does not, so a real write is at least possible (not a promise that one happens). `none` — no defender at all. On a surface that is entirely `mocked`, a clean probe is not evidence of persistence; it is evidence that nothing could have measured it.",
124
+ "properties": {
125
+ "mocked": {
126
+ "type": "integer",
127
+ "minimum": 0
128
+ },
129
+ "unmocked": {
130
+ "type": "integer",
131
+ "minimum": 0
132
+ },
133
+ "none": {
134
+ "type": "integer",
135
+ "minimum": 0
136
+ }
137
+ }
110
138
  }
111
139
  }
112
140
  },
@@ -165,6 +193,11 @@
165
193
  }
166
194
  }
167
195
  }
196
+ },
197
+ "outOfScope": {
198
+ "type": "integer",
199
+ "minimum": 0,
200
+ "description": "Proposals the concern's fault classes excluded. Reported so a narrow concern cannot look like a quiet one: the producers found these, and this concern is not about them."
168
201
  }
169
202
  }
170
203
  },
@@ -305,6 +338,29 @@
305
338
  "items": {
306
339
  "type": "object"
307
340
  }
341
+ },
342
+ "ordering": {
343
+ "type": "object",
344
+ "required": [
345
+ "observed",
346
+ "sources"
347
+ ],
348
+ "additionalProperties": false,
349
+ "description": "Where the candidate ordering came from. `observed` is how many of this project's OWN probed faults informed it; 0 means the shipped productivity prior was used unchanged. An ordering nobody can trace is a number nobody should trust.",
350
+ "properties": {
351
+ "observed": {
352
+ "type": "integer",
353
+ "minimum": 0
354
+ },
355
+ "sources": {
356
+ "type": "array",
357
+ "items": {
358
+ "type": "string",
359
+ "minLength": 1
360
+ },
361
+ "description": "The evidence documents the counts were read from."
362
+ }
363
+ }
308
364
  }
309
365
  }
310
366
  }
package/src/cli.mjs CHANGED
@@ -19,6 +19,7 @@ import { replayCommand } from './commands/replay.mjs';
19
19
  import { mcpCommand } from './commands/mcp.mjs';
20
20
  import { admitCommand } from './commands/admit.mjs';
21
21
  import { sweepCommand } from './commands/sweep.mjs';
22
+ import { concernsCommand } from './commands/concerns.mjs';
22
23
 
23
24
  const VERSION = JSON.parse(readFileSync(join(fileURLToPath(import.meta.url), '..', '..', 'package.json'), 'utf8')).version;
24
25
  const ISSUES = 'https://github.com/raccioly/testguard/issues';
@@ -34,6 +35,7 @@ export const USAGE = `testguard ${VERSION} — proves a test suite defends the c
34
35
  testguard gate [dir] fail when a changed source file carries no claim and no excusing ignore entry
35
36
  testguard scaffold <file> propose faults mechanically for one source file, as a draft claims document
36
37
  testguard sweep [dir] propose faults for the changed source files that carry no claim, probe a bounded selection, and report what nothing noticed
38
+ testguard concerns [dir] the one-sentence scopes a sweep can be aimed by: where to look, and which shapes of break are relevant
37
39
  testguard mcp serve the read-only status/brief/claims/evidence tools over MCP on stdio, so the loop works in any agent harness
38
40
  testguard replay [dir] would this suite have caught the bugs that already escaped? Replays real fix commits and calibrates fault classes against them
39
41
  testguard admit <test> --claim <ID> the two-gate rule as one verb: is this test green on HEAD and does it fail on every fault of the claim?
@@ -77,6 +79,8 @@ probe
77
79
 
78
80
  scaffold --claim <ID> (put every proposal under this claim; copies it if it exists) --out <path> --json
79
81
  sweep --changed <ref> (required unless --save-paths; CI bases and a safe local default are detected) --cap <n>
82
+ --concern <ID> aim the sweep by one of this project's concerns: its targets and its fault classes.
83
+ testguard concerns lists them. --save-paths is sugar for the built-in SAVE-PERSISTS
80
84
  --save-paths sweep every file that WRITES TO STORAGE instead of the diff, and report the surface as the
81
85
  denominator: "120 writes across 38 files, swept 12". A clean sweep over an unstated
82
86
  denominator is a sample of unknown size, not a guarantee that saving works
@@ -126,7 +130,7 @@ brief --evidence <path> --baseline <path> --max <n> --text (print only;
126
130
  exit codes: 0 nothing new to prove · 1 unproven claims (or claim drift, or unclaimed changes) · 2 precondition failed · 3 usage
127
131
  `;
128
132
 
129
- const COMMANDS = { mcp: mcpCommand, replay: replayCommand, probe: probeCommand, claims: claimsCommand, baseline: baselineCommand, brief: briefCommand, scaffold: scaffoldCommand, status: statusCommand, init: initCommand, gate: gateCommand, admit: admitCommand, sweep: sweepCommand };
133
+ const COMMANDS = { mcp: mcpCommand, replay: replayCommand, probe: probeCommand, claims: claimsCommand, baseline: baselineCommand, brief: briefCommand, scaffold: scaffoldCommand, status: statusCommand, init: initCommand, gate: gateCommand, admit: admitCommand, sweep: sweepCommand, concerns: concernsCommand };
130
134
 
131
135
  export async function main(argv, io = { out: (s) => process.stdout.write(s + '\n'), err: (s) => process.stderr.write(s + '\n') }) {
132
136
  let parsed;
@@ -154,6 +158,8 @@ export async function main(argv, io = { out: (s) => process.stdout.write(s + '\n
154
158
  exclude: { type: 'string', multiple: true },
155
159
  cap: { type: 'string' },
156
160
  'save-paths': { type: 'boolean', default: false },
161
+ concern: { type: 'string' },
162
+ concerns: { type: 'string' },
157
163
  strict: { type: 'boolean', default: false },
158
164
  explain: { type: 'boolean', default: false },
159
165
  ignore: { type: 'string' },
@@ -0,0 +1,32 @@
1
+ import { resolve } from 'node:path';
2
+ import { loadConcerns, renderConcerns } from '../supply/concerns.mjs';
3
+ import { validate } from '../../spec/lib/validate.mjs';
4
+
5
+ /**
6
+ * `testguard concerns [dir]`: the one-sentence scopes a sweep can be aimed by.
7
+ *
8
+ * A concern says WHERE to look and WHICH shapes of break are relevant. It is
9
+ * never a promise about the code — only a claim is that — so this command
10
+ * validates and lists, and has no exit code that means "something is wrong
11
+ * with your project". Exit 2 is a malformed concerns file, which is wrong with
12
+ * the FILE.
13
+ */
14
+ export async function concernsCommand({ projectDir, values, version }, io) {
15
+ const path = values.concerns ? resolve(values.concerns) : undefined;
16
+ const loaded = loadConcerns(projectDir, { path });
17
+
18
+ if (loaded.source) {
19
+ const result = validate('concerns', { schemaVersion: 1, concerns: loaded.raw });
20
+ if (!result.ok) {
21
+ io.err(`${loaded.source} does not conform to the spec:`);
22
+ for (const e of result.errors) io.err(` ${e.path}: ${e.message}`);
23
+ return 2;
24
+ }
25
+ }
26
+ if (values.json) {
27
+ io.out(JSON.stringify({ schemaVersion: 1, tool: { name: 'testguard', version }, source: loaded.source, shadowed: loaded.shadowed, concerns: loaded.concerns }, null, 2));
28
+ } else {
29
+ io.out(renderConcerns(loaded));
30
+ }
31
+ return 0;
32
+ }
@@ -1,9 +1,17 @@
1
1
  import { join, resolve } from 'node:path';
2
- import { sweep, renderSweep } from '../sweep/sweep.mjs';
2
+ import { sweep, renderSweep, ConcernError } from '../sweep/sweep.mjs';
3
3
  import { resolveChangedRef } from '../gate/changed.mjs';
4
4
  import { writeSpecDoc } from '../evidence/writer.mjs';
5
5
 
6
6
  export const sweepPath = (projectDir) => join(projectDir, '.testguard', 'sweep.json');
7
+ /**
8
+ * A sweep's own evidence, beside the canonical one and never replacing it.
9
+ * The next sweep reads it to learn which fault classes are productive HERE,
10
+ * which is the feedback half of the selection Google reports taking from 15%
11
+ * to 89% productive. Machine-proposed faults, so it is the closest observation
12
+ * of the distribution the ranker actually orders.
13
+ */
14
+ export const sweepEvidencePath = (projectDir) => join(projectDir, '.testguard', 'sweep-evidence.json');
7
15
 
8
16
  /**
9
17
  * `testguard sweep [dir] --changed <ref>`: propose faults for the changed
@@ -21,11 +29,14 @@ export const sweepPath = (projectDir) => join(projectDir, '.testguard', 'sweep.j
21
29
  */
22
30
  export async function sweepCommand({ projectDir, values, version }, io) {
23
31
  const mode = values['save-paths'] ? 'save-paths' : 'changed';
32
+ // A concern that aims at the write surface or at a glob needs no diff, the
33
+ // same as --save-paths. Only a concern that defers to the gate does.
34
+ const aimed = Boolean(values.concern) || mode === 'save-paths';
24
35
  // `--save-paths` scans the whole write surface, so it needs no diff. The gate
25
36
  // is still computed underneath (the document reports `changed`), and HEAD is
26
37
  // a reference every repository has.
27
38
  const resolved = resolveChangedRef({ explicit: values.changed, projectDir })
28
- ?? (mode === 'save-paths' ? { ref: 'HEAD', from: 'save-paths', required: false } : null);
39
+ ?? (aimed ? { ref: 'HEAD', from: values.concern ? `concern ${values.concern}` : 'save-paths', required: false } : null);
29
40
  if (!resolved) {
30
41
  io.err('sweep needs a reference to measure the change against: --changed <ref> (e.g. origin/main), or set TESTGUARD_CHANGED_REF. CI bases and a safe local remote default or differently named upstream are detected automatically. Or sweep the write surface instead with --save-paths.');
31
42
  return 3;
@@ -46,34 +57,52 @@ export async function sweepCommand({ projectDir, values, version }, io) {
46
57
  }
47
58
  if (mode === 'changed' && !resolved.required && !values.json && !values.quiet) io.err(`sweep: comparing against ${resolved.ref} (${resolved.from})`);
48
59
 
49
- const doc = await sweep({
50
- projectDir,
51
- ref: resolved.ref,
52
- mode,
53
- includeDirty: values['include-dirty'],
54
- exclude: values.exclude ?? [],
55
- cap,
56
- confirmRuns,
57
- budgetMs,
58
- runnerCommand: values['runner-cmd'],
59
- runnerName: values.runner,
60
- nodeModules: values['node-modules'] ? resolve(values['node-modules']) : process.env.TESTGUARD_NODE_MODULES,
61
- toolVersion: version,
62
- claimsPath: values.claims ? resolve(values.claims) : undefined,
63
- ignorePath: values.ignore ? resolve(values.ignore) : undefined,
64
- onStage: !values.quiet && !values.json && process.stderr.isTTY
60
+ let doc;
61
+ try {
62
+ doc = await sweep({
63
+ projectDir,
64
+ ref: resolved.ref,
65
+ mode,
66
+ concern: values.concern,
67
+ concernsPath: values.concerns ? resolve(values.concerns) : undefined,
68
+ includeDirty: values['include-dirty'],
69
+ exclude: values.exclude ?? [],
70
+ cap,
71
+ confirmRuns,
72
+ budgetMs,
73
+ runnerCommand: values['runner-cmd'],
74
+ runnerName: values.runner,
75
+ nodeModules: values['node-modules'] ? resolve(values['node-modules']) : process.env.TESTGUARD_NODE_MODULES,
76
+ toolVersion: version,
77
+ claimsPath: values.claims ? resolve(values.claims) : undefined,
78
+ ignorePath: values.ignore ? resolve(values.ignore) : undefined,
79
+ onStage: !values.quiet && !values.json && process.stderr.isTTY
65
80
  ? ({ claimId, faultId, stage, i, n }) => process.stderr.write(`\r\x1b[K … ${claimId}/${faultId} ${stage} ${i}/${n}`)
66
81
  : undefined,
67
- onWarn: (m) => { if (!values.quiet) io.err(`warning: ${m}`); },
68
- });
82
+ onWarn: (m) => { if (!values.quiet) io.err(`warning: ${m}`); },
83
+ });
84
+ } catch (e) {
85
+ // A concern named on the command line that this project does not declare
86
+ // is a usage error, not a precondition failure: nothing about the
87
+ // repository is wrong.
88
+ if (e instanceof ConcernError) {
89
+ io.err(e.message);
90
+ return 3;
91
+ }
92
+ throw e;
93
+ }
69
94
  if (process.stderr.isTTY && !values.quiet && !values.json) process.stderr.write('\r\x1b[K');
70
95
 
96
+ const { evidence, ...document } = doc;
71
97
  const outPath = values.out ? resolve(values.out) : sweepPath(projectDir);
72
- writeSpecDoc('sweep', outPath, doc);
98
+ writeSpecDoc('sweep', outPath, document);
99
+ // Written only when something was actually probed: an empty document would
100
+ // teach the next run that every class is unproductive.
101
+ if (evidence?.records?.length) writeSpecDoc('evidence', sweepEvidencePath(projectDir), evidence);
73
102
  if (values.json) {
74
- io.out(JSON.stringify(doc, null, 2));
103
+ io.out(JSON.stringify(document, null, 2));
75
104
  } else {
76
- io.out(renderSweep(doc, { limit: Number(values.max) || 20 }));
105
+ io.out(renderSweep(document, { limit: Number(values.max) || 20 }));
77
106
  if (!values.quiet) io.out(`\nsweep: ${outPath}`);
78
107
  }
79
108
  return doc.exitCode;
@@ -0,0 +1,162 @@
1
+ /**
2
+ * A concern: one sentence a human writes, that generates faults across a whole
3
+ * surface.
4
+ *
5
+ * The rung between "nothing written" and "a claim per behaviour". A claim names
6
+ * one promise and the faults that would break it; that is precise and it is why
7
+ * a repository with two hundred screens never finishes writing them. A concern
8
+ * names a KIND of promise — "every write persists the data it was given" — and
9
+ * says where to look for it and which shapes of break are relevant. One
10
+ * sentence, two hundred screens.
11
+ *
12
+ * The idea is Meta's. ACH has an engineer describe an area of concern in plain
13
+ * text, generates faults specific to it, and gates every step by execution
14
+ * ("Mutation-Guided LLM-based Test Generation at Meta", Foster et al., FSE
15
+ * 2025). What is taken here is the UNIT — the concern as the thing a human
16
+ * writes — not their generator.
17
+ *
18
+ * NO MODEL IS REQUIRED, and that is deliberate. A concern's useful core is a
19
+ * named scope plus a producer selection, both of which the mechanical
20
+ * producers already satisfy. An LLM can later propose faults a line-oriented
21
+ * producer cannot see, as an ADDITIONAL source for a concern that already
22
+ * works — rather than as the thing the rung depends on. That keeps every
23
+ * verifying command offline, which is the property this tool is least willing
24
+ * to trade.
25
+ *
26
+ * A concern is NOT a claim and never becomes one by itself. It says where to
27
+ * look; the probe says what it found; a human states the sentence worth
28
+ * defending. Nothing here is written to `testguard.claims.json`.
29
+ */
30
+ import { existsSync, readFileSync } from 'node:fs';
31
+ import { join } from 'node:path';
32
+ import { saveSurface } from './savepath.mjs';
33
+ import { globToRegExp, walk } from '../util/glob.mjs';
34
+
35
+ export const defaultConcernsPath = (projectDir) => join(projectDir, 'testguard.concerns.json');
36
+
37
+ /**
38
+ * The concerns every project gets without writing anything.
39
+ *
40
+ * Two, because two are the ones whose targets can be enumerated honestly. An
41
+ * authorization concern would need a heuristic for "which files check
42
+ * permissions", and a guess there produces exactly the non-actionable noise
43
+ * that gets a check switched off. A project that knows its own auth layer can
44
+ * say so with a `glob` target in three lines; this file will not guess for it.
45
+ */
46
+ export const BUILTIN_CONCERNS = Object.freeze([
47
+ Object.freeze({
48
+ id: 'SAVE-PERSISTS',
49
+ statement: 'Every write to storage persists the data it was given.',
50
+ severity: 'high',
51
+ targets: { kind: 'write-sites' },
52
+ // The shapes that break a persisted payload. Measured on a real
53
+ // application: 12 of 12 surviving faults on the persistence path were a
54
+ // field dropped from the payload, so the list leads with that.
55
+ faultClasses: ['field-dropped', 'call-removed', 'statement-deleted', 'literal-changed'],
56
+ builtin: true,
57
+ }),
58
+ Object.freeze({
59
+ id: 'CHANGED-CODE',
60
+ statement: 'Code changed in this branch is defended by something.',
61
+ severity: 'medium',
62
+ targets: { kind: 'changed' },
63
+ // Everything: a diff has no shape to narrow by.
64
+ faultClasses: null,
65
+ builtin: true,
66
+ }),
67
+ ]);
68
+
69
+ /**
70
+ * Load the project's concerns, with the built-ins behind them.
71
+ *
72
+ * A project concern with the same id as a built-in REPLACES it, so a team can
73
+ * retune `SAVE-PERSISTS` for an ORM this tool does not recognise without
74
+ * forking anything. Shadowing is recorded rather than silent: a concern that
75
+ * quietly replaced a default would be the kind of surprise that costs an
76
+ * afternoon.
77
+ */
78
+ export function loadConcerns(projectDir, { path } = {}) {
79
+ const p = path ?? defaultConcernsPath(projectDir);
80
+ let project = [];
81
+ let source = null;
82
+ if (existsSync(p)) {
83
+ const doc = JSON.parse(readFileSync(p, 'utf8'));
84
+ project = doc.concerns ?? [];
85
+ source = p;
86
+ }
87
+ const overridden = new Set(project.map((c) => c.id));
88
+ const concerns = [
89
+ ...project.map((c) => ({ ...c, builtin: false })),
90
+ ...BUILTIN_CONCERNS.filter((c) => !overridden.has(c.id)),
91
+ ];
92
+ return {
93
+ concerns,
94
+ // The project's own entries, exactly as written. Validation runs against
95
+ // these: the enriched objects above carry a `builtin` marker this tool
96
+ // added, and a schema that refuses unknown properties is right to reject
97
+ // them. Validating what we constructed instead of what the user wrote
98
+ // would report OUR bug as THEIR malformed file.
99
+ raw: project.map(({ builtin, ...c }) => c),
100
+ source,
101
+ shadowed: BUILTIN_CONCERNS.filter((c) => overridden.has(c.id)).map((c) => c.id),
102
+ };
103
+ }
104
+
105
+ /** One concern by id, or null. Ids are matched exactly: a near-miss is a typo, not an intention. */
106
+ export const concernById = (concerns, id) => concerns.find((c) => c.id === id) ?? null;
107
+
108
+ /**
109
+ * The files a concern applies to.
110
+ *
111
+ * `write-sites` asks the enumerator; `glob` matches the project's own paths;
112
+ * `changed` returns null, meaning "the caller's diff decides" — a concern does
113
+ * not get to re-derive what the gate already computed.
114
+ */
115
+ export function targetsFor(projectDir, concern, { changedFiles } = {}) {
116
+ const t = concern.targets ?? { kind: 'changed' };
117
+ if (t.kind === 'write-sites') return saveSurface(projectDir).files.map((f) => f.file);
118
+ if (t.kind === 'glob') {
119
+ // Against the whole project, never the diff. A concern like "every admin
120
+ // route refuses an unauthorised caller" is about the admin routes, not
121
+ // about the ones that happened to change today — and pooling from the diff
122
+ // made such a concern report clean on a clean branch, which is the exact
123
+ // false reassurance a concern exists to prevent.
124
+ const res = (t.globs ?? []).map(globToRegExp);
125
+ return walk(projectDir).filter((f) => res.some((re) => re.test(f)));
126
+ }
127
+ return changedFiles ?? null;
128
+ }
129
+
130
+ /**
131
+ * Keep only the faults whose class this concern cares about.
132
+ *
133
+ * `faultClasses: null` means every class — the honest default for a concern
134
+ * with no shape to narrow by. Narrowing is what makes one sentence useful
135
+ * across two hundred files: a persistence concern that also reported every
136
+ * altered return value would bury the payload findings it exists to surface.
137
+ */
138
+ export function filterByConcern(candidates, concern) {
139
+ if (!concern?.faultClasses) return candidates;
140
+ const keep = new Set(concern.faultClasses);
141
+ return candidates.filter((c) => keep.has(c.fault.faultClass));
142
+ }
143
+
144
+ /** The concerns as text, for someone deciding which to sweep. */
145
+ export function renderConcerns({ concerns, source, shadowed }) {
146
+ const out = [];
147
+ out.push(source ? `${concerns.length} concern${concerns.length === 1 ? '' : 's'} (${source}, plus the built-ins)` : `${concerns.length} built-in concern${concerns.length === 1 ? '' : 's'} — none declared in this project.`);
148
+ out.push('');
149
+ for (const c of concerns) {
150
+ const shape = c.targets?.kind === 'glob' ? `glob ${(c.targets.globs ?? []).join(', ')}` : c.targets?.kind ?? 'changed';
151
+ out.push(` ${c.id}${c.builtin ? ' (built-in)' : ''}`);
152
+ out.push(` ${c.statement}`);
153
+ out.push(` targets: ${shape} · faults: ${c.faultClasses ? c.faultClasses.join(', ') : 'every class'}`);
154
+ out.push('');
155
+ }
156
+ if (shadowed.length) out.push(`This project replaces the built-in ${shadowed.join(', ')}.`);
157
+ if (!source) {
158
+ out.push('Declare your own in testguard.concerns.json. A concern is one sentence and a');
159
+ out.push('scope: it says where to look, never what is true. Only a claim does that.');
160
+ }
161
+ return out.join('\n');
162
+ }
@@ -0,0 +1,146 @@
1
+ /**
2
+ * What this project's own runs say about which faults are worth proposing.
3
+ *
4
+ * `select.mjs` ships a productivity prior measured on one real application.
5
+ * That is a reasonable starting point and a poor stopping point: the classes
6
+ * that survive a Next.js suite are not the classes that survive a Python one,
7
+ * and Google's mutation service got from a 15% productive rate to 89% on
8
+ * exactly this — feedback, accumulated over years, refining which candidates
9
+ * are worth a developer's attention ("Practical Mutation Testing at Scale",
10
+ * Petrović et al., 2021). The static table is the part of that idea this tool
11
+ * had not yet taken.
12
+ *
13
+ * THE FEEDBACK IS ALREADY ON DISK. Every evidence record carries a
14
+ * `faultClass` and a `verdict`, so "how often does a fault of this class
15
+ * survive here" is a tally, not a new thing to capture. There is no click to
16
+ * collect and no state to maintain: a project that has probed anything has
17
+ * already said what its own productive classes are.
18
+ *
19
+ * WHY THIS ESTIMATOR AND NOT A BETTER-SOUNDING ONE. The combination below is a
20
+ * Beta-Binomial posterior mean, and a posterior mean is what minimises Brier
21
+ * and log score in expectation — it is a STRICTLY PROPER estimate, which is
22
+ * the only kind that cannot be improved by reporting something other than an
23
+ * honest probability. Optimising the ordering for accuracy or F1 instead would
24
+ * maximise the number of survivors found while destroying the calibration of
25
+ * the number attached to each, which is the documented failure mode of naive
26
+ * optimisation and the reason Laya trains against proper scoring rules. If
27
+ * this file is ever changed to rank by anything but an honest probability,
28
+ * that is the invariant being broken.
29
+ *
30
+ * Pure apart from reading the documents it is given.
31
+ */
32
+ import { existsSync, readFileSync } from 'node:fs';
33
+ import { join } from 'node:path';
34
+ import { MEASURED_PRODUCTIVITY } from './select.mjs';
35
+
36
+ /** The evidence a project may have produced, in the order they are most relevant. */
37
+ export const EVIDENCE_PATHS = Object.freeze([
38
+ // A sweep probes MACHINE-proposed faults, the same distribution the ranker
39
+ // orders, so it is the closest observation available.
40
+ '.testguard/sweep-evidence.json',
41
+ // The canonical probe is human-authored faults: a real observation of the
42
+ // same classes, drawn slightly differently, and still worth counting.
43
+ '.testguard/evidence.json',
44
+ ]);
45
+
46
+ /**
47
+ * Survival counts per fault class from one evidence document.
48
+ *
49
+ * Only `killed` and `survived` are counted. `nocover`, `unverifiable`,
50
+ * `fault-invalid`, `timeout` and `flaky-defender` say nothing about whether a
51
+ * fault of that class would be NOTICED — they say the question could not be
52
+ * asked — and folding them in either direction would be inventing data.
53
+ */
54
+ export function tallyEvidence(doc) {
55
+ const by = {};
56
+ for (const r of doc?.records ?? []) {
57
+ const cls = r.subject?.faultClass;
58
+ if (!cls) continue;
59
+ if (r.verdict !== 'killed' && r.verdict !== 'survived') continue;
60
+ const e = (by[cls] ??= { survived: 0, n: 0 });
61
+ e.n += 1;
62
+ if (r.verdict === 'survived') e.survived += 1;
63
+ }
64
+ return by;
65
+ }
66
+
67
+ /** Merge tallies from several documents. Counts add; nothing is averaged twice. */
68
+ export function mergeTallies(tallies) {
69
+ const by = {};
70
+ for (const t of tallies) {
71
+ for (const [cls, e] of Object.entries(t)) {
72
+ const m = (by[cls] ??= { survived: 0, n: 0 });
73
+ m.survived += e.survived;
74
+ m.n += e.n;
75
+ }
76
+ }
77
+ return by;
78
+ }
79
+
80
+ /**
81
+ * The productivity table to rank with: this project's own observations, with
82
+ * the shipped measurement as the prior they are shrunk toward.
83
+ *
84
+ * The shipped numbers become pseudo-counts rather than a competing estimate —
85
+ * `n` of them, so a class measured on twelve faults elsewhere carries the
86
+ * weight of twelve observations here and no more. A project with its own
87
+ * hundred records therefore overrides the shipping default; a project with
88
+ * three nudges it.
89
+ *
90
+ * `shipped` is injectable so the combination can be tested without depending
91
+ * on whatever the current measurement happens to be.
92
+ */
93
+ export function combinedProductivity(local, shipped = MEASURED_PRODUCTIVITY) {
94
+ const out = {};
95
+ for (const cls of new Set([...Object.keys(shipped), ...Object.keys(local)])) {
96
+ const s = shipped[cls];
97
+ const l = local[cls];
98
+ const priorSurvived = s ? s.p * s.n : 0;
99
+ const priorN = s ? s.n : 0;
100
+ const n = priorN + (l?.n ?? 0);
101
+ if (n === 0) continue;
102
+ const survived = priorSurvived + (l?.survived ?? 0);
103
+ out[cls] = { p: survived / n, n, observed: l?.n ?? 0 };
104
+ }
105
+ return out;
106
+ }
107
+
108
+ /** Read whichever evidence documents this project has. Missing or malformed ones are skipped, never guessed at. */
109
+ export function readEvidence(projectDir, paths = EVIDENCE_PATHS) {
110
+ const docs = [];
111
+ for (const rel of paths) {
112
+ const p = join(projectDir, rel);
113
+ if (!existsSync(p)) continue;
114
+ try {
115
+ docs.push({ path: rel, doc: JSON.parse(readFileSync(p, 'utf8')) });
116
+ } catch {
117
+ // A document this tool cannot parse is not evidence of anything. It is
118
+ // skipped rather than defaulted, because a ranking built on a guess
119
+ // about a corrupt file is worse than one built on the shipped prior.
120
+ }
121
+ }
122
+ return docs;
123
+ }
124
+
125
+ /**
126
+ * The learned table plus what it was learned from, so a report can say whether
127
+ * an ordering came from this project or from the shipping default. An ordering
128
+ * nobody can trace is a number nobody should trust.
129
+ */
130
+ export function learnedProductivity(projectDir, { paths, shipped } = {}) {
131
+ const docs = readEvidence(projectDir, paths);
132
+ const local = mergeTallies(docs.map((d) => tallyEvidence(d.doc)));
133
+ const observed = Object.values(local).reduce((a, e) => a + e.n, 0);
134
+ return {
135
+ table: combinedProductivity(local, shipped),
136
+ sources: docs.map((d) => d.path),
137
+ observed,
138
+ classes: Object.keys(local).sort(),
139
+ };
140
+ }
141
+
142
+ /** One line for a report: where the ordering came from. */
143
+ export function renderLearned(learned) {
144
+ if (learned.observed === 0) return 'ordering: the shipped productivity prior — this project has no probed evidence yet.';
145
+ return `ordering: ${learned.observed} probed fault${learned.observed === 1 ? '' : 's'} of this project's own, across ${learned.classes.length} class${learned.classes.length === 1 ? '' : 'es'} (${learned.sources.join(', ')}), shrunk toward the shipped prior.`;
146
+ }
@@ -171,3 +171,56 @@ export function persistenceHint(signal, faultLine) {
171
171
  const field = /^\s*([A-Za-z_$][\w$]*)\s*(?::|,\s*$)/.exec(faultLine ?? '')?.[1];
172
172
  return `${signal.file} ${signal.reason} Assert the whole object written by the call this fault changes${field ? `, including \`${field}\`` : ''} — or read the record back and assert on what was stored.`;
173
173
  }
174
+
175
+ /**
176
+ * What this claim's defenders are CAPABLE of proving about a write.
177
+ *
178
+ * The read-back question — "did the row actually land?" — cannot be answered by
179
+ * a test that replaced the database with a mock. Such a test can prove the call
180
+ * shape and nothing beyond it: the `where` may match no rows, the transaction
181
+ * may roll back, a constraint may reject the write, and every one of those
182
+ * passes against a mock that recorded the arguments and returned a plausible
183
+ * object.
184
+ *
185
+ * So rather than pretend to measure persistence, this states the LIMIT. It is
186
+ * the same discipline as `nocover` — which says "no test imports this" instead
187
+ * of guessing — one level finer: these tests exist, they run, and there is a
188
+ * class of failure they structurally cannot see.
189
+ *
190
+ * `mocked` every defender replaces the persistence layer. Call shape only.
191
+ * `unmocked` at least one does not, so a real write is at least possible.
192
+ * NOT a promise that one happens: a test may simply never reach
193
+ * the database. Possible is the honest word.
194
+ * `none` no defender at all; `nocover` already covers this.
195
+ *
196
+ * Measured on a real AI-authored application: of 37 files that write to
197
+ * storage, 32 were defended only by mocking tests, 4 had no defender, and 1
198
+ * had an unmocked defender — a mailer, not a database write. Nothing in that
199
+ * suite could prove any save persisted.
200
+ */
201
+ export function persistenceProvability(projectDir, defenders) {
202
+ if (!defenders || defenders.length === 0) return { class: 'none', mocking: [], unmocked: [] };
203
+ const mocking = [];
204
+ const unmocked = [];
205
+ for (const d of defenders) (analyzePersistence(projectDir, d) ? mocking : unmocked).push(d);
206
+ return { class: unmocked.length ? 'unmocked' : 'mocked', mocking, unmocked };
207
+ }
208
+
209
+ /**
210
+ * The provability of a whole surface, as counts. The headline a save-path
211
+ * report opens with, because "how many of these could be proven at all" is a
212
+ * different question from "how many were probed" and a far more alarming one.
213
+ */
214
+ export function provabilitySummary(projectDir, targets, defendersOf) {
215
+ const counts = { mocked: 0, unmocked: 0, none: 0 };
216
+ for (const file of targets) {
217
+ let defenders = [];
218
+ try {
219
+ defenders = defendersOf(file);
220
+ } catch {
221
+ defenders = [];
222
+ }
223
+ counts[persistenceProvability(projectDir, defenders).class] += 1;
224
+ }
225
+ return counts;
226
+ }
@@ -35,9 +35,12 @@ import { loadClaims, defaultClaimsPath } from '../claims/load.mjs';
35
35
  import { computeChangedGate } from '../gate/changed.mjs';
36
36
  import { scaffoldFile } from '../scaffold/scaffold.mjs';
37
37
  import { selectFaults, onWritePath, capFor } from '../supply/select.mjs';
38
+ import { learnedProductivity } from '../supply/feedback.mjs';
39
+ import { loadConcerns, concernById, targetsFor, filterByConcern } from '../supply/concerns.mjs';
38
40
  import { saveSurface } from '../supply/savepath.mjs';
39
- import { persistenceSignals, persistenceHint } from '../supply/persistence.mjs';
41
+ import { persistenceSignals, persistenceHint, provabilitySummary } from '../supply/persistence.mjs';
40
42
  import { probe } from '../probe/probe.mjs';
43
+ import { discoverDefenders } from '../probe/discover.mjs';
41
44
  import { hintFor } from '../brief/brief.mjs';
42
45
 
43
46
  /**
@@ -106,6 +109,9 @@ export function sortFindings(findings) {
106
109
  /** How many findings a sweep is willing to fail on. */
107
110
  export const gatingCount = (findings) => findings.filter((f) => GATING.has(f.verdict)).length;
108
111
 
112
+ /** A concern named on the command line that this project does not declare. */
113
+ export class ConcernError extends Error {}
114
+
109
115
  /** Rebuild claim documents from a flat selection, keeping each claim's own faults together. */
110
116
  function claimsFromSelection(selected) {
111
117
  const byClaim = new Map();
@@ -128,6 +134,8 @@ export async function sweep({
128
134
  projectDir,
129
135
  ref,
130
136
  mode = 'changed',
137
+ concern: concernId,
138
+ concernsPath,
131
139
  includeDirty = false,
132
140
  exclude = [],
133
141
  cap,
@@ -154,10 +162,23 @@ export async function sweep({
154
162
  //
155
163
  // A test file with no claim is a different problem — it has nothing to
156
164
  // falsify — and the gate already names it. Sweep proposes against source.
157
- const surface = mode === 'save-paths' ? saveSurface(projectDir) : null;
158
- const targets = surface
159
- ? surface.files.map((f) => f.file)
160
- : gate.uncovered.filter((u) => u.kind === 'source').map((u) => u.file);
165
+ // `--save-paths` is sugar for the built-in SAVE-PERSISTS concern: the same
166
+ // targets and the same producer selection, named so a project can retune it
167
+ // for an ORM this tool does not recognise without forking anything.
168
+ const loaded = loadConcerns(projectDir, { path: concernsPath });
169
+ const concern = concernId
170
+ ? concernById(loaded.concerns, concernId)
171
+ : mode === 'save-paths' ? concernById(loaded.concerns, 'SAVE-PERSISTS') : null;
172
+ if (concernId && !concern) {
173
+ throw new ConcernError(`unknown concern ${concernId}. Declared: ${loaded.concerns.map((c) => c.id).join(', ')}`);
174
+ }
175
+ const changedTargets = gate.uncovered.filter((u) => u.kind === 'source').map((u) => u.file);
176
+ // A concern whose targets are the diff defers to the gate rather than
177
+ // re-deriving what it already computed.
178
+ const concernTargets = concern ? targetsFor(projectDir, concern, { changedFiles: changedTargets }) : null;
179
+ const usesSurface = concern?.targets?.kind === 'write-sites' || mode === 'save-paths';
180
+ const surface = usesSurface ? saveSurface(projectDir) : null;
181
+ const targets = concernTargets ?? changedTargets;
161
182
 
162
183
  const cPath = claimsPath ?? defaultClaimsPath(projectDir);
163
184
  const existingClaims = existsSync(cPath) ? loadClaims(cPath) : { claims: [] };
@@ -184,8 +205,27 @@ export async function sweep({
184
205
  for (const claim of result.doc.claims) for (const fault of claim.faults) candidates.push({ claim, fault });
185
206
  }
186
207
 
208
+ // What the defenders of this surface are CAPABLE of proving, before anything
209
+ // is probed. A test that replaced the database can prove the call shape and
210
+ // never that the row landed, and that limit is a property of the suite rather
211
+ // than of any verdict — so it is measured once, over the whole surface, and
212
+ // reported whether or not a single fault survives.
213
+ const provability = surface
214
+ ? provabilitySummary(projectDir, targets, (f) => discoverDefenders(projectDir, f))
215
+ : null;
216
+
217
+ // Order by what THIS project's own runs have shown, not only by what was
218
+ // measured elsewhere. A project that has probed nothing falls back to the
219
+ // shipped prior; one with its own records overrides it in proportion to how
220
+ // many it has. This is the feedback half of Google's 15%-to-89%, using data
221
+ // the project already produced rather than a new thing to collect.
222
+ const learned = learnedProductivity(projectDir);
223
+ // Narrowing is what makes one sentence useful across two hundred files: a
224
+ // persistence concern that also reported every altered return value would
225
+ // bury the payload findings it exists to surface.
226
+ const relevant = filterByConcern(candidates, concern);
187
227
  const limit = cap ?? capFor(targets.length);
188
- const selection = selectFaults(candidates, { cap: limit });
228
+ const selection = selectFaults(relevant, { cap: limit, table: learned.table });
189
229
  const claims = claimsFromSelection(selection.selected);
190
230
 
191
231
  let evidence = { records: [] };
@@ -234,20 +274,28 @@ export async function sweep({
234
274
  includeDirty,
235
275
  scope: {
236
276
  mode,
277
+ ...(concern ? { concern: concern.id } : {}),
237
278
  changed: gate.changed,
238
- ...(surface ? { writeSites: surface.siteCount, payloadFields: surface.keyCount } : {}),
279
+ ...(surface ? { writeSites: surface.siteCount, payloadFields: surface.keyCount, provability } : {}),
239
280
  targets: targets.length,
240
281
  swept: drafts.length,
241
282
  ...(skipped.length ? { skipped } : {}),
242
283
  },
243
284
  selection: {
244
- proposed: candidates.length,
285
+ proposed: relevant.length,
286
+ ...(candidates.length !== relevant.length ? { outOfScope: candidates.length - relevant.length } : {}),
245
287
  selected: selection.selected.length,
246
288
  deferred: selection.deferred.length,
247
289
  cap: selection.cap,
248
290
  byClass: selection.byClass,
249
291
  },
250
292
  counts,
293
+ ordering: { observed: learned.observed, sources: learned.sources },
294
+ // Carried, not written here: the caller owns I/O. Persisting it is what
295
+ // closes the feedback loop — the next sweep learns from these verdicts,
296
+ // which are MACHINE-proposed faults and so the closest observation of the
297
+ // distribution the ranker orders.
298
+ evidence,
251
299
  findings: ordered,
252
300
  exitCode: gating > 0 ? 1 : 0,
253
301
  drafts,
@@ -262,16 +310,44 @@ export function renderSweep(doc, { limit = 20 } = {}) {
262
310
  if (scope.targets === 0) {
263
311
  out.push(saves
264
312
  ? 'sweep: no write to storage found in this project. Nothing to propose.'
265
- : `sweep: no changed source file is without a claim against ${doc.ref}. Nothing to propose.`);
313
+ : scope.concern
314
+ ? `sweep: concern ${scope.concern} matches no file in this project. Nothing to propose.`
315
+ : `sweep: no changed source file is without a claim against ${doc.ref}. Nothing to propose.`);
266
316
  return out.join('\n');
267
317
  }
268
318
  // The denominator first. A report that lists findings without saying how much
269
319
  // was looked at invites the reader to assume the rest is fine.
320
+ // Say how the sweep was aimed. A glob concern that reports "unclaimed changed
321
+ // files" is describing a diff it never looked at.
270
322
  out.push(saves
271
323
  ? `${scope.writeSites} write${scope.writeSites === 1 ? '' : 's'} to storage across ${scope.targets} file${scope.targets === 1 ? '' : 's'}, carrying ${scope.payloadFields} payload field${scope.payloadFields === 1 ? '' : 's'}. Swept ${scope.swept}.`
272
- : `swept ${scope.swept} of ${scope.targets} unclaimed changed file${scope.targets === 1 ? '' : 's'} against ${doc.ref}.`);
324
+ : scope.concern
325
+ ? `concern ${scope.concern} matches ${scope.targets} file${scope.targets === 1 ? '' : 's'}. Swept ${scope.swept}.`
326
+ : `swept ${scope.swept} of ${scope.targets} unclaimed changed file${scope.targets === 1 ? '' : 's'} against ${doc.ref}.`);
327
+ if (saves && scope.provability) {
328
+ // The limit, before any verdict. A test that replaced the database can
329
+ // prove the call shape and never that the row landed, so on a surface
330
+ // defended entirely by such tests a clean probe is not evidence of
331
+ // persistence — it is evidence that nothing could have measured it.
332
+ const { mocked, unmocked, none } = scope.provability;
333
+ out.push(` of those ${scope.targets} file${scope.targets === 1 ? '' : 's'}: ${mocked} defended only by tests that mock the persistence layer, ${unmocked} with an unmocked defender, ${none} with no defender at all.`);
334
+ if (unmocked === 0 && mocked > 0) {
335
+ out.push(' Nothing in this suite can prove a write reached storage. A mocked test proves the');
336
+ out.push(' call shape; a where-clause that matches nothing, a rolled-back transaction and a');
337
+ out.push(' rejected constraint all pass against it.');
338
+ }
339
+ }
273
340
  out.push(`proposed ${selection.proposed} fault${selection.proposed === 1 ? '' : 's'}, probed ${selection.selected} (cap ${selection.cap})${selection.deferred ? `, deferred ${selection.deferred}` : ''}.`);
274
- if (selection.deferred) out.push(` The ${selection.deferred} deferred are not a verdict: raise --cap${saves ? '.' : ', or sweep a smaller change.'}`);
341
+ if (selection.deferred) out.push(` The ${selection.deferred} deferred are not a verdict: raise --cap${saves || scope.concern ? '.' : ', or sweep a smaller change.'}`);
342
+ // A narrow concern must not look like a quiet one: the producers found these,
343
+ // and this concern is not about them.
344
+ if (selection.outOfScope) out.push(` ${selection.outOfScope} further proposal${selection.outOfScope === 1 ? ' was' : 's were'} outside this concern's fault classes.`);
345
+ // An ordering nobody can trace is a number nobody should trust.
346
+ if (doc.ordering) {
347
+ out.push(doc.ordering.observed === 0
348
+ ? ' ordering: the shipped productivity prior — this project has no probed evidence yet.'
349
+ : ` ordering: ${doc.ordering.observed} probed fault${doc.ordering.observed === 1 ? '' : 's'} of this project's own (${doc.ordering.sources.join(', ')}), shrunk toward the shipped prior.`);
350
+ }
275
351
  for (const s of scope.skipped ?? []) out.push(` skipped ${s.file}: ${s.reason}`);
276
352
  out.push('');
277
353