testguard-cli 0.12.0 → 0.13.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +57 -0
- package/README.md +86 -8
- package/package.json +1 -1
- package/spec/lib/validate.mjs +38 -2
- package/spec/schemas/concerns.schema.json +41 -0
- package/spec/schemas/sweep.schema.json +56 -0
- package/src/cli.mjs +7 -1
- package/src/commands/concerns.mjs +32 -0
- package/src/commands/sweep.mjs +52 -23
- package/src/supply/concerns.mjs +162 -0
- package/src/supply/feedback.mjs +146 -0
- package/src/supply/persistence.mjs +53 -0
- package/src/sweep/sweep.mjs +87 -11
package/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [0.13.0] - 2026-09-22
|
|
11
|
+
|
|
12
|
+
Automated weekly release — everything merged since `v0.12.0`.
|
|
13
|
+
|
|
14
|
+
### Changed
|
|
15
|
+
|
|
16
|
+
- chore(homebrew): sha256 for v0.12.0 (#159)
|
|
17
|
+
- feat: persistence provability, and an ordering that learns from this project's own verdicts (#157)
|
|
18
|
+
|
|
19
|
+
|
|
10
20
|
## [0.12.0] - 2026-09-22
|
|
11
21
|
|
|
12
22
|
Automated weekly release — everything merged since `v0.11.0`.
|
|
@@ -18,6 +28,53 @@ Automated weekly release — everything merged since `v0.11.0`.
|
|
|
18
28
|
- chore(homebrew): sha256 for v0.11.0 (#153)
|
|
19
29
|
|
|
20
30
|
|
|
31
|
+
### Added
|
|
32
|
+
|
|
33
|
+
- **Concerns** — one sentence a human writes that generates faults across a
|
|
34
|
+
whole surface, the rung between writing nothing and writing a claim per
|
|
35
|
+
behaviour. A concern says *where* to look and *which shapes of break* are
|
|
36
|
+
relevant: `"every write persists the data it was given"` targets the write
|
|
37
|
+
sites and applies the payload producers. `testguard concerns` lists them,
|
|
38
|
+
`sweep --concern <ID>` aims a sweep by one, and `--save-paths` is now sugar
|
|
39
|
+
for the built-in `SAVE-PERSISTS`.
|
|
40
|
+
- Two built-ins ship, and only two: an authorization concern would need a
|
|
41
|
+
heuristic for "which files check permissions", and a guess there is the
|
|
42
|
+
non-actionable noise that gets a check switched off. A project that knows
|
|
43
|
+
its own auth layer says so with a `glob` target in three lines.
|
|
44
|
+
- A project concern with the same id **replaces** a built-in, so a team can
|
|
45
|
+
retune `SAVE-PERSISTS` for an ORM this tool does not recognise without
|
|
46
|
+
forking anything. The replacement is reported, never silent.
|
|
47
|
+
- `urn:claimspec:v1:concerns`, with rules that refuse a glob target naming no
|
|
48
|
+
globs (it would match nothing and report clean) and an empty fault-class
|
|
49
|
+
list (a typo for "every class", every time).
|
|
50
|
+
- **No model is required, deliberately.** A concern's useful core is a named
|
|
51
|
+
scope plus a producer selection, which the mechanical producers already
|
|
52
|
+
satisfy — so every verifying command stays offline and NFR-01 is unchanged.
|
|
53
|
+
An LLM can later propose faults a line-oriented producer cannot see, as an
|
|
54
|
+
additional source for a concern that already works.
|
|
55
|
+
|
|
56
|
+
### Added
|
|
57
|
+
|
|
58
|
+
- A save-path sweep now reports **what its defenders could prove**, before it
|
|
59
|
+
reports what they did: per file, whether every defender mocks the persistence
|
|
60
|
+
layer (call shape only), at least one does not (a real write is possible), or
|
|
61
|
+
there is no defender at all. When nothing is unmocked the report says so
|
|
62
|
+
outright — a clean probe there is not evidence of persistence, it is evidence
|
|
63
|
+
that nothing could have measured it. Measured on a real AI-authored
|
|
64
|
+
application: of 37 files that write to storage, 32 were defended only by
|
|
65
|
+
mocking tests.
|
|
66
|
+
- `scope.provability` on the sweep document, with a validator rule that every
|
|
67
|
+
target is classified exactly once.
|
|
68
|
+
- The candidate ordering now **learns from this project's own probed verdicts**
|
|
69
|
+
instead of only the shipped productivity prior. Every evidence record already
|
|
70
|
+
carries a `faultClass` and a `verdict`, so "how often does a fault of this
|
|
71
|
+
class survive *here*" is a tally rather than a new thing to capture. A sweep
|
|
72
|
+
persists its own evidence to `.testguard/sweep-evidence.json` — beside the
|
|
73
|
+
canonical one, never replacing it — so the next run learns from
|
|
74
|
+
machine-proposed faults, the same distribution the ranker orders. The sweep
|
|
75
|
+
document records `ordering.observed` and `ordering.sources`, because an
|
|
76
|
+
ordering nobody can trace is a number nobody should trust.
|
|
77
|
+
|
|
21
78
|
### Added
|
|
22
79
|
|
|
23
80
|
- `sweep --save-paths` — sweep every file that writes to storage instead of the
|
package/README.md
CHANGED
|
@@ -127,9 +127,9 @@ raises coverage. TestGuard admits it because it fails when the claim is false.
|
|
|
127
127
|
| npm | `npm i -D testguard-cli` then `npx testguard probe` |
|
|
128
128
|
| pip | `pip install testguard-cli` then `testguard probe` (needs Node ≥ 20) |
|
|
129
129
|
| Homebrew | `brew tap raccioly/tap && brew install testguard` |
|
|
130
|
-
| GitHub Action | `uses: raccioly/testguard@v0.
|
|
130
|
+
| GitHub Action | `uses: raccioly/testguard@v0.13.0` — see [`action.yml`](./action.yml) |
|
|
131
131
|
| pre-commit | `repo: https://github.com/raccioly/testguard`, hooks `testguard-claims`, `testguard-probe` |
|
|
132
|
-
| GitLab CI | `include: - remote: https://raw.githubusercontent.com/raccioly/testguard/v0.
|
|
132
|
+
| GitLab CI | `include: - remote: https://raw.githubusercontent.com/raccioly/testguard/v0.13.0/packaging/gitlab/testguard.gitlab-ci.yml` with `inputs:` — see [`packaging/gitlab/`](./packaging/gitlab/testguard.gitlab-ci.yml) |
|
|
133
133
|
|
|
134
134
|
Projects that set `min-release-age` in `.npmrc` cannot see a version published
|
|
135
135
|
less than that many days ago (`ENOVERSIONS`); install that one with
|
|
@@ -451,6 +451,14 @@ Three rules make it safe to run on a repository that has never seen this tool:
|
|
|
451
451
|
is reported and never gates — a tool that fails because its own guess was bad
|
|
452
452
|
is a tool people switch off.
|
|
453
453
|
|
|
454
|
+
The ordering **learns from your own runs**. Every evidence record already
|
|
455
|
+
carries a fault class and a verdict, so "how often does a fault of this class
|
|
456
|
+
survive here" is a tally, not a new thing to collect — a sweep persists its
|
|
457
|
+
evidence and the next one reads it. A project with no evidence falls back to
|
|
458
|
+
the shipped prior and says so; one with its own records overrides it in
|
|
459
|
+
proportion to how many it has. The report names which, because an ordering
|
|
460
|
+
nobody can trace is a number nobody should trust.
|
|
461
|
+
|
|
454
462
|
The cap and the ordering are Google's: their mutation service surfaces at most
|
|
455
463
|
7 × |files| mutants per change and orders candidates on the measured
|
|
456
464
|
productivity of their operator in similar context, which took their productive
|
|
@@ -466,6 +474,60 @@ A sweep is weaker evidence than a probe, and says so. Nobody stated that the
|
|
|
466
474
|
behaviour mattered. It is stronger than nothing, which is what a repository
|
|
467
475
|
with no claims has — and the survivors worth defending become the first claims.
|
|
468
476
|
|
|
477
|
+
#### Concerns: one sentence, two hundred screens
|
|
478
|
+
|
|
479
|
+
A claim names one promise precisely, which is why a repository with two hundred
|
|
480
|
+
screens never finishes writing them. A **concern** names a *kind* of promise and
|
|
481
|
+
says where to look for it:
|
|
482
|
+
|
|
483
|
+
```bash
|
|
484
|
+
npx testguard-cli concerns # what this project can be swept by
|
|
485
|
+
npx testguard-cli sweep --concern ADMIN-GUARDS # aim a sweep at one
|
|
486
|
+
```
|
|
487
|
+
|
|
488
|
+
```json
|
|
489
|
+
{
|
|
490
|
+
"schemaVersion": 1,
|
|
491
|
+
"concerns": [
|
|
492
|
+
{
|
|
493
|
+
"id": "ADMIN-GUARDS",
|
|
494
|
+
"statement": "Every admin route refuses a caller without the admin role.",
|
|
495
|
+
"severity": "critical",
|
|
496
|
+
"targets": { "kind": "glob", "globs": ["src/app/api/admin/**"] },
|
|
497
|
+
"faultClasses": ["guard-removed", "condition-forced", "return-altered"]
|
|
498
|
+
}
|
|
499
|
+
]
|
|
500
|
+
}
|
|
501
|
+
```
|
|
502
|
+
|
|
503
|
+
Five lines, and the sweep reports:
|
|
504
|
+
|
|
505
|
+
```
|
|
506
|
+
concern ADMIN-GUARDS matches 24 files. Swept 23.
|
|
507
|
+
158 further proposals were outside this concern's fault classes.
|
|
508
|
+
|
|
509
|
+
SURVIVED src/app/api/admin/claims/[id]/route.ts
|
|
510
|
+
[line 41] Guard never triggers: `if (!membership || (membership.role !== 'owner'
|
|
511
|
+
&& membership.role !== 'manager'))` becomes `if (false)`.
|
|
512
|
+
```
|
|
513
|
+
|
|
514
|
+
The idea is Meta's: ACH has an engineer describe an area of concern in plain
|
|
515
|
+
text and gates every generated step by execution. What is taken here is the
|
|
516
|
+
**unit** — the concern as the thing a human writes — not their generator, and
|
|
517
|
+
**no model is required**. A concern's useful core is a named scope plus a
|
|
518
|
+
producer selection, which the mechanical producers already satisfy, so every
|
|
519
|
+
verifying command stays offline.
|
|
520
|
+
|
|
521
|
+
Two concerns ship built in, and only two. An authorization concern would need a
|
|
522
|
+
heuristic for *"which files check permissions"*, and a guess there is exactly
|
|
523
|
+
the noise that gets a check switched off — so `SAVE-PERSISTS` and
|
|
524
|
+
`CHANGED-CODE` are the defaults, and a project that knows its own auth layer
|
|
525
|
+
says so in three lines. Declaring a concern with a built-in's id replaces it,
|
|
526
|
+
and the replacement is reported rather than silent.
|
|
527
|
+
|
|
528
|
+
A concern is **not** a claim and never becomes one by itself. It says where to
|
|
529
|
+
look; the probe says what it found; a human states the sentence worth defending.
|
|
530
|
+
|
|
469
531
|
#### Sweeping the save surface instead of the diff
|
|
470
532
|
|
|
471
533
|
A diff cannot answer *"when the user clicks save, does it actually save?"* — it
|
|
@@ -494,10 +556,26 @@ denominator comes first and the document carries `writeSites` and
|
|
|
494
556
|
AI-authored application: **12 of 26 probed persistence faults survived a fully
|
|
495
557
|
green suite, and all 12 were a dropped payload field.**
|
|
496
558
|
|
|
497
|
-
|
|
498
|
-
|
|
499
|
-
|
|
500
|
-
|
|
559
|
+
It also states, before any verdict, **what the defenders could prove at all**:
|
|
560
|
+
|
|
561
|
+
```
|
|
562
|
+
of those 37 files: 32 defended only by tests that mock the persistence layer,
|
|
563
|
+
1 with an unmocked defender, 4 with no defender at all.
|
|
564
|
+
```
|
|
565
|
+
|
|
566
|
+
That line is the read-back answer. A test that replaced the database can prove
|
|
567
|
+
the *call shape* and nothing beyond it — a `where` that matches no rows, a
|
|
568
|
+
rolled-back transaction and a rejected constraint all pass against a mock that
|
|
569
|
+
recorded the arguments and returned a plausible object. So rather than pretend
|
|
570
|
+
to measure persistence, TestGuard states the limit, the same way `nocover` says
|
|
571
|
+
"no test imports this" instead of guessing.
|
|
572
|
+
|
|
573
|
+
When every defender mocks the layer, the report says so outright: *nothing in
|
|
574
|
+
this suite can prove a write reached storage*. A clean probe there is not
|
|
575
|
+
evidence of persistence — it is evidence that nothing could have measured it.
|
|
576
|
+
|
|
577
|
+
`unmocked` is deliberately weaker than "proves persistence": a test that does
|
|
578
|
+
not replace the database may simply never reach it. Possible is the honest word.
|
|
501
579
|
|
|
502
580
|
### Every change needs a claim
|
|
503
581
|
|
|
@@ -580,12 +658,12 @@ detected base that does not resolve is a warning for `status` and `brief`
|
|
|
580
658
|
# GitHub Actions
|
|
581
659
|
- uses: actions/checkout@v4
|
|
582
660
|
with: { fetch-depth: 0 }
|
|
583
|
-
- uses: raccioly/testguard@v0.
|
|
661
|
+
- uses: raccioly/testguard@v0.13.0
|
|
584
662
|
with: { command: gate }
|
|
585
663
|
|
|
586
664
|
# GitLab CI — the component-shaped template: gate + probe, brief as an artifact and, opted in, as a merge-request note
|
|
587
665
|
include:
|
|
588
|
-
- remote: 'https://raw.githubusercontent.com/raccioly/testguard/v0.
|
|
666
|
+
- remote: 'https://raw.githubusercontent.com/raccioly/testguard/v0.13.0/packaging/gitlab/testguard.gitlab-ci.yml'
|
|
589
667
|
inputs: { dir: backend, post_note: true } # post_note needs TESTGUARD_GITLAB_TOKEN (api scope); one note, updated in place
|
|
590
668
|
```
|
|
591
669
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "testguard-cli",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.13.0",
|
|
4
4
|
"description": "Breaks your code on purpose and reports every promise your tests did not notice breaking. Mutation testing bound to the claims a project makes — vitest, jest, Playwright, pytest and unittest.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|
package/spec/lib/validate.mjs
CHANGED
|
@@ -7,7 +7,7 @@ import { reproduces, decimals, MAX_PLACES } from './wilson.mjs';
|
|
|
7
7
|
|
|
8
8
|
const schemaDir = join(dirname(fileURLToPath(import.meta.url)), '..', 'schemas');
|
|
9
9
|
|
|
10
|
-
export const KINDS = Object.freeze(['claims', 'evidence', 'baseline', 'ignore', 'calibration', 'brief', 'status', 'gate', 'replay', 'sweep']);
|
|
10
|
+
export const KINDS = Object.freeze(['claims', 'evidence', 'baseline', 'ignore', 'calibration', 'brief', 'status', 'gate', 'replay', 'sweep', 'concerns']);
|
|
11
11
|
/**
|
|
12
12
|
* How a document says it tried to falsify its claims. Absent means
|
|
13
13
|
* `fault-injection`, so every document written before the field existed is
|
|
@@ -305,6 +305,29 @@ const semantic = {
|
|
|
305
305
|
return errors;
|
|
306
306
|
},
|
|
307
307
|
|
|
308
|
+
concerns(doc) {
|
|
309
|
+
const errors = [];
|
|
310
|
+
const seen = new Set();
|
|
311
|
+
doc.concerns.forEach((c, i) => {
|
|
312
|
+
if (seen.has(c.id)) errors.push({ path: `/concerns/${i}/id`, message: `duplicate concern id "${c.id}"` });
|
|
313
|
+
seen.add(c.id);
|
|
314
|
+
// A glob target with no globs matches nothing, which reads as "this
|
|
315
|
+
// concern found no problems" rather than "this concern was never aimed".
|
|
316
|
+
if (c.targets?.kind === 'glob' && !(c.targets.globs?.length > 0)) {
|
|
317
|
+
errors.push({ path: `/concerns/${i}/targets/globs`, message: `concern "${c.id}" targets a glob but names none; it would match nothing and report clean` });
|
|
318
|
+
}
|
|
319
|
+
if (c.targets?.kind !== 'glob' && c.targets?.globs) {
|
|
320
|
+
errors.push({ path: `/concerns/${i}/targets/globs`, message: `concern "${c.id}" names globs but does not target them` });
|
|
321
|
+
}
|
|
322
|
+
// An empty list is not "every class"; it is a concern that can never
|
|
323
|
+
// propose anything, which is a typo for `null` every time.
|
|
324
|
+
if (Array.isArray(c.faultClasses) && c.faultClasses.length === 0) {
|
|
325
|
+
errors.push({ path: `/concerns/${i}/faultClasses`, message: `concern "${c.id}" allows no fault class; omit the field for every class` });
|
|
326
|
+
}
|
|
327
|
+
});
|
|
328
|
+
return errors;
|
|
329
|
+
},
|
|
330
|
+
|
|
308
331
|
sweep(doc) {
|
|
309
332
|
const errors = [];
|
|
310
333
|
const s = doc.selection;
|
|
@@ -318,10 +341,17 @@ const semantic = {
|
|
|
318
341
|
// measured, and its absence on a save-paths sweep hides the denominator
|
|
319
342
|
// entirely — which is the whole reason that mode exists.
|
|
320
343
|
const saves = doc.scope.mode === 'save-paths';
|
|
321
|
-
for (const k of ['writeSites', 'payloadFields']) {
|
|
344
|
+
for (const k of ['writeSites', 'payloadFields', 'provability']) {
|
|
322
345
|
if (saves && doc.scope[k] === undefined) errors.push({ path: `/scope/${k}`, message: `a save-paths sweep must report ${k}: the surface is the denominator its findings are read against` });
|
|
323
346
|
if (!saves && doc.scope[k] !== undefined) errors.push({ path: `/scope/${k}`, message: `${k} belongs to a save-paths sweep; a changed sweep did not measure the write surface` });
|
|
324
347
|
}
|
|
348
|
+
if (saves && doc.scope.provability) {
|
|
349
|
+
const pv = doc.scope.provability;
|
|
350
|
+
const total = pv.mocked + pv.unmocked + pv.none;
|
|
351
|
+
// Every target is classified exactly once. A surface whose provability
|
|
352
|
+
// does not account for all of it has a bucket nobody looked in.
|
|
353
|
+
if (total !== doc.scope.targets) errors.push({ path: '/scope/provability', message: `provability accounts for ${total} files; targets is ${doc.scope.targets}` });
|
|
354
|
+
}
|
|
325
355
|
if (saves && doc.scope.writeSites !== undefined && doc.scope.targets > doc.scope.writeSites) {
|
|
326
356
|
errors.push({ path: '/scope/targets', message: `targets (${doc.scope.targets}) exceeds writeSites (${doc.scope.writeSites}); every target is a file with at least one write` });
|
|
327
357
|
}
|
|
@@ -343,6 +373,12 @@ const semantic = {
|
|
|
343
373
|
const notKilled = probed - (doc.counts.killed ?? 0);
|
|
344
374
|
if (doc.findings.length !== notKilled) errors.push({ path: '/findings', message: `${doc.findings.length} findings for ${notKilled} records that were not killed; findings are every non-killed record` });
|
|
345
375
|
|
|
376
|
+
// An ordering that claims to have learned from something must name it.
|
|
377
|
+
if (doc.ordering) {
|
|
378
|
+
if (doc.ordering.observed > 0 && doc.ordering.sources.length === 0) errors.push({ path: '/ordering/sources', message: 'an ordering learned from evidence must name the documents it was learned from' });
|
|
379
|
+
if (doc.ordering.observed === 0 && doc.ordering.sources.length > 0) errors.push({ path: '/ordering/observed', message: 'sources are named but nothing was observed; the ordering used the shipped prior' });
|
|
380
|
+
}
|
|
381
|
+
|
|
346
382
|
const gating = doc.findings.filter((f) => f.verdict === 'survived' || f.verdict === 'nocover').length;
|
|
347
383
|
if (gating > 0 && doc.exitCode !== 1) errors.push({ path: '/exitCode', message: 'a survived or nocover finding must exit 1; a sweep never passes over a deliberate break nothing noticed' });
|
|
348
384
|
if (gating === 0 && doc.exitCode !== 0) errors.push({ path: '/exitCode', message: 'exit 1 requires a survived or nocover finding; a sweep does not fail on its own unanchorable proposal' });
|
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
{
|
|
2
|
+
"$id": "urn:claimspec:v1:concerns",
|
|
3
|
+
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
4
|
+
"title": "claimspec v1 — concerns",
|
|
5
|
+
"description": "A concern is one sentence a human writes that generates faults across a whole surface: it says WHERE to look and WHICH shapes of break are relevant. It is the rung between writing nothing and writing a claim per behaviour. A concern is never a claim and never becomes one by itself — it says where to look, the probe says what it found, and a human states the sentence worth defending. Nothing in this document is a promise about the code; only a claim is that.",
|
|
6
|
+
"type": "object",
|
|
7
|
+
"required": ["schemaVersion", "concerns"],
|
|
8
|
+
"additionalProperties": false,
|
|
9
|
+
"properties": {
|
|
10
|
+
"$schema": { "type": "string" },
|
|
11
|
+
"schemaVersion": { "const": 1 },
|
|
12
|
+
"concerns": {
|
|
13
|
+
"type": "array",
|
|
14
|
+
"items": {
|
|
15
|
+
"type": "object",
|
|
16
|
+
"required": ["id", "statement"],
|
|
17
|
+
"additionalProperties": false,
|
|
18
|
+
"properties": {
|
|
19
|
+
"id": { "$ref": "urn:claimspec:v1:common#/$defs/identifier", "description": "Declaring an id that matches a built-in REPLACES it, which is how a project retunes a default for an ORM or a layout this tool does not recognise. The replacement is reported, never silent." },
|
|
20
|
+
"statement": { "type": "string", "minLength": 1, "description": "The sentence, in plain language. It describes a KIND of promise, not one behaviour: \"every write persists the data it was given\", not \"the register action stores password_hash\". The second is a claim." },
|
|
21
|
+
"severity": { "$ref": "urn:claimspec:v1:common#/$defs/severity" },
|
|
22
|
+
"targets": {
|
|
23
|
+
"type": "object",
|
|
24
|
+
"required": ["kind"],
|
|
25
|
+
"additionalProperties": false,
|
|
26
|
+
"description": "Where this concern applies. `write-sites` asks the write enumerator. `glob` matches paths. `changed` defers to whatever diff the caller is already measuring — a concern does not re-derive what the gate computed.",
|
|
27
|
+
"properties": {
|
|
28
|
+
"kind": { "enum": ["write-sites", "glob", "changed"] },
|
|
29
|
+
"globs": { "type": "array", "items": { "type": "string", "minLength": 1 } }
|
|
30
|
+
}
|
|
31
|
+
},
|
|
32
|
+
"faultClasses": {
|
|
33
|
+
"type": ["array", "null"],
|
|
34
|
+
"items": { "$ref": "urn:claimspec:v1:common#/$defs/faultClass" },
|
|
35
|
+
"description": "Which producers are relevant. `null` or absent means every class, the honest default for a concern with no shape to narrow by. Narrowing is what makes one sentence useful across two hundred files: a persistence concern that also reported every altered return value would bury the payload findings it exists to surface."
|
|
36
|
+
}
|
|
37
|
+
}
|
|
38
|
+
}
|
|
39
|
+
}
|
|
40
|
+
}
|
|
41
|
+
}
|
|
@@ -62,6 +62,10 @@
|
|
|
62
62
|
],
|
|
63
63
|
"description": "What the sweep was pointed at. `changed` proposes for the changed source files that carry no claim. `save-paths` proposes for every file that writes to storage, because a diff cannot answer how many save paths exist — and a clean report over an unstated denominator is a sample of unknown size, not a guarantee."
|
|
64
64
|
},
|
|
65
|
+
"concern": {
|
|
66
|
+
"$ref": "urn:claimspec:v1:common#/$defs/identifier",
|
|
67
|
+
"description": "The concern this sweep was aimed by, when one was. A concern narrows both the targets and the fault classes, so the same surface swept under two concerns is two different questions."
|
|
68
|
+
},
|
|
65
69
|
"changed": {
|
|
66
70
|
"type": "integer",
|
|
67
71
|
"minimum": 0,
|
|
@@ -107,6 +111,30 @@
|
|
|
107
111
|
"type": "integer",
|
|
108
112
|
"minimum": 0,
|
|
109
113
|
"description": "save-paths only: payload keys sitting on their own line, which is what a field-dropped fault can anchor to."
|
|
114
|
+
},
|
|
115
|
+
"provability": {
|
|
116
|
+
"type": "object",
|
|
117
|
+
"required": [
|
|
118
|
+
"mocked",
|
|
119
|
+
"unmocked",
|
|
120
|
+
"none"
|
|
121
|
+
],
|
|
122
|
+
"additionalProperties": false,
|
|
123
|
+
"description": "save-paths only: what the defenders of this surface are CAPABLE of proving, before anything is probed. `mocked` — every defender replaces the persistence layer, so it can prove the call shape and never that the row landed. `unmocked` — at least one does not, so a real write is at least possible (not a promise that one happens). `none` — no defender at all. On a surface that is entirely `mocked`, a clean probe is not evidence of persistence; it is evidence that nothing could have measured it.",
|
|
124
|
+
"properties": {
|
|
125
|
+
"mocked": {
|
|
126
|
+
"type": "integer",
|
|
127
|
+
"minimum": 0
|
|
128
|
+
},
|
|
129
|
+
"unmocked": {
|
|
130
|
+
"type": "integer",
|
|
131
|
+
"minimum": 0
|
|
132
|
+
},
|
|
133
|
+
"none": {
|
|
134
|
+
"type": "integer",
|
|
135
|
+
"minimum": 0
|
|
136
|
+
}
|
|
137
|
+
}
|
|
110
138
|
}
|
|
111
139
|
}
|
|
112
140
|
},
|
|
@@ -165,6 +193,11 @@
|
|
|
165
193
|
}
|
|
166
194
|
}
|
|
167
195
|
}
|
|
196
|
+
},
|
|
197
|
+
"outOfScope": {
|
|
198
|
+
"type": "integer",
|
|
199
|
+
"minimum": 0,
|
|
200
|
+
"description": "Proposals the concern's fault classes excluded. Reported so a narrow concern cannot look like a quiet one: the producers found these, and this concern is not about them."
|
|
168
201
|
}
|
|
169
202
|
}
|
|
170
203
|
},
|
|
@@ -305,6 +338,29 @@
|
|
|
305
338
|
"items": {
|
|
306
339
|
"type": "object"
|
|
307
340
|
}
|
|
341
|
+
},
|
|
342
|
+
"ordering": {
|
|
343
|
+
"type": "object",
|
|
344
|
+
"required": [
|
|
345
|
+
"observed",
|
|
346
|
+
"sources"
|
|
347
|
+
],
|
|
348
|
+
"additionalProperties": false,
|
|
349
|
+
"description": "Where the candidate ordering came from. `observed` is how many of this project's OWN probed faults informed it; 0 means the shipped productivity prior was used unchanged. An ordering nobody can trace is a number nobody should trust.",
|
|
350
|
+
"properties": {
|
|
351
|
+
"observed": {
|
|
352
|
+
"type": "integer",
|
|
353
|
+
"minimum": 0
|
|
354
|
+
},
|
|
355
|
+
"sources": {
|
|
356
|
+
"type": "array",
|
|
357
|
+
"items": {
|
|
358
|
+
"type": "string",
|
|
359
|
+
"minLength": 1
|
|
360
|
+
},
|
|
361
|
+
"description": "The evidence documents the counts were read from."
|
|
362
|
+
}
|
|
363
|
+
}
|
|
308
364
|
}
|
|
309
365
|
}
|
|
310
366
|
}
|
package/src/cli.mjs
CHANGED
|
@@ -19,6 +19,7 @@ import { replayCommand } from './commands/replay.mjs';
|
|
|
19
19
|
import { mcpCommand } from './commands/mcp.mjs';
|
|
20
20
|
import { admitCommand } from './commands/admit.mjs';
|
|
21
21
|
import { sweepCommand } from './commands/sweep.mjs';
|
|
22
|
+
import { concernsCommand } from './commands/concerns.mjs';
|
|
22
23
|
|
|
23
24
|
const VERSION = JSON.parse(readFileSync(join(fileURLToPath(import.meta.url), '..', '..', 'package.json'), 'utf8')).version;
|
|
24
25
|
const ISSUES = 'https://github.com/raccioly/testguard/issues';
|
|
@@ -34,6 +35,7 @@ export const USAGE = `testguard ${VERSION} — proves a test suite defends the c
|
|
|
34
35
|
testguard gate [dir] fail when a changed source file carries no claim and no excusing ignore entry
|
|
35
36
|
testguard scaffold <file> propose faults mechanically for one source file, as a draft claims document
|
|
36
37
|
testguard sweep [dir] propose faults for the changed source files that carry no claim, probe a bounded selection, and report what nothing noticed
|
|
38
|
+
testguard concerns [dir] the one-sentence scopes a sweep can be aimed by: where to look, and which shapes of break are relevant
|
|
37
39
|
testguard mcp serve the read-only status/brief/claims/evidence tools over MCP on stdio, so the loop works in any agent harness
|
|
38
40
|
testguard replay [dir] would this suite have caught the bugs that already escaped? Replays real fix commits and calibrates fault classes against them
|
|
39
41
|
testguard admit <test> --claim <ID> the two-gate rule as one verb: is this test green on HEAD and does it fail on every fault of the claim?
|
|
@@ -77,6 +79,8 @@ probe
|
|
|
77
79
|
|
|
78
80
|
scaffold --claim <ID> (put every proposal under this claim; copies it if it exists) --out <path> --json
|
|
79
81
|
sweep --changed <ref> (required unless --save-paths; CI bases and a safe local default are detected) --cap <n>
|
|
82
|
+
--concern <ID> aim the sweep by one of this project's concerns: its targets and its fault classes.
|
|
83
|
+
testguard concerns lists them. --save-paths is sugar for the built-in SAVE-PERSISTS
|
|
80
84
|
--save-paths sweep every file that WRITES TO STORAGE instead of the diff, and report the surface as the
|
|
81
85
|
denominator: "120 writes across 38 files, swept 12". A clean sweep over an unstated
|
|
82
86
|
denominator is a sample of unknown size, not a guarantee that saving works
|
|
@@ -126,7 +130,7 @@ brief --evidence <path> --baseline <path> --max <n> --text (print only;
|
|
|
126
130
|
exit codes: 0 nothing new to prove · 1 unproven claims (or claim drift, or unclaimed changes) · 2 precondition failed · 3 usage
|
|
127
131
|
`;
|
|
128
132
|
|
|
129
|
-
const COMMANDS = { mcp: mcpCommand, replay: replayCommand, probe: probeCommand, claims: claimsCommand, baseline: baselineCommand, brief: briefCommand, scaffold: scaffoldCommand, status: statusCommand, init: initCommand, gate: gateCommand, admit: admitCommand, sweep: sweepCommand };
|
|
133
|
+
const COMMANDS = { mcp: mcpCommand, replay: replayCommand, probe: probeCommand, claims: claimsCommand, baseline: baselineCommand, brief: briefCommand, scaffold: scaffoldCommand, status: statusCommand, init: initCommand, gate: gateCommand, admit: admitCommand, sweep: sweepCommand, concerns: concernsCommand };
|
|
130
134
|
|
|
131
135
|
export async function main(argv, io = { out: (s) => process.stdout.write(s + '\n'), err: (s) => process.stderr.write(s + '\n') }) {
|
|
132
136
|
let parsed;
|
|
@@ -154,6 +158,8 @@ export async function main(argv, io = { out: (s) => process.stdout.write(s + '\n
|
|
|
154
158
|
exclude: { type: 'string', multiple: true },
|
|
155
159
|
cap: { type: 'string' },
|
|
156
160
|
'save-paths': { type: 'boolean', default: false },
|
|
161
|
+
concern: { type: 'string' },
|
|
162
|
+
concerns: { type: 'string' },
|
|
157
163
|
strict: { type: 'boolean', default: false },
|
|
158
164
|
explain: { type: 'boolean', default: false },
|
|
159
165
|
ignore: { type: 'string' },
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
import { resolve } from 'node:path';
|
|
2
|
+
import { loadConcerns, renderConcerns } from '../supply/concerns.mjs';
|
|
3
|
+
import { validate } from '../../spec/lib/validate.mjs';
|
|
4
|
+
|
|
5
|
+
/**
|
|
6
|
+
* `testguard concerns [dir]`: the one-sentence scopes a sweep can be aimed by.
|
|
7
|
+
*
|
|
8
|
+
* A concern says WHERE to look and WHICH shapes of break are relevant. It is
|
|
9
|
+
* never a promise about the code — only a claim is that — so this command
|
|
10
|
+
* validates and lists, and has no exit code that means "something is wrong
|
|
11
|
+
* with your project". Exit 2 is a malformed concerns file, which is wrong with
|
|
12
|
+
* the FILE.
|
|
13
|
+
*/
|
|
14
|
+
export async function concernsCommand({ projectDir, values, version }, io) {
|
|
15
|
+
const path = values.concerns ? resolve(values.concerns) : undefined;
|
|
16
|
+
const loaded = loadConcerns(projectDir, { path });
|
|
17
|
+
|
|
18
|
+
if (loaded.source) {
|
|
19
|
+
const result = validate('concerns', { schemaVersion: 1, concerns: loaded.raw });
|
|
20
|
+
if (!result.ok) {
|
|
21
|
+
io.err(`${loaded.source} does not conform to the spec:`);
|
|
22
|
+
for (const e of result.errors) io.err(` ${e.path}: ${e.message}`);
|
|
23
|
+
return 2;
|
|
24
|
+
}
|
|
25
|
+
}
|
|
26
|
+
if (values.json) {
|
|
27
|
+
io.out(JSON.stringify({ schemaVersion: 1, tool: { name: 'testguard', version }, source: loaded.source, shadowed: loaded.shadowed, concerns: loaded.concerns }, null, 2));
|
|
28
|
+
} else {
|
|
29
|
+
io.out(renderConcerns(loaded));
|
|
30
|
+
}
|
|
31
|
+
return 0;
|
|
32
|
+
}
|
package/src/commands/sweep.mjs
CHANGED
|
@@ -1,9 +1,17 @@
|
|
|
1
1
|
import { join, resolve } from 'node:path';
|
|
2
|
-
import { sweep, renderSweep } from '../sweep/sweep.mjs';
|
|
2
|
+
import { sweep, renderSweep, ConcernError } from '../sweep/sweep.mjs';
|
|
3
3
|
import { resolveChangedRef } from '../gate/changed.mjs';
|
|
4
4
|
import { writeSpecDoc } from '../evidence/writer.mjs';
|
|
5
5
|
|
|
6
6
|
export const sweepPath = (projectDir) => join(projectDir, '.testguard', 'sweep.json');
|
|
7
|
+
/**
|
|
8
|
+
* A sweep's own evidence, beside the canonical one and never replacing it.
|
|
9
|
+
* The next sweep reads it to learn which fault classes are productive HERE,
|
|
10
|
+
* which is the feedback half of the selection Google reports taking from 15%
|
|
11
|
+
* to 89% productive. Machine-proposed faults, so it is the closest observation
|
|
12
|
+
* of the distribution the ranker actually orders.
|
|
13
|
+
*/
|
|
14
|
+
export const sweepEvidencePath = (projectDir) => join(projectDir, '.testguard', 'sweep-evidence.json');
|
|
7
15
|
|
|
8
16
|
/**
|
|
9
17
|
* `testguard sweep [dir] --changed <ref>`: propose faults for the changed
|
|
@@ -21,11 +29,14 @@ export const sweepPath = (projectDir) => join(projectDir, '.testguard', 'sweep.j
|
|
|
21
29
|
*/
|
|
22
30
|
export async function sweepCommand({ projectDir, values, version }, io) {
|
|
23
31
|
const mode = values['save-paths'] ? 'save-paths' : 'changed';
|
|
32
|
+
// A concern that aims at the write surface or at a glob needs no diff, the
|
|
33
|
+
// same as --save-paths. Only a concern that defers to the gate does.
|
|
34
|
+
const aimed = Boolean(values.concern) || mode === 'save-paths';
|
|
24
35
|
// `--save-paths` scans the whole write surface, so it needs no diff. The gate
|
|
25
36
|
// is still computed underneath (the document reports `changed`), and HEAD is
|
|
26
37
|
// a reference every repository has.
|
|
27
38
|
const resolved = resolveChangedRef({ explicit: values.changed, projectDir })
|
|
28
|
-
?? (
|
|
39
|
+
?? (aimed ? { ref: 'HEAD', from: values.concern ? `concern ${values.concern}` : 'save-paths', required: false } : null);
|
|
29
40
|
if (!resolved) {
|
|
30
41
|
io.err('sweep needs a reference to measure the change against: --changed <ref> (e.g. origin/main), or set TESTGUARD_CHANGED_REF. CI bases and a safe local remote default or differently named upstream are detected automatically. Or sweep the write surface instead with --save-paths.');
|
|
31
42
|
return 3;
|
|
@@ -46,34 +57,52 @@ export async function sweepCommand({ projectDir, values, version }, io) {
|
|
|
46
57
|
}
|
|
47
58
|
if (mode === 'changed' && !resolved.required && !values.json && !values.quiet) io.err(`sweep: comparing against ${resolved.ref} (${resolved.from})`);
|
|
48
59
|
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
60
|
+
let doc;
|
|
61
|
+
try {
|
|
62
|
+
doc = await sweep({
|
|
63
|
+
projectDir,
|
|
64
|
+
ref: resolved.ref,
|
|
65
|
+
mode,
|
|
66
|
+
concern: values.concern,
|
|
67
|
+
concernsPath: values.concerns ? resolve(values.concerns) : undefined,
|
|
68
|
+
includeDirty: values['include-dirty'],
|
|
69
|
+
exclude: values.exclude ?? [],
|
|
70
|
+
cap,
|
|
71
|
+
confirmRuns,
|
|
72
|
+
budgetMs,
|
|
73
|
+
runnerCommand: values['runner-cmd'],
|
|
74
|
+
runnerName: values.runner,
|
|
75
|
+
nodeModules: values['node-modules'] ? resolve(values['node-modules']) : process.env.TESTGUARD_NODE_MODULES,
|
|
76
|
+
toolVersion: version,
|
|
77
|
+
claimsPath: values.claims ? resolve(values.claims) : undefined,
|
|
78
|
+
ignorePath: values.ignore ? resolve(values.ignore) : undefined,
|
|
79
|
+
onStage: !values.quiet && !values.json && process.stderr.isTTY
|
|
65
80
|
? ({ claimId, faultId, stage, i, n }) => process.stderr.write(`\r\x1b[K … ${claimId}/${faultId} ${stage} ${i}/${n}`)
|
|
66
81
|
: undefined,
|
|
67
|
-
|
|
68
|
-
|
|
82
|
+
onWarn: (m) => { if (!values.quiet) io.err(`warning: ${m}`); },
|
|
83
|
+
});
|
|
84
|
+
} catch (e) {
|
|
85
|
+
// A concern named on the command line that this project does not declare
|
|
86
|
+
// is a usage error, not a precondition failure: nothing about the
|
|
87
|
+
// repository is wrong.
|
|
88
|
+
if (e instanceof ConcernError) {
|
|
89
|
+
io.err(e.message);
|
|
90
|
+
return 3;
|
|
91
|
+
}
|
|
92
|
+
throw e;
|
|
93
|
+
}
|
|
69
94
|
if (process.stderr.isTTY && !values.quiet && !values.json) process.stderr.write('\r\x1b[K');
|
|
70
95
|
|
|
96
|
+
const { evidence, ...document } = doc;
|
|
71
97
|
const outPath = values.out ? resolve(values.out) : sweepPath(projectDir);
|
|
72
|
-
writeSpecDoc('sweep', outPath,
|
|
98
|
+
writeSpecDoc('sweep', outPath, document);
|
|
99
|
+
// Written only when something was actually probed: an empty document would
|
|
100
|
+
// teach the next run that every class is unproductive.
|
|
101
|
+
if (evidence?.records?.length) writeSpecDoc('evidence', sweepEvidencePath(projectDir), evidence);
|
|
73
102
|
if (values.json) {
|
|
74
|
-
io.out(JSON.stringify(
|
|
103
|
+
io.out(JSON.stringify(document, null, 2));
|
|
75
104
|
} else {
|
|
76
|
-
io.out(renderSweep(
|
|
105
|
+
io.out(renderSweep(document, { limit: Number(values.max) || 20 }));
|
|
77
106
|
if (!values.quiet) io.out(`\nsweep: ${outPath}`);
|
|
78
107
|
}
|
|
79
108
|
return doc.exitCode;
|
|
@@ -0,0 +1,162 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* A concern: one sentence a human writes, that generates faults across a whole
|
|
3
|
+
* surface.
|
|
4
|
+
*
|
|
5
|
+
* The rung between "nothing written" and "a claim per behaviour". A claim names
|
|
6
|
+
* one promise and the faults that would break it; that is precise and it is why
|
|
7
|
+
* a repository with two hundred screens never finishes writing them. A concern
|
|
8
|
+
* names a KIND of promise — "every write persists the data it was given" — and
|
|
9
|
+
* says where to look for it and which shapes of break are relevant. One
|
|
10
|
+
* sentence, two hundred screens.
|
|
11
|
+
*
|
|
12
|
+
* The idea is Meta's. ACH has an engineer describe an area of concern in plain
|
|
13
|
+
* text, generates faults specific to it, and gates every step by execution
|
|
14
|
+
* ("Mutation-Guided LLM-based Test Generation at Meta", Foster et al., FSE
|
|
15
|
+
* 2025). What is taken here is the UNIT — the concern as the thing a human
|
|
16
|
+
* writes — not their generator.
|
|
17
|
+
*
|
|
18
|
+
* NO MODEL IS REQUIRED, and that is deliberate. A concern's useful core is a
|
|
19
|
+
* named scope plus a producer selection, both of which the mechanical
|
|
20
|
+
* producers already satisfy. An LLM can later propose faults a line-oriented
|
|
21
|
+
* producer cannot see, as an ADDITIONAL source for a concern that already
|
|
22
|
+
* works — rather than as the thing the rung depends on. That keeps every
|
|
23
|
+
* verifying command offline, which is the property this tool is least willing
|
|
24
|
+
* to trade.
|
|
25
|
+
*
|
|
26
|
+
* A concern is NOT a claim and never becomes one by itself. It says where to
|
|
27
|
+
* look; the probe says what it found; a human states the sentence worth
|
|
28
|
+
* defending. Nothing here is written to `testguard.claims.json`.
|
|
29
|
+
*/
|
|
30
|
+
import { existsSync, readFileSync } from 'node:fs';
|
|
31
|
+
import { join } from 'node:path';
|
|
32
|
+
import { saveSurface } from './savepath.mjs';
|
|
33
|
+
import { globToRegExp, walk } from '../util/glob.mjs';
|
|
34
|
+
|
|
35
|
+
export const defaultConcernsPath = (projectDir) => join(projectDir, 'testguard.concerns.json');
|
|
36
|
+
|
|
37
|
+
/**
|
|
38
|
+
* The concerns every project gets without writing anything.
|
|
39
|
+
*
|
|
40
|
+
* Two, because two are the ones whose targets can be enumerated honestly. An
|
|
41
|
+
* authorization concern would need a heuristic for "which files check
|
|
42
|
+
* permissions", and a guess there produces exactly the non-actionable noise
|
|
43
|
+
* that gets a check switched off. A project that knows its own auth layer can
|
|
44
|
+
* say so with a `glob` target in three lines; this file will not guess for it.
|
|
45
|
+
*/
|
|
46
|
+
export const BUILTIN_CONCERNS = Object.freeze([
|
|
47
|
+
Object.freeze({
|
|
48
|
+
id: 'SAVE-PERSISTS',
|
|
49
|
+
statement: 'Every write to storage persists the data it was given.',
|
|
50
|
+
severity: 'high',
|
|
51
|
+
targets: { kind: 'write-sites' },
|
|
52
|
+
// The shapes that break a persisted payload. Measured on a real
|
|
53
|
+
// application: 12 of 12 surviving faults on the persistence path were a
|
|
54
|
+
// field dropped from the payload, so the list leads with that.
|
|
55
|
+
faultClasses: ['field-dropped', 'call-removed', 'statement-deleted', 'literal-changed'],
|
|
56
|
+
builtin: true,
|
|
57
|
+
}),
|
|
58
|
+
Object.freeze({
|
|
59
|
+
id: 'CHANGED-CODE',
|
|
60
|
+
statement: 'Code changed in this branch is defended by something.',
|
|
61
|
+
severity: 'medium',
|
|
62
|
+
targets: { kind: 'changed' },
|
|
63
|
+
// Everything: a diff has no shape to narrow by.
|
|
64
|
+
faultClasses: null,
|
|
65
|
+
builtin: true,
|
|
66
|
+
}),
|
|
67
|
+
]);
|
|
68
|
+
|
|
69
|
+
/**
|
|
70
|
+
* Load the project's concerns, with the built-ins behind them.
|
|
71
|
+
*
|
|
72
|
+
* A project concern with the same id as a built-in REPLACES it, so a team can
|
|
73
|
+
* retune `SAVE-PERSISTS` for an ORM this tool does not recognise without
|
|
74
|
+
* forking anything. Shadowing is recorded rather than silent: a concern that
|
|
75
|
+
* quietly replaced a default would be the kind of surprise that costs an
|
|
76
|
+
* afternoon.
|
|
77
|
+
*/
|
|
78
|
+
export function loadConcerns(projectDir, { path } = {}) {
|
|
79
|
+
const p = path ?? defaultConcernsPath(projectDir);
|
|
80
|
+
let project = [];
|
|
81
|
+
let source = null;
|
|
82
|
+
if (existsSync(p)) {
|
|
83
|
+
const doc = JSON.parse(readFileSync(p, 'utf8'));
|
|
84
|
+
project = doc.concerns ?? [];
|
|
85
|
+
source = p;
|
|
86
|
+
}
|
|
87
|
+
const overridden = new Set(project.map((c) => c.id));
|
|
88
|
+
const concerns = [
|
|
89
|
+
...project.map((c) => ({ ...c, builtin: false })),
|
|
90
|
+
...BUILTIN_CONCERNS.filter((c) => !overridden.has(c.id)),
|
|
91
|
+
];
|
|
92
|
+
return {
|
|
93
|
+
concerns,
|
|
94
|
+
// The project's own entries, exactly as written. Validation runs against
|
|
95
|
+
// these: the enriched objects above carry a `builtin` marker this tool
|
|
96
|
+
// added, and a schema that refuses unknown properties is right to reject
|
|
97
|
+
// them. Validating what we constructed instead of what the user wrote
|
|
98
|
+
// would report OUR bug as THEIR malformed file.
|
|
99
|
+
raw: project.map(({ builtin, ...c }) => c),
|
|
100
|
+
source,
|
|
101
|
+
shadowed: BUILTIN_CONCERNS.filter((c) => overridden.has(c.id)).map((c) => c.id),
|
|
102
|
+
};
|
|
103
|
+
}
|
|
104
|
+
|
|
105
|
+
/** One concern by id, or null. Ids are matched exactly: a near-miss is a typo, not an intention. */
|
|
106
|
+
export const concernById = (concerns, id) => concerns.find((c) => c.id === id) ?? null;
|
|
107
|
+
|
|
108
|
+
/**
|
|
109
|
+
* The files a concern applies to.
|
|
110
|
+
*
|
|
111
|
+
* `write-sites` asks the enumerator; `glob` matches the project's own paths;
|
|
112
|
+
* `changed` returns null, meaning "the caller's diff decides" — a concern does
|
|
113
|
+
* not get to re-derive what the gate already computed.
|
|
114
|
+
*/
|
|
115
|
+
export function targetsFor(projectDir, concern, { changedFiles } = {}) {
|
|
116
|
+
const t = concern.targets ?? { kind: 'changed' };
|
|
117
|
+
if (t.kind === 'write-sites') return saveSurface(projectDir).files.map((f) => f.file);
|
|
118
|
+
if (t.kind === 'glob') {
|
|
119
|
+
// Against the whole project, never the diff. A concern like "every admin
|
|
120
|
+
// route refuses an unauthorised caller" is about the admin routes, not
|
|
121
|
+
// about the ones that happened to change today — and pooling from the diff
|
|
122
|
+
// made such a concern report clean on a clean branch, which is the exact
|
|
123
|
+
// false reassurance a concern exists to prevent.
|
|
124
|
+
const res = (t.globs ?? []).map(globToRegExp);
|
|
125
|
+
return walk(projectDir).filter((f) => res.some((re) => re.test(f)));
|
|
126
|
+
}
|
|
127
|
+
return changedFiles ?? null;
|
|
128
|
+
}
|
|
129
|
+
|
|
130
|
+
/**
|
|
131
|
+
* Keep only the faults whose class this concern cares about.
|
|
132
|
+
*
|
|
133
|
+
* `faultClasses: null` means every class — the honest default for a concern
|
|
134
|
+
* with no shape to narrow by. Narrowing is what makes one sentence useful
|
|
135
|
+
* across two hundred files: a persistence concern that also reported every
|
|
136
|
+
* altered return value would bury the payload findings it exists to surface.
|
|
137
|
+
*/
|
|
138
|
+
export function filterByConcern(candidates, concern) {
|
|
139
|
+
if (!concern?.faultClasses) return candidates;
|
|
140
|
+
const keep = new Set(concern.faultClasses);
|
|
141
|
+
return candidates.filter((c) => keep.has(c.fault.faultClass));
|
|
142
|
+
}
|
|
143
|
+
|
|
144
|
+
/** The concerns as text, for someone deciding which to sweep. */
|
|
145
|
+
export function renderConcerns({ concerns, source, shadowed }) {
|
|
146
|
+
const out = [];
|
|
147
|
+
out.push(source ? `${concerns.length} concern${concerns.length === 1 ? '' : 's'} (${source}, plus the built-ins)` : `${concerns.length} built-in concern${concerns.length === 1 ? '' : 's'} — none declared in this project.`);
|
|
148
|
+
out.push('');
|
|
149
|
+
for (const c of concerns) {
|
|
150
|
+
const shape = c.targets?.kind === 'glob' ? `glob ${(c.targets.globs ?? []).join(', ')}` : c.targets?.kind ?? 'changed';
|
|
151
|
+
out.push(` ${c.id}${c.builtin ? ' (built-in)' : ''}`);
|
|
152
|
+
out.push(` ${c.statement}`);
|
|
153
|
+
out.push(` targets: ${shape} · faults: ${c.faultClasses ? c.faultClasses.join(', ') : 'every class'}`);
|
|
154
|
+
out.push('');
|
|
155
|
+
}
|
|
156
|
+
if (shadowed.length) out.push(`This project replaces the built-in ${shadowed.join(', ')}.`);
|
|
157
|
+
if (!source) {
|
|
158
|
+
out.push('Declare your own in testguard.concerns.json. A concern is one sentence and a');
|
|
159
|
+
out.push('scope: it says where to look, never what is true. Only a claim does that.');
|
|
160
|
+
}
|
|
161
|
+
return out.join('\n');
|
|
162
|
+
}
|
|
@@ -0,0 +1,146 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* What this project's own runs say about which faults are worth proposing.
|
|
3
|
+
*
|
|
4
|
+
* `select.mjs` ships a productivity prior measured on one real application.
|
|
5
|
+
* That is a reasonable starting point and a poor stopping point: the classes
|
|
6
|
+
* that survive a Next.js suite are not the classes that survive a Python one,
|
|
7
|
+
* and Google's mutation service got from a 15% productive rate to 89% on
|
|
8
|
+
* exactly this — feedback, accumulated over years, refining which candidates
|
|
9
|
+
* are worth a developer's attention ("Practical Mutation Testing at Scale",
|
|
10
|
+
* Petrović et al., 2021). The static table is the part of that idea this tool
|
|
11
|
+
* had not yet taken.
|
|
12
|
+
*
|
|
13
|
+
* THE FEEDBACK IS ALREADY ON DISK. Every evidence record carries a
|
|
14
|
+
* `faultClass` and a `verdict`, so "how often does a fault of this class
|
|
15
|
+
* survive here" is a tally, not a new thing to capture. There is no click to
|
|
16
|
+
* collect and no state to maintain: a project that has probed anything has
|
|
17
|
+
* already said what its own productive classes are.
|
|
18
|
+
*
|
|
19
|
+
* WHY THIS ESTIMATOR AND NOT A BETTER-SOUNDING ONE. The combination below is a
|
|
20
|
+
* Beta-Binomial posterior mean, and a posterior mean is what minimises Brier
|
|
21
|
+
* and log score in expectation — it is a STRICTLY PROPER estimate, which is
|
|
22
|
+
* the only kind that cannot be improved by reporting something other than an
|
|
23
|
+
* honest probability. Optimising the ordering for accuracy or F1 instead would
|
|
24
|
+
* maximise the number of survivors found while destroying the calibration of
|
|
25
|
+
* the number attached to each, which is the documented failure mode of naive
|
|
26
|
+
* optimisation and the reason Laya trains against proper scoring rules. If
|
|
27
|
+
* this file is ever changed to rank by anything but an honest probability,
|
|
28
|
+
* that is the invariant being broken.
|
|
29
|
+
*
|
|
30
|
+
* Pure apart from reading the documents it is given.
|
|
31
|
+
*/
|
|
32
|
+
import { existsSync, readFileSync } from 'node:fs';
|
|
33
|
+
import { join } from 'node:path';
|
|
34
|
+
import { MEASURED_PRODUCTIVITY } from './select.mjs';
|
|
35
|
+
|
|
36
|
+
/** The evidence a project may have produced, in the order they are most relevant. */
|
|
37
|
+
export const EVIDENCE_PATHS = Object.freeze([
|
|
38
|
+
// A sweep probes MACHINE-proposed faults, the same distribution the ranker
|
|
39
|
+
// orders, so it is the closest observation available.
|
|
40
|
+
'.testguard/sweep-evidence.json',
|
|
41
|
+
// The canonical probe is human-authored faults: a real observation of the
|
|
42
|
+
// same classes, drawn slightly differently, and still worth counting.
|
|
43
|
+
'.testguard/evidence.json',
|
|
44
|
+
]);
|
|
45
|
+
|
|
46
|
+
/**
|
|
47
|
+
* Survival counts per fault class from one evidence document.
|
|
48
|
+
*
|
|
49
|
+
* Only `killed` and `survived` are counted. `nocover`, `unverifiable`,
|
|
50
|
+
* `fault-invalid`, `timeout` and `flaky-defender` say nothing about whether a
|
|
51
|
+
* fault of that class would be NOTICED — they say the question could not be
|
|
52
|
+
* asked — and folding them in either direction would be inventing data.
|
|
53
|
+
*/
|
|
54
|
+
export function tallyEvidence(doc) {
|
|
55
|
+
const by = {};
|
|
56
|
+
for (const r of doc?.records ?? []) {
|
|
57
|
+
const cls = r.subject?.faultClass;
|
|
58
|
+
if (!cls) continue;
|
|
59
|
+
if (r.verdict !== 'killed' && r.verdict !== 'survived') continue;
|
|
60
|
+
const e = (by[cls] ??= { survived: 0, n: 0 });
|
|
61
|
+
e.n += 1;
|
|
62
|
+
if (r.verdict === 'survived') e.survived += 1;
|
|
63
|
+
}
|
|
64
|
+
return by;
|
|
65
|
+
}
|
|
66
|
+
|
|
67
|
+
/** Merge tallies from several documents. Counts add; nothing is averaged twice. */
|
|
68
|
+
export function mergeTallies(tallies) {
|
|
69
|
+
const by = {};
|
|
70
|
+
for (const t of tallies) {
|
|
71
|
+
for (const [cls, e] of Object.entries(t)) {
|
|
72
|
+
const m = (by[cls] ??= { survived: 0, n: 0 });
|
|
73
|
+
m.survived += e.survived;
|
|
74
|
+
m.n += e.n;
|
|
75
|
+
}
|
|
76
|
+
}
|
|
77
|
+
return by;
|
|
78
|
+
}
|
|
79
|
+
|
|
80
|
+
/**
|
|
81
|
+
* The productivity table to rank with: this project's own observations, with
|
|
82
|
+
* the shipped measurement as the prior they are shrunk toward.
|
|
83
|
+
*
|
|
84
|
+
* The shipped numbers become pseudo-counts rather than a competing estimate —
|
|
85
|
+
* `n` of them, so a class measured on twelve faults elsewhere carries the
|
|
86
|
+
* weight of twelve observations here and no more. A project with its own
|
|
87
|
+
* hundred records therefore overrides the shipping default; a project with
|
|
88
|
+
* three nudges it.
|
|
89
|
+
*
|
|
90
|
+
* `shipped` is injectable so the combination can be tested without depending
|
|
91
|
+
* on whatever the current measurement happens to be.
|
|
92
|
+
*/
|
|
93
|
+
export function combinedProductivity(local, shipped = MEASURED_PRODUCTIVITY) {
|
|
94
|
+
const out = {};
|
|
95
|
+
for (const cls of new Set([...Object.keys(shipped), ...Object.keys(local)])) {
|
|
96
|
+
const s = shipped[cls];
|
|
97
|
+
const l = local[cls];
|
|
98
|
+
const priorSurvived = s ? s.p * s.n : 0;
|
|
99
|
+
const priorN = s ? s.n : 0;
|
|
100
|
+
const n = priorN + (l?.n ?? 0);
|
|
101
|
+
if (n === 0) continue;
|
|
102
|
+
const survived = priorSurvived + (l?.survived ?? 0);
|
|
103
|
+
out[cls] = { p: survived / n, n, observed: l?.n ?? 0 };
|
|
104
|
+
}
|
|
105
|
+
return out;
|
|
106
|
+
}
|
|
107
|
+
|
|
108
|
+
/** Read whichever evidence documents this project has. Missing or malformed ones are skipped, never guessed at. */
|
|
109
|
+
export function readEvidence(projectDir, paths = EVIDENCE_PATHS) {
|
|
110
|
+
const docs = [];
|
|
111
|
+
for (const rel of paths) {
|
|
112
|
+
const p = join(projectDir, rel);
|
|
113
|
+
if (!existsSync(p)) continue;
|
|
114
|
+
try {
|
|
115
|
+
docs.push({ path: rel, doc: JSON.parse(readFileSync(p, 'utf8')) });
|
|
116
|
+
} catch {
|
|
117
|
+
// A document this tool cannot parse is not evidence of anything. It is
|
|
118
|
+
// skipped rather than defaulted, because a ranking built on a guess
|
|
119
|
+
// about a corrupt file is worse than one built on the shipped prior.
|
|
120
|
+
}
|
|
121
|
+
}
|
|
122
|
+
return docs;
|
|
123
|
+
}
|
|
124
|
+
|
|
125
|
+
/**
|
|
126
|
+
* The learned table plus what it was learned from, so a report can say whether
|
|
127
|
+
* an ordering came from this project or from the shipping default. An ordering
|
|
128
|
+
* nobody can trace is a number nobody should trust.
|
|
129
|
+
*/
|
|
130
|
+
export function learnedProductivity(projectDir, { paths, shipped } = {}) {
|
|
131
|
+
const docs = readEvidence(projectDir, paths);
|
|
132
|
+
const local = mergeTallies(docs.map((d) => tallyEvidence(d.doc)));
|
|
133
|
+
const observed = Object.values(local).reduce((a, e) => a + e.n, 0);
|
|
134
|
+
return {
|
|
135
|
+
table: combinedProductivity(local, shipped),
|
|
136
|
+
sources: docs.map((d) => d.path),
|
|
137
|
+
observed,
|
|
138
|
+
classes: Object.keys(local).sort(),
|
|
139
|
+
};
|
|
140
|
+
}
|
|
141
|
+
|
|
142
|
+
/** One line for a report: where the ordering came from. */
|
|
143
|
+
export function renderLearned(learned) {
|
|
144
|
+
if (learned.observed === 0) return 'ordering: the shipped productivity prior — this project has no probed evidence yet.';
|
|
145
|
+
return `ordering: ${learned.observed} probed fault${learned.observed === 1 ? '' : 's'} of this project's own, across ${learned.classes.length} class${learned.classes.length === 1 ? '' : 'es'} (${learned.sources.join(', ')}), shrunk toward the shipped prior.`;
|
|
146
|
+
}
|
|
@@ -171,3 +171,56 @@ export function persistenceHint(signal, faultLine) {
|
|
|
171
171
|
const field = /^\s*([A-Za-z_$][\w$]*)\s*(?::|,\s*$)/.exec(faultLine ?? '')?.[1];
|
|
172
172
|
return `${signal.file} ${signal.reason} Assert the whole object written by the call this fault changes${field ? `, including \`${field}\`` : ''} — or read the record back and assert on what was stored.`;
|
|
173
173
|
}
|
|
174
|
+
|
|
175
|
+
/**
|
|
176
|
+
* What this claim's defenders are CAPABLE of proving about a write.
|
|
177
|
+
*
|
|
178
|
+
* The read-back question — "did the row actually land?" — cannot be answered by
|
|
179
|
+
* a test that replaced the database with a mock. Such a test can prove the call
|
|
180
|
+
* shape and nothing beyond it: the `where` may match no rows, the transaction
|
|
181
|
+
* may roll back, a constraint may reject the write, and every one of those
|
|
182
|
+
* passes against a mock that recorded the arguments and returned a plausible
|
|
183
|
+
* object.
|
|
184
|
+
*
|
|
185
|
+
* So rather than pretend to measure persistence, this states the LIMIT. It is
|
|
186
|
+
* the same discipline as `nocover` — which says "no test imports this" instead
|
|
187
|
+
* of guessing — one level finer: these tests exist, they run, and there is a
|
|
188
|
+
* class of failure they structurally cannot see.
|
|
189
|
+
*
|
|
190
|
+
* `mocked` every defender replaces the persistence layer. Call shape only.
|
|
191
|
+
* `unmocked` at least one does not, so a real write is at least possible.
|
|
192
|
+
* NOT a promise that one happens: a test may simply never reach
|
|
193
|
+
* the database. Possible is the honest word.
|
|
194
|
+
* `none` no defender at all; `nocover` already covers this.
|
|
195
|
+
*
|
|
196
|
+
* Measured on a real AI-authored application: of 37 files that write to
|
|
197
|
+
* storage, 32 were defended only by mocking tests, 4 had no defender, and 1
|
|
198
|
+
* had an unmocked defender — a mailer, not a database write. Nothing in that
|
|
199
|
+
* suite could prove any save persisted.
|
|
200
|
+
*/
|
|
201
|
+
export function persistenceProvability(projectDir, defenders) {
|
|
202
|
+
if (!defenders || defenders.length === 0) return { class: 'none', mocking: [], unmocked: [] };
|
|
203
|
+
const mocking = [];
|
|
204
|
+
const unmocked = [];
|
|
205
|
+
for (const d of defenders) (analyzePersistence(projectDir, d) ? mocking : unmocked).push(d);
|
|
206
|
+
return { class: unmocked.length ? 'unmocked' : 'mocked', mocking, unmocked };
|
|
207
|
+
}
|
|
208
|
+
|
|
209
|
+
/**
|
|
210
|
+
* The provability of a whole surface, as counts. The headline a save-path
|
|
211
|
+
* report opens with, because "how many of these could be proven at all" is a
|
|
212
|
+
* different question from "how many were probed" and a far more alarming one.
|
|
213
|
+
*/
|
|
214
|
+
export function provabilitySummary(projectDir, targets, defendersOf) {
|
|
215
|
+
const counts = { mocked: 0, unmocked: 0, none: 0 };
|
|
216
|
+
for (const file of targets) {
|
|
217
|
+
let defenders = [];
|
|
218
|
+
try {
|
|
219
|
+
defenders = defendersOf(file);
|
|
220
|
+
} catch {
|
|
221
|
+
defenders = [];
|
|
222
|
+
}
|
|
223
|
+
counts[persistenceProvability(projectDir, defenders).class] += 1;
|
|
224
|
+
}
|
|
225
|
+
return counts;
|
|
226
|
+
}
|
package/src/sweep/sweep.mjs
CHANGED
|
@@ -35,9 +35,12 @@ import { loadClaims, defaultClaimsPath } from '../claims/load.mjs';
|
|
|
35
35
|
import { computeChangedGate } from '../gate/changed.mjs';
|
|
36
36
|
import { scaffoldFile } from '../scaffold/scaffold.mjs';
|
|
37
37
|
import { selectFaults, onWritePath, capFor } from '../supply/select.mjs';
|
|
38
|
+
import { learnedProductivity } from '../supply/feedback.mjs';
|
|
39
|
+
import { loadConcerns, concernById, targetsFor, filterByConcern } from '../supply/concerns.mjs';
|
|
38
40
|
import { saveSurface } from '../supply/savepath.mjs';
|
|
39
|
-
import { persistenceSignals, persistenceHint } from '../supply/persistence.mjs';
|
|
41
|
+
import { persistenceSignals, persistenceHint, provabilitySummary } from '../supply/persistence.mjs';
|
|
40
42
|
import { probe } from '../probe/probe.mjs';
|
|
43
|
+
import { discoverDefenders } from '../probe/discover.mjs';
|
|
41
44
|
import { hintFor } from '../brief/brief.mjs';
|
|
42
45
|
|
|
43
46
|
/**
|
|
@@ -106,6 +109,9 @@ export function sortFindings(findings) {
|
|
|
106
109
|
/** How many findings a sweep is willing to fail on. */
|
|
107
110
|
export const gatingCount = (findings) => findings.filter((f) => GATING.has(f.verdict)).length;
|
|
108
111
|
|
|
112
|
+
/** A concern named on the command line that this project does not declare. */
|
|
113
|
+
export class ConcernError extends Error {}
|
|
114
|
+
|
|
109
115
|
/** Rebuild claim documents from a flat selection, keeping each claim's own faults together. */
|
|
110
116
|
function claimsFromSelection(selected) {
|
|
111
117
|
const byClaim = new Map();
|
|
@@ -128,6 +134,8 @@ export async function sweep({
|
|
|
128
134
|
projectDir,
|
|
129
135
|
ref,
|
|
130
136
|
mode = 'changed',
|
|
137
|
+
concern: concernId,
|
|
138
|
+
concernsPath,
|
|
131
139
|
includeDirty = false,
|
|
132
140
|
exclude = [],
|
|
133
141
|
cap,
|
|
@@ -154,10 +162,23 @@ export async function sweep({
|
|
|
154
162
|
//
|
|
155
163
|
// A test file with no claim is a different problem — it has nothing to
|
|
156
164
|
// falsify — and the gate already names it. Sweep proposes against source.
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
165
|
+
// `--save-paths` is sugar for the built-in SAVE-PERSISTS concern: the same
|
|
166
|
+
// targets and the same producer selection, named so a project can retune it
|
|
167
|
+
// for an ORM this tool does not recognise without forking anything.
|
|
168
|
+
const loaded = loadConcerns(projectDir, { path: concernsPath });
|
|
169
|
+
const concern = concernId
|
|
170
|
+
? concernById(loaded.concerns, concernId)
|
|
171
|
+
: mode === 'save-paths' ? concernById(loaded.concerns, 'SAVE-PERSISTS') : null;
|
|
172
|
+
if (concernId && !concern) {
|
|
173
|
+
throw new ConcernError(`unknown concern ${concernId}. Declared: ${loaded.concerns.map((c) => c.id).join(', ')}`);
|
|
174
|
+
}
|
|
175
|
+
const changedTargets = gate.uncovered.filter((u) => u.kind === 'source').map((u) => u.file);
|
|
176
|
+
// A concern whose targets are the diff defers to the gate rather than
|
|
177
|
+
// re-deriving what it already computed.
|
|
178
|
+
const concernTargets = concern ? targetsFor(projectDir, concern, { changedFiles: changedTargets }) : null;
|
|
179
|
+
const usesSurface = concern?.targets?.kind === 'write-sites' || mode === 'save-paths';
|
|
180
|
+
const surface = usesSurface ? saveSurface(projectDir) : null;
|
|
181
|
+
const targets = concernTargets ?? changedTargets;
|
|
161
182
|
|
|
162
183
|
const cPath = claimsPath ?? defaultClaimsPath(projectDir);
|
|
163
184
|
const existingClaims = existsSync(cPath) ? loadClaims(cPath) : { claims: [] };
|
|
@@ -184,8 +205,27 @@ export async function sweep({
|
|
|
184
205
|
for (const claim of result.doc.claims) for (const fault of claim.faults) candidates.push({ claim, fault });
|
|
185
206
|
}
|
|
186
207
|
|
|
208
|
+
// What the defenders of this surface are CAPABLE of proving, before anything
|
|
209
|
+
// is probed. A test that replaced the database can prove the call shape and
|
|
210
|
+
// never that the row landed, and that limit is a property of the suite rather
|
|
211
|
+
// than of any verdict — so it is measured once, over the whole surface, and
|
|
212
|
+
// reported whether or not a single fault survives.
|
|
213
|
+
const provability = surface
|
|
214
|
+
? provabilitySummary(projectDir, targets, (f) => discoverDefenders(projectDir, f))
|
|
215
|
+
: null;
|
|
216
|
+
|
|
217
|
+
// Order by what THIS project's own runs have shown, not only by what was
|
|
218
|
+
// measured elsewhere. A project that has probed nothing falls back to the
|
|
219
|
+
// shipped prior; one with its own records overrides it in proportion to how
|
|
220
|
+
// many it has. This is the feedback half of Google's 15%-to-89%, using data
|
|
221
|
+
// the project already produced rather than a new thing to collect.
|
|
222
|
+
const learned = learnedProductivity(projectDir);
|
|
223
|
+
// Narrowing is what makes one sentence useful across two hundred files: a
|
|
224
|
+
// persistence concern that also reported every altered return value would
|
|
225
|
+
// bury the payload findings it exists to surface.
|
|
226
|
+
const relevant = filterByConcern(candidates, concern);
|
|
187
227
|
const limit = cap ?? capFor(targets.length);
|
|
188
|
-
const selection = selectFaults(
|
|
228
|
+
const selection = selectFaults(relevant, { cap: limit, table: learned.table });
|
|
189
229
|
const claims = claimsFromSelection(selection.selected);
|
|
190
230
|
|
|
191
231
|
let evidence = { records: [] };
|
|
@@ -234,20 +274,28 @@ export async function sweep({
|
|
|
234
274
|
includeDirty,
|
|
235
275
|
scope: {
|
|
236
276
|
mode,
|
|
277
|
+
...(concern ? { concern: concern.id } : {}),
|
|
237
278
|
changed: gate.changed,
|
|
238
|
-
...(surface ? { writeSites: surface.siteCount, payloadFields: surface.keyCount } : {}),
|
|
279
|
+
...(surface ? { writeSites: surface.siteCount, payloadFields: surface.keyCount, provability } : {}),
|
|
239
280
|
targets: targets.length,
|
|
240
281
|
swept: drafts.length,
|
|
241
282
|
...(skipped.length ? { skipped } : {}),
|
|
242
283
|
},
|
|
243
284
|
selection: {
|
|
244
|
-
proposed:
|
|
285
|
+
proposed: relevant.length,
|
|
286
|
+
...(candidates.length !== relevant.length ? { outOfScope: candidates.length - relevant.length } : {}),
|
|
245
287
|
selected: selection.selected.length,
|
|
246
288
|
deferred: selection.deferred.length,
|
|
247
289
|
cap: selection.cap,
|
|
248
290
|
byClass: selection.byClass,
|
|
249
291
|
},
|
|
250
292
|
counts,
|
|
293
|
+
ordering: { observed: learned.observed, sources: learned.sources },
|
|
294
|
+
// Carried, not written here: the caller owns I/O. Persisting it is what
|
|
295
|
+
// closes the feedback loop — the next sweep learns from these verdicts,
|
|
296
|
+
// which are MACHINE-proposed faults and so the closest observation of the
|
|
297
|
+
// distribution the ranker orders.
|
|
298
|
+
evidence,
|
|
251
299
|
findings: ordered,
|
|
252
300
|
exitCode: gating > 0 ? 1 : 0,
|
|
253
301
|
drafts,
|
|
@@ -262,16 +310,44 @@ export function renderSweep(doc, { limit = 20 } = {}) {
|
|
|
262
310
|
if (scope.targets === 0) {
|
|
263
311
|
out.push(saves
|
|
264
312
|
? 'sweep: no write to storage found in this project. Nothing to propose.'
|
|
265
|
-
:
|
|
313
|
+
: scope.concern
|
|
314
|
+
? `sweep: concern ${scope.concern} matches no file in this project. Nothing to propose.`
|
|
315
|
+
: `sweep: no changed source file is without a claim against ${doc.ref}. Nothing to propose.`);
|
|
266
316
|
return out.join('\n');
|
|
267
317
|
}
|
|
268
318
|
// The denominator first. A report that lists findings without saying how much
|
|
269
319
|
// was looked at invites the reader to assume the rest is fine.
|
|
320
|
+
// Say how the sweep was aimed. A glob concern that reports "unclaimed changed
|
|
321
|
+
// files" is describing a diff it never looked at.
|
|
270
322
|
out.push(saves
|
|
271
323
|
? `${scope.writeSites} write${scope.writeSites === 1 ? '' : 's'} to storage across ${scope.targets} file${scope.targets === 1 ? '' : 's'}, carrying ${scope.payloadFields} payload field${scope.payloadFields === 1 ? '' : 's'}. Swept ${scope.swept}.`
|
|
272
|
-
:
|
|
324
|
+
: scope.concern
|
|
325
|
+
? `concern ${scope.concern} matches ${scope.targets} file${scope.targets === 1 ? '' : 's'}. Swept ${scope.swept}.`
|
|
326
|
+
: `swept ${scope.swept} of ${scope.targets} unclaimed changed file${scope.targets === 1 ? '' : 's'} against ${doc.ref}.`);
|
|
327
|
+
if (saves && scope.provability) {
|
|
328
|
+
// The limit, before any verdict. A test that replaced the database can
|
|
329
|
+
// prove the call shape and never that the row landed, so on a surface
|
|
330
|
+
// defended entirely by such tests a clean probe is not evidence of
|
|
331
|
+
// persistence — it is evidence that nothing could have measured it.
|
|
332
|
+
const { mocked, unmocked, none } = scope.provability;
|
|
333
|
+
out.push(` of those ${scope.targets} file${scope.targets === 1 ? '' : 's'}: ${mocked} defended only by tests that mock the persistence layer, ${unmocked} with an unmocked defender, ${none} with no defender at all.`);
|
|
334
|
+
if (unmocked === 0 && mocked > 0) {
|
|
335
|
+
out.push(' Nothing in this suite can prove a write reached storage. A mocked test proves the');
|
|
336
|
+
out.push(' call shape; a where-clause that matches nothing, a rolled-back transaction and a');
|
|
337
|
+
out.push(' rejected constraint all pass against it.');
|
|
338
|
+
}
|
|
339
|
+
}
|
|
273
340
|
out.push(`proposed ${selection.proposed} fault${selection.proposed === 1 ? '' : 's'}, probed ${selection.selected} (cap ${selection.cap})${selection.deferred ? `, deferred ${selection.deferred}` : ''}.`);
|
|
274
|
-
if (selection.deferred) out.push(` The ${selection.deferred} deferred are not a verdict: raise --cap${saves ? '.' : ', or sweep a smaller change.'}`);
|
|
341
|
+
if (selection.deferred) out.push(` The ${selection.deferred} deferred are not a verdict: raise --cap${saves || scope.concern ? '.' : ', or sweep a smaller change.'}`);
|
|
342
|
+
// A narrow concern must not look like a quiet one: the producers found these,
|
|
343
|
+
// and this concern is not about them.
|
|
344
|
+
if (selection.outOfScope) out.push(` ${selection.outOfScope} further proposal${selection.outOfScope === 1 ? ' was' : 's were'} outside this concern's fault classes.`);
|
|
345
|
+
// An ordering nobody can trace is a number nobody should trust.
|
|
346
|
+
if (doc.ordering) {
|
|
347
|
+
out.push(doc.ordering.observed === 0
|
|
348
|
+
? ' ordering: the shipped productivity prior — this project has no probed evidence yet.'
|
|
349
|
+
: ` ordering: ${doc.ordering.observed} probed fault${doc.ordering.observed === 1 ? '' : 's'} of this project's own (${doc.ordering.sources.join(', ')}), shrunk toward the shipped prior.`);
|
|
350
|
+
}
|
|
275
351
|
for (const s of scope.skipped ?? []) out.push(` skipped ${s.file}: ${s.reason}`);
|
|
276
352
|
out.push('');
|
|
277
353
|
|