testguard-cli 0.11.0 → 0.12.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -7,6 +7,57 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
7
7
 
8
8
  ## [Unreleased]
9
9
 
10
+ ## [0.12.0] - 2026-09-22
11
+
12
+ Automated weekly release — everything merged since `v0.11.0`.
13
+
14
+ ### Changed
15
+
16
+ - feat(sweep): --save-paths, and the denominator a save report is read against (#156)
17
+ - docs: make the brief describe what the tool actually does, and record the prior art (#155)
18
+ - chore(homebrew): sha256 for v0.11.0 (#153)
19
+
20
+
21
+ ### Added
22
+
23
+ - `sweep --save-paths` — sweep every file that writes to storage instead of the
24
+ diff, and report the surface as the denominator the findings are read
25
+ against. A diff cannot answer how many save paths exist, and a clean report
26
+ over an unstated denominator is a sample of unknown size. The sweep document
27
+ gains `scope.mode`, `scope.writeSites` and `scope.payloadFields`, and the
28
+ validator refuses a save-paths sweep that omits its surface.
29
+ - `src/supply/savepath.mjs` — the write-site enumerator: which calls put data
30
+ into storage, and which payload keys sit on their own line (the ones a
31
+ `field-dropped` fault can anchor to). Measured on a real AI-authored
32
+ application: 119 writes across 37 files carrying 226 payload fields.
33
+
34
+ ### Fixed
35
+
36
+ - `walk()` skipped `dist` and `coverage` but not `.next`, `build`, `out`,
37
+ `.svelte-kit`, `.turbo` and their siblings, while the claim above it already
38
+ promised that a build's test files cannot be picked up. A scan of a real
39
+ Next.js application counted 104 files instead of 32, because `.next/` carries
40
+ compiled copies of the application — its writes and its tests. The claim was
41
+ right; the list was short.
42
+
43
+ ### Added
44
+
45
+ - `docs-canonical/PRIOR-ART.md` — the canonical record of what was taken from
46
+ Google's and Meta's mutation-testing programmes, what was deliberately not
47
+ taken, the measured reason no model decides a verdict, and the known gaps in
48
+ what was borrowed. Registered with DocGuard.
49
+ - README prior-art section pointing at it.
50
+
51
+ ### Changed
52
+
53
+ - The technical brief now describes what the tool actually does. A new
54
+ *Starting from zero* page carries Field report C (the persistence-path
55
+ measurement: 12 of 26 faults survived a green suite, all of them a dropped
56
+ payload field) and the claim-supply ladder; prior art names Google and Meta
57
+ and what each solved; the decision-model result is stated as a measurement
58
+ rather than a design assertion; and a *what it deliberately is not* section
59
+ closes it. `sweep` added to the verb table. Six pages to eight, re-rendered.
60
+
10
61
  ## [0.11.0] - 2026-09-21
11
62
 
12
63
  Automated weekly release — everything merged since `v0.10.4`.
package/README.md CHANGED
@@ -127,9 +127,9 @@ raises coverage. TestGuard admits it because it fails when the claim is false.
127
127
  | npm | `npm i -D testguard-cli` then `npx testguard probe` |
128
128
  | pip | `pip install testguard-cli` then `testguard probe` (needs Node ≥ 20) |
129
129
  | Homebrew | `brew tap raccioly/tap && brew install testguard` |
130
- | GitHub Action | `uses: raccioly/testguard@v0.11.0` — see [`action.yml`](./action.yml) |
130
+ | GitHub Action | `uses: raccioly/testguard@v0.12.0` — see [`action.yml`](./action.yml) |
131
131
  | pre-commit | `repo: https://github.com/raccioly/testguard`, hooks `testguard-claims`, `testguard-probe` |
132
- | GitLab CI | `include: - remote: https://raw.githubusercontent.com/raccioly/testguard/v0.11.0/packaging/gitlab/testguard.gitlab-ci.yml` with `inputs:` — see [`packaging/gitlab/`](./packaging/gitlab/testguard.gitlab-ci.yml) |
132
+ | GitLab CI | `include: - remote: https://raw.githubusercontent.com/raccioly/testguard/v0.12.0/packaging/gitlab/testguard.gitlab-ci.yml` with `inputs:` — see [`packaging/gitlab/`](./packaging/gitlab/testguard.gitlab-ci.yml) |
133
133
 
134
134
  Projects that set `min-release-age` in `.npmrc` cannot see a version published
135
135
  less than that many days ago (`ENOVERSIONS`); install that one with
@@ -466,6 +466,39 @@ A sweep is weaker evidence than a probe, and says so. Nobody stated that the
466
466
  behaviour mattered. It is stronger than nothing, which is what a repository
467
467
  with no claims has — and the survivors worth defending become the first claims.
468
468
 
469
+ #### Sweeping the save surface instead of the diff
470
+
471
+ A diff cannot answer *"when the user clicks save, does it actually save?"* — it
472
+ only knows what changed today. `--save-paths` points the same sweep at every
473
+ file that writes to storage:
474
+
475
+ ```bash
476
+ npx testguard-cli sweep --save-paths --cap 20
477
+ ```
478
+
479
+ ```
480
+ 119 writes to storage across 37 files, carrying 226 payload fields. Swept 36.
481
+ proposed 762 faults, probed 20 (cap 20), deferred 742.
482
+
483
+ SURVIVED src/app/actions/register.ts [write path]
484
+ [line 59] Field dropped: `password_hash` is no longer written.
485
+ src/tests/actions/register.test.ts mocks @/lib/prisma; 1 exact, 8 partial,
486
+ 31 argument-free call assertions in the file. A field dropped from the write
487
+ payload fails only against an assertion that names that field exactly.
488
+ ```
489
+
490
+ The first line is the point. A report that lists findings without saying how
491
+ much surface it looked at invites you to assume the rest is fine, so the
492
+ denominator comes first and the document carries `writeSites` and
493
+ `payloadFields` whether or not anything survived. Measured on a real
494
+ AI-authored application: **12 of 26 probed persistence faults survived a fully
495
+ green suite, and all 12 were a dropped payload field.**
496
+
497
+ This is a *sample* of the surface, and says so. It does not yet prove a write
498
+ reached storage — only that a test would notice if the payload changed.
499
+ Read-back oracles are the next rung
500
+ ([#154](https://github.com/raccioly/testguard/issues/154)).
501
+
469
502
  ### Every change needs a claim
470
503
 
471
504
  `probe` can only verify claims that exist. Every escaped defect in the field
@@ -547,12 +580,12 @@ detected base that does not resolve is a warning for `status` and `brief`
547
580
  # GitHub Actions
548
581
  - uses: actions/checkout@v4
549
582
  with: { fetch-depth: 0 }
550
- - uses: raccioly/testguard@v0.11.0
583
+ - uses: raccioly/testguard@v0.12.0
551
584
  with: { command: gate }
552
585
 
553
586
  # GitLab CI — the component-shaped template: gate + probe, brief as an artifact and, opted in, as a merge-request note
554
587
  include:
555
- - remote: 'https://raw.githubusercontent.com/raccioly/testguard/v0.11.0/packaging/gitlab/testguard.gitlab-ci.yml'
588
+ - remote: 'https://raw.githubusercontent.com/raccioly/testguard/v0.12.0/packaging/gitlab/testguard.gitlab-ci.yml'
556
589
  inputs: { dir: backend, post_note: true } # post_note needs TESTGUARD_GITLAB_TOKEN (api scope); one note, updated in place
557
590
  ```
558
591
 
@@ -580,6 +613,32 @@ talks to the network; only the job posts, and only when told to.
580
613
  toward green. The one pass is `killed`; everything else gates.
581
614
  - **One exact-pinned runtime dependency** (`ajv`), Node ≥ 20, MIT.
582
615
 
616
+ ### Prior art, and what was taken from where
617
+
618
+ Breaking code to test the tests is mutation testing, and it dates to the 1970s.
619
+ Two industrial programmes solved the parts that make it usable, and this tool
620
+ takes from both.
621
+
622
+ **Google** made it affordable: mutate only *changed* lines, cap what is
623
+ surfaced at seven per file, and order candidates by the measured productivity
624
+ of the operator in similar context — developer feedback took their productive
625
+ rate from 15% to 89%
626
+ ([Petrović et al., 2021](https://arxiv.org/abs/2102.11378)). `sweep`'s diff
627
+ scope, its cap and its ordering are theirs; the productivity prior here is
628
+ measured on real probed faults rather than assumed.
629
+
630
+ **Meta** made it specific: an engineer writes a *concern* in plain text, a
631
+ model drafts faults for it, and every step is gated by execution before a
632
+ human ever sees it — 73% of the resulting tests were accepted
633
+ ([Foster et al., FSE 2025](https://arxiv.org/abs/2501.12862)). The rule that
634
+ nothing a model proposes reaches the claims file without passing `probe` and a
635
+ human keep is theirs.
636
+
637
+ What neither does is bind a fault to a **stated promise**, which is what this
638
+ tool is for. The full record — including what was deliberately *not* taken, the
639
+ measured reason no model decides a verdict, and the gaps in what we borrowed —
640
+ is in [`docs-canonical/PRIOR-ART.md`](docs-canonical/PRIOR-ART.md).
641
+
583
642
  ### What TestGuard is not
584
643
 
585
644
  - **Not a test generator.** It judges a test the agent wrote (`admit`); the
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "testguard-cli",
3
- "version": "0.11.0",
3
+ "version": "0.12.0",
4
4
  "description": "Breaks your code on purpose and reports every promise your tests did not notice breaking. Mutation testing bound to the claims a project makes — vitest, jest, Playwright, pytest and unittest.",
5
5
  "type": "module",
6
6
  "license": "MIT",
@@ -313,6 +313,18 @@ const semantic = {
313
313
  if (s.proposed !== s.selected + s.deferred) errors.push({ path: '/selection/proposed', message: `proposed (${s.proposed}) must equal selected (${s.selected}) + deferred (${s.deferred})` });
314
314
  if (s.selected > s.cap) errors.push({ path: '/selection/selected', message: `selected (${s.selected}) exceeds the cap (${s.cap})` });
315
315
  if (doc.scope.swept > doc.scope.targets) errors.push({ path: '/scope/swept', message: `swept (${doc.scope.swept}) cannot exceed targets (${doc.scope.targets})` });
316
+ // The counters that only mean something in one mode must not appear in the
317
+ // other: `writeSites` on a diff sweep would be a denominator nobody
318
+ // measured, and its absence on a save-paths sweep hides the denominator
319
+ // entirely — which is the whole reason that mode exists.
320
+ const saves = doc.scope.mode === 'save-paths';
321
+ for (const k of ['writeSites', 'payloadFields']) {
322
+ if (saves && doc.scope[k] === undefined) errors.push({ path: `/scope/${k}`, message: `a save-paths sweep must report ${k}: the surface is the denominator its findings are read against` });
323
+ if (!saves && doc.scope[k] !== undefined) errors.push({ path: `/scope/${k}`, message: `${k} belongs to a save-paths sweep; a changed sweep did not measure the write surface` });
324
+ }
325
+ if (saves && doc.scope.writeSites !== undefined && doc.scope.targets > doc.scope.writeSites) {
326
+ errors.push({ path: '/scope/targets', message: `targets (${doc.scope.targets}) exceeds writeSites (${doc.scope.writeSites}); every target is a file with at least one write` });
327
+ }
316
328
 
317
329
  const byClass = Object.values(s.byClass);
318
330
  const proposedByClass = byClass.reduce((a, c) => a + c.proposed, 0);
@@ -4,61 +4,165 @@
4
4
  "title": "claimspec v1 — sweep",
5
5
  "description": "What a sweep proposed and what survived it. A sweep probes faults NOBODY STATED: it takes the changed source files that carry no claim, proposes faults with the mechanical producers, and reports the ones a green suite did not notice. Its findings are therefore weaker evidence than a probe's — no human said the behaviour mattered — and stronger than nothing, which is what a repository with no claims has. The drafts it carries are proposals for a human to keep or drop; they are not claims, and nothing here belongs in a claims file until someone states the sentence it defends.",
6
6
  "type": "object",
7
- "required": ["schemaVersion", "tool", "generatedAt", "ref", "base", "head", "includeDirty", "scope", "selection", "counts", "findings", "exitCode"],
7
+ "required": [
8
+ "schemaVersion",
9
+ "tool",
10
+ "generatedAt",
11
+ "ref",
12
+ "base",
13
+ "head",
14
+ "includeDirty",
15
+ "scope",
16
+ "selection",
17
+ "counts",
18
+ "findings",
19
+ "exitCode"
20
+ ],
8
21
  "additionalProperties": false,
9
22
  "properties": {
10
- "$schema": { "type": "string" },
11
- "schemaVersion": { "const": 1 },
12
- "tool": { "$ref": "urn:claimspec:v1:common#/$defs/tool" },
13
- "generatedAt": { "$ref": "urn:claimspec:v1:common#/$defs/isoDateTime" },
14
- "ref": { "type": "string", "description": "The reference the change was measured against." },
15
- "base": { "$ref": "urn:claimspec:v1:common#/$defs/gitSha" },
16
- "head": { "$ref": "urn:claimspec:v1:common#/$defs/gitSha" },
17
- "includeDirty": { "type": "boolean" },
23
+ "$schema": {
24
+ "type": "string"
25
+ },
26
+ "schemaVersion": {
27
+ "const": 1
28
+ },
29
+ "tool": {
30
+ "$ref": "urn:claimspec:v1:common#/$defs/tool"
31
+ },
32
+ "generatedAt": {
33
+ "$ref": "urn:claimspec:v1:common#/$defs/isoDateTime"
34
+ },
35
+ "ref": {
36
+ "type": "string",
37
+ "description": "The reference the change was measured against."
38
+ },
39
+ "base": {
40
+ "$ref": "urn:claimspec:v1:common#/$defs/gitSha"
41
+ },
42
+ "head": {
43
+ "$ref": "urn:claimspec:v1:common#/$defs/gitSha"
44
+ },
45
+ "includeDirty": {
46
+ "type": "boolean"
47
+ },
18
48
  "scope": {
19
49
  "type": "object",
20
- "required": ["changed", "targets", "swept"],
50
+ "required": [
51
+ "mode",
52
+ "changed",
53
+ "targets",
54
+ "swept"
55
+ ],
21
56
  "additionalProperties": false,
22
57
  "properties": {
23
- "changed": { "type": "integer", "minimum": 0, "description": "Files in the change, before any filter." },
24
- "targets": { "type": "integer", "minimum": 0, "description": "Changed SOURCE files carrying no claim — what the gate calls uncovered. A sweep proposes against these and nothing else." },
25
- "swept": { "type": "integer", "minimum": 0, "description": "Targets that yielded at least one proposal. Never more than `targets`: a file the producers cannot touch is skipped, not swept." },
58
+ "mode": {
59
+ "enum": [
60
+ "changed",
61
+ "save-paths"
62
+ ],
63
+ "description": "What the sweep was pointed at. `changed` proposes for the changed source files that carry no claim. `save-paths` proposes for every file that writes to storage, because a diff cannot answer how many save paths exist — and a clean report over an unstated denominator is a sample of unknown size, not a guarantee."
64
+ },
65
+ "changed": {
66
+ "type": "integer",
67
+ "minimum": 0,
68
+ "description": "Files in the change, before any filter."
69
+ },
70
+ "targets": {
71
+ "type": "integer",
72
+ "minimum": 0,
73
+ "description": "Changed SOURCE files carrying no claim — what the gate calls uncovered. A sweep proposes against these and nothing else."
74
+ },
75
+ "swept": {
76
+ "type": "integer",
77
+ "minimum": 0,
78
+ "description": "Targets that yielded at least one proposal. Never more than `targets`: a file the producers cannot touch is skipped, not swept."
79
+ },
26
80
  "skipped": {
27
81
  "type": "array",
28
82
  "description": "Targets that produced nothing, and why. Recorded rather than dropped: a file that silently yielded no proposal reads as a file with no findings.",
29
83
  "items": {
30
84
  "type": "object",
31
- "required": ["file", "reason"],
85
+ "required": [
86
+ "file",
87
+ "reason"
88
+ ],
32
89
  "additionalProperties": false,
33
90
  "properties": {
34
- "file": { "$ref": "urn:claimspec:v1:common#/$defs/repoPath" },
35
- "reason": { "type": "string", "minLength": 1 }
91
+ "file": {
92
+ "$ref": "urn:claimspec:v1:common#/$defs/repoPath"
93
+ },
94
+ "reason": {
95
+ "type": "string",
96
+ "minLength": 1
97
+ }
36
98
  }
37
99
  }
100
+ },
101
+ "writeSites": {
102
+ "type": "integer",
103
+ "minimum": 0,
104
+ "description": "save-paths only: calls that write to storage across the whole surface, not only the ones probed."
105
+ },
106
+ "payloadFields": {
107
+ "type": "integer",
108
+ "minimum": 0,
109
+ "description": "save-paths only: payload keys sitting on their own line, which is what a field-dropped fault can anchor to."
38
110
  }
39
111
  }
40
112
  },
41
113
  "selection": {
42
114
  "type": "object",
43
115
  "description": "What was proposed and what was probed. A sweep is capped — past a few findings per file nobody reads them, and the runner is spent either way — so the count NOT probed is reported beside the count that was. A cap that hides its own remainder is a coverage claim nobody made.",
44
- "required": ["proposed", "selected", "deferred", "cap", "byClass"],
116
+ "required": [
117
+ "proposed",
118
+ "selected",
119
+ "deferred",
120
+ "cap",
121
+ "byClass"
122
+ ],
45
123
  "additionalProperties": false,
46
124
  "properties": {
47
- "proposed": { "type": "integer", "minimum": 0 },
48
- "selected": { "type": "integer", "minimum": 0 },
49
- "deferred": { "type": "integer", "minimum": 0, "description": "Proposed and not probed. Not a verdict of any kind." },
50
- "cap": { "type": "integer", "minimum": 1 },
125
+ "proposed": {
126
+ "type": "integer",
127
+ "minimum": 0
128
+ },
129
+ "selected": {
130
+ "type": "integer",
131
+ "minimum": 0
132
+ },
133
+ "deferred": {
134
+ "type": "integer",
135
+ "minimum": 0,
136
+ "description": "Proposed and not probed. Not a verdict of any kind."
137
+ },
138
+ "cap": {
139
+ "type": "integer",
140
+ "minimum": 1
141
+ },
51
142
  "byClass": {
52
143
  "type": "object",
53
144
  "description": "Per fault class: how many were proposed, how many probed, and how many sat on a write payload.",
54
145
  "additionalProperties": {
55
146
  "type": "object",
56
- "required": ["proposed", "selected", "writePath"],
147
+ "required": [
148
+ "proposed",
149
+ "selected",
150
+ "writePath"
151
+ ],
57
152
  "additionalProperties": false,
58
153
  "properties": {
59
- "proposed": { "type": "integer", "minimum": 0 },
60
- "selected": { "type": "integer", "minimum": 0 },
61
- "writePath": { "type": "integer", "minimum": 0 }
154
+ "proposed": {
155
+ "type": "integer",
156
+ "minimum": 0
157
+ },
158
+ "selected": {
159
+ "type": "integer",
160
+ "minimum": 0
161
+ },
162
+ "writePath": {
163
+ "type": "integer",
164
+ "minimum": 0
165
+ }
62
166
  }
63
167
  }
64
168
  }
@@ -67,48 +171,120 @@
67
171
  "counts": {
68
172
  "type": "object",
69
173
  "description": "Every verdict the probe reached, including `killed`. A sweep whose faults were all caught is a real result and must be readable as one.",
70
- "additionalProperties": { "type": "integer", "minimum": 0 },
71
- "propertyNames": { "$ref": "urn:claimspec:v1:common#/$defs/verdict" }
174
+ "additionalProperties": {
175
+ "type": "integer",
176
+ "minimum": 0
177
+ },
178
+ "propertyNames": {
179
+ "$ref": "urn:claimspec:v1:common#/$defs/verdict"
180
+ }
72
181
  },
73
182
  "findings": {
74
183
  "type": "array",
75
184
  "description": "Every record that was not killed, most actionable first. `survived` and `nocover` are statements about the test suite; the rest are statements about the PROPOSAL and do not gate, because a sweep's faults are machine-made and a tool that fails on its own bad guess gets switched off.",
76
185
  "items": {
77
186
  "type": "object",
78
- "required": ["file", "claimId", "faultId", "faultClass", "verdict", "description", "writePath", "hint"],
187
+ "required": [
188
+ "file",
189
+ "claimId",
190
+ "faultId",
191
+ "faultClass",
192
+ "verdict",
193
+ "description",
194
+ "writePath",
195
+ "hint"
196
+ ],
79
197
  "additionalProperties": false,
80
198
  "properties": {
81
- "file": { "$ref": "urn:claimspec:v1:common#/$defs/repoPath" },
82
- "line": { "type": "integer", "minimum": 1 },
83
- "claimId": { "$ref": "urn:claimspec:v1:common#/$defs/identifier" },
84
- "faultId": { "$ref": "urn:claimspec:v1:common#/$defs/identifier" },
85
- "faultClass": { "$ref": "urn:claimspec:v1:common#/$defs/faultClass" },
86
- "verdict": { "$ref": "urn:claimspec:v1:common#/$defs/verdict" },
87
- "reason": { "type": "string", "minLength": 1 },
88
- "description": { "type": "string", "minLength": 1 },
89
- "writePath": { "type": "boolean", "description": "The fault sits on a database write or the payload of one. Every surviving fault in the measurement that prompted this field was such a line." },
90
- "hint": { "type": "string", "minLength": 1, "description": "One line naming the mechanism, never only the verdict." },
199
+ "file": {
200
+ "$ref": "urn:claimspec:v1:common#/$defs/repoPath"
201
+ },
202
+ "line": {
203
+ "type": "integer",
204
+ "minimum": 1
205
+ },
206
+ "claimId": {
207
+ "$ref": "urn:claimspec:v1:common#/$defs/identifier"
208
+ },
209
+ "faultId": {
210
+ "$ref": "urn:claimspec:v1:common#/$defs/identifier"
211
+ },
212
+ "faultClass": {
213
+ "$ref": "urn:claimspec:v1:common#/$defs/faultClass"
214
+ },
215
+ "verdict": {
216
+ "$ref": "urn:claimspec:v1:common#/$defs/verdict"
217
+ },
218
+ "reason": {
219
+ "type": "string",
220
+ "minLength": 1
221
+ },
222
+ "description": {
223
+ "type": "string",
224
+ "minLength": 1
225
+ },
226
+ "writePath": {
227
+ "type": "boolean",
228
+ "description": "The fault sits on a database write or the payload of one. Every surviving fault in the measurement that prompted this field was such a line."
229
+ },
230
+ "hint": {
231
+ "type": "string",
232
+ "minLength": 1,
233
+ "description": "One line naming the mechanism, never only the verdict."
234
+ },
91
235
  "signals": {
92
236
  "type": "array",
93
237
  "minItems": 1,
94
238
  "description": "Static, annotation-grade signals about the defenders of a SURVIVOR on the write path. They never change a verdict. `persistence-payload-unasserted`: the defender mocks the persistence layer and the file's call assertions never name an exact payload, so a field dropped from a write cannot fail them. Descriptive, never predictive: no threshold on the assertion mix separates the files whose payload faults survive from the files whose do not.",
95
239
  "items": {
96
240
  "type": "object",
97
- "required": ["file", "signal", "reason"],
241
+ "required": [
242
+ "file",
243
+ "signal",
244
+ "reason"
245
+ ],
98
246
  "additionalProperties": false,
99
247
  "properties": {
100
- "file": { "$ref": "urn:claimspec:v1:common#/$defs/repoPath" },
101
- "signal": { "enum": ["persistence-payload-unasserted"] },
102
- "reason": { "type": "string", "minLength": 1 },
103
- "specifiers": { "type": "array", "items": { "type": "string", "minLength": 1 } },
248
+ "file": {
249
+ "$ref": "urn:claimspec:v1:common#/$defs/repoPath"
250
+ },
251
+ "signal": {
252
+ "enum": [
253
+ "persistence-payload-unasserted"
254
+ ]
255
+ },
256
+ "reason": {
257
+ "type": "string",
258
+ "minLength": 1
259
+ },
260
+ "specifiers": {
261
+ "type": "array",
262
+ "items": {
263
+ "type": "string",
264
+ "minLength": 1
265
+ }
266
+ },
104
267
  "counts": {
105
268
  "type": "object",
106
- "required": ["exact", "partial", "argless"],
269
+ "required": [
270
+ "exact",
271
+ "partial",
272
+ "argless"
273
+ ],
107
274
  "additionalProperties": false,
108
275
  "properties": {
109
- "exact": { "type": "integer", "minimum": 0 },
110
- "partial": { "type": "integer", "minimum": 0 },
111
- "argless": { "type": "integer", "minimum": 0 }
276
+ "exact": {
277
+ "type": "integer",
278
+ "minimum": 0
279
+ },
280
+ "partial": {
281
+ "type": "integer",
282
+ "minimum": 0
283
+ },
284
+ "argless": {
285
+ "type": "integer",
286
+ "minimum": 0
287
+ }
112
288
  }
113
289
  }
114
290
  }
@@ -117,11 +293,18 @@
117
293
  }
118
294
  }
119
295
  },
120
- "exitCode": { "enum": [0, 1] },
296
+ "exitCode": {
297
+ "enum": [
298
+ 0,
299
+ 1
300
+ ]
301
+ },
121
302
  "drafts": {
122
303
  "type": "array",
123
304
  "description": "The draft claims documents the proposals came from, carried so a human can keep one without re-running the sweep. Drafts, never claims: every statement is a TODO nobody has written yet.",
124
- "items": { "type": "object" }
305
+ "items": {
306
+ "type": "object"
307
+ }
125
308
  }
126
309
  }
127
310
  }
package/src/cli.mjs CHANGED
@@ -76,7 +76,10 @@ probe
76
76
  --json the status document plus this run's result (records, newSinceBaseline, exitCode)
77
77
 
78
78
  scaffold --claim <ID> (put every proposal under this claim; copies it if it exists) --out <path> --json
79
- sweep --changed <ref> (required; CI bases and a safe local default are detected) --cap <n> (default 7 x unclaimed files)
79
+ sweep --changed <ref> (required unless --save-paths; CI bases and a safe local default are detected) --cap <n>
80
+ --save-paths sweep every file that WRITES TO STORAGE instead of the diff, and report the surface as the
81
+ denominator: "120 writes across 38 files, swept 12". A clean sweep over an unstated
82
+ denominator is a sample of unknown size, not a guarantee that saving works
80
83
  --include-dirty --exclude <glob> --confirm <n> --budget <ms> --max <n> --out <path> --json
81
84
  the cold start: no claim and no concern needed. Everything it proposes is a DRAFT — it never writes testguard.claims.json,
82
85
  and its evidence never replaces .testguard/evidence.json. exit 1 on a fault that survived, or a file no test imports
@@ -150,6 +153,7 @@ export async function main(argv, io = { out: (s) => process.stdout.write(s + '\n
150
153
  since: { type: 'string' },
151
154
  exclude: { type: 'string', multiple: true },
152
155
  cap: { type: 'string' },
156
+ 'save-paths': { type: 'boolean', default: false },
153
157
  strict: { type: 'boolean', default: false },
154
158
  explain: { type: 'boolean', default: false },
155
159
  ignore: { type: 'string' },
@@ -20,9 +20,14 @@ export const sweepPath = (projectDir) => join(projectDir, '.testguard', 'sweep.j
20
20
  * read.
21
21
  */
22
22
  export async function sweepCommand({ projectDir, values, version }, io) {
23
- const resolved = resolveChangedRef({ explicit: values.changed, projectDir });
23
+ const mode = values['save-paths'] ? 'save-paths' : 'changed';
24
+ // `--save-paths` scans the whole write surface, so it needs no diff. The gate
25
+ // is still computed underneath (the document reports `changed`), and HEAD is
26
+ // a reference every repository has.
27
+ const resolved = resolveChangedRef({ explicit: values.changed, projectDir })
28
+ ?? (mode === 'save-paths' ? { ref: 'HEAD', from: 'save-paths', required: false } : null);
24
29
  if (!resolved) {
25
- io.err('sweep needs a reference to measure the change against: --changed <ref> (e.g. origin/main), or set TESTGUARD_CHANGED_REF. CI bases and a safe local remote default or differently named upstream are detected automatically.');
30
+ io.err('sweep needs a reference to measure the change against: --changed <ref> (e.g. origin/main), or set TESTGUARD_CHANGED_REF. CI bases and a safe local remote default or differently named upstream are detected automatically. Or sweep the write surface instead with --save-paths.');
26
31
  return 3;
27
32
  }
28
33
  const confirmRuns = Number(values.confirm);
@@ -39,11 +44,12 @@ export async function sweepCommand({ projectDir, values, version }, io) {
39
44
  return 3;
40
45
  }
41
46
  }
42
- if (!resolved.required && !values.json && !values.quiet) io.err(`sweep: comparing against ${resolved.ref} (${resolved.from})`);
47
+ if (mode === 'changed' && !resolved.required && !values.json && !values.quiet) io.err(`sweep: comparing against ${resolved.ref} (${resolved.from})`);
43
48
 
44
49
  const doc = await sweep({
45
50
  projectDir,
46
51
  ref: resolved.ref,
52
+ mode,
47
53
  includeDirty: values['include-dirty'],
48
54
  exclude: values.exclude ?? [],
49
55
  cap,
@@ -0,0 +1,216 @@
1
+ /**
2
+ * Where this project writes to storage, and what it writes.
3
+ *
4
+ * A probe answers "would a test notice if this were wrong". It cannot answer
5
+ * "how many places write to the database, and how many of those has anyone
6
+ * looked at" — and on an application with two hundred screens that second
7
+ * question is the one that decides whether a clean report means anything. A
8
+ * verdict with no denominator is not a guarantee; it is a sample of unknown
9
+ * size.
10
+ *
11
+ * So this enumerates the surface. Measured on a real AI-authored Next.js
12
+ * application: 115 write sites across 34 source files, of which 46% of the
13
+ * probed persistence faults survived a fully green suite, and every survivor
14
+ * was a field dropped from the payload. The faults are already produced by
15
+ * `field-dropped` and already ranked up by `onWritePath`; what was missing was
16
+ * the list of places to point them at.
17
+ *
18
+ * DELIBERATELY SHALLOW. This finds call sites, not data flow. It does not know
19
+ * whether a write is reachable, whether two writes are the same transaction,
20
+ * or whether a helper wraps one. A site it cannot see is reported as nothing
21
+ * at all rather than as an absence of risk — the same rule the scaffold
22
+ * producers follow, and the reason `skipped` exists on a sweep. Its output is
23
+ * a denominator and a set of targets, never a verdict.
24
+ */
25
+ import { readFileSync } from 'node:fs';
26
+ import { join, extname } from 'node:path';
27
+ import { walk } from '../util/glob.mjs';
28
+
29
+ /**
30
+ * A call that puts data into storage.
31
+ *
32
+ * The receiver is matched by NAME because that is how these layers are bound
33
+ * in practice — `prisma.user.create`, `tx.invitation.upsert`, `db.orders.insertOne`.
34
+ * Matching by import would be more correct and would miss the common case
35
+ * where the client is a module-scope singleton imported under a project alias.
36
+ *
37
+ * `delete` is included: a save path that stops deleting is as broken as one
38
+ * that stops writing, and account deletion is where that matters most.
39
+ */
40
+ const WRITE_CALL = /\b(prisma|tx|trx|db|database|knex|sequelize|mongoose|supabase|drizzle|client|conn)\s*\.\s*(?:([A-Za-z_$][\w$]*)\s*\.\s*)?(create|createMany|createOne|update|updateMany|updateOne|upsert|insert|insertOne|insertMany|save|delete|deleteMany|deleteOne|destroy|remove|set|put)\s*\(/g;
41
+
42
+ /**
43
+ * Raw SQL that writes. A template literal or a string argument, either way.
44
+ *
45
+ * Two regexes for one pattern, deliberately. A `g`-flagged RegExp carries
46
+ * `lastIndex` across calls, so `WRITE_SQL.test(a)` followed by
47
+ * `WRITE_SQL.test(b)` answers the second question from where the first one
48
+ * stopped — which makes a scan's result depend on the order the files were
49
+ * read. `matchAll` needs the flag; the cheap reject must not have it.
50
+ */
51
+ const WRITE_SQL = /\b(INSERT\s+INTO|UPDATE\s+[A-Za-z_"`[]|DELETE\s+FROM|UPSERT\s+INTO)\b/gi;
52
+ const HAS_SQL = new RegExp(WRITE_SQL.source, 'i');
53
+
54
+ /**
55
+ * Every operation name the call pattern accepts, as one alternation. Derived
56
+ * rather than written twice: the cheap reject below used a shorter list, and a
57
+ * `deleteMany(` call matched the full pattern while failing the reject, so
58
+ * every file whose only write was a `...Many` or `...One` call was dropped
59
+ * from the surface before it was ever scanned.
60
+ */
61
+ const OPERATIONS = 'create|createMany|createOne|update|updateMany|updateOne|upsert|insert|insertOne|insertMany|save|delete|deleteMany|deleteOne|destroy|remove|set|put';
62
+ const HAS_WRITE_CALL = new RegExp(`\\b(?:${OPERATIONS})\\s*\\(`);
63
+
64
+ /**
65
+ * Files that contain writes but are never the product's own save path.
66
+ *
67
+ * Seed and migration scripts write, and a real run surfaced
68
+ * `prisma/seed-badges.ts` as NOCOVER — true, and not a finding: nobody tests a
69
+ * seed script, and reporting it teaches the reader to skim. The directory
70
+ * forms are matched as well as the file stems, because a seed lives in
71
+ * `prisma/seed.ts` as often as in `seeds/`.
72
+ */
73
+ const NOT_SOURCE = /\.(test|spec)\.[cm]?[jt]sx?$|(^|\/)(tests?|__tests__|__mocks__|__fixtures__|fixtures|migrations|seeds?|scripts?|prisma|drizzle)\/|(^|\/)seeds?[.-][^/]*$|(^|\/)seeds?\.[cm]?[jt]sx?$/;
74
+
75
+ const SOURCE_EXT = new Set(['.js', '.mjs', '.cjs', '.ts', '.mts', '.cts', '.jsx', '.tsx']);
76
+
77
+ /**
78
+ * The text of one call's arguments, from its open parenthesis to the matching
79
+ * close, with quotes tracked so a parenthesis inside a string is not counted.
80
+ *
81
+ * The same scan `payloadAssertions` uses, and for the same reason: a
82
+ * fixed-width window reads into the NEXT call, and on a file with several
83
+ * writes in a row that silently attributes one write's payload to another.
84
+ */
85
+ export function balancedArgs(source, open) {
86
+ let depth = 0;
87
+ let quote = null;
88
+ for (let i = open; i < source.length; i++) {
89
+ const c = source[i];
90
+ if (quote) {
91
+ if (c === '\\') i += 1;
92
+ else if (c === quote) quote = null;
93
+ continue;
94
+ }
95
+ if (c === '"' || c === "'" || c === '`') quote = c;
96
+ else if (c === '(') depth += 1;
97
+ else if (c === ')') {
98
+ depth -= 1;
99
+ if (depth === 0) return { start: open + 1, end: i, text: source.slice(open + 1, i) };
100
+ }
101
+ }
102
+ return { start: open + 1, end: source.length, text: source.slice(open + 1) };
103
+ }
104
+
105
+ /**
106
+ * The keys of a write's payload, each with the line it sits on.
107
+ *
108
+ * Both property forms, because shorthand is not an edge case here: every
109
+ * surviving payload fault measured on the register action was shorthand
110
+ * (`email,` `username,` `password_hash,`). A pattern requiring a colon would
111
+ * miss exactly the fields this exists to find.
112
+ *
113
+ * Keys are read only where they sit on their own line, which is what the
114
+ * `field-dropped` producer can anchor to. A single-line write like
115
+ * `update({ where, data })` therefore reports no keys — correctly, because
116
+ * there is no line to delete.
117
+ */
118
+ export function payloadKeys(argsText, firstLine) {
119
+ const keys = [];
120
+ argsText.split('\n').forEach((line, i) => {
121
+ const m = /^\s*(?:['"`])?([A-Za-z_$][\w$]*)(?:['"`])?\s*(?::|,\s*$)/.exec(line);
122
+ if (m) keys.push({ key: m[1], line: firstLine + i });
123
+ });
124
+ return keys;
125
+ }
126
+
127
+ /** Line number (1-based) of an index into a source string. */
128
+ const lineAt = (source, index) => source.slice(0, index).split('\n').length;
129
+
130
+ /**
131
+ * Every write this file performs. Pure apart from the read; exported so the
132
+ * shape can be tested against a string without a project on disk.
133
+ */
134
+ export function writeSitesIn(source, file) {
135
+ const sites = [];
136
+ for (const m of source.matchAll(WRITE_CALL)) {
137
+ const open = m.index + m[0].length - 1;
138
+ const args = balancedArgs(source, open);
139
+ const line = lineAt(source, m.index);
140
+ sites.push({
141
+ file,
142
+ line,
143
+ endLine: lineAt(source, args.end),
144
+ receiver: m[1],
145
+ model: m[2] ?? null,
146
+ operation: m[3],
147
+ keys: payloadKeys(args.text, lineAt(source, args.start)),
148
+ });
149
+ }
150
+ for (const m of source.matchAll(WRITE_SQL)) {
151
+ sites.push({
152
+ file,
153
+ line: lineAt(source, m.index),
154
+ endLine: lineAt(source, m.index),
155
+ receiver: 'sql',
156
+ model: null,
157
+ operation: m[1].split(/\s+/)[0].toLowerCase(),
158
+ keys: [],
159
+ });
160
+ }
161
+ // Deterministic: a surface that reorders between runs is not a denominator.
162
+ return sites.sort((a, b) => a.line - b.line || a.operation.localeCompare(b.operation));
163
+ }
164
+
165
+ /** Is this path part of the product's own save surface? */
166
+ export const isSaveSource = (rel) => SOURCE_EXT.has(extname(rel)) && !NOT_SOURCE.test(rel);
167
+
168
+ /**
169
+ * The project's whole write surface: every source file that writes, and every
170
+ * write in it.
171
+ *
172
+ * `files` may be given to scope the scan (a diff, a directory); the default is
173
+ * every source file the glob walker can see.
174
+ */
175
+ export function saveSurface(projectDir, { files } = {}) {
176
+ const candidates = (files ?? walk(projectDir)).filter(isSaveSource);
177
+ const byFile = [];
178
+ let siteCount = 0;
179
+ let keyCount = 0;
180
+ for (const rel of candidates) {
181
+ let source;
182
+ try {
183
+ source = readFileSync(join(projectDir, rel), 'utf8');
184
+ } catch {
185
+ continue;
186
+ }
187
+ // Cheap reject before the balanced scan: most files never write.
188
+ if (!HAS_WRITE_CALL.test(source) && !HAS_SQL.test(source)) continue;
189
+ const sites = writeSitesIn(source, rel);
190
+ if (!sites.length) continue;
191
+ byFile.push({ file: rel, sites });
192
+ siteCount += sites.length;
193
+ keyCount += sites.reduce((a, s) => a + s.keys.length, 0);
194
+ }
195
+ byFile.sort((a, b) => a.file.localeCompare(b.file));
196
+ return { files: byFile, fileCount: byFile.length, siteCount, keyCount };
197
+ }
198
+
199
+ /**
200
+ * The surface as text — the denominator, said out loud.
201
+ *
202
+ * A report that lists findings without saying how much was looked at invites
203
+ * the reader to assume the rest is fine, which is the assumption this whole
204
+ * tool exists to remove.
205
+ */
206
+ export function renderSurface(surface, { limit = 10 } = {}) {
207
+ if (surface.fileCount === 0) return 'no write to storage found in this project.';
208
+ const out = [`${surface.siteCount} write${surface.siteCount === 1 ? '' : 's'} to storage across ${surface.fileCount} file${surface.fileCount === 1 ? '' : 's'}, carrying ${surface.keyCount} payload field${surface.keyCount === 1 ? '' : 's'} on their own lines.`];
209
+ const ranked = [...surface.files].sort((a, b) => b.sites.length - a.sites.length || a.file.localeCompare(b.file));
210
+ for (const f of ranked.slice(0, limit)) {
211
+ const ops = [...new Set(f.sites.map((s) => s.operation))].sort().join(', ');
212
+ out.push(` ${String(f.sites.length).padStart(3)} ${f.file} (${ops})`);
213
+ }
214
+ if (ranked.length > limit) out.push(` … ${ranked.length - limit} more files`);
215
+ return out.join('\n');
216
+ }
@@ -35,6 +35,7 @@ import { loadClaims, defaultClaimsPath } from '../claims/load.mjs';
35
35
  import { computeChangedGate } from '../gate/changed.mjs';
36
36
  import { scaffoldFile } from '../scaffold/scaffold.mjs';
37
37
  import { selectFaults, onWritePath, capFor } from '../supply/select.mjs';
38
+ import { saveSurface } from '../supply/savepath.mjs';
38
39
  import { persistenceSignals, persistenceHint } from '../supply/persistence.mjs';
39
40
  import { probe } from '../probe/probe.mjs';
40
41
  import { hintFor } from '../brief/brief.mjs';
@@ -126,6 +127,7 @@ function claimsFromSelection(selected) {
126
127
  export async function sweep({
127
128
  projectDir,
128
129
  ref,
130
+ mode = 'changed',
129
131
  includeDirty = false,
130
132
  exclude = [],
131
133
  cap,
@@ -142,9 +144,20 @@ export async function sweep({
142
144
  onWarn = () => {},
143
145
  }) {
144
146
  const gate = computeChangedGate({ projectDir, ref, includeDirty, exclude, toolVersion, claimsPath, ignorePath });
147
+
148
+ // ── What the sweep is pointed at. ──
149
+ // `changed` asks "what did this change leave unclaimed". `save-paths` asks a
150
+ // question the diff cannot: "of every place this project writes to storage,
151
+ // how many would notice if the write stopped carrying a field". The second
152
+ // needs the whole surface, because a clean report over an unstated
153
+ // denominator is a sample of unknown size, not a guarantee.
154
+ //
145
155
  // A test file with no claim is a different problem — it has nothing to
146
156
  // falsify — and the gate already names it. Sweep proposes against source.
147
- const targets = gate.uncovered.filter((u) => u.kind === 'source').map((u) => u.file);
157
+ const surface = mode === 'save-paths' ? saveSurface(projectDir) : null;
158
+ const targets = surface
159
+ ? surface.files.map((f) => f.file)
160
+ : gate.uncovered.filter((u) => u.kind === 'source').map((u) => u.file);
148
161
 
149
162
  const cPath = claimsPath ?? defaultClaimsPath(projectDir);
150
163
  const existingClaims = existsSync(cPath) ? loadClaims(cPath) : { claims: [] };
@@ -220,7 +233,9 @@ export async function sweep({
220
233
  head: gate.head,
221
234
  includeDirty,
222
235
  scope: {
236
+ mode,
223
237
  changed: gate.changed,
238
+ ...(surface ? { writeSites: surface.siteCount, payloadFields: surface.keyCount } : {}),
224
239
  targets: targets.length,
225
240
  swept: drafts.length,
226
241
  ...(skipped.length ? { skipped } : {}),
@@ -243,13 +258,20 @@ export async function sweep({
243
258
  export function renderSweep(doc, { limit = 20 } = {}) {
244
259
  const out = [];
245
260
  const { scope, selection } = doc;
261
+ const saves = scope.mode === 'save-paths';
246
262
  if (scope.targets === 0) {
247
- out.push(`sweep: no changed source file is without a claim against ${doc.ref}. Nothing to propose.`);
263
+ out.push(saves
264
+ ? 'sweep: no write to storage found in this project. Nothing to propose.'
265
+ : `sweep: no changed source file is without a claim against ${doc.ref}. Nothing to propose.`);
248
266
  return out.join('\n');
249
267
  }
250
- out.push(`swept ${scope.swept} of ${scope.targets} unclaimed changed file${scope.targets === 1 ? '' : 's'} against ${doc.ref}.`);
268
+ // The denominator first. A report that lists findings without saying how much
269
+ // was looked at invites the reader to assume the rest is fine.
270
+ out.push(saves
271
+ ? `${scope.writeSites} write${scope.writeSites === 1 ? '' : 's'} to storage across ${scope.targets} file${scope.targets === 1 ? '' : 's'}, carrying ${scope.payloadFields} payload field${scope.payloadFields === 1 ? '' : 's'}. Swept ${scope.swept}.`
272
+ : `swept ${scope.swept} of ${scope.targets} unclaimed changed file${scope.targets === 1 ? '' : 's'} against ${doc.ref}.`);
251
273
  out.push(`proposed ${selection.proposed} fault${selection.proposed === 1 ? '' : 's'}, probed ${selection.selected} (cap ${selection.cap})${selection.deferred ? `, deferred ${selection.deferred}` : ''}.`);
252
- if (selection.deferred) out.push(` The ${selection.deferred} deferred are not a verdict: raise --cap, or sweep a smaller change.`);
274
+ if (selection.deferred) out.push(` The ${selection.deferred} deferred are not a verdict: raise --cap${saves ? '.' : ', or sweep a smaller change.'}`);
253
275
  for (const s of scope.skipped ?? []) out.push(` skipped ${s.file}: ${s.reason}`);
254
276
  out.push('');
255
277
 
@@ -268,6 +290,12 @@ export function renderSweep(doc, { limit = 20 } = {}) {
268
290
  }
269
291
  if (doc.findings.length > limit) out.push(` … ${doc.findings.length - limit} more in the sweep document`);
270
292
  }
293
+ if (saves) {
294
+ out.push('');
295
+ out.push(`This is a sample of the write surface, not a verdict over it: ${selection.selected} of`);
296
+ out.push(`${selection.proposed} proposed faults were probed. The ${scope.writeSites} writes above are the`);
297
+ out.push('denominator — a clean sweep says nothing about the ones nobody pointed a fault at.');
298
+ }
271
299
  out.push('These are PROPOSALS, not claims. Keep the ones worth defending: state the');
272
300
  out.push('claim, copy its fault into testguard.claims.json, and probe it from then on.');
273
301
  return out.join('\n');
package/src/util/glob.mjs CHANGED
@@ -5,7 +5,15 @@ import { join } from 'node:path';
5
5
  // virtualenv inside the project holds thousands of `test_*.py` files belonging
6
6
  // to installed packages, and without skipping it every one of them would be
7
7
  // collected as a test file of the project under probe.
8
+ // `dist` and `coverage` were here from the start; the framework build outputs
9
+ // were not, and the claim above them already promised that "a build's test
10
+ // files" are not this project's. A Next.js `.next/` carries compiled copies of
11
+ // the application — including its writes and its tests — so a surface scan of a
12
+ // real app counted the same code twice and a defender glob could match a
13
+ // compiled chunk. The promise was right and the list was short.
8
14
  const SKIP_DIRS = new Set(['node_modules', '.git', 'dist', 'coverage', '.testguard',
15
+ '.next', '.nuxt', '.svelte-kit', '.output', '.turbo', '.parcel-cache', '.angular', '.astro',
16
+ 'build', 'out', '.cache', '.vercel', '.netlify',
9
17
  '__pycache__', '.venv', 'venv', 'site-packages', '.tox', '.nox',
10
18
  '.pytest_cache', '.mypy_cache', '.ruff_cache', '.eggs']);
11
19