testguard-cli 0.13.0 → 0.14.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +87 -0
- package/README.md +13 -7
- package/package.json +1 -1
- package/spec/GATE-SEMANTICS.md +8 -0
- package/spec/lib/validate.mjs +12 -1
- package/spec/schemas/evidence.schema.json +35 -2
- package/spec/schemas/sweep.schema.json +5 -0
- package/src/probe/probe.mjs +8 -1
- package/src/probe/pyimports.mjs +9 -2
- package/src/scaffold/producers.python.mjs +54 -2
- package/src/supply/persistence.mjs +20 -0
- package/src/supply/select.mjs +39 -6
- package/src/sweep/sweep.mjs +30 -10
- package/src/util/glob.mjs +5 -1
package/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,93 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [0.14.0] - 2026-09-22
|
|
11
|
+
|
|
12
|
+
Automated weekly release — everything merged since `v0.13.0`.
|
|
13
|
+
|
|
14
|
+
### Changed
|
|
15
|
+
|
|
16
|
+
- fix: three defects the arid measurement found, presentational JSX set aside, and the claimspec signal bump (#164)
|
|
17
|
+
- docs: carry the 0.12 and 0.13 results into the brief and the prior-art record (#162)
|
|
18
|
+
- chore(homebrew): sha256 for v0.13.0 (#161)
|
|
19
|
+
|
|
20
|
+
|
|
21
|
+
### Added
|
|
22
|
+
|
|
23
|
+
- **claimspec v1 evidence admits `persistence-payload-unasserted`** to
|
|
24
|
+
`defenders.signals`, the enum shared with every claimspec consumer. The
|
|
25
|
+
signal had lived only in the sweep document, so `probe` could not surface
|
|
26
|
+
why a save-path survivor was missed. Now a `survived` record whose fault
|
|
27
|
+
sits on the write path carries it for each resolved defender that mocks the
|
|
28
|
+
persistence layer, with the mocked `specifiers` and the file's assertion
|
|
29
|
+
`counts` (exact, partial, argument-free). The validator refuses it on any
|
|
30
|
+
other verdict, on a file that is not a resolved defender, or without its
|
|
31
|
+
reason — the same rule the sweep document already enforced on its
|
|
32
|
+
findings, and the sweep now repeats what the record says rather than
|
|
33
|
+
re-deriving it. Conformance carries a valid example and three invalid
|
|
34
|
+
cases. No consumer outside this repository validates claimspec evidence
|
|
35
|
+
today; DocGuard's adoption (docguard#420) is notified rather than broken.
|
|
36
|
+
- `sweep` sets aside a purely presentational JSX element — an icon, or a
|
|
37
|
+
static wrapper with no expression and no handler — instead of spending a
|
|
38
|
+
probe run on it, and says so: `selection.presentational` in the document, a
|
|
39
|
+
line in the report, and the remainder arithmetic now proves
|
|
40
|
+
proposed = probed + deferred + set aside. Measured on a 14-file UI diff:
|
|
41
|
+
150 of 510 proposals, 29 of 30 probe slots and 7 of 10 survivors were of
|
|
42
|
+
this shape — Google's definition of an arid node, in a syntax Google never
|
|
43
|
+
mutated. Because the ordering learns from survival, those survivors were
|
|
44
|
+
also teaching it to rank the class higher; set-aside proposals never enter
|
|
45
|
+
the evidence, so they no longer can. Google's own five arid categories were
|
|
46
|
+
measured at under 1% of proposals and are not suppressed.
|
|
47
|
+
|
|
48
|
+
### Fixed
|
|
49
|
+
|
|
50
|
+
- Python defender discovery lost the last name on an import line that ends in
|
|
51
|
+
a comment: `from pkg import cli, demo # noqa: E402` found only `cli`, and
|
|
52
|
+
`import pkg.demo # noqa` found nothing. On a real suite where `# noqa` is
|
|
53
|
+
the idiom of every src-layout test, 12 of 30 probed faults were reported
|
|
54
|
+
NOCOVER for modules whose tests import them — a gating verdict in the
|
|
55
|
+
forbidden direction. Comments are now stripped before names are read.
|
|
56
|
+
- File walking descended into `.claude/`, so a nested agent worktree under
|
|
57
|
+
`.claude/worktrees/` contributed another branch's tests as defenders of this
|
|
58
|
+
tree — files the scratch worktree cannot even contain, because the directory
|
|
59
|
+
is gitignored. `.claude` joins the tool-directory skip list, and the claim
|
|
60
|
+
that promised this now names it.
|
|
61
|
+
- The Python `statement-deleted` producer proposed on lines inside a bracket
|
|
62
|
+
opened earlier — `help="…")` inside an `add_argument(` — and none of those
|
|
63
|
+
faults compiled: 81 of 600 proposals on a real CLI (13.5%). The producer now
|
|
64
|
+
tracks bracket depth across lines (triple-quoted strings included, so a
|
|
65
|
+
docstring's parentheses cannot leave the rest of the file "inside") and
|
|
66
|
+
refuses the `pass` shapes there; dict entries and call arguments keep their
|
|
67
|
+
own producers. Re-measured: 7 of 526 (1.3%).
|
|
68
|
+
|
|
69
|
+
### Changed
|
|
70
|
+
|
|
71
|
+
- The self-probe cost budget is 1650 s, from 1440. Six new claims (85 s on
|
|
72
|
+
the Node 22 PR leg, worst 22 s, all under the 100 s per-claim ceiling) took
|
|
73
|
+
that leg from 1367 s to 1457 s, over by 17 s. The v0.13.0 release job
|
|
74
|
+
measured 1113 s from scratch, but the PR leg runs the same claims 25-30%
|
|
75
|
+
slower and is where the ceiling is checked, so 1650 restores ~13% headroom
|
|
76
|
+
over the leg that binds. The per-claim ceiling is unchanged.
|
|
77
|
+
- The technical brief carries the 0.12 and 0.13 results it was missing: the
|
|
78
|
+
whole-surface provability measurement (32 of 37 files that write are defended
|
|
79
|
+
only by tests that mock the database) replaces the sample-level number in
|
|
80
|
+
Fig. 7; the ladder's second rung is now a shipped verb with a real finding
|
|
81
|
+
(a five-line concern found an admin route whose owner/manager check can be
|
|
82
|
+
forced to `false` with no test failing) rather than a description of intent;
|
|
83
|
+
and the first rung says its ordering learns from the project's own runs.
|
|
84
|
+
Re-rendered; still eight sheets from eight pages.
|
|
85
|
+
- `docs-canonical/PRIOR-ART.md` records the arid-node measurement instead of
|
|
86
|
+
a promise: every proposal from four sweeps over two of the maintainer's
|
|
87
|
+
repositories was classified by hand against Google's five arid categories,
|
|
88
|
+
and they account for under 1% of what the producers propose. No suppressor
|
|
89
|
+
is built for them. The same pass found what IS material — presentational
|
|
90
|
+
JSX taking 29 of 30 probe slots on a UI diff, and 13.5% of Python proposals
|
|
91
|
+
that cannot compile — and lists both as open, with numbers.
|
|
92
|
+
- `docs-canonical/PRIOR-ART.md` no longer lists the feedback loop and concerns
|
|
93
|
+
as unbuilt. The two gaps that remain are restated precisely: the loop learns
|
|
94
|
+
from verdicts rather than explicit developer clicks, and concerns have no
|
|
95
|
+
model-backed producer.
|
|
96
|
+
|
|
10
97
|
## [0.13.0] - 2026-09-22
|
|
11
98
|
|
|
12
99
|
Automated weekly release — everything merged since `v0.12.0`.
|
package/README.md
CHANGED
|
@@ -127,9 +127,9 @@ raises coverage. TestGuard admits it because it fails when the claim is false.
|
|
|
127
127
|
| npm | `npm i -D testguard-cli` then `npx testguard probe` |
|
|
128
128
|
| pip | `pip install testguard-cli` then `testguard probe` (needs Node ≥ 20) |
|
|
129
129
|
| Homebrew | `brew tap raccioly/tap && brew install testguard` |
|
|
130
|
-
| GitHub Action | `uses: raccioly/testguard@v0.
|
|
130
|
+
| GitHub Action | `uses: raccioly/testguard@v0.14.0` — see [`action.yml`](./action.yml) |
|
|
131
131
|
| pre-commit | `repo: https://github.com/raccioly/testguard`, hooks `testguard-claims`, `testguard-probe` |
|
|
132
|
-
| GitLab CI | `include: - remote: https://raw.githubusercontent.com/raccioly/testguard/v0.
|
|
132
|
+
| GitLab CI | `include: - remote: https://raw.githubusercontent.com/raccioly/testguard/v0.14.0/packaging/gitlab/testguard.gitlab-ci.yml` with `inputs:` — see [`packaging/gitlab/`](./packaging/gitlab/testguard.gitlab-ci.yml) |
|
|
133
133
|
|
|
134
134
|
Projects that set `min-release-age` in `.npmrc` cannot see a version published
|
|
135
135
|
less than that many days ago (`ENOVERSIONS`); install that one with
|
|
@@ -234,6 +234,10 @@ npx testguard-cli admit test/x.test.ts --claim X # is this test green on HEAD
|
|
|
234
234
|
static signal `mocked-never-asserted` — the cheapest blind-spot signal
|
|
235
235
|
there is, and the exact signature of one escaped bug in the field
|
|
236
236
|
reports. Silence it, visibly, with `// unasserted: <why>` above the mock.
|
|
237
|
+
Its sibling `persistence-payload-unasserted` is attached to a **survivor
|
|
238
|
+
on a write path** whose defender mocks the database: the test can prove
|
|
239
|
+
the call shape and never that the row landed, and the signal names the
|
|
240
|
+
mocked layer and the file's assertion mix (exact, partial, argument-free).
|
|
237
241
|
|
|
238
242
|
Runners: **vitest**, **jest**, **Playwright** and **Python** (`--runner
|
|
239
243
|
auto` picks the first of vitest, jest, python that resolves). A JavaScript
|
|
@@ -423,8 +427,9 @@ alarming: something was deliberately broken and not one test failed.
|
|
|
423
427
|
|
|
424
428
|
```
|
|
425
429
|
swept 13 of 14 unclaimed changed files against origin/main.
|
|
426
|
-
proposed 510 faults, probed 10 (cap 10), deferred
|
|
427
|
-
The
|
|
430
|
+
proposed 510 faults, probed 10 (cap 10), deferred 350.
|
|
431
|
+
The 350 deferred are not a verdict: raise --cap, or sweep a smaller change.
|
|
432
|
+
150 presentational elements (icons, static wrappers) set aside: nobody writes a test for those, and a survivor there would teach the ordering the wrong lesson.
|
|
428
433
|
|
|
429
434
|
9 findings — a deliberate break that no test noticed:
|
|
430
435
|
|
|
@@ -658,12 +663,12 @@ detected base that does not resolve is a warning for `status` and `brief`
|
|
|
658
663
|
# GitHub Actions
|
|
659
664
|
- uses: actions/checkout@v4
|
|
660
665
|
with: { fetch-depth: 0 }
|
|
661
|
-
- uses: raccioly/testguard@v0.
|
|
666
|
+
- uses: raccioly/testguard@v0.14.0
|
|
662
667
|
with: { command: gate }
|
|
663
668
|
|
|
664
669
|
# GitLab CI — the component-shaped template: gate + probe, brief as an artifact and, opted in, as a merge-request note
|
|
665
670
|
include:
|
|
666
|
-
- remote: 'https://raw.githubusercontent.com/raccioly/testguard/v0.
|
|
671
|
+
- remote: 'https://raw.githubusercontent.com/raccioly/testguard/v0.14.0/packaging/gitlab/testguard.gitlab-ci.yml'
|
|
667
672
|
inputs: { dir: backend, post_note: true } # post_note needs TESTGUARD_GITLAB_TOKEN (api scope); one note, updated in place
|
|
668
673
|
```
|
|
669
674
|
|
|
@@ -981,7 +986,8 @@ parameter-derived argument swapped for `None`, a key dropped from a payload
|
|
|
981
986
|
`dict` or an allow-list. Two differences are deliberate. A statement is
|
|
982
987
|
removed by replacing it with `pass`, never by deleting the line, because a
|
|
983
988
|
block whose only statement is gone is an `IndentationError`; and a line that
|
|
984
|
-
leaves a bracket open (`COLOURS = {`)
|
|
989
|
+
leaves a bracket open (`COLOURS = {`), or sits inside one opened on an earlier
|
|
990
|
+
line (`help="…"` in an `add_argument(`), is never removed at all. Both exist
|
|
985
991
|
because a fault that cannot compile is a `fault-invalid` verdict — a probe run
|
|
986
992
|
spent saying nothing about the tests. The two JSX shapes (`element-removed`,
|
|
987
993
|
`handler-dropped`) have no Python meaning and are absent rather than faked.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "testguard-cli",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.14.0",
|
|
4
4
|
"description": "Breaks your code on purpose and reports every promise your tests did not notice breaking. Mutation testing bound to the claims a project makes — vitest, jest, Playwright, pytest and unittest.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|
package/spec/GATE-SEMANTICS.md
CHANGED
|
@@ -343,6 +343,14 @@ the suppress-up-to-count semantics are identical.
|
|
|
343
343
|
tool then records `unasserted-annotated` with the reason — silenced, never
|
|
344
344
|
hidden. A file that mocks the subject is never a discovered defender: a
|
|
345
345
|
mock cannot detect a fault in what it replaced.
|
|
346
|
+
`persistence-payload-unasserted`: a resolved defender mocks the persistence
|
|
347
|
+
layer, so it can prove the call shape and never that a row landed; the
|
|
348
|
+
reason names the layer and the file's assertion mix. It is attached only to
|
|
349
|
+
a **survived** record whose fault sits on the write path — a field of a
|
|
350
|
+
persisted payload or the write call itself — because there it says why the
|
|
351
|
+
survivor was missed, and anywhere else it says nothing. A conforming
|
|
352
|
+
document never carries it on another verdict, on a file that is not a
|
|
353
|
+
resolved defender, or without its reason.
|
|
346
354
|
|
|
347
355
|
## Claimed source surface
|
|
348
356
|
|
package/spec/lib/validate.mjs
CHANGED
|
@@ -144,6 +144,16 @@ const semantic = {
|
|
|
144
144
|
if (s.signal === 'target-attribute-patched') {
|
|
145
145
|
if (!s.reason) errors.push({ path: `${p}/defenders/signals`, message: `target-attribute-patched on ${s.file} requires the patched attributes as its reason` });
|
|
146
146
|
if (!r.defenders.resolved.includes(s.file)) errors.push({ path: `${p}/defenders/signals`, message: `${s.file} patches attributes of the subject but is not a resolved defender` });
|
|
147
|
+
} else if (s.signal === 'persistence-payload-unasserted') {
|
|
148
|
+
// The same rule the sweep document enforces on its findings: the
|
|
149
|
+
// signal explains a SURVIVOR on the write path. A record does not
|
|
150
|
+
// carry the fault's line, so the write-path half is the producer's
|
|
151
|
+
// to keep (`persistenceSignalsFor`); the verdict half is checked here.
|
|
152
|
+
// The file mocks the persistence layer, not the subject, so it is a
|
|
153
|
+
// resolved defender and never a mocking one.
|
|
154
|
+
if (!s.reason) errors.push({ path: `${p}/defenders/signals`, message: `persistence-payload-unasserted on ${s.file} requires the mocked layer and assertion mix as its reason` });
|
|
155
|
+
if (r.verdict !== 'survived') errors.push({ path: `${p}/defenders/signals`, message: `persistence-payload-unasserted explains a survivor; on a ${r.verdict} record it explains nothing` });
|
|
156
|
+
if (!r.defenders.resolved.includes(s.file)) errors.push({ path: `${p}/defenders/signals`, message: `${s.file} mocks the persistence layer but is not a resolved defender` });
|
|
147
157
|
} else if (!(r.defenders.mocking ?? []).includes(s.file)) {
|
|
148
158
|
errors.push({ path: `${p}/defenders/signals`, message: `${s.file} carries a mock signal but is not listed in defenders.mocking` });
|
|
149
159
|
}
|
|
@@ -333,7 +343,8 @@ const semantic = {
|
|
|
333
343
|
const s = doc.selection;
|
|
334
344
|
// A cap that hides its remainder is a coverage claim nobody made, so the
|
|
335
345
|
// arithmetic that proves nothing was dropped is checked, not trusted.
|
|
336
|
-
|
|
346
|
+
const setAside = s.presentational ?? 0;
|
|
347
|
+
if (s.proposed !== s.selected + s.deferred + setAside) errors.push({ path: '/selection/proposed', message: `proposed (${s.proposed}) must equal selected (${s.selected}) + deferred (${s.deferred})${setAside ? ` + presentational (${setAside})` : ''}` });
|
|
337
348
|
if (s.selected > s.cap) errors.push({ path: '/selection/selected', message: `selected (${s.selected}) exceeds the cap (${s.cap})` });
|
|
338
349
|
if (doc.scope.swept > doc.scope.targets) errors.push({ path: '/scope/swept', message: `swept (${doc.scope.swept}) cannot exceed targets (${doc.scope.targets})` });
|
|
339
350
|
// The counters that only mean something in one mode must not appear in the
|
|
@@ -547,7 +547,7 @@
|
|
|
547
547
|
}
|
|
548
548
|
},
|
|
549
549
|
"signals": {
|
|
550
|
-
"description": "Static, annotation-grade signals about the defenders. They never change a verdict. `mocked-never-asserted`: the file mocks the subject and never asserts on anything imported from it. `unasserted-annotated`: the same, but the author annotated the mock with a reason (`// unasserted: <why>`), recorded here \u2014 silenced, never hidden. `target-attribute-patched`: a Python defender that patches individual attributes of the subject (`patch(\"pkg.mod.fn\")`) rather than the module. Unlike a JavaScript module mock it does NOT disqualify the file \u2014 a fault anywhere else in the module is still detectable \u2014 so the file stays in `resolved`; `reason` names the patched attributes, which are where that test is least likely to notice anything.",
|
|
550
|
+
"description": "Static, annotation-grade signals about the defenders. They never change a verdict. `mocked-never-asserted`: the file mocks the subject and never asserts on anything imported from it. `unasserted-annotated`: the same, but the author annotated the mock with a reason (`// unasserted: <why>`), recorded here \u2014 silenced, never hidden. `target-attribute-patched`: a Python defender that patches individual attributes of the subject (`patch(\"pkg.mod.fn\")`) rather than the module. Unlike a JavaScript module mock it does NOT disqualify the file \u2014 a fault anywhere else in the module is still detectable \u2014 so the file stays in `resolved`; `reason` names the patched attributes, which are where that test is least likely to notice anything. `persistence-payload-unasserted`: a resolved defender that mocks the persistence layer (`vi.mock('@/lib/prisma')`) and so can prove the call shape but never that a row landed; `reason` names the mocked layer and the file's assertion mix (exact, partial, argument-free), `counts` carries the mix as numbers and `specifiers` the mocked module paths. It belongs only to a `survived` record whose fault sits on the write path \u2014 a field of a persisted payload, or the write call itself \u2014 because there it explains why the survivor was missed and anywhere else it explains nothing. Measured on a real application: 12 of 12 survivors on the save path were a dropped payload field, and every one of their defenders asserted with a partial matcher.",
|
|
551
551
|
"type": "array",
|
|
552
552
|
"items": {
|
|
553
553
|
"type": "object",
|
|
@@ -563,13 +563,46 @@
|
|
|
563
563
|
"enum": [
|
|
564
564
|
"mocked-never-asserted",
|
|
565
565
|
"unasserted-annotated",
|
|
566
|
-
"target-attribute-patched"
|
|
566
|
+
"target-attribute-patched",
|
|
567
|
+
"persistence-payload-unasserted"
|
|
567
568
|
]
|
|
568
569
|
},
|
|
569
570
|
"reason": {
|
|
570
571
|
"type": "string",
|
|
571
572
|
"minLength": 1,
|
|
572
573
|
"maxLength": 512
|
|
574
|
+
},
|
|
575
|
+
"specifiers": {
|
|
576
|
+
"description": "persistence-payload-unasserted only: the persistence-layer module specifiers the file mocks.",
|
|
577
|
+
"type": "array",
|
|
578
|
+
"items": {
|
|
579
|
+
"type": "string",
|
|
580
|
+
"minLength": 1
|
|
581
|
+
}
|
|
582
|
+
},
|
|
583
|
+
"counts": {
|
|
584
|
+
"description": "persistence-payload-unasserted only: the file's call-shape assertions by kind. `exact` is the only kind a dropped field can fail.",
|
|
585
|
+
"type": "object",
|
|
586
|
+
"required": [
|
|
587
|
+
"exact",
|
|
588
|
+
"partial",
|
|
589
|
+
"argless"
|
|
590
|
+
],
|
|
591
|
+
"additionalProperties": false,
|
|
592
|
+
"properties": {
|
|
593
|
+
"exact": {
|
|
594
|
+
"type": "integer",
|
|
595
|
+
"minimum": 0
|
|
596
|
+
},
|
|
597
|
+
"partial": {
|
|
598
|
+
"type": "integer",
|
|
599
|
+
"minimum": 0
|
|
600
|
+
},
|
|
601
|
+
"argless": {
|
|
602
|
+
"type": "integer",
|
|
603
|
+
"minimum": 0
|
|
604
|
+
}
|
|
605
|
+
}
|
|
573
606
|
}
|
|
574
607
|
},
|
|
575
608
|
"additionalProperties": false
|
|
@@ -198,6 +198,11 @@
|
|
|
198
198
|
"type": "integer",
|
|
199
199
|
"minimum": 0,
|
|
200
200
|
"description": "Proposals the concern's fault classes excluded. Reported so a narrow concern cannot look like a quiet one: the producers found these, and this concern is not about them."
|
|
201
|
+
},
|
|
202
|
+
"presentational": {
|
|
203
|
+
"type": "integer",
|
|
204
|
+
"minimum": 0,
|
|
205
|
+
"description": "Proposals set aside because they remove a purely presentational JSX element — an icon, or a static wrapper with no expression and no handler. Nobody writes a test for those, and a survivor on one would teach the survival-learned ordering to prefer the class. Counted in `proposed` and in `byClass`; never probed, never a finding. Absent when zero."
|
|
201
206
|
}
|
|
202
207
|
}
|
|
203
208
|
},
|
package/src/probe/probe.mjs
CHANGED
|
@@ -16,6 +16,7 @@ import { classifyIndependence } from './independence.mjs';
|
|
|
16
16
|
import { escalationStart, foldEscalationRun, escalationResult, flakeRate, killersFromRuns, subjectOf, isReusable } from './attribution.mjs';
|
|
17
17
|
import { hashFile, sha256 } from '../util/hash.mjs';
|
|
18
18
|
import { fingerprint } from '../../spec/lib/fingerprint.mjs';
|
|
19
|
+
import { persistenceSignalsFor } from '../supply/persistence.mjs';
|
|
19
20
|
|
|
20
21
|
|
|
21
22
|
/** Is `child` the same file as `root`, or under it? Both must already be real paths. */
|
|
@@ -485,13 +486,19 @@ async function probeOne({ claim, fault, defenders, discovered, allTests, iso, is
|
|
|
485
486
|
detail.independence = classifyIndependence({ dir: historyDir, ref: historyRef, targetFile: toRepoPath(fault.file), defenders: (killers.length ? killers : defenders).map(toRepoPath) });
|
|
486
487
|
}
|
|
487
488
|
|
|
489
|
+
// Why a survivor on a write path was missed: each defender that mocks the
|
|
490
|
+
// persistence layer, with its assertion mix. Descriptive, never predictive —
|
|
491
|
+
// it is attached to a verdict already reached and selects nothing. The sweep
|
|
492
|
+
// document carried this alone until claimspec v1 admitted it to the evidence.
|
|
493
|
+
const allSignals = [...signals, ...persistenceSignalsFor({ verdict, fault, defenders, projectDir: iso.projectDir })];
|
|
494
|
+
|
|
488
495
|
return {
|
|
489
496
|
fingerprint: fingerprint({ claimId: claim.id, subjectId: fault.id, file: fault.file, verdict }),
|
|
490
497
|
claim: { id: claim.id, statement: claim.statement, severity: claim.severity, source: claim.source, producedBy: claim.producedBy },
|
|
491
498
|
subject,
|
|
492
499
|
verdict,
|
|
493
500
|
detail,
|
|
494
|
-
defenders: { requested: claim.defendedBy ?? [], resolved: defenders, nocover: defenders.length === 0, ...(discovered ? { discovered: true } : {}), ...(byRunner ? { byRunner } : {}), ...(mocking.length ? { mocking } : {}), ...(
|
|
501
|
+
defenders: { requested: claim.defendedBy ?? [], resolved: defenders, nocover: defenders.length === 0, ...(discovered ? { discovered: true } : {}), ...(byRunner ? { byRunner } : {}), ...(mocking.length ? { mocking } : {}), ...(allSignals.length ? { signals: allSignals } : {}) },
|
|
495
502
|
inputs,
|
|
496
503
|
rank: rank({ severity: claim.severity, sourceKind: claim.source.kind, blast, independence: detail.independence?.class }),
|
|
497
504
|
};
|
package/src/probe/pyimports.mjs
CHANGED
|
@@ -49,11 +49,18 @@ export function pyModuleName(projectDir, rel) {
|
|
|
49
49
|
* `from x import (a, b)` collapsed onto one line so the names are visible.
|
|
50
50
|
*/
|
|
51
51
|
function logicalLines(source) {
|
|
52
|
-
|
|
52
|
+
// A trailing comment is stripped from every physical line BEFORE anything
|
|
53
|
+
// is joined or split. An import statement carries no string literal, so a
|
|
54
|
+
// `#` on one always starts a comment — and `from pkg import cli, demo # noqa: E402`
|
|
55
|
+
// is the idiom of every src-layout test that inserts into sys.path before
|
|
56
|
+
// importing. Measured on a real suite: the comment was kept inside the LAST
|
|
57
|
+
// name, which then failed the identifier test, so `demo` was never seen and
|
|
58
|
+
// its tests were reported as no cover. 72 import lines in one suite.
|
|
59
|
+
const raw = source.split('\n').map((l) => l.replace(/#.*$/, ''));
|
|
53
60
|
const out = [];
|
|
54
61
|
for (let i = 0; i < raw.length; i++) {
|
|
55
62
|
let line = raw[i];
|
|
56
|
-
if (
|
|
63
|
+
if (!line.trim()) continue;
|
|
57
64
|
while (/\\\s*$/.test(line) && i + 1 < raw.length) line = line.replace(/\\\s*$/, ' ') + raw[++i];
|
|
58
65
|
if (/^\s*from\s/.test(line) && line.includes('(') && !line.includes(')')) {
|
|
59
66
|
// Bounded: a malformed file must not make this loop to the end of a large source.
|
|
@@ -60,6 +60,54 @@ function isIncomplete(line) {
|
|
|
60
60
|
return depth > 0;
|
|
61
61
|
}
|
|
62
62
|
|
|
63
|
+
/**
|
|
64
|
+
* The bracket depth at the START of every line, for the whole file.
|
|
65
|
+
*
|
|
66
|
+
* `isIncomplete()` answers "does this line leave a bracket open"; this answers
|
|
67
|
+
* the other half, "does this line sit inside one opened above it". Measured on
|
|
68
|
+
* a real Python CLI: 77 of 600 proposals were a `statement-deleted` on a
|
|
69
|
+
* continuation line — `help="...")` inside an `add_argument(` — and none of
|
|
70
|
+
* them compiled. The tokenizer's exact depth agreed on 74 of the 77 and
|
|
71
|
+
* flagged no valid proposal: a dict entry or a call argument is inside a
|
|
72
|
+
* bracket by definition and keeps its own producer.
|
|
73
|
+
*
|
|
74
|
+
* Line-oriented, no AST: triple-quoted strings are tracked across lines so a
|
|
75
|
+
* docstring's parentheses cannot leave the rest of the file "inside" forever,
|
|
76
|
+
* and single-line strings and comments are skipped. Memoised per `lines`
|
|
77
|
+
* array because the scaffold calls the producer once per line.
|
|
78
|
+
*/
|
|
79
|
+
const DEPTHS = new WeakMap();
|
|
80
|
+
export function depthBefore(lines) {
|
|
81
|
+
if (DEPTHS.has(lines)) return DEPTHS.get(lines);
|
|
82
|
+
const out = new Array(lines.length);
|
|
83
|
+
let depth = 0;
|
|
84
|
+
let triple = null;
|
|
85
|
+
for (let i = 0; i < lines.length; i++) {
|
|
86
|
+
out[i] = depth;
|
|
87
|
+
const line = lines[i];
|
|
88
|
+
for (let j = 0; j < line.length;) {
|
|
89
|
+
if (triple) {
|
|
90
|
+
if (line.startsWith(triple, j)) { triple = null; j += 3; } else j += line[j] === '\\' ? 2 : 1;
|
|
91
|
+
continue;
|
|
92
|
+
}
|
|
93
|
+
const ch = line[j];
|
|
94
|
+
if (ch === '#') break;
|
|
95
|
+
if (line.startsWith('"""', j) || line.startsWith("'''", j)) { triple = line.slice(j, j + 3); j += 3; continue; }
|
|
96
|
+
if (ch === '"' || ch === "'") {
|
|
97
|
+
let k = j + 1;
|
|
98
|
+
while (k < line.length && line[k] !== ch) k += line[k] === '\\' ? 2 : 1;
|
|
99
|
+
j = k + 1; // an unterminated string runs to the end of the line
|
|
100
|
+
continue;
|
|
101
|
+
}
|
|
102
|
+
if (ch === '(' || ch === '[' || ch === '{') depth++;
|
|
103
|
+
else if (ch === ')' || ch === ']' || ch === '}') depth = Math.max(0, depth - 1);
|
|
104
|
+
j++;
|
|
105
|
+
}
|
|
106
|
+
}
|
|
107
|
+
DEPTHS.set(lines, out);
|
|
108
|
+
return out;
|
|
109
|
+
}
|
|
110
|
+
|
|
63
111
|
/** The function a `def` line opens, or null. */
|
|
64
112
|
export function functionHead(line) {
|
|
65
113
|
const m = DEF_LINE.exec(line);
|
|
@@ -166,9 +214,13 @@ export function proposalsForLine(lines, i, ctx = {}) {
|
|
|
166
214
|
if (!line.trim() || COMMENT.test(line) || DEFINITION.test(line)) return out;
|
|
167
215
|
const params = ctx.params ?? new Set();
|
|
168
216
|
const indent = ' '.repeat(indentOf(line) ?? 0);
|
|
217
|
+
// A line inside a bracket opened above it is an argument or an entry, never
|
|
218
|
+
// a statement: `pass` there does not parse. The shapes that keep their own
|
|
219
|
+
// producers on such a line (a dropped field, a swapped argument) are below.
|
|
220
|
+
const inside = depthBefore(lines)[i] > 0;
|
|
169
221
|
|
|
170
222
|
const ifm = IF_LINE.exec(line);
|
|
171
|
-
if (ifm && isGuard(ifm[4], lines, i)) {
|
|
223
|
+
if (!inside && ifm && isGuard(ifm[4], lines, i)) {
|
|
172
224
|
if (ifm[4]) {
|
|
173
225
|
// `if not allowed: raise …` — the whole guard is this one line.
|
|
174
226
|
out.push({ faultClass: 'statement-deleted', description: `Guard removed: \`${line.trim()}\` no longer runs.`, replace: `${indent}pass` });
|
|
@@ -183,7 +235,7 @@ export function proposalsForLine(lines, i, ctx = {}) {
|
|
|
183
235
|
out.push({ faultClass: 'return-altered', description: `Check always passes: \`return ${rm[2]}\` becomes \`return True\`.`, replace: `${rm[1]}return True` });
|
|
184
236
|
}
|
|
185
237
|
|
|
186
|
-
const complete = !isIncomplete(line);
|
|
238
|
+
const complete = !inside && !isIncomplete(line);
|
|
187
239
|
if (complete && CHECK_CALL.test(line)) {
|
|
188
240
|
out.push({ faultClass: 'call-removed', description: `Check call removed: \`${line.trim()}\` no longer runs.`, replace: `${indent}pass` });
|
|
189
241
|
} else if (complete && MUTATION.test(line)) {
|
|
@@ -38,6 +38,7 @@
|
|
|
38
38
|
*/
|
|
39
39
|
import { readFileSync } from 'node:fs';
|
|
40
40
|
import { join } from 'node:path';
|
|
41
|
+
import { onWritePath } from './select.mjs';
|
|
41
42
|
|
|
42
43
|
/**
|
|
43
44
|
* Module specifiers that mean "the database". A project-local path counts when
|
|
@@ -158,6 +159,25 @@ export function persistenceSignals(projectDir, defenders) {
|
|
|
158
159
|
return defenders.map((d) => analyzePersistence(projectDir, d)).filter(Boolean);
|
|
159
160
|
}
|
|
160
161
|
|
|
162
|
+
/**
|
|
163
|
+
* The persistence signals ONE EVIDENCE RECORD may carry, or none.
|
|
164
|
+
*
|
|
165
|
+
* The rule the sweep document already enforces, now applied where the record
|
|
166
|
+
* is written: the signal explains a SURVIVOR whose fault sits on the write
|
|
167
|
+
* path, and nothing else. On a kill it explains nothing; off the write path a
|
|
168
|
+
* loose mock assertion is not evidence of anything; on `nocover` or
|
|
169
|
+
* `unverifiable` the question was never asked. Reporting it anywhere else is
|
|
170
|
+
* the non-actionable noise that gets a check switched off.
|
|
171
|
+
*
|
|
172
|
+
* `analyze` is injected so the rule is testable without a filesystem; the
|
|
173
|
+
* probe passes the scratch tree, where the defenders are the project's own.
|
|
174
|
+
*/
|
|
175
|
+
export function persistenceSignalsFor({ verdict, fault, defenders, projectDir }, analyze = persistenceSignals) {
|
|
176
|
+
if (verdict !== 'survived') return [];
|
|
177
|
+
if (!onWritePath(fault)) return [];
|
|
178
|
+
return analyze(projectDir, defenders);
|
|
179
|
+
}
|
|
180
|
+
|
|
161
181
|
/**
|
|
162
182
|
* One line an agent can act on, in the register `hintFor()` uses.
|
|
163
183
|
*
|
package/src/supply/select.mjs
CHANGED
|
@@ -89,6 +89,35 @@ export function onWritePath(fault) {
|
|
|
89
89
|
return PERSISTENCE_CALL.test(line) || PAYLOAD_KEY.test(line) || PERSISTENCE_CALL.test(fault.description ?? '');
|
|
90
90
|
}
|
|
91
91
|
|
|
92
|
+
/**
|
|
93
|
+
* Is this proposal a purely presentational JSX element — the arid class this
|
|
94
|
+
* tool actually has?
|
|
95
|
+
*
|
|
96
|
+
* Google suppresses "arid" nodes nobody would write a test for (logging,
|
|
97
|
+
* timeouts, flags). Measured here on 2026-09-22, those account for under 1% of
|
|
98
|
+
* proposals: the producers are shape-targeted and rarely land on a log line.
|
|
99
|
+
* What IS material is this: on a 14-file UI diff, 150 of 510 proposals were an
|
|
100
|
+
* `element-removed` on an icon (`<X />`) or a static wrapper (`<div>`, `<p>`
|
|
101
|
+
* with no expression and no handler). The ranker spent 29 of 30 slots on that
|
|
102
|
+
* class, 7 of the 10 survivors were of this shape, and because the ordering
|
|
103
|
+
* learns from survival those survivors then teach it to rank the class higher.
|
|
104
|
+
*
|
|
105
|
+
* Line-oriented, like every producer. An icon is a self-closing PascalCase
|
|
106
|
+
* element whose props are only sizing and styling; a static wrapper is a
|
|
107
|
+
* layout or text tag whose own line carries no `{expression}` and no
|
|
108
|
+
* interactive or test-facing attribute. Anything else — a component, an
|
|
109
|
+
* `<img>`, a `<label>`, an element with a handler or dynamic children — is a
|
|
110
|
+
* real proposal and stays.
|
|
111
|
+
*/
|
|
112
|
+
const ICON_ELEMENT = /^\s*<[A-Z]\w*(?:\s+(?:size|className|strokeWidth|aria-hidden|color|width|height|fill)(?:=(?:"[^"]*"|\{[^}]*\}))?)*\s*\/>\s*$/;
|
|
113
|
+
const WRAPPER_ELEMENT = /^\s*<(?:div|span|p|h[1-6]|section|header|footer|main|nav|ul|ol|li|small|strong|em|b|i|hr|br)\b/;
|
|
114
|
+
const INTERACTIVE_ATTR = /\b(?:on[A-Z]\w*=|href=|role=|aria-(?!hidden)\w+=|data-testid=|htmlFor=|tabIndex=)/;
|
|
115
|
+
export function presentational(fault) {
|
|
116
|
+
if (fault.faultClass !== 'element-removed') return false;
|
|
117
|
+
const line = fault.find ?? '';
|
|
118
|
+
return ICON_ELEMENT.test(line) || (WRAPPER_ELEMENT.test(line) && !line.includes('{') && !INTERACTIVE_ATTR.test(line));
|
|
119
|
+
}
|
|
120
|
+
|
|
92
121
|
/** The boost a write-path fault gets. Additive, not multiplicative: it must not be able to overtake a measured 1.0. */
|
|
93
122
|
const WRITE_PATH_BONUS = 0.25;
|
|
94
123
|
|
|
@@ -109,13 +138,17 @@ export const capFor = (fileCount, perFile = 7) => Math.max(perFile, fileCount *
|
|
|
109
138
|
*
|
|
110
139
|
* `candidates` are `{ claim, fault }` pairs — the claim is carried so the
|
|
111
140
|
* caller can rebuild a claims document from the selection without a second
|
|
112
|
-
* lookup. Returns the selection, everything deferred,
|
|
113
|
-
* a report needs to explain itself.
|
|
141
|
+
* lookup. Returns the selection, everything deferred, everything set aside as
|
|
142
|
+
* presentational, and the per-class tally a report needs to explain itself.
|
|
143
|
+
* The three lists partition the candidates: nothing is dropped without a name.
|
|
114
144
|
*/
|
|
115
145
|
export function selectFaults(candidates, { cap, table } = {}) {
|
|
116
146
|
const limit = Number.isInteger(cap) && cap > 0 ? cap : capFor(new Set(candidates.map((c) => c.fault.file)).size);
|
|
117
|
-
const
|
|
118
|
-
|
|
147
|
+
const all = candidates.map((c) => ({ ...c, score: scoreOf(c.fault, { table }), writePath: onWritePath(c.fault) }));
|
|
148
|
+
// Set aside, never silently: a presentational element is not worth a probe
|
|
149
|
+
// run, and a survivor on one would teach the ordering the wrong lesson.
|
|
150
|
+
const presentationalList = all.filter((c) => presentational(c.fault));
|
|
151
|
+
const ranked = all.filter((c) => !presentational(c.fault))
|
|
119
152
|
// Deterministic to the last tie: score, then file, then line, then id.
|
|
120
153
|
// Two runs of the same sweep on the same tree are the same sweep.
|
|
121
154
|
.sort((a, b) =>
|
|
@@ -158,11 +191,11 @@ export function selectFaults(candidates, { cap, table } = {}) {
|
|
|
158
191
|
const taken = new Set(selected);
|
|
159
192
|
const deferred = ranked.filter((r) => !taken.has(r));
|
|
160
193
|
const byClass = {};
|
|
161
|
-
for (const r of
|
|
194
|
+
for (const r of all) {
|
|
162
195
|
const e = (byClass[r.fault.faultClass] ??= { proposed: 0, selected: 0, writePath: 0 });
|
|
163
196
|
e.proposed += 1;
|
|
164
197
|
if (r.writePath) e.writePath += 1;
|
|
165
198
|
}
|
|
166
199
|
for (const r of selected) byClass[r.fault.faultClass].selected += 1;
|
|
167
|
-
return { selected, deferred, cap: limit, byClass };
|
|
200
|
+
return { selected, deferred, presentational: presentationalList, cap: limit, byClass };
|
|
168
201
|
}
|
package/src/sweep/sweep.mjs
CHANGED
|
@@ -38,7 +38,7 @@ import { selectFaults, onWritePath, capFor } from '../supply/select.mjs';
|
|
|
38
38
|
import { learnedProductivity } from '../supply/feedback.mjs';
|
|
39
39
|
import { loadConcerns, concernById, targetsFor, filterByConcern } from '../supply/concerns.mjs';
|
|
40
40
|
import { saveSurface } from '../supply/savepath.mjs';
|
|
41
|
-
import {
|
|
41
|
+
import { persistenceHint, provabilitySummary } from '../supply/persistence.mjs';
|
|
42
42
|
import { probe } from '../probe/probe.mjs';
|
|
43
43
|
import { discoverDefenders } from '../probe/discover.mjs';
|
|
44
44
|
import { hintFor } from '../brief/brief.mjs';
|
|
@@ -109,6 +109,27 @@ export function sortFindings(findings) {
|
|
|
109
109
|
/** How many findings a sweep is willing to fail on. */
|
|
110
110
|
export const gatingCount = (findings) => findings.filter((f) => GATING.has(f.verdict)).length;
|
|
111
111
|
|
|
112
|
+
/**
|
|
113
|
+
* The `selection` block of the document: every proposal accounted for.
|
|
114
|
+
*
|
|
115
|
+
* Pure and exported for the same reason as `findingFrom`: the arithmetic that
|
|
116
|
+
* proves nothing was hidden — proposed = probed + deferred + set aside — is a
|
|
117
|
+
* rule you can state in one line, and the validator that checks it is not the
|
|
118
|
+
* engine that has to satisfy it.
|
|
119
|
+
*/
|
|
120
|
+
export function summarizeSelection(selection, { proposed, outOfScope = 0 }) {
|
|
121
|
+
const presentationalCount = selection.presentational?.length ?? 0;
|
|
122
|
+
return {
|
|
123
|
+
proposed,
|
|
124
|
+
...(outOfScope ? { outOfScope } : {}),
|
|
125
|
+
selected: selection.selected.length,
|
|
126
|
+
deferred: selection.deferred.length,
|
|
127
|
+
...(presentationalCount ? { presentational: presentationalCount } : {}),
|
|
128
|
+
cap: selection.cap,
|
|
129
|
+
byClass: selection.byClass,
|
|
130
|
+
};
|
|
131
|
+
}
|
|
132
|
+
|
|
112
133
|
/** A concern named on the command line that this project does not declare. */
|
|
113
134
|
export class ConcernError extends Error {}
|
|
114
135
|
|
|
@@ -254,7 +275,10 @@ export async function sweep({
|
|
|
254
275
|
// second scan; `line` and `find` live on the draft, not on the evidence.
|
|
255
276
|
const draftOf = new Map(selection.selected.map(({ claim, fault }) => [`${claim.id}/${fault.id}`, fault]));
|
|
256
277
|
|
|
257
|
-
|
|
278
|
+
// The probe attaches the persistence signal to the record itself now
|
|
279
|
+
// (claimspec v1 evidence, `defenders.signals`); the finding repeats what the
|
|
280
|
+
// record says rather than re-deriving it, so the two can never disagree.
|
|
281
|
+
const signalsFor = (r) => (r.defenders?.signals ?? []).filter((s) => s.signal === 'persistence-payload-unasserted');
|
|
258
282
|
const counts = {};
|
|
259
283
|
const findings = [];
|
|
260
284
|
for (const r of evidence.records) {
|
|
@@ -281,14 +305,7 @@ export async function sweep({
|
|
|
281
305
|
swept: drafts.length,
|
|
282
306
|
...(skipped.length ? { skipped } : {}),
|
|
283
307
|
},
|
|
284
|
-
selection: {
|
|
285
|
-
proposed: relevant.length,
|
|
286
|
-
...(candidates.length !== relevant.length ? { outOfScope: candidates.length - relevant.length } : {}),
|
|
287
|
-
selected: selection.selected.length,
|
|
288
|
-
deferred: selection.deferred.length,
|
|
289
|
-
cap: selection.cap,
|
|
290
|
-
byClass: selection.byClass,
|
|
291
|
-
},
|
|
308
|
+
selection: summarizeSelection(selection, { proposed: relevant.length, outOfScope: candidates.length - relevant.length }),
|
|
292
309
|
counts,
|
|
293
310
|
ordering: { observed: learned.observed, sources: learned.sources },
|
|
294
311
|
// Carried, not written here: the caller owns I/O. Persisting it is what
|
|
@@ -342,6 +359,9 @@ export function renderSweep(doc, { limit = 20 } = {}) {
|
|
|
342
359
|
// A narrow concern must not look like a quiet one: the producers found these,
|
|
343
360
|
// and this concern is not about them.
|
|
344
361
|
if (selection.outOfScope) out.push(` ${selection.outOfScope} further proposal${selection.outOfScope === 1 ? ' was' : 's were'} outside this concern's fault classes.`);
|
|
362
|
+
// The arid class this tool has. Set aside and said so: a count nobody prints
|
|
363
|
+
// is a proposal nobody made.
|
|
364
|
+
if (selection.presentational) out.push(` ${selection.presentational} presentational element${selection.presentational === 1 ? '' : 's'} (icons, static wrappers) set aside: nobody writes a test for those, and a survivor there would teach the ordering the wrong lesson.`);
|
|
345
365
|
// An ordering nobody can trace is a number nobody should trust.
|
|
346
366
|
if (doc.ordering) {
|
|
347
367
|
out.push(doc.ordering.observed === 0
|
package/src/util/glob.mjs
CHANGED
|
@@ -11,7 +11,11 @@ import { join } from 'node:path';
|
|
|
11
11
|
// the application — including its writes and its tests — so a surface scan of a
|
|
12
12
|
// real app counted the same code twice and a defender glob could match a
|
|
13
13
|
// compiled chunk. The promise was right and the list was short.
|
|
14
|
-
|
|
14
|
+
// `.claude` holds agent tooling and, under `.claude/worktrees/`, whole nested
|
|
15
|
+
// checkouts of OTHER branches. Measured: discovery on a real repository returned
|
|
16
|
+
// another branch's test file as a defender — a file the scratch tree cannot
|
|
17
|
+
// even contain, because the directory is gitignored.
|
|
18
|
+
const SKIP_DIRS = new Set(['node_modules', '.git', 'dist', 'coverage', '.testguard', '.claude',
|
|
15
19
|
'.next', '.nuxt', '.svelte-kit', '.output', '.turbo', '.parcel-cache', '.angular', '.astro',
|
|
16
20
|
'build', 'out', '.cache', '.vercel', '.netlify',
|
|
17
21
|
'__pycache__', '.venv', 'venv', 'site-packages', '.tox', '.nox',
|