testguard-cli 0.14.0 → 0.14.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +81 -0
- package/README.md +13 -6
- package/package.json +1 -1
- package/spec/GATE-SEMANTICS.md +12 -3
- package/spec/lib/validate.mjs +8 -0
- package/spec/schemas/replay.schema.json +94 -22
- package/src/probe/runners/index.mjs +21 -2
- package/src/replay/replay.mjs +22 -1
- package/src/scaffold/producers.python.mjs +20 -1
- package/src/supply/feedback.mjs +14 -2
package/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,87 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [0.14.2] - 2026-09-28
|
|
11
|
+
|
|
12
|
+
Automated weekly release — everything merged since `v0.14.1`.
|
|
13
|
+
|
|
14
|
+
### Changed
|
|
15
|
+
|
|
16
|
+
- chore(homebrew): sha256 for v0.14.1 (#171)
|
|
17
|
+
|
|
18
|
+
|
|
19
|
+
## [0.14.1] - 2026-09-27
|
|
20
|
+
|
|
21
|
+
Automated weekly release — everything merged since `v0.14.0`.
|
|
22
|
+
|
|
23
|
+
### Changed
|
|
24
|
+
|
|
25
|
+
- fix: four defects that made `nocover` report discovery failures as findings (#169)
|
|
26
|
+
- chore: untrack local agent tooling and a generated fixture artifact (#168)
|
|
27
|
+
- chore(homebrew): sha256 for v0.14.0 (#167)
|
|
28
|
+
- docs(brief): page 6 carries what 0.14.0 changed there (#166)
|
|
29
|
+
|
|
30
|
+
|
|
31
|
+
### Fixed
|
|
32
|
+
|
|
33
|
+
- **`sweep` no longer refuses to write after a sweep that learned nothing.**
|
|
34
|
+
`learnedProductivity` named every evidence document it *read* as an ordering
|
|
35
|
+
source, while `observed` counts only `killed` and `survived` records. A
|
|
36
|
+
document whose records are all `nocover` — the ordinary shape of a first
|
|
37
|
+
sweep — therefore produced `sources: [...]` with `observed: 0`, which the
|
|
38
|
+
sweep validator correctly refuses. Because that document is written to
|
|
39
|
+
`.testguard/sweep-evidence.json`, **every later sweep failed the same way,
|
|
40
|
+
permanently**, until someone deleted a gitignored file nothing mentioned. An
|
|
41
|
+
ordering now names only the documents that contributed an observation, which
|
|
42
|
+
is what "the documents it was learned from" always meant. Measured on an
|
|
43
|
+
external corpus: this fired on 10 of 34 sweep invocations.
|
|
44
|
+
- **`replay` resolves Python importers with the Python resolver.**
|
|
45
|
+
`replay` matched test files against a reverted source file with the
|
|
46
|
+
JavaScript importer for every language. `from pkg.mod import x` carries no
|
|
47
|
+
quoted specifier, so nothing ever matched, the defender set was always empty
|
|
48
|
+
for Python, and **every Python verdict was `nocover`** — "no test imports the
|
|
49
|
+
reverted source" — however complete the suite. `discoverDefendersDetailed`
|
|
50
|
+
has always branched on the target's language; `replay` now does the same.
|
|
51
|
+
- **A runner that matches no test file is no longer selected silently.**
|
|
52
|
+
`--runner auto` tries vitest, jest, then python. A TypeScript project on
|
|
53
|
+
Playwright resolves neither JavaScript runner, so `auto` reached python
|
|
54
|
+
wherever a `python3` existed, globbed for `.py`, found none, and reported
|
|
55
|
+
every fault as `nocover`. `auto` now prefers a resolvable runner that can see
|
|
56
|
+
test files, and when none can it says so (`no runner matched a test file …
|
|
57
|
+
every verdict will be nocover`) instead of letting discovery's failure read
|
|
58
|
+
as a finding about the project. An explicitly named `--runner` is still
|
|
59
|
+
honoured; every selection now reports `testFiles`.
|
|
60
|
+
|
|
61
|
+
- **`replay` no longer reports `nocover` when it removed the only test file.**
|
|
62
|
+
Removing the fix's own test removes the whole FILE, and on a project with
|
|
63
|
+
few, large test files that also removes tests which pre-dated the fix and
|
|
64
|
+
might have caught the bug. Nothing is left to run, so nothing can be
|
|
65
|
+
concluded: the verdict is now `unverifiable` with reason
|
|
66
|
+
`the-fix-shipped-the-only-test-file`. `nocover` is a statement about the
|
|
67
|
+
PROJECT — "no test exercises this" — and it enters the calibration as a
|
|
68
|
+
miss, so the old behaviour charged a project for evidence the measurement
|
|
69
|
+
itself destroyed. The validator now refuses the mislabelled form, and
|
|
70
|
+
`GATE-SEMANTICS.md` carries it as a third rule beside de-duplication and
|
|
71
|
+
test removal.
|
|
72
|
+
|
|
73
|
+
### Changed
|
|
74
|
+
|
|
75
|
+
- **Python: a module-level dunder assignment is no longer proposed as a
|
|
76
|
+
fault.** `__version__`, `__all__` and `__author__` at module level are
|
|
77
|
+
metadata nobody writes a test for, so a survivor on one teaches the
|
|
78
|
+
survival-learned ordering to prefer a barren class — the same reasoning that
|
|
79
|
+
already sets a presentational JSX element aside. Narrow on purpose: a
|
|
80
|
+
module-level *constant* such as `DEFAULT_MAX_SIZE = 100` is still proposed,
|
|
81
|
+
because removing a real default changes a real default.
|
|
82
|
+
|
|
83
|
+
### Changed
|
|
84
|
+
|
|
85
|
+
- The brief's page 6 says what 0.14.0 changed there: the evidence now names
|
|
86
|
+
the mocked layer and the assertion shape beside a save-path survivor's
|
|
87
|
+
verdict, and the fourth rule of the lower rungs — whatever a sweep does not
|
|
88
|
+
probe, it counts, presentational elements included. Re-rendered; still eight
|
|
89
|
+
sheets from eight pages.
|
|
90
|
+
|
|
10
91
|
## [0.14.0] - 2026-09-22
|
|
11
92
|
|
|
12
93
|
Automated weekly release — everything merged since `v0.13.0`.
|
package/README.md
CHANGED
|
@@ -127,9 +127,9 @@ raises coverage. TestGuard admits it because it fails when the claim is false.
|
|
|
127
127
|
| npm | `npm i -D testguard-cli` then `npx testguard probe` |
|
|
128
128
|
| pip | `pip install testguard-cli` then `testguard probe` (needs Node ≥ 20) |
|
|
129
129
|
| Homebrew | `brew tap raccioly/tap && brew install testguard` |
|
|
130
|
-
| GitHub Action | `uses: raccioly/testguard@v0.14.
|
|
130
|
+
| GitHub Action | `uses: raccioly/testguard@v0.14.2` — see [`action.yml`](./action.yml) |
|
|
131
131
|
| pre-commit | `repo: https://github.com/raccioly/testguard`, hooks `testguard-claims`, `testguard-probe` |
|
|
132
|
-
| GitLab CI | `include: - remote: https://raw.githubusercontent.com/raccioly/testguard/v0.14.
|
|
132
|
+
| GitLab CI | `include: - remote: https://raw.githubusercontent.com/raccioly/testguard/v0.14.2/packaging/gitlab/testguard.gitlab-ci.yml` with `inputs:` — see [`packaging/gitlab/`](./packaging/gitlab/testguard.gitlab-ci.yml) |
|
|
133
133
|
|
|
134
134
|
Projects that set `min-release-age` in `.npmrc` cannot see a version published
|
|
135
135
|
less than that many days ago (`ENOVERSIONS`); install that one with
|
|
@@ -663,12 +663,12 @@ detected base that does not resolve is a warning for `status` and `brief`
|
|
|
663
663
|
# GitHub Actions
|
|
664
664
|
- uses: actions/checkout@v4
|
|
665
665
|
with: { fetch-depth: 0 }
|
|
666
|
-
- uses: raccioly/testguard@v0.14.
|
|
666
|
+
- uses: raccioly/testguard@v0.14.2
|
|
667
667
|
with: { command: gate }
|
|
668
668
|
|
|
669
669
|
# GitLab CI — the component-shaped template: gate + probe, brief as an artifact and, opted in, as a merge-request note
|
|
670
670
|
include:
|
|
671
|
-
- remote: 'https://raw.githubusercontent.com/raccioly/testguard/v0.14.
|
|
671
|
+
- remote: 'https://raw.githubusercontent.com/raccioly/testguard/v0.14.2/packaging/gitlab/testguard.gitlab-ci.yml'
|
|
672
672
|
inputs: { dir: backend, post_note: true } # post_note needs TESTGUARD_GITLAB_TOKEN (api scope); one note, updated in place
|
|
673
673
|
```
|
|
674
674
|
|
|
@@ -983,13 +983,20 @@ The table above is JavaScript. `scaffold` reads Python too, and proposes the
|
|
|
983
983
|
same fault classes in Python syntax: `if <guard>:` → `if False:`,
|
|
984
984
|
`return <check>` → `return True`, `verify=True` → `verify=False`, a
|
|
985
985
|
parameter-derived argument swapped for `None`, a key dropped from a payload
|
|
986
|
-
`dict` or an allow-list.
|
|
986
|
+
`dict` or an allow-list. Three differences are deliberate. A statement is
|
|
987
987
|
removed by replacing it with `pass`, never by deleting the line, because a
|
|
988
988
|
block whose only statement is gone is an `IndentationError`; and a line that
|
|
989
989
|
leaves a bracket open (`COLOURS = {`), or sits inside one opened on an earlier
|
|
990
990
|
line (`help="…"` in an `add_argument(`), is never removed at all. Both exist
|
|
991
991
|
because a fault that cannot compile is a `fault-invalid` verdict — a probe run
|
|
992
|
-
spent saying nothing about the tests.
|
|
992
|
+
spent saying nothing about the tests. Third, a **module-level dunder
|
|
993
|
+
assignment** (`__version__`, `__all__`, `__author__`) is not proposed. Deleting
|
|
994
|
+
one does change behaviour, so it is not an equivalent mutant; it is something
|
|
995
|
+
nobody writes a test for, which is the disqualification an icon already gets on
|
|
996
|
+
the JSX side — a survivor there teaches the survival-learned ordering to prefer
|
|
997
|
+
a barren class. The rule is narrow on purpose: a module-level *constant*
|
|
998
|
+
(`DEFAULT_MAX_SIZE = 100`) is still proposed, because removing a real default
|
|
999
|
+
changes a real default. The two JSX shapes (`element-removed`,
|
|
993
1000
|
`handler-dropped`) have no Python meaning and are absent rather than faked.
|
|
994
1001
|
|
|
995
1002
|
Every proposal's `find` is the exact line with `expectHits`/`occurrence`
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "testguard-cli",
|
|
3
|
-
"version": "0.14.
|
|
3
|
+
"version": "0.14.2",
|
|
4
4
|
"description": "Breaks your code on purpose and reports every promise your tests did not notice breaking. Mutation testing bound to the claims a project makes — vitest, jest, Playwright, pytest and unittest.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|
package/spec/GATE-SEMANTICS.md
CHANGED
|
@@ -197,8 +197,8 @@ Its verdicts mirror the probe's, for the same reasons:
|
|
|
197
197
|
|---|---|
|
|
198
198
|
| `caught` | a remaining test failed **by assertion** on the reverted source, every run. The suite knew. |
|
|
199
199
|
| `blind` | the suite stayed green on known-broken code. |
|
|
200
|
-
| `nocover` | no test imports the reverted files
|
|
201
|
-
| `unverifiable` | the revert did not apply, the suite failed to load,
|
|
200
|
+
| `nocover` | no test imports the reverted files, **and at least one test file remained to ask**. Worse than `blind`: nothing was even tried. |
|
|
201
|
+
| `unverifiable` | the revert did not apply, the suite failed to load, it timed out, or removing the fix's own test left **no test file at all** (`the-fix-shipped-the-only-test-file`). Carries a `reason`. |
|
|
202
202
|
| `flaky` | the runs disagreed. A flaky failure reads as "the suite caught it", so flakiness biases this metric **optimistically** — mixed runs are never `caught`. |
|
|
203
203
|
|
|
204
204
|
Two rules that follow:
|
|
@@ -208,6 +208,13 @@ Two rules that follow:
|
|
|
208
208
|
twice corrupts the corpus a calibration is computed from.
|
|
209
209
|
2. **The fix's own test is removed before the run.** It proves nothing about
|
|
210
210
|
what the suite knew before the fix existed.
|
|
211
|
+
3. **If that leaves no test file, the run is `unverifiable`, not `nocover`.**
|
|
212
|
+
Removal takes the whole FILE, so on a project with few, large test files it
|
|
213
|
+
also removes tests that pre-dated the fix and might have caught the bug. The
|
|
214
|
+
suite the measurement needed no longer exists, so nothing can be concluded.
|
|
215
|
+
`nocover` is a statement about the PROJECT — "no test exercises this" — and
|
|
216
|
+
using it here would charge the project, in the calibration, for evidence the
|
|
217
|
+
method itself destroyed.
|
|
211
218
|
|
|
212
219
|
`caught`, `blind` and `nocover` are measurements and enter the calibration
|
|
213
220
|
ratio: `caught` as a hit, the other two as misses. `nocover` counts as a miss
|
|
@@ -216,7 +223,9 @@ because it is one — the worst kind. Leaving it out would let a project with
|
|
|
216
223
|
since its worst outcomes would leave the denominator before the ratio is
|
|
217
224
|
taken; Stryker and PIT count no-coverage mutants against the headline score
|
|
218
225
|
for the same reason. `flaky` and `unverifiable` are failed measurements and
|
|
219
|
-
enter neither side
|
|
226
|
+
enter neither side — which is why rule 3 above matters: a run with no test
|
|
227
|
+
file left is a failed measurement wearing `nocover`'s clothes, and left
|
|
228
|
+
unclassified it would inflate every miss rate computed from the corpus.
|
|
220
229
|
|
|
221
230
|
### Calibration arithmetic is reproducible
|
|
222
231
|
|
package/spec/lib/validate.mjs
CHANGED
|
@@ -297,6 +297,14 @@ const semantic = {
|
|
|
297
297
|
}
|
|
298
298
|
if (r.verdict === 'nocover' && r.ranTests) errors.push({ path: `${p}/ranTests`, message: 'nocover means no test exercises the reverted files; it cannot have run tests' });
|
|
299
299
|
if (r.verdict === 'unverifiable' && !r.reason) errors.push({ path: `${p}/reason`, message: 'unverifiable requires a reason (e.g. revert-did-not-apply, suite-failed-to-load)' });
|
|
300
|
+
// A run that had no test file left is a FAILED MEASUREMENT, never a
|
|
301
|
+
// finding about the project: removing the fix's test removed the file it
|
|
302
|
+
// lived in, and with it any pre-existing test that might have caught the
|
|
303
|
+
// bug. Recording that as `nocover` would put it in the calibration as a
|
|
304
|
+
// miss and charge the project for evidence the method destroyed.
|
|
305
|
+
if (r.reason === 'the-fix-shipped-the-only-test-file' && r.verdict !== 'unverifiable') {
|
|
306
|
+
errors.push({ path: `${p}/verdict`, message: 'no test file remained after removing the fix\'s own, so nothing could be concluded: that is unverifiable, not a finding about the project' });
|
|
307
|
+
}
|
|
300
308
|
});
|
|
301
309
|
return errors;
|
|
302
310
|
},
|
|
@@ -1,29 +1,70 @@
|
|
|
1
1
|
{
|
|
2
2
|
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
3
3
|
"$id": "urn:claimspec:v1:replay",
|
|
4
|
-
"title": "claimspec v1
|
|
5
|
-
"description": "Would this suite have caught the bugs that actually escaped? For each fix commit in a range: revert only the source to its parent, remove the test the fix shipped, run what remains. `blind` means the suite was blind to a bug that really shipped. Unlike an injected fault, a replayed bug is ground truth
|
|
4
|
+
"title": "claimspec v1 \u2014 bug replay",
|
|
5
|
+
"description": "Would this suite have caught the bugs that actually escaped? For each fix commit in a range: revert only the source to its parent, remove the test the fix shipped, run what remains. `blind` means the suite was blind to a bug that really shipped. Unlike an injected fault, a replayed bug is ground truth \u2014 a human already confirmed it was a defect \u2014 which is what makes this the oracle a fault model is calibrated against.",
|
|
6
6
|
"type": "object",
|
|
7
|
-
"required": [
|
|
7
|
+
"required": [
|
|
8
|
+
"schemaVersion",
|
|
9
|
+
"tool",
|
|
10
|
+
"run",
|
|
11
|
+
"records"
|
|
12
|
+
],
|
|
8
13
|
"properties": {
|
|
9
|
-
"$schema": {
|
|
10
|
-
|
|
11
|
-
|
|
14
|
+
"$schema": {
|
|
15
|
+
"type": "string"
|
|
16
|
+
},
|
|
17
|
+
"schemaVersion": {
|
|
18
|
+
"const": 1
|
|
19
|
+
},
|
|
20
|
+
"tool": {
|
|
21
|
+
"$ref": "urn:claimspec:v1:common#/$defs/tool"
|
|
22
|
+
},
|
|
12
23
|
"run": {
|
|
13
24
|
"type": "object",
|
|
14
|
-
"required": [
|
|
25
|
+
"required": [
|
|
26
|
+
"id",
|
|
27
|
+
"startedAt",
|
|
28
|
+
"range",
|
|
29
|
+
"confirmRuns"
|
|
30
|
+
],
|
|
15
31
|
"properties": {
|
|
16
|
-
"id": {
|
|
17
|
-
|
|
18
|
-
|
|
32
|
+
"id": {
|
|
33
|
+
"type": "string",
|
|
34
|
+
"minLength": 1,
|
|
35
|
+
"maxLength": 128
|
|
36
|
+
},
|
|
37
|
+
"startedAt": {
|
|
38
|
+
"$ref": "urn:claimspec:v1:common#/$defs/isoDateTime"
|
|
39
|
+
},
|
|
40
|
+
"finishedAt": {
|
|
41
|
+
"$ref": "urn:claimspec:v1:common#/$defs/isoDateTime"
|
|
42
|
+
},
|
|
19
43
|
"range": {
|
|
20
44
|
"description": "The commit range searched for fix commits, as given.",
|
|
21
45
|
"type": "string",
|
|
22
46
|
"minLength": 1,
|
|
23
47
|
"maxLength": 256
|
|
24
48
|
},
|
|
25
|
-
"runner": {
|
|
26
|
-
|
|
49
|
+
"runner": {
|
|
50
|
+
"type": "object",
|
|
51
|
+
"required": [
|
|
52
|
+
"name"
|
|
53
|
+
],
|
|
54
|
+
"properties": {
|
|
55
|
+
"name": {
|
|
56
|
+
"type": "string"
|
|
57
|
+
},
|
|
58
|
+
"version": {
|
|
59
|
+
"type": "string"
|
|
60
|
+
}
|
|
61
|
+
},
|
|
62
|
+
"additionalProperties": false
|
|
63
|
+
},
|
|
64
|
+
"confirmRuns": {
|
|
65
|
+
"type": "integer",
|
|
66
|
+
"minimum": 1
|
|
67
|
+
},
|
|
27
68
|
"candidates": {
|
|
28
69
|
"description": "Fix commits found in the range before de-duplication by patch-id.",
|
|
29
70
|
"type": "integer",
|
|
@@ -41,32 +82,61 @@
|
|
|
41
82
|
"type": "array",
|
|
42
83
|
"items": {
|
|
43
84
|
"type": "object",
|
|
44
|
-
"required": [
|
|
85
|
+
"required": [
|
|
86
|
+
"commit",
|
|
87
|
+
"patchId",
|
|
88
|
+
"verdict",
|
|
89
|
+
"source",
|
|
90
|
+
"removedTests",
|
|
91
|
+
"runs"
|
|
92
|
+
],
|
|
45
93
|
"properties": {
|
|
46
|
-
"commit": {
|
|
94
|
+
"commit": {
|
|
95
|
+
"$ref": "urn:claimspec:v1:common#/$defs/gitSha"
|
|
96
|
+
},
|
|
47
97
|
"patchId": {
|
|
48
98
|
"description": "`git patch-id --stable` of the fix: the identity of the CHANGE rather than of the commit. A dual-branch topology carries the same fix under several shas, and counting it twice corrupts the corpus. Hex, 40 characters for a SHA-1 repository and 64 for a SHA-256 one; the commit sha when git could not compute one.",
|
|
49
99
|
"type": "string",
|
|
50
100
|
"pattern": "^[a-f0-9]{40}(?:[a-f0-9]{24})?$"
|
|
51
101
|
},
|
|
52
|
-
"subject": {
|
|
102
|
+
"subject": {
|
|
103
|
+
"description": "The fix commit's subject line, for a human reading the corpus.",
|
|
104
|
+
"type": "string",
|
|
105
|
+
"maxLength": 512
|
|
106
|
+
},
|
|
53
107
|
"verdict": {
|
|
54
|
-
"description": "`caught` = a remaining test failed by assertion on the reverted source, every run. `blind` = the suite stayed green on known-broken code. `nocover` = no test exercises the reverted files at all, which is worse than blind
|
|
55
|
-
"enum": [
|
|
108
|
+
"description": "`caught` = a remaining test failed by assertion on the reverted source, every run. `blind` = the suite stayed green on known-broken code. `nocover` = no test exercises the reverted files at all, which is worse than blind; it requires that at least one test file REMAINED after the fix's own were removed, otherwise nothing could be concluded and the verdict is `unverifiable` with reason `the-fix-shipped-the-only-test-file`. `unverifiable` = nothing could be concluded: the revert did not apply, the suite could not load, it timed out, or the commit only ADDS source (`no-prior-version` \u2014 there is no earlier behaviour to have been blind to, so it is not a bug the suite could have caught and it never enters a calibration). `flaky` = the runs disagreed, so nothing can be concluded.",
|
|
109
|
+
"enum": [
|
|
110
|
+
"caught",
|
|
111
|
+
"blind",
|
|
112
|
+
"nocover",
|
|
113
|
+
"unverifiable",
|
|
114
|
+
"flaky"
|
|
115
|
+
]
|
|
116
|
+
},
|
|
117
|
+
"reason": {
|
|
118
|
+
"type": "string",
|
|
119
|
+
"maxLength": 256
|
|
56
120
|
},
|
|
57
|
-
"reason": { "type": "string", "maxLength": 256 },
|
|
58
121
|
"source": {
|
|
59
122
|
"description": "Source files reverted to the fix's parent.",
|
|
60
123
|
"type": "array",
|
|
61
124
|
"minItems": 1,
|
|
62
|
-
"items": {
|
|
125
|
+
"items": {
|
|
126
|
+
"$ref": "urn:claimspec:v1:common#/$defs/repoPath"
|
|
127
|
+
}
|
|
63
128
|
},
|
|
64
129
|
"removedTests": {
|
|
65
130
|
"description": "Test files the fix shipped, removed before the run: the fix's own test proves nothing about what the suite knew before it.",
|
|
66
131
|
"type": "array",
|
|
67
|
-
"items": {
|
|
132
|
+
"items": {
|
|
133
|
+
"$ref": "urn:claimspec:v1:common#/$defs/repoPath"
|
|
134
|
+
}
|
|
135
|
+
},
|
|
136
|
+
"ranTests": {
|
|
137
|
+
"type": "integer",
|
|
138
|
+
"minimum": 0
|
|
68
139
|
},
|
|
69
|
-
"ranTests": { "type": "integer", "minimum": 0 },
|
|
70
140
|
"faultClass": {
|
|
71
141
|
"description": "The injected-fault class this real diff most resembles, derived mechanically from the diff shape. This is the join key between ground truth and a fault model: it is what lets P(this fault class is a real bug) be computed.",
|
|
72
142
|
"$ref": "urn:claimspec:v1:common#/$defs/faultClass"
|
|
@@ -74,7 +144,9 @@
|
|
|
74
144
|
"runs": {
|
|
75
145
|
"type": "array",
|
|
76
146
|
"minItems": 1,
|
|
77
|
-
"items": {
|
|
147
|
+
"items": {
|
|
148
|
+
"$ref": "urn:claimspec:v1:evidence#/$defs/testRun"
|
|
149
|
+
}
|
|
78
150
|
}
|
|
79
151
|
},
|
|
80
152
|
"additionalProperties": false
|
|
@@ -39,11 +39,30 @@ export async function selectRunner({ projectDir, sourceDir, python, name = 'auto
|
|
|
39
39
|
const candidates = name === 'auto' ? [vitest, jest, pythonRunner] : [RUNNERS[name]];
|
|
40
40
|
if (!candidates[0]) return { error: `unknown runner "${name}"; use ${RUNNER_NAMES.join(', ')} or auto` };
|
|
41
41
|
const messages = [];
|
|
42
|
+
// A runner that resolves but whose globs match ZERO test files is not an
|
|
43
|
+
// answer. A pure TypeScript project using Playwright resolves neither vitest
|
|
44
|
+
// nor jest, and `auto` then reaches python — which resolves wherever a
|
|
45
|
+
// python3 exists — globs for `.py`, finds none, and reports every fault as
|
|
46
|
+
// `nocover`: a damning statement about the project produced by a runner that
|
|
47
|
+
// could not have seen its tests.
|
|
48
|
+
//
|
|
49
|
+
// So `auto` prefers a candidate that can actually see tests. An explicitly
|
|
50
|
+
// named runner is still honoured, because the caller asked for it; it is
|
|
51
|
+
// reported with `testFiles` so a caller can say what it is working with.
|
|
52
|
+
let blind = null;
|
|
42
53
|
for (const r of candidates) {
|
|
43
54
|
const c = await r.check({ projectDir, sourceDir, python });
|
|
44
|
-
if (c.ok)
|
|
45
|
-
|
|
55
|
+
if (!c.ok) { messages.push(`${r.name}: ${c.message}`); continue; }
|
|
56
|
+
const selected = { runner: r, version: c.version, source: c.source, engine: c.engine };
|
|
57
|
+
let testFiles = 0;
|
|
58
|
+
try { testFiles = r.tests(projectDir).length; } catch { testFiles = 0; }
|
|
59
|
+
if (testFiles > 0 || name !== 'auto') return { ...selected, testFiles };
|
|
60
|
+
blind ??= { ...selected, testFiles: 0 };
|
|
61
|
+
messages.push(`${r.name}: resolved but matched no test files`);
|
|
46
62
|
}
|
|
63
|
+
// Nothing saw a test file. Return the first that resolved, and say plainly
|
|
64
|
+
// what that means, rather than letting every verdict read as a finding.
|
|
65
|
+
if (blind) return { ...blind, warning: `no runner matched a test file in this project; using ${blind.runner.name}, which sees none — every verdict will be nocover` };
|
|
47
66
|
return { error: messages.join('; ') };
|
|
48
67
|
}
|
|
49
68
|
|
package/src/replay/replay.mjs
CHANGED
|
@@ -7,6 +7,7 @@ import { selectRunner, RUNNERS } from '../probe/runners/index.mjs';
|
|
|
7
7
|
import { parseCommandTemplate } from '../probe/runners/shared.mjs';
|
|
8
8
|
import { fileImports } from '../probe/rank.mjs';
|
|
9
9
|
import { IS_PY_TEST, pyFileImports } from '../probe/pyimports.mjs';
|
|
10
|
+
import { isPython } from '../probe/discover.mjs';
|
|
10
11
|
import { labelDiff } from './label.mjs';
|
|
11
12
|
import { wilsonInterval, roundTo } from '../../spec/lib/wilson.mjs';
|
|
12
13
|
|
|
@@ -211,7 +212,27 @@ async function replayOne({ fix, iso, root, projectDir, confirmRuns, budgetMs, co
|
|
|
211
212
|
// reverted source file. `nocover` means nothing does — worse than blind,
|
|
212
213
|
// and a coverage report shows it as a red line you can ignore.
|
|
213
214
|
const all = runner.tests(iso.projectDir).filter((t) => !fix.tests.some((ft) => rel(ft) === t));
|
|
214
|
-
|
|
215
|
+
// Removing the fix's test removes the whole FILE, which on a project with
|
|
216
|
+
// few, large test files takes pre-existing tests with it — tests that did
|
|
217
|
+
// exist before the fix and might have caught the bug. When nothing is left,
|
|
218
|
+
// the suite this measurement needed no longer exists, so nothing can be
|
|
219
|
+
// concluded. That is `unverifiable`, not `nocover`: `nocover` is a finding
|
|
220
|
+
// about the project ("no test exercises this"), and counting it as a miss
|
|
221
|
+
// would charge the project for evidence the method itself destroyed.
|
|
222
|
+
if (all.length === 0) {
|
|
223
|
+
return { ...base, faultClass, verdict: 'unverifiable', reason: 'the-fix-shipped-the-only-test-file', ranTests: 0, runs: [{ outcome: 'error', durationMs: 0 }] };
|
|
224
|
+
}
|
|
225
|
+
// A target's language decides how its importers are found, exactly as
|
|
226
|
+
// `discoverDefendersDetailed` decides it. Resolving a `.py` target with the
|
|
227
|
+
// JavaScript resolver matches nothing — `IMPORT_RE` wants a quoted
|
|
228
|
+
// specifier and `from pkg.mod import x` has none — so `related` was always
|
|
229
|
+
// empty for Python and EVERY verdict was `nocover`, however good the suite.
|
|
230
|
+
const related = all.filter((t) => fix.source.some((sf) => {
|
|
231
|
+
const target = rel(sf);
|
|
232
|
+
return isPython(target)
|
|
233
|
+
? pyFileImports(iso.projectDir, join(iso.projectDir, t), target, t)
|
|
234
|
+
: fileImports(iso.projectDir, join(iso.projectDir, t), target);
|
|
235
|
+
}));
|
|
215
236
|
if (related.length === 0) {
|
|
216
237
|
return { ...base, faultClass, verdict: 'nocover', reason: 'no-test-imports-the-reverted-source', ranTests: 0, runs: [{ outcome: 'error', durationMs: 0 }] };
|
|
217
238
|
}
|
|
@@ -208,6 +208,25 @@ function argumentSwapProposals(lines, i, params) {
|
|
|
208
208
|
* parameter names; `ctx.fieldDrops` disables payload shapes in files where a
|
|
209
209
|
* dict literal is data rather than something the product sends.
|
|
210
210
|
*/
|
|
211
|
+
/**
|
|
212
|
+
* A module-level dunder assignment — `__version__`, `__all__`, `__author__`.
|
|
213
|
+
*
|
|
214
|
+
* Deleting one changes behaviour (an attribute stops existing, a star-import
|
|
215
|
+
* exports differently), so it is not an equivalent mutant. It is something
|
|
216
|
+
* nobody writes a test for, which is a different disqualification and the same
|
|
217
|
+
* one the JSX side already applies to an icon: a survivor there teaches the
|
|
218
|
+
* survival-learned ordering to prefer the class, and the class is barren.
|
|
219
|
+
*
|
|
220
|
+
* Measured on this corpus: every false positive in the adjudicated sample was
|
|
221
|
+
* a module-level metadata assignment, and `statement-deleted` already took 55%
|
|
222
|
+
* of a 3% sample of proposals, so the class does not need the help.
|
|
223
|
+
*
|
|
224
|
+
* Deliberately NARROW. Only a dunder, and only at module level. A module-level
|
|
225
|
+
* CONSTANT (`DEFAULT_MAX_SIZE = 100`) is excluded from this rule on purpose:
|
|
226
|
+
* removing one changes a real default and a test may well be owed.
|
|
227
|
+
*/
|
|
228
|
+
const MODULE_DUNDER = /^__\w+__\s*(?::[^=]+)?=[^=]/;
|
|
229
|
+
|
|
211
230
|
export function proposalsForLine(lines, i, ctx = {}) {
|
|
212
231
|
const out = [];
|
|
213
232
|
const line = lines[i];
|
|
@@ -238,7 +257,7 @@ export function proposalsForLine(lines, i, ctx = {}) {
|
|
|
238
257
|
const complete = !inside && !isIncomplete(line);
|
|
239
258
|
if (complete && CHECK_CALL.test(line)) {
|
|
240
259
|
out.push({ faultClass: 'call-removed', description: `Check call removed: \`${line.trim()}\` no longer runs.`, replace: `${indent}pass` });
|
|
241
|
-
} else if (complete && MUTATION.test(line)) {
|
|
260
|
+
} else if (complete && MUTATION.test(line) && !(indent === '' && MODULE_DUNDER.test(line))) {
|
|
242
261
|
out.push({ faultClass: 'statement-deleted', description: `State change removed: \`${line.trim()}\` no longer runs.`, replace: `${indent}pass` });
|
|
243
262
|
}
|
|
244
263
|
|
package/src/supply/feedback.mjs
CHANGED
|
@@ -129,11 +129,23 @@ export function readEvidence(projectDir, paths = EVIDENCE_PATHS) {
|
|
|
129
129
|
*/
|
|
130
130
|
export function learnedProductivity(projectDir, { paths, shipped } = {}) {
|
|
131
131
|
const docs = readEvidence(projectDir, paths);
|
|
132
|
-
const
|
|
132
|
+
const tallied = docs.map((d) => ({ path: d.path, tally: tallyEvidence(d.doc) }));
|
|
133
|
+
// A document is a SOURCE only if it contributed an observation. One that was
|
|
134
|
+
// read but yielded nothing countable is not something an ordering was
|
|
135
|
+
// "learned from", and naming it anyway is what produced a sweep document the
|
|
136
|
+
// validator correctly refused to write: sources named, observed zero.
|
|
137
|
+
//
|
|
138
|
+
// That state is the ORDINARY shape of a first sweep. `tallyEvidence` counts
|
|
139
|
+
// only `killed` and `survived`, so evidence whose records are all `nocover`
|
|
140
|
+
// teaches nothing — and once such a document was on disk, EVERY later sweep
|
|
141
|
+
// failed the same check, permanently, until someone deleted a gitignored
|
|
142
|
+
// file nothing told them about.
|
|
143
|
+
const contributing = tallied.filter((t) => Object.keys(t.tally).length > 0);
|
|
144
|
+
const local = mergeTallies(contributing.map((t) => t.tally));
|
|
133
145
|
const observed = Object.values(local).reduce((a, e) => a + e.n, 0);
|
|
134
146
|
return {
|
|
135
147
|
table: combinedProductivity(local, shipped),
|
|
136
|
-
sources:
|
|
148
|
+
sources: contributing.map((t) => t.path),
|
|
137
149
|
observed,
|
|
138
150
|
classes: Object.keys(local).sort(),
|
|
139
151
|
};
|