testguard-cli 0.12.0 → 0.14.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +144 -0
- package/README.md +95 -11
- package/package.json +1 -1
- package/spec/GATE-SEMANTICS.md +8 -0
- package/spec/lib/validate.mjs +50 -3
- package/spec/schemas/concerns.schema.json +41 -0
- package/spec/schemas/evidence.schema.json +35 -2
- package/spec/schemas/sweep.schema.json +61 -0
- package/src/cli.mjs +7 -1
- package/src/commands/concerns.mjs +32 -0
- package/src/commands/sweep.mjs +52 -23
- package/src/probe/probe.mjs +8 -1
- package/src/probe/pyimports.mjs +9 -2
- package/src/scaffold/producers.python.mjs +54 -2
- package/src/supply/concerns.mjs +162 -0
- package/src/supply/feedback.mjs +146 -0
- package/src/supply/persistence.mjs +73 -0
- package/src/supply/select.mjs +39 -6
- package/src/sweep/sweep.mjs +114 -18
- package/src/util/glob.mjs +5 -1
package/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,103 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [0.14.0] - 2026-09-22
|
|
11
|
+
|
|
12
|
+
Automated weekly release — everything merged since `v0.13.0`.
|
|
13
|
+
|
|
14
|
+
### Changed
|
|
15
|
+
|
|
16
|
+
- fix: three defects the arid measurement found, presentational JSX set aside, and the claimspec signal bump (#164)
|
|
17
|
+
- docs: carry the 0.12 and 0.13 results into the brief and the prior-art record (#162)
|
|
18
|
+
- chore(homebrew): sha256 for v0.13.0 (#161)
|
|
19
|
+
|
|
20
|
+
|
|
21
|
+
### Added
|
|
22
|
+
|
|
23
|
+
- **claimspec v1 evidence admits `persistence-payload-unasserted`** to
|
|
24
|
+
`defenders.signals`, the enum shared with every claimspec consumer. The
|
|
25
|
+
signal had lived only in the sweep document, so `probe` could not surface
|
|
26
|
+
why a save-path survivor was missed. Now a `survived` record whose fault
|
|
27
|
+
sits on the write path carries it for each resolved defender that mocks the
|
|
28
|
+
persistence layer, with the mocked `specifiers` and the file's assertion
|
|
29
|
+
`counts` (exact, partial, argument-free). The validator refuses it on any
|
|
30
|
+
other verdict, on a file that is not a resolved defender, or without its
|
|
31
|
+
reason — the same rule the sweep document already enforced on its
|
|
32
|
+
findings, and the sweep now repeats what the record says rather than
|
|
33
|
+
re-deriving it. Conformance carries a valid example and three invalid
|
|
34
|
+
cases. No consumer outside this repository validates claimspec evidence
|
|
35
|
+
today; DocGuard's adoption (docguard#420) is notified rather than broken.
|
|
36
|
+
- `sweep` sets aside a purely presentational JSX element — an icon, or a
|
|
37
|
+
static wrapper with no expression and no handler — instead of spending a
|
|
38
|
+
probe run on it, and says so: `selection.presentational` in the document, a
|
|
39
|
+
line in the report, and the remainder arithmetic now proves
|
|
40
|
+
proposed = probed + deferred + set aside. Measured on a 14-file UI diff:
|
|
41
|
+
150 of 510 proposals, 29 of 30 probe slots and 7 of 10 survivors were of
|
|
42
|
+
this shape — Google's definition of an arid node, in a syntax Google never
|
|
43
|
+
mutated. Because the ordering learns from survival, those survivors were
|
|
44
|
+
also teaching it to rank the class higher; set-aside proposals never enter
|
|
45
|
+
the evidence, so they no longer can. Google's own five arid categories were
|
|
46
|
+
measured at under 1% of proposals and are not suppressed.
|
|
47
|
+
|
|
48
|
+
### Fixed
|
|
49
|
+
|
|
50
|
+
- Python defender discovery lost the last name on an import line that ends in
|
|
51
|
+
a comment: `from pkg import cli, demo # noqa: E402` found only `cli`, and
|
|
52
|
+
`import pkg.demo # noqa` found nothing. On a real suite where `# noqa` is
|
|
53
|
+
the idiom of every src-layout test, 12 of 30 probed faults were reported
|
|
54
|
+
NOCOVER for modules whose tests import them — a gating verdict in the
|
|
55
|
+
forbidden direction. Comments are now stripped before names are read.
|
|
56
|
+
- File walking descended into `.claude/`, so a nested agent worktree under
|
|
57
|
+
`.claude/worktrees/` contributed another branch's tests as defenders of this
|
|
58
|
+
tree — files the scratch worktree cannot even contain, because the directory
|
|
59
|
+
is gitignored. `.claude` joins the tool-directory skip list, and the claim
|
|
60
|
+
that promised this now names it.
|
|
61
|
+
- The Python `statement-deleted` producer proposed on lines inside a bracket
|
|
62
|
+
opened earlier — `help="…")` inside an `add_argument(` — and none of those
|
|
63
|
+
faults compiled: 81 of 600 proposals on a real CLI (13.5%). The producer now
|
|
64
|
+
tracks bracket depth across lines (triple-quoted strings included, so a
|
|
65
|
+
docstring's parentheses cannot leave the rest of the file "inside") and
|
|
66
|
+
refuses the `pass` shapes there; dict entries and call arguments keep their
|
|
67
|
+
own producers. Re-measured: 7 of 526 (1.3%).
|
|
68
|
+
|
|
69
|
+
### Changed
|
|
70
|
+
|
|
71
|
+
- The self-probe cost budget is 1650 s, from 1440. Six new claims (85 s on
|
|
72
|
+
the Node 22 PR leg, worst 22 s, all under the 100 s per-claim ceiling) took
|
|
73
|
+
that leg from 1367 s to 1457 s, over by 17 s. The v0.13.0 release job
|
|
74
|
+
measured 1113 s from scratch, but the PR leg runs the same claims 25-30%
|
|
75
|
+
slower and is where the ceiling is checked, so 1650 restores ~13% headroom
|
|
76
|
+
over the leg that binds. The per-claim ceiling is unchanged.
|
|
77
|
+
- The technical brief carries the 0.12 and 0.13 results it was missing: the
|
|
78
|
+
whole-surface provability measurement (32 of 37 files that write are defended
|
|
79
|
+
only by tests that mock the database) replaces the sample-level number in
|
|
80
|
+
Fig. 7; the ladder's second rung is now a shipped verb with a real finding
|
|
81
|
+
(a five-line concern found an admin route whose owner/manager check can be
|
|
82
|
+
forced to `false` with no test failing) rather than a description of intent;
|
|
83
|
+
and the first rung says its ordering learns from the project's own runs.
|
|
84
|
+
Re-rendered; still eight sheets from eight pages.
|
|
85
|
+
- `docs-canonical/PRIOR-ART.md` records the arid-node measurement instead of
|
|
86
|
+
a promise: every proposal from four sweeps over two of the maintainer's
|
|
87
|
+
repositories was classified by hand against Google's five arid categories,
|
|
88
|
+
and they account for under 1% of what the producers propose. No suppressor
|
|
89
|
+
is built for them. The same pass found what IS material — presentational
|
|
90
|
+
JSX taking 29 of 30 probe slots on a UI diff, and 13.5% of Python proposals
|
|
91
|
+
that cannot compile — and lists both as open, with numbers.
|
|
92
|
+
- `docs-canonical/PRIOR-ART.md` no longer lists the feedback loop and concerns
|
|
93
|
+
as unbuilt. The two gaps that remain are restated precisely: the loop learns
|
|
94
|
+
from verdicts rather than explicit developer clicks, and concerns have no
|
|
95
|
+
model-backed producer.
|
|
96
|
+
|
|
97
|
+
## [0.13.0] - 2026-09-22
|
|
98
|
+
|
|
99
|
+
Automated weekly release — everything merged since `v0.12.0`.
|
|
100
|
+
|
|
101
|
+
### Changed
|
|
102
|
+
|
|
103
|
+
- chore(homebrew): sha256 for v0.12.0 (#159)
|
|
104
|
+
- feat: persistence provability, and an ordering that learns from this project's own verdicts (#157)
|
|
105
|
+
|
|
106
|
+
|
|
10
107
|
## [0.12.0] - 2026-09-22
|
|
11
108
|
|
|
12
109
|
Automated weekly release — everything merged since `v0.11.0`.
|
|
@@ -18,6 +115,53 @@ Automated weekly release — everything merged since `v0.11.0`.
|
|
|
18
115
|
- chore(homebrew): sha256 for v0.11.0 (#153)
|
|
19
116
|
|
|
20
117
|
|
|
118
|
+
### Added
|
|
119
|
+
|
|
120
|
+
- **Concerns** — one sentence a human writes that generates faults across a
|
|
121
|
+
whole surface, the rung between writing nothing and writing a claim per
|
|
122
|
+
behaviour. A concern says *where* to look and *which shapes of break* are
|
|
123
|
+
relevant: `"every write persists the data it was given"` targets the write
|
|
124
|
+
sites and applies the payload producers. `testguard concerns` lists them,
|
|
125
|
+
`sweep --concern <ID>` aims a sweep by one, and `--save-paths` is now sugar
|
|
126
|
+
for the built-in `SAVE-PERSISTS`.
|
|
127
|
+
- Two built-ins ship, and only two: an authorization concern would need a
|
|
128
|
+
heuristic for "which files check permissions", and a guess there is the
|
|
129
|
+
non-actionable noise that gets a check switched off. A project that knows
|
|
130
|
+
its own auth layer says so with a `glob` target in three lines.
|
|
131
|
+
- A project concern with the same id **replaces** a built-in, so a team can
|
|
132
|
+
retune `SAVE-PERSISTS` for an ORM this tool does not recognise without
|
|
133
|
+
forking anything. The replacement is reported, never silent.
|
|
134
|
+
- `urn:claimspec:v1:concerns`, with rules that refuse a glob target naming no
|
|
135
|
+
globs (it would match nothing and report clean) and an empty fault-class
|
|
136
|
+
list (a typo for "every class", every time).
|
|
137
|
+
- **No model is required, deliberately.** A concern's useful core is a named
|
|
138
|
+
scope plus a producer selection, which the mechanical producers already
|
|
139
|
+
satisfy — so every verifying command stays offline and NFR-01 is unchanged.
|
|
140
|
+
An LLM can later propose faults a line-oriented producer cannot see, as an
|
|
141
|
+
additional source for a concern that already works.
|
|
142
|
+
|
|
143
|
+
### Added
|
|
144
|
+
|
|
145
|
+
- A save-path sweep now reports **what its defenders could prove**, before it
|
|
146
|
+
reports what they did: per file, whether every defender mocks the persistence
|
|
147
|
+
layer (call shape only), at least one does not (a real write is possible), or
|
|
148
|
+
there is no defender at all. When nothing is unmocked the report says so
|
|
149
|
+
outright — a clean probe there is not evidence of persistence, it is evidence
|
|
150
|
+
that nothing could have measured it. Measured on a real AI-authored
|
|
151
|
+
application: of 37 files that write to storage, 32 were defended only by
|
|
152
|
+
mocking tests.
|
|
153
|
+
- `scope.provability` on the sweep document, with a validator rule that every
|
|
154
|
+
target is classified exactly once.
|
|
155
|
+
- The candidate ordering now **learns from this project's own probed verdicts**
|
|
156
|
+
instead of only the shipped productivity prior. Every evidence record already
|
|
157
|
+
carries a `faultClass` and a `verdict`, so "how often does a fault of this
|
|
158
|
+
class survive *here*" is a tally rather than a new thing to capture. A sweep
|
|
159
|
+
persists its own evidence to `.testguard/sweep-evidence.json` — beside the
|
|
160
|
+
canonical one, never replacing it — so the next run learns from
|
|
161
|
+
machine-proposed faults, the same distribution the ranker orders. The sweep
|
|
162
|
+
document records `ordering.observed` and `ordering.sources`, because an
|
|
163
|
+
ordering nobody can trace is a number nobody should trust.
|
|
164
|
+
|
|
21
165
|
### Added
|
|
22
166
|
|
|
23
167
|
- `sweep --save-paths` — sweep every file that writes to storage instead of the
|
package/README.md
CHANGED
|
@@ -127,9 +127,9 @@ raises coverage. TestGuard admits it because it fails when the claim is false.
|
|
|
127
127
|
| npm | `npm i -D testguard-cli` then `npx testguard probe` |
|
|
128
128
|
| pip | `pip install testguard-cli` then `testguard probe` (needs Node ≥ 20) |
|
|
129
129
|
| Homebrew | `brew tap raccioly/tap && brew install testguard` |
|
|
130
|
-
| GitHub Action | `uses: raccioly/testguard@v0.
|
|
130
|
+
| GitHub Action | `uses: raccioly/testguard@v0.14.0` — see [`action.yml`](./action.yml) |
|
|
131
131
|
| pre-commit | `repo: https://github.com/raccioly/testguard`, hooks `testguard-claims`, `testguard-probe` |
|
|
132
|
-
| GitLab CI | `include: - remote: https://raw.githubusercontent.com/raccioly/testguard/v0.
|
|
132
|
+
| GitLab CI | `include: - remote: https://raw.githubusercontent.com/raccioly/testguard/v0.14.0/packaging/gitlab/testguard.gitlab-ci.yml` with `inputs:` — see [`packaging/gitlab/`](./packaging/gitlab/testguard.gitlab-ci.yml) |
|
|
133
133
|
|
|
134
134
|
Projects that set `min-release-age` in `.npmrc` cannot see a version published
|
|
135
135
|
less than that many days ago (`ENOVERSIONS`); install that one with
|
|
@@ -234,6 +234,10 @@ npx testguard-cli admit test/x.test.ts --claim X # is this test green on HEAD
|
|
|
234
234
|
static signal `mocked-never-asserted` — the cheapest blind-spot signal
|
|
235
235
|
there is, and the exact signature of one escaped bug in the field
|
|
236
236
|
reports. Silence it, visibly, with `// unasserted: <why>` above the mock.
|
|
237
|
+
Its sibling `persistence-payload-unasserted` is attached to a **survivor
|
|
238
|
+
on a write path** whose defender mocks the database: the test can prove
|
|
239
|
+
the call shape and never that the row landed, and the signal names the
|
|
240
|
+
mocked layer and the file's assertion mix (exact, partial, argument-free).
|
|
237
241
|
|
|
238
242
|
Runners: **vitest**, **jest**, **Playwright** and **Python** (`--runner
|
|
239
243
|
auto` picks the first of vitest, jest, python that resolves). A JavaScript
|
|
@@ -423,8 +427,9 @@ alarming: something was deliberately broken and not one test failed.
|
|
|
423
427
|
|
|
424
428
|
```
|
|
425
429
|
swept 13 of 14 unclaimed changed files against origin/main.
|
|
426
|
-
proposed 510 faults, probed 10 (cap 10), deferred
|
|
427
|
-
The
|
|
430
|
+
proposed 510 faults, probed 10 (cap 10), deferred 350.
|
|
431
|
+
The 350 deferred are not a verdict: raise --cap, or sweep a smaller change.
|
|
432
|
+
150 presentational elements (icons, static wrappers) set aside: nobody writes a test for those, and a survivor there would teach the ordering the wrong lesson.
|
|
428
433
|
|
|
429
434
|
9 findings — a deliberate break that no test noticed:
|
|
430
435
|
|
|
@@ -451,6 +456,14 @@ Three rules make it safe to run on a repository that has never seen this tool:
|
|
|
451
456
|
is reported and never gates — a tool that fails because its own guess was bad
|
|
452
457
|
is a tool people switch off.
|
|
453
458
|
|
|
459
|
+
The ordering **learns from your own runs**. Every evidence record already
|
|
460
|
+
carries a fault class and a verdict, so "how often does a fault of this class
|
|
461
|
+
survive here" is a tally, not a new thing to collect — a sweep persists its
|
|
462
|
+
evidence and the next one reads it. A project with no evidence falls back to
|
|
463
|
+
the shipped prior and says so; one with its own records overrides it in
|
|
464
|
+
proportion to how many it has. The report names which, because an ordering
|
|
465
|
+
nobody can trace is a number nobody should trust.
|
|
466
|
+
|
|
454
467
|
The cap and the ordering are Google's: their mutation service surfaces at most
|
|
455
468
|
7 × |files| mutants per change and orders candidates on the measured
|
|
456
469
|
productivity of their operator in similar context, which took their productive
|
|
@@ -466,6 +479,60 @@ A sweep is weaker evidence than a probe, and says so. Nobody stated that the
|
|
|
466
479
|
behaviour mattered. It is stronger than nothing, which is what a repository
|
|
467
480
|
with no claims has — and the survivors worth defending become the first claims.
|
|
468
481
|
|
|
482
|
+
#### Concerns: one sentence, two hundred screens
|
|
483
|
+
|
|
484
|
+
A claim names one promise precisely, which is why a repository with two hundred
|
|
485
|
+
screens never finishes writing them. A **concern** names a *kind* of promise and
|
|
486
|
+
says where to look for it:
|
|
487
|
+
|
|
488
|
+
```bash
|
|
489
|
+
npx testguard-cli concerns # what this project can be swept by
|
|
490
|
+
npx testguard-cli sweep --concern ADMIN-GUARDS # aim a sweep at one
|
|
491
|
+
```
|
|
492
|
+
|
|
493
|
+
```json
|
|
494
|
+
{
|
|
495
|
+
"schemaVersion": 1,
|
|
496
|
+
"concerns": [
|
|
497
|
+
{
|
|
498
|
+
"id": "ADMIN-GUARDS",
|
|
499
|
+
"statement": "Every admin route refuses a caller without the admin role.",
|
|
500
|
+
"severity": "critical",
|
|
501
|
+
"targets": { "kind": "glob", "globs": ["src/app/api/admin/**"] },
|
|
502
|
+
"faultClasses": ["guard-removed", "condition-forced", "return-altered"]
|
|
503
|
+
}
|
|
504
|
+
]
|
|
505
|
+
}
|
|
506
|
+
```
|
|
507
|
+
|
|
508
|
+
Five lines, and the sweep reports:
|
|
509
|
+
|
|
510
|
+
```
|
|
511
|
+
concern ADMIN-GUARDS matches 24 files. Swept 23.
|
|
512
|
+
158 further proposals were outside this concern's fault classes.
|
|
513
|
+
|
|
514
|
+
SURVIVED src/app/api/admin/claims/[id]/route.ts
|
|
515
|
+
[line 41] Guard never triggers: `if (!membership || (membership.role !== 'owner'
|
|
516
|
+
&& membership.role !== 'manager'))` becomes `if (false)`.
|
|
517
|
+
```
|
|
518
|
+
|
|
519
|
+
The idea is Meta's: ACH has an engineer describe an area of concern in plain
|
|
520
|
+
text and gates every generated step by execution. What is taken here is the
|
|
521
|
+
**unit** — the concern as the thing a human writes — not their generator, and
|
|
522
|
+
**no model is required**. A concern's useful core is a named scope plus a
|
|
523
|
+
producer selection, which the mechanical producers already satisfy, so every
|
|
524
|
+
verifying command stays offline.
|
|
525
|
+
|
|
526
|
+
Two concerns ship built in, and only two. An authorization concern would need a
|
|
527
|
+
heuristic for *"which files check permissions"*, and a guess there is exactly
|
|
528
|
+
the noise that gets a check switched off — so `SAVE-PERSISTS` and
|
|
529
|
+
`CHANGED-CODE` are the defaults, and a project that knows its own auth layer
|
|
530
|
+
says so in three lines. Declaring a concern with a built-in's id replaces it,
|
|
531
|
+
and the replacement is reported rather than silent.
|
|
532
|
+
|
|
533
|
+
A concern is **not** a claim and never becomes one by itself. It says where to
|
|
534
|
+
look; the probe says what it found; a human states the sentence worth defending.
|
|
535
|
+
|
|
469
536
|
#### Sweeping the save surface instead of the diff
|
|
470
537
|
|
|
471
538
|
A diff cannot answer *"when the user clicks save, does it actually save?"* — it
|
|
@@ -494,10 +561,26 @@ denominator comes first and the document carries `writeSites` and
|
|
|
494
561
|
AI-authored application: **12 of 26 probed persistence faults survived a fully
|
|
495
562
|
green suite, and all 12 were a dropped payload field.**
|
|
496
563
|
|
|
497
|
-
|
|
498
|
-
|
|
499
|
-
|
|
500
|
-
|
|
564
|
+
It also states, before any verdict, **what the defenders could prove at all**:
|
|
565
|
+
|
|
566
|
+
```
|
|
567
|
+
of those 37 files: 32 defended only by tests that mock the persistence layer,
|
|
568
|
+
1 with an unmocked defender, 4 with no defender at all.
|
|
569
|
+
```
|
|
570
|
+
|
|
571
|
+
That line is the read-back answer. A test that replaced the database can prove
|
|
572
|
+
the *call shape* and nothing beyond it — a `where` that matches no rows, a
|
|
573
|
+
rolled-back transaction and a rejected constraint all pass against a mock that
|
|
574
|
+
recorded the arguments and returned a plausible object. So rather than pretend
|
|
575
|
+
to measure persistence, TestGuard states the limit, the same way `nocover` says
|
|
576
|
+
"no test imports this" instead of guessing.
|
|
577
|
+
|
|
578
|
+
When every defender mocks the layer, the report says so outright: *nothing in
|
|
579
|
+
this suite can prove a write reached storage*. A clean probe there is not
|
|
580
|
+
evidence of persistence — it is evidence that nothing could have measured it.
|
|
581
|
+
|
|
582
|
+
`unmocked` is deliberately weaker than "proves persistence": a test that does
|
|
583
|
+
not replace the database may simply never reach it. Possible is the honest word.
|
|
501
584
|
|
|
502
585
|
### Every change needs a claim
|
|
503
586
|
|
|
@@ -580,12 +663,12 @@ detected base that does not resolve is a warning for `status` and `brief`
|
|
|
580
663
|
# GitHub Actions
|
|
581
664
|
- uses: actions/checkout@v4
|
|
582
665
|
with: { fetch-depth: 0 }
|
|
583
|
-
- uses: raccioly/testguard@v0.
|
|
666
|
+
- uses: raccioly/testguard@v0.14.0
|
|
584
667
|
with: { command: gate }
|
|
585
668
|
|
|
586
669
|
# GitLab CI — the component-shaped template: gate + probe, brief as an artifact and, opted in, as a merge-request note
|
|
587
670
|
include:
|
|
588
|
-
- remote: 'https://raw.githubusercontent.com/raccioly/testguard/v0.
|
|
671
|
+
- remote: 'https://raw.githubusercontent.com/raccioly/testguard/v0.14.0/packaging/gitlab/testguard.gitlab-ci.yml'
|
|
589
672
|
inputs: { dir: backend, post_note: true } # post_note needs TESTGUARD_GITLAB_TOKEN (api scope); one note, updated in place
|
|
590
673
|
```
|
|
591
674
|
|
|
@@ -903,7 +986,8 @@ parameter-derived argument swapped for `None`, a key dropped from a payload
|
|
|
903
986
|
`dict` or an allow-list. Two differences are deliberate. A statement is
|
|
904
987
|
removed by replacing it with `pass`, never by deleting the line, because a
|
|
905
988
|
block whose only statement is gone is an `IndentationError`; and a line that
|
|
906
|
-
leaves a bracket open (`COLOURS = {`)
|
|
989
|
+
leaves a bracket open (`COLOURS = {`), or sits inside one opened on an earlier
|
|
990
|
+
line (`help="…"` in an `add_argument(`), is never removed at all. Both exist
|
|
907
991
|
because a fault that cannot compile is a `fault-invalid` verdict — a probe run
|
|
908
992
|
spent saying nothing about the tests. The two JSX shapes (`element-removed`,
|
|
909
993
|
`handler-dropped`) have no Python meaning and are absent rather than faked.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "testguard-cli",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.14.0",
|
|
4
4
|
"description": "Breaks your code on purpose and reports every promise your tests did not notice breaking. Mutation testing bound to the claims a project makes — vitest, jest, Playwright, pytest and unittest.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|
package/spec/GATE-SEMANTICS.md
CHANGED
|
@@ -343,6 +343,14 @@ the suppress-up-to-count semantics are identical.
|
|
|
343
343
|
tool then records `unasserted-annotated` with the reason — silenced, never
|
|
344
344
|
hidden. A file that mocks the subject is never a discovered defender: a
|
|
345
345
|
mock cannot detect a fault in what it replaced.
|
|
346
|
+
`persistence-payload-unasserted`: a resolved defender mocks the persistence
|
|
347
|
+
layer, so it can prove the call shape and never that a row landed; the
|
|
348
|
+
reason names the layer and the file's assertion mix. It is attached only to
|
|
349
|
+
a **survived** record whose fault sits on the write path — a field of a
|
|
350
|
+
persisted payload or the write call itself — because there it says why the
|
|
351
|
+
survivor was missed, and anywhere else it says nothing. A conforming
|
|
352
|
+
document never carries it on another verdict, on a file that is not a
|
|
353
|
+
resolved defender, or without its reason.
|
|
346
354
|
|
|
347
355
|
## Claimed source surface
|
|
348
356
|
|
package/spec/lib/validate.mjs
CHANGED
|
@@ -7,7 +7,7 @@ import { reproduces, decimals, MAX_PLACES } from './wilson.mjs';
|
|
|
7
7
|
|
|
8
8
|
const schemaDir = join(dirname(fileURLToPath(import.meta.url)), '..', 'schemas');
|
|
9
9
|
|
|
10
|
-
export const KINDS = Object.freeze(['claims', 'evidence', 'baseline', 'ignore', 'calibration', 'brief', 'status', 'gate', 'replay', 'sweep']);
|
|
10
|
+
export const KINDS = Object.freeze(['claims', 'evidence', 'baseline', 'ignore', 'calibration', 'brief', 'status', 'gate', 'replay', 'sweep', 'concerns']);
|
|
11
11
|
/**
|
|
12
12
|
* How a document says it tried to falsify its claims. Absent means
|
|
13
13
|
* `fault-injection`, so every document written before the field existed is
|
|
@@ -144,6 +144,16 @@ const semantic = {
|
|
|
144
144
|
if (s.signal === 'target-attribute-patched') {
|
|
145
145
|
if (!s.reason) errors.push({ path: `${p}/defenders/signals`, message: `target-attribute-patched on ${s.file} requires the patched attributes as its reason` });
|
|
146
146
|
if (!r.defenders.resolved.includes(s.file)) errors.push({ path: `${p}/defenders/signals`, message: `${s.file} patches attributes of the subject but is not a resolved defender` });
|
|
147
|
+
} else if (s.signal === 'persistence-payload-unasserted') {
|
|
148
|
+
// The same rule the sweep document enforces on its findings: the
|
|
149
|
+
// signal explains a SURVIVOR on the write path. A record does not
|
|
150
|
+
// carry the fault's line, so the write-path half is the producer's
|
|
151
|
+
// to keep (`persistenceSignalsFor`); the verdict half is checked here.
|
|
152
|
+
// The file mocks the persistence layer, not the subject, so it is a
|
|
153
|
+
// resolved defender and never a mocking one.
|
|
154
|
+
if (!s.reason) errors.push({ path: `${p}/defenders/signals`, message: `persistence-payload-unasserted on ${s.file} requires the mocked layer and assertion mix as its reason` });
|
|
155
|
+
if (r.verdict !== 'survived') errors.push({ path: `${p}/defenders/signals`, message: `persistence-payload-unasserted explains a survivor; on a ${r.verdict} record it explains nothing` });
|
|
156
|
+
if (!r.defenders.resolved.includes(s.file)) errors.push({ path: `${p}/defenders/signals`, message: `${s.file} mocks the persistence layer but is not a resolved defender` });
|
|
147
157
|
} else if (!(r.defenders.mocking ?? []).includes(s.file)) {
|
|
148
158
|
errors.push({ path: `${p}/defenders/signals`, message: `${s.file} carries a mock signal but is not listed in defenders.mocking` });
|
|
149
159
|
}
|
|
@@ -305,12 +315,36 @@ const semantic = {
|
|
|
305
315
|
return errors;
|
|
306
316
|
},
|
|
307
317
|
|
|
318
|
+
concerns(doc) {
|
|
319
|
+
const errors = [];
|
|
320
|
+
const seen = new Set();
|
|
321
|
+
doc.concerns.forEach((c, i) => {
|
|
322
|
+
if (seen.has(c.id)) errors.push({ path: `/concerns/${i}/id`, message: `duplicate concern id "${c.id}"` });
|
|
323
|
+
seen.add(c.id);
|
|
324
|
+
// A glob target with no globs matches nothing, which reads as "this
|
|
325
|
+
// concern found no problems" rather than "this concern was never aimed".
|
|
326
|
+
if (c.targets?.kind === 'glob' && !(c.targets.globs?.length > 0)) {
|
|
327
|
+
errors.push({ path: `/concerns/${i}/targets/globs`, message: `concern "${c.id}" targets a glob but names none; it would match nothing and report clean` });
|
|
328
|
+
}
|
|
329
|
+
if (c.targets?.kind !== 'glob' && c.targets?.globs) {
|
|
330
|
+
errors.push({ path: `/concerns/${i}/targets/globs`, message: `concern "${c.id}" names globs but does not target them` });
|
|
331
|
+
}
|
|
332
|
+
// An empty list is not "every class"; it is a concern that can never
|
|
333
|
+
// propose anything, which is a typo for `null` every time.
|
|
334
|
+
if (Array.isArray(c.faultClasses) && c.faultClasses.length === 0) {
|
|
335
|
+
errors.push({ path: `/concerns/${i}/faultClasses`, message: `concern "${c.id}" allows no fault class; omit the field for every class` });
|
|
336
|
+
}
|
|
337
|
+
});
|
|
338
|
+
return errors;
|
|
339
|
+
},
|
|
340
|
+
|
|
308
341
|
sweep(doc) {
|
|
309
342
|
const errors = [];
|
|
310
343
|
const s = doc.selection;
|
|
311
344
|
// A cap that hides its remainder is a coverage claim nobody made, so the
|
|
312
345
|
// arithmetic that proves nothing was dropped is checked, not trusted.
|
|
313
|
-
|
|
346
|
+
const setAside = s.presentational ?? 0;
|
|
347
|
+
if (s.proposed !== s.selected + s.deferred + setAside) errors.push({ path: '/selection/proposed', message: `proposed (${s.proposed}) must equal selected (${s.selected}) + deferred (${s.deferred})${setAside ? ` + presentational (${setAside})` : ''}` });
|
|
314
348
|
if (s.selected > s.cap) errors.push({ path: '/selection/selected', message: `selected (${s.selected}) exceeds the cap (${s.cap})` });
|
|
315
349
|
if (doc.scope.swept > doc.scope.targets) errors.push({ path: '/scope/swept', message: `swept (${doc.scope.swept}) cannot exceed targets (${doc.scope.targets})` });
|
|
316
350
|
// The counters that only mean something in one mode must not appear in the
|
|
@@ -318,10 +352,17 @@ const semantic = {
|
|
|
318
352
|
// measured, and its absence on a save-paths sweep hides the denominator
|
|
319
353
|
// entirely — which is the whole reason that mode exists.
|
|
320
354
|
const saves = doc.scope.mode === 'save-paths';
|
|
321
|
-
for (const k of ['writeSites', 'payloadFields']) {
|
|
355
|
+
for (const k of ['writeSites', 'payloadFields', 'provability']) {
|
|
322
356
|
if (saves && doc.scope[k] === undefined) errors.push({ path: `/scope/${k}`, message: `a save-paths sweep must report ${k}: the surface is the denominator its findings are read against` });
|
|
323
357
|
if (!saves && doc.scope[k] !== undefined) errors.push({ path: `/scope/${k}`, message: `${k} belongs to a save-paths sweep; a changed sweep did not measure the write surface` });
|
|
324
358
|
}
|
|
359
|
+
if (saves && doc.scope.provability) {
|
|
360
|
+
const pv = doc.scope.provability;
|
|
361
|
+
const total = pv.mocked + pv.unmocked + pv.none;
|
|
362
|
+
// Every target is classified exactly once. A surface whose provability
|
|
363
|
+
// does not account for all of it has a bucket nobody looked in.
|
|
364
|
+
if (total !== doc.scope.targets) errors.push({ path: '/scope/provability', message: `provability accounts for ${total} files; targets is ${doc.scope.targets}` });
|
|
365
|
+
}
|
|
325
366
|
if (saves && doc.scope.writeSites !== undefined && doc.scope.targets > doc.scope.writeSites) {
|
|
326
367
|
errors.push({ path: '/scope/targets', message: `targets (${doc.scope.targets}) exceeds writeSites (${doc.scope.writeSites}); every target is a file with at least one write` });
|
|
327
368
|
}
|
|
@@ -343,6 +384,12 @@ const semantic = {
|
|
|
343
384
|
const notKilled = probed - (doc.counts.killed ?? 0);
|
|
344
385
|
if (doc.findings.length !== notKilled) errors.push({ path: '/findings', message: `${doc.findings.length} findings for ${notKilled} records that were not killed; findings are every non-killed record` });
|
|
345
386
|
|
|
387
|
+
// An ordering that claims to have learned from something must name it.
|
|
388
|
+
if (doc.ordering) {
|
|
389
|
+
if (doc.ordering.observed > 0 && doc.ordering.sources.length === 0) errors.push({ path: '/ordering/sources', message: 'an ordering learned from evidence must name the documents it was learned from' });
|
|
390
|
+
if (doc.ordering.observed === 0 && doc.ordering.sources.length > 0) errors.push({ path: '/ordering/observed', message: 'sources are named but nothing was observed; the ordering used the shipped prior' });
|
|
391
|
+
}
|
|
392
|
+
|
|
346
393
|
const gating = doc.findings.filter((f) => f.verdict === 'survived' || f.verdict === 'nocover').length;
|
|
347
394
|
if (gating > 0 && doc.exitCode !== 1) errors.push({ path: '/exitCode', message: 'a survived or nocover finding must exit 1; a sweep never passes over a deliberate break nothing noticed' });
|
|
348
395
|
if (gating === 0 && doc.exitCode !== 0) errors.push({ path: '/exitCode', message: 'exit 1 requires a survived or nocover finding; a sweep does not fail on its own unanchorable proposal' });
|
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
{
|
|
2
|
+
"$id": "urn:claimspec:v1:concerns",
|
|
3
|
+
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
4
|
+
"title": "claimspec v1 — concerns",
|
|
5
|
+
"description": "A concern is one sentence a human writes that generates faults across a whole surface: it says WHERE to look and WHICH shapes of break are relevant. It is the rung between writing nothing and writing a claim per behaviour. A concern is never a claim and never becomes one by itself — it says where to look, the probe says what it found, and a human states the sentence worth defending. Nothing in this document is a promise about the code; only a claim is that.",
|
|
6
|
+
"type": "object",
|
|
7
|
+
"required": ["schemaVersion", "concerns"],
|
|
8
|
+
"additionalProperties": false,
|
|
9
|
+
"properties": {
|
|
10
|
+
"$schema": { "type": "string" },
|
|
11
|
+
"schemaVersion": { "const": 1 },
|
|
12
|
+
"concerns": {
|
|
13
|
+
"type": "array",
|
|
14
|
+
"items": {
|
|
15
|
+
"type": "object",
|
|
16
|
+
"required": ["id", "statement"],
|
|
17
|
+
"additionalProperties": false,
|
|
18
|
+
"properties": {
|
|
19
|
+
"id": { "$ref": "urn:claimspec:v1:common#/$defs/identifier", "description": "Declaring an id that matches a built-in REPLACES it, which is how a project retunes a default for an ORM or a layout this tool does not recognise. The replacement is reported, never silent." },
|
|
20
|
+
"statement": { "type": "string", "minLength": 1, "description": "The sentence, in plain language. It describes a KIND of promise, not one behaviour: \"every write persists the data it was given\", not \"the register action stores password_hash\". The second is a claim." },
|
|
21
|
+
"severity": { "$ref": "urn:claimspec:v1:common#/$defs/severity" },
|
|
22
|
+
"targets": {
|
|
23
|
+
"type": "object",
|
|
24
|
+
"required": ["kind"],
|
|
25
|
+
"additionalProperties": false,
|
|
26
|
+
"description": "Where this concern applies. `write-sites` asks the write enumerator. `glob` matches paths. `changed` defers to whatever diff the caller is already measuring — a concern does not re-derive what the gate computed.",
|
|
27
|
+
"properties": {
|
|
28
|
+
"kind": { "enum": ["write-sites", "glob", "changed"] },
|
|
29
|
+
"globs": { "type": "array", "items": { "type": "string", "minLength": 1 } }
|
|
30
|
+
}
|
|
31
|
+
},
|
|
32
|
+
"faultClasses": {
|
|
33
|
+
"type": ["array", "null"],
|
|
34
|
+
"items": { "$ref": "urn:claimspec:v1:common#/$defs/faultClass" },
|
|
35
|
+
"description": "Which producers are relevant. `null` or absent means every class, the honest default for a concern with no shape to narrow by. Narrowing is what makes one sentence useful across two hundred files: a persistence concern that also reported every altered return value would bury the payload findings it exists to surface."
|
|
36
|
+
}
|
|
37
|
+
}
|
|
38
|
+
}
|
|
39
|
+
}
|
|
40
|
+
}
|
|
41
|
+
}
|
|
@@ -547,7 +547,7 @@
|
|
|
547
547
|
}
|
|
548
548
|
},
|
|
549
549
|
"signals": {
|
|
550
|
-
"description": "Static, annotation-grade signals about the defenders. They never change a verdict. `mocked-never-asserted`: the file mocks the subject and never asserts on anything imported from it. `unasserted-annotated`: the same, but the author annotated the mock with a reason (`// unasserted: <why>`), recorded here \u2014 silenced, never hidden. `target-attribute-patched`: a Python defender that patches individual attributes of the subject (`patch(\"pkg.mod.fn\")`) rather than the module. Unlike a JavaScript module mock it does NOT disqualify the file \u2014 a fault anywhere else in the module is still detectable \u2014 so the file stays in `resolved`; `reason` names the patched attributes, which are where that test is least likely to notice anything.",
|
|
550
|
+
"description": "Static, annotation-grade signals about the defenders. They never change a verdict. `mocked-never-asserted`: the file mocks the subject and never asserts on anything imported from it. `unasserted-annotated`: the same, but the author annotated the mock with a reason (`// unasserted: <why>`), recorded here \u2014 silenced, never hidden. `target-attribute-patched`: a Python defender that patches individual attributes of the subject (`patch(\"pkg.mod.fn\")`) rather than the module. Unlike a JavaScript module mock it does NOT disqualify the file \u2014 a fault anywhere else in the module is still detectable \u2014 so the file stays in `resolved`; `reason` names the patched attributes, which are where that test is least likely to notice anything. `persistence-payload-unasserted`: a resolved defender that mocks the persistence layer (`vi.mock('@/lib/prisma')`) and so can prove the call shape but never that a row landed; `reason` names the mocked layer and the file's assertion mix (exact, partial, argument-free), `counts` carries the mix as numbers and `specifiers` the mocked module paths. It belongs only to a `survived` record whose fault sits on the write path \u2014 a field of a persisted payload, or the write call itself \u2014 because there it explains why the survivor was missed and anywhere else it explains nothing. Measured on a real application: 12 of 12 survivors on the save path were a dropped payload field, and every one of their defenders asserted with a partial matcher.",
|
|
551
551
|
"type": "array",
|
|
552
552
|
"items": {
|
|
553
553
|
"type": "object",
|
|
@@ -563,13 +563,46 @@
|
|
|
563
563
|
"enum": [
|
|
564
564
|
"mocked-never-asserted",
|
|
565
565
|
"unasserted-annotated",
|
|
566
|
-
"target-attribute-patched"
|
|
566
|
+
"target-attribute-patched",
|
|
567
|
+
"persistence-payload-unasserted"
|
|
567
568
|
]
|
|
568
569
|
},
|
|
569
570
|
"reason": {
|
|
570
571
|
"type": "string",
|
|
571
572
|
"minLength": 1,
|
|
572
573
|
"maxLength": 512
|
|
574
|
+
},
|
|
575
|
+
"specifiers": {
|
|
576
|
+
"description": "persistence-payload-unasserted only: the persistence-layer module specifiers the file mocks.",
|
|
577
|
+
"type": "array",
|
|
578
|
+
"items": {
|
|
579
|
+
"type": "string",
|
|
580
|
+
"minLength": 1
|
|
581
|
+
}
|
|
582
|
+
},
|
|
583
|
+
"counts": {
|
|
584
|
+
"description": "persistence-payload-unasserted only: the file's call-shape assertions by kind. `exact` is the only kind a dropped field can fail.",
|
|
585
|
+
"type": "object",
|
|
586
|
+
"required": [
|
|
587
|
+
"exact",
|
|
588
|
+
"partial",
|
|
589
|
+
"argless"
|
|
590
|
+
],
|
|
591
|
+
"additionalProperties": false,
|
|
592
|
+
"properties": {
|
|
593
|
+
"exact": {
|
|
594
|
+
"type": "integer",
|
|
595
|
+
"minimum": 0
|
|
596
|
+
},
|
|
597
|
+
"partial": {
|
|
598
|
+
"type": "integer",
|
|
599
|
+
"minimum": 0
|
|
600
|
+
},
|
|
601
|
+
"argless": {
|
|
602
|
+
"type": "integer",
|
|
603
|
+
"minimum": 0
|
|
604
|
+
}
|
|
605
|
+
}
|
|
573
606
|
}
|
|
574
607
|
},
|
|
575
608
|
"additionalProperties": false
|
|
@@ -62,6 +62,10 @@
|
|
|
62
62
|
],
|
|
63
63
|
"description": "What the sweep was pointed at. `changed` proposes for the changed source files that carry no claim. `save-paths` proposes for every file that writes to storage, because a diff cannot answer how many save paths exist — and a clean report over an unstated denominator is a sample of unknown size, not a guarantee."
|
|
64
64
|
},
|
|
65
|
+
"concern": {
|
|
66
|
+
"$ref": "urn:claimspec:v1:common#/$defs/identifier",
|
|
67
|
+
"description": "The concern this sweep was aimed by, when one was. A concern narrows both the targets and the fault classes, so the same surface swept under two concerns is two different questions."
|
|
68
|
+
},
|
|
65
69
|
"changed": {
|
|
66
70
|
"type": "integer",
|
|
67
71
|
"minimum": 0,
|
|
@@ -107,6 +111,30 @@
|
|
|
107
111
|
"type": "integer",
|
|
108
112
|
"minimum": 0,
|
|
109
113
|
"description": "save-paths only: payload keys sitting on their own line, which is what a field-dropped fault can anchor to."
|
|
114
|
+
},
|
|
115
|
+
"provability": {
|
|
116
|
+
"type": "object",
|
|
117
|
+
"required": [
|
|
118
|
+
"mocked",
|
|
119
|
+
"unmocked",
|
|
120
|
+
"none"
|
|
121
|
+
],
|
|
122
|
+
"additionalProperties": false,
|
|
123
|
+
"description": "save-paths only: what the defenders of this surface are CAPABLE of proving, before anything is probed. `mocked` — every defender replaces the persistence layer, so it can prove the call shape and never that the row landed. `unmocked` — at least one does not, so a real write is at least possible (not a promise that one happens). `none` — no defender at all. On a surface that is entirely `mocked`, a clean probe is not evidence of persistence; it is evidence that nothing could have measured it.",
|
|
124
|
+
"properties": {
|
|
125
|
+
"mocked": {
|
|
126
|
+
"type": "integer",
|
|
127
|
+
"minimum": 0
|
|
128
|
+
},
|
|
129
|
+
"unmocked": {
|
|
130
|
+
"type": "integer",
|
|
131
|
+
"minimum": 0
|
|
132
|
+
},
|
|
133
|
+
"none": {
|
|
134
|
+
"type": "integer",
|
|
135
|
+
"minimum": 0
|
|
136
|
+
}
|
|
137
|
+
}
|
|
110
138
|
}
|
|
111
139
|
}
|
|
112
140
|
},
|
|
@@ -165,6 +193,16 @@
|
|
|
165
193
|
}
|
|
166
194
|
}
|
|
167
195
|
}
|
|
196
|
+
},
|
|
197
|
+
"outOfScope": {
|
|
198
|
+
"type": "integer",
|
|
199
|
+
"minimum": 0,
|
|
200
|
+
"description": "Proposals the concern's fault classes excluded. Reported so a narrow concern cannot look like a quiet one: the producers found these, and this concern is not about them."
|
|
201
|
+
},
|
|
202
|
+
"presentational": {
|
|
203
|
+
"type": "integer",
|
|
204
|
+
"minimum": 0,
|
|
205
|
+
"description": "Proposals set aside because they remove a purely presentational JSX element — an icon, or a static wrapper with no expression and no handler. Nobody writes a test for those, and a survivor on one would teach the survival-learned ordering to prefer the class. Counted in `proposed` and in `byClass`; never probed, never a finding. Absent when zero."
|
|
168
206
|
}
|
|
169
207
|
}
|
|
170
208
|
},
|
|
@@ -305,6 +343,29 @@
|
|
|
305
343
|
"items": {
|
|
306
344
|
"type": "object"
|
|
307
345
|
}
|
|
346
|
+
},
|
|
347
|
+
"ordering": {
|
|
348
|
+
"type": "object",
|
|
349
|
+
"required": [
|
|
350
|
+
"observed",
|
|
351
|
+
"sources"
|
|
352
|
+
],
|
|
353
|
+
"additionalProperties": false,
|
|
354
|
+
"description": "Where the candidate ordering came from. `observed` is how many of this project's OWN probed faults informed it; 0 means the shipped productivity prior was used unchanged. An ordering nobody can trace is a number nobody should trust.",
|
|
355
|
+
"properties": {
|
|
356
|
+
"observed": {
|
|
357
|
+
"type": "integer",
|
|
358
|
+
"minimum": 0
|
|
359
|
+
},
|
|
360
|
+
"sources": {
|
|
361
|
+
"type": "array",
|
|
362
|
+
"items": {
|
|
363
|
+
"type": "string",
|
|
364
|
+
"minLength": 1
|
|
365
|
+
},
|
|
366
|
+
"description": "The evidence documents the counts were read from."
|
|
367
|
+
}
|
|
368
|
+
}
|
|
308
369
|
}
|
|
309
370
|
}
|
|
310
371
|
}
|