testguard-cli 0.6.0 → 0.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +489 -0
- package/README.md +287 -34
- package/package.json +2 -2
- package/spec/GATE-SEMANTICS.md +210 -8
- package/spec/README.md +32 -11
- package/spec/lib/validate.mjs +125 -14
- package/spec/lib/wilson.mjs +112 -0
- package/spec/schemas/baseline.schema.json +7 -7
- package/spec/schemas/brief.schema.json +12 -12
- package/spec/schemas/calibration.schema.json +96 -26
- package/spec/schemas/claims.schema.json +129 -25
- package/spec/schemas/common.schema.json +127 -29
- package/spec/schemas/evidence.schema.json +175 -73
- package/spec/schemas/gate.schema.json +12 -12
- package/spec/schemas/ignore.schema.json +4 -4
- package/spec/schemas/replay.schema.json +10 -10
- package/spec/schemas/status.schema.json +25 -25
- package/src/cli.mjs +20 -5
- package/src/commands/claims.mjs +22 -2
- package/src/commands/probe.mjs +24 -4
- package/src/commands/replay.mjs +1 -1
- package/src/gate/changed.mjs +9 -2
- package/src/probe/attribution.mjs +123 -0
- package/src/probe/classify.mjs +18 -2
- package/src/probe/cost.mjs +205 -0
- package/src/probe/discover.mjs +14 -6
- package/src/probe/inject.mjs +41 -9
- package/src/probe/mocks.mjs +2 -0
- package/src/probe/probe.mjs +235 -52
- package/src/probe/progress.mjs +87 -0
- package/src/probe/pyimports.mjs +287 -0
- package/src/probe/rank.mjs +2 -0
- package/src/probe/runners/index.mjs +38 -11
- package/src/probe/runners/jest.mjs +13 -1
- package/src/probe/runners/playwright.mjs +9 -0
- package/src/probe/runners/python/_testguard_pytest_plugin.py +132 -0
- package/src/probe/runners/python/_testguard_report.py +141 -0
- package/src/probe/runners/python/_testguard_unittest_main.py +168 -0
- package/src/probe/runners/python.mjs +316 -0
- package/src/probe/runners/shared.mjs +10 -1
- package/src/probe/runners/vitest.mjs +13 -1
- package/src/replay/replay.mjs +50 -22
- package/src/scaffold/producers.python.mjs +209 -0
- package/src/scaffold/scaffold.mjs +86 -32
- package/src/util/glob.mjs +7 -1
package/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,495 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [0.8.0] - 2026-09-18
|
|
11
|
+
|
|
12
|
+
A calibration you can quote.
|
|
13
|
+
|
|
14
|
+
A calibration document now says what its number means and where its labels
|
|
15
|
+
came from, and it reproduces: `p` and every interval are recomputed by the
|
|
16
|
+
validator from `n`, `positives`, `method` and `confidence`, at the document's
|
|
17
|
+
own precision — the spec's own example had carried a truncated bound since
|
|
18
|
+
the day it was written, and nothing could notice. `nocover` counts as the
|
|
19
|
+
miss it is, so a project with no tests at all for a subsystem no longer
|
|
20
|
+
outscores one with weak tests. The format also carries what
|
|
21
|
+
websec-validator's shipped table already said — corpus, caveat, a producer
|
|
22
|
+
floor, backoff tiers, a labelled prior — so a second tool's honesty survives
|
|
23
|
+
translation; that table is now a conformance example, and "adoptable" means
|
|
24
|
+
the file validates rather than a sentence saying so. Underneath: the
|
|
25
|
+
self-probe gate is held to a cost budget (#89), and a reserved method has
|
|
26
|
+
somewhere to put its data (#91).
|
|
27
|
+
|
|
28
|
+
### Added
|
|
29
|
+
- **A calibration can now say what it measures and where it came from** (#82).
|
|
30
|
+
websec-validator's shipped `calibration.json` already carried `corpus`,
|
|
31
|
+
`min_n`, `caveat`, `evidence_status` and `limitation` — the fields that say
|
|
32
|
+
"indicative, do not quote this number" — and claimspec had nowhere to put
|
|
33
|
+
them, so adopting it would have stripped that honesty in translation. It
|
|
34
|
+
also had nowhere to say what `p` *was*: testguard's is a miss rate where
|
|
35
|
+
high is bad, websec's is P(real) where high is good, and both validated
|
|
36
|
+
identically. The schema gains `measures` (a registry: `escape-missed`,
|
|
37
|
+
`finding-real`), provenance in `source` (`corpus`, `caveat`, `limitation`,
|
|
38
|
+
`evidenceStatus`, a `tool-oracle` kind, and an unconstrained `detail` on the
|
|
39
|
+
`methodDetail` pattern), a producer floor `minN` that consumers may raise
|
|
40
|
+
and never lower, ordered `backoff` tiers so a class→label→prior cascade is
|
|
41
|
+
expressible, a `fallback` whose shape says it is a guess, a per-cell
|
|
42
|
+
`breakdown` that sums to `n`, `p: null` at `n = 0`, and `|`-compound bucket
|
|
43
|
+
keys whose arity the validator checks. Every field is optional and absence
|
|
44
|
+
means *unattributed* — readable, not quotable — so no existing document
|
|
45
|
+
breaks; testguard's emitter always writes them and a self-claim guards it.
|
|
46
|
+
The proof of adoption is a file, not a sentence: websec-validator's table,
|
|
47
|
+
translated field for field, is now `spec/conformance/examples/calibration-websec.json`
|
|
48
|
+
and validates with every number reproducing. `GATE-SEMANTICS.md` records
|
|
49
|
+
the merge rule (sum counts, recompute, never average; only when `measures`
|
|
50
|
+
and `bucketBy` agree) that websec's shipped-plus-local overlay already
|
|
51
|
+
depends on.
|
|
52
|
+
- **The gate is now gated on its own cost** (#89). `--cost` has printed the
|
|
53
|
+
self-probe's wall clock into every CI log since #67, and the gate still went
|
|
54
|
+
from 9.6 to 23.6 minutes across one merged change without anything saying a
|
|
55
|
+
word. A measurement nothing gates on is not a check — which is the premise
|
|
56
|
+
this tool rests on, applied to itself.
|
|
57
|
+
`testguard.cost-budget.json` carries the ceiling, its measurement and why it
|
|
58
|
+
is set where it is. CI fails when the probe exceeds it and opens with the
|
|
59
|
+
most expensive claims, so the first thing a failure shows is where to look.
|
|
60
|
+
Raising it is a committed diff with a reason: growth is allowed, going
|
|
61
|
+
unnoticed is not.
|
|
62
|
+
The budget is on the **total**, never a per-fault average. The regression
|
|
63
|
+
that prompted this took the corpus from 59 faults to 91 while per-fault cost
|
|
64
|
+
barely moved — an average would have reported everything fine while the gate
|
|
65
|
+
tripled, and a test says so. A budget that is missing or not a positive
|
|
66
|
+
number is refused rather than treated as unlimited, so a typo cannot quietly
|
|
67
|
+
disable the gate.
|
|
68
|
+
|
|
69
|
+
|
|
70
|
+
### Fixed
|
|
71
|
+
- **The cost budget is set against what CI measures.** 950 s was 18% over
|
|
72
|
+
the local 804 s figure and only 4.6% over the 906 s CI already measured on
|
|
73
|
+
`main`, so the first pull request to add any claim tripped it — #94 passed
|
|
74
|
+
its own leg only because the cache reused verdicts, and `main` went red at
|
|
75
|
+
981 s from scratch the moment it merged. Raised to 1100 s (~14% over the
|
|
76
|
+
962 s CI measures with #82's re-pointed defenders), with the CI figures
|
|
77
|
+
recorded beside the local one so the next person budgets against the
|
|
78
|
+
number the check actually sees. The heavy claims the failure now names are
|
|
79
|
+
the next #73-style cut.
|
|
80
|
+
- **Calibration's pure tests moved off the replay fixture** (the #73 pattern).
|
|
81
|
+
Three calibration claims were defended by `test/replay.test.mjs`, which
|
|
82
|
+
builds a scripted git corpus on every run; their kills come from pure
|
|
83
|
+
`calibrationFrom` tests that need none of it. The cost budget fired on the
|
|
84
|
+
first full run — 1022 s against 950 s, the top rows being exactly those
|
|
85
|
+
claims — so the tests now live in `test/calibration.test.mjs`, the claims
|
|
86
|
+
point there, and every fault was re-probed after the move, not assumed.
|
|
87
|
+
- **A cost-budget failure names its claims.** `checkCostBudget` read `id`
|
|
88
|
+
from a report whose claims carry `claimId`, so the one line meant to say
|
|
89
|
+
*which* claims to look at printed `undefined` five times — and its unit test
|
|
90
|
+
fed the shape the code wished for, so it passed. Found the first time the
|
|
91
|
+
budget actually fired. The test now goes through `costReport`, and
|
|
92
|
+
`TG-COST-BUDGET-NAMES-THE-CLAIM` guards it.
|
|
93
|
+
- **A calibration counts `nocover` as a miss** (#82). `nocover` — no test
|
|
94
|
+
even imports the broken file — is the worst replay outcome, and the ratio
|
|
95
|
+
excluded it: a project with no tests at all for a subsystem scored *better*
|
|
96
|
+
than one with weak tests, because its worst outcomes left the denominator
|
|
97
|
+
before the ratio was taken. It now enters both sides, as it does in Stryker
|
|
98
|
+
and PIT, and the replay summary says how many misses were blind and how many
|
|
99
|
+
had no test at all. The schema description, the emitter's docstring and
|
|
100
|
+
`GATE-SEMANTICS.md` had also disagreed about what `p` was — "P(finding is
|
|
101
|
+
real)" in one place, the miss rate in another, and the docstring's two
|
|
102
|
+
sentences conditioning on different denominators — and now say one thing:
|
|
103
|
+
for testguard, the share of real escaped bugs the suite failed to catch.
|
|
104
|
+
- **A calibration's arithmetic is checked, not trusted.** The validator
|
|
105
|
+
recomputed nothing: it checked that `p` sat inside `ci` and would pass a
|
|
106
|
+
document with every number invented. It now recomputes `p` from
|
|
107
|
+
`positives / n` and `ci` from `n`, `positives`, `method` and `confidence`
|
|
108
|
+
with `spec/lib/wilson.mjs` — one implementation, shared with the emitter —
|
|
109
|
+
at the document's own precision, accepting either the exact quantile or the
|
|
110
|
+
textbook 1.96. The spec's own example had carried a truncated lower bound
|
|
111
|
+
(6/30 → `0.09`; Wilson gives 0.0950 → `0.10`) since the day it was written,
|
|
112
|
+
and websec-validator's shipped table reproduces exactly.
|
|
113
|
+
- **A reserved method can now say something.** #81 reserved `assertion` and
|
|
114
|
+
`scan` without required fields, and `fault` / `detail` are both
|
|
115
|
+
`additionalProperties: false` — so a probe using a reserved method could
|
|
116
|
+
declare that it existed and **nothing whatsoever about it**: not which rule
|
|
117
|
+
was checked, not where. That is not a reservation, it is a dead end, and
|
|
118
|
+
DocGuard's writer would have hit it on its first line.
|
|
119
|
+
Both now carry `methodDetail`: an object the spec deliberately does not
|
|
120
|
+
constrain, on the probe and on the record. It is **forbidden under
|
|
121
|
+
`fault-injection`**, whose shape is specified and must not acquire a junk
|
|
122
|
+
drawer — a self-claim fails if that ever stops being enforced — and it is a
|
|
123
|
+
staging area rather than a permanent home. What the first real consumer puts
|
|
124
|
+
there is the evidence for what the specified shape should become.
|
|
125
|
+
|
|
126
|
+
## [0.7.0] - 2026-09-18
|
|
127
|
+
|
|
128
|
+
Python, and a fault that cannot hide.
|
|
129
|
+
|
|
130
|
+
TestGuard now probes Python projects with nothing installed into them, and it
|
|
131
|
+
no longer takes a green suite at its word: a fault applied to code the tests
|
|
132
|
+
never execute used to be reported as SURVIVED — an audit finding that reads as
|
|
133
|
+
devastating and is entirely false — and is now `unverifiable`, because the
|
|
134
|
+
defenders are made to prove they can fail because of that file before a
|
|
135
|
+
survival about it is allowed to stand.
|
|
136
|
+
|
|
137
|
+
Alongside: the shared spec is named `claimspec` and no longer assumes fault
|
|
138
|
+
injection is the only way to falsify a claim, one bad fault can no longer cost
|
|
139
|
+
the evidence for the whole run, and the self-probe gate was measured rather
|
|
140
|
+
than guessed at.
|
|
141
|
+
|
|
142
|
+
### Upgrading
|
|
143
|
+
|
|
144
|
+
**Some claims that were `survived` will become `unverifiable`, and your frozen
|
|
145
|
+
baseline will not suppress them.** This is the intended consequence of the
|
|
146
|
+
negative control, and it is worth reading before you re-freeze anything.
|
|
147
|
+
|
|
148
|
+
A fingerprint is derived from the claim, the subject, the file and **the
|
|
149
|
+
verdict**. A claim whose subject the defenders never execute was reported
|
|
150
|
+
`survived`; it is now `unverifiable` with reason `subject-not-executed`. That
|
|
151
|
+
is a different fingerprint, so a baseline frozen before this release does not
|
|
152
|
+
contain it, and it gates as new.
|
|
153
|
+
|
|
154
|
+
**The finding is real.** It says the defenders do not execute that file at all,
|
|
155
|
+
so every earlier verdict about it was a statement about their reach rather than
|
|
156
|
+
their assertions — including the `survived` your baseline was suppressing. A
|
|
157
|
+
`survived` you had accepted as known debt was, in these cases, not a measurement
|
|
158
|
+
of anything.
|
|
159
|
+
|
|
160
|
+
What to do, in order:
|
|
161
|
+
|
|
162
|
+
1. `testguard probe` and read the records carrying
|
|
163
|
+
`detail.negativeControl: "not-reached"`. Each names a file no defender
|
|
164
|
+
loads.
|
|
165
|
+
2. **Prefer fixing the defenders.** A claim whose subject is never imported is
|
|
166
|
+
the cheapest blind spot you will ever find — cheaper than a survivor,
|
|
167
|
+
because nothing was even tried.
|
|
168
|
+
3. If you are accepting it as existing debt for now, `testguard baseline`
|
|
169
|
+
re-freezes today's findings so only newer ones gate. Do this **after**
|
|
170
|
+
looking at them, not instead of.
|
|
171
|
+
|
|
172
|
+
`baseline --restamp` is **not** the command for this: it moves a baseline's
|
|
173
|
+
`head` onto a clean identical probe, and it requires the same fingerprints.
|
|
174
|
+
|
|
175
|
+
Nothing else in this release requires migration. The `claimspec` rename changes
|
|
176
|
+
no emitted document, the `method` field defaults to `fault-injection` so every
|
|
177
|
+
existing claims and evidence file stays valid, and the gate work changed which
|
|
178
|
+
tests defend which claims, never a verdict.
|
|
179
|
+
|
|
180
|
+
### Added
|
|
181
|
+
|
|
182
|
+
- **A five-page technical brief**, [`docs/testguard-explained.pdf`](docs/testguard-explained.pdf):
|
|
183
|
+
the whole idea on page one, the field measurements on page two, then
|
|
184
|
+
mechanics, architecture and prior art. Six diagrams, including a dot matrix
|
|
185
|
+
of the 39 injected faults from the second field report — 21 survived a fully
|
|
186
|
+
green suite, 39/39 killed after tests were written against the survivors. Built from
|
|
187
|
+
`docs/testguard-explained.html` with `node .github/scripts/build-one-pager.mjs`
|
|
188
|
+
(headless Chrome, so the one pinned runtime dependency stays one).
|
|
189
|
+
|
|
190
|
+
### Changed
|
|
191
|
+
- **A probe says which method produced it, and fault injection is no longer
|
|
192
|
+
assumed** (#81). `claims` required `file`, `find` and `replace` on every
|
|
193
|
+
fault; `evidence` required `confirmRuns`, `mode`, `defenders`, `inputs` and
|
|
194
|
+
the baseline and probe runs on every record. All of those are what *fault
|
|
195
|
+
injection* means by showing its work — and DocGuard verifies claims by
|
|
196
|
+
reading code while websec-validator scans a surface, so neither could
|
|
197
|
+
conform without emitting empty arrays, which is lying, and the one thing
|
|
198
|
+
this format exists to make impossible.
|
|
199
|
+
A probe now carries `method`, and a run says which method produced it.
|
|
200
|
+
**It defaults to `fault-injection`, so every document written before the
|
|
201
|
+
field existed validates unchanged** — verified against this repository's own
|
|
202
|
+
88-claim file and every conformance example, none of which carries the field.
|
|
203
|
+
The injection requirements did not disappear, they moved: the schema asks
|
|
204
|
+
for them only under `fault-injection`, and the validator enforces them there
|
|
205
|
+
in full. Relaxing the schema so a scanning tool can conform is not a licence
|
|
206
|
+
for an injecting tool to stop showing its work, and a self-claim now says so.
|
|
207
|
+
`assertion` and `scan` are **reserved**: their required fields are
|
|
208
|
+
deliberately unspecified, to be defined by their first real consumer with
|
|
209
|
+
conformance examples written from the shape that tool actually has.
|
|
210
|
+
Designing them for an absent tool is exactly the mistake that made this
|
|
211
|
+
change necessary — the spec was drafted against a single consumer and
|
|
212
|
+
hard-coded its assumptions as requirements.
|
|
213
|
+
- **The self-probe gate, re-measured and cut** (#73). `--cost` at `ffaaa81`
|
|
214
|
+
reported **1414 s across 546 defender runs for 91 faults** — back where #67
|
|
215
|
+
started, because #77 took the corpus from 59 faults to 91. The gate did not
|
|
216
|
+
decay; the workload grew, and nothing gated on that.
|
|
217
|
+
The issue's original table pointed at `test/replay.test.mjs`. Measurement
|
|
218
|
+
disagreed: that file is 138 s across *six* claims, about 23 s each, the
|
|
219
|
+
cheapest per-claim cost in the top ten — the table had read a cumulative
|
|
220
|
+
column as a per-claim one. The real cost was four claims paying an
|
|
221
|
+
end-to-end fixture probe, 33 to 37 s a run, to falsify decisions that are
|
|
222
|
+
pure functions.
|
|
223
|
+
Each was re-pointed at a test that does not need the expensive setup, and
|
|
224
|
+
**each was then re-probed rather than assumed**: 7 faults, 7 killed, 92 s
|
|
225
|
+
where they had cost about 860 s.
|
|
226
|
+
- `checkProvenance` is now exported and unit-tested. It was already a
|
|
227
|
+
standalone function, and only a 33 s Python fixture probe could falsify it.
|
|
228
|
+
- `argvFor` is pure, and its own comment says it is "exported so it is
|
|
229
|
+
falsifiable without spawning anything" — while the claim still paid two
|
|
230
|
+
fixture probes. `test/runner-argv.test.mjs` asserts the argv directly in
|
|
231
|
+
126 ms, and covers `selectRunner`'s unknown-name guard too.
|
|
232
|
+
- The isolation fault needed new work before it could move: forcing
|
|
233
|
+
isolation to in-place left every fast test green, because they all probe
|
|
234
|
+
in place anyway, so only the 37 s fixture noticed. Caught by injecting the
|
|
235
|
+
fault by hand before trusting the re-point — the same trap that made two
|
|
236
|
+
of five claims survive in #67.
|
|
237
|
+
- **`TG-PIPELINE-REPRODUCES-THE-ORACLE` is restated as
|
|
238
|
+
`TG-PROBE-NEVER-EDITS-THE-TREE-IT-MEASURES`**, both faults carried over
|
|
239
|
+
unchanged, the old id retired through the ignore file with a reason.
|
|
240
|
+
The old statement promised that a probe "reproduces the known-answer
|
|
241
|
+
fixture's verdicts exactly", and **no fault in it ever falsified that
|
|
242
|
+
half**: one removes the restore, the other forces in-place isolation, and
|
|
243
|
+
both are about tree safety. A claim that is defended and still over-promises
|
|
244
|
+
is the one thing a probe cannot catch, so the statement was narrowed to what
|
|
245
|
+
its faults actually prove — swapping the defender alone would have kept the
|
|
246
|
+
over-promise and merely made it cheaper. The fixture oracle still runs on
|
|
247
|
+
every `npm test` and in CI; it is no longer the defender of a claim whose
|
|
248
|
+
cost it dominated.
|
|
249
|
+
- **The shared spec is named `claimspec`.** Schemas are now identified as
|
|
250
|
+
`urn:claimspec:v1:<kind>` rather than `urn:guard-spec:v1:<kind>`.
|
|
251
|
+
**No emitted document changes**: the identifier lives only in the schemas'
|
|
252
|
+
own `$id`/`$ref`, the validator's registry and the documentation, so every
|
|
253
|
+
evidence file, baseline, brief and status already written stays valid and
|
|
254
|
+
keeps validating. Nothing needs migrating.
|
|
255
|
+
The old name said which family of tools the formats came from. The name that
|
|
256
|
+
matters is what they are *for* — claims, the evidence that tried to falsify
|
|
257
|
+
them, and gating only the delta — and most of the tools that could adopt
|
|
258
|
+
them (a scanner with an existing-violations problem, a mutation tester, an
|
|
259
|
+
agent harness that needs `brief` and `status`) do no fault injection at all.
|
|
260
|
+
`spec/README.md` now says so, and says plainly which kinds a second tool can
|
|
261
|
+
adopt today and which cannot (#81).
|
|
262
|
+
- **The README leads with the plain explanation.** The research evidence is
|
|
263
|
+
still there and still matters, but it is not the first thing a newcomer
|
|
264
|
+
meets: an "In one minute" section now opens with what coverage cannot tell
|
|
265
|
+
you, the `objectContaining` exhibit, killed vs SURVIVED, the seven verdicts,
|
|
266
|
+
and — stated plainly — that mutation testing is from the 1970s and the novel
|
|
267
|
+
part is binding every fault to a claim.
|
|
268
|
+
- The one-line descriptions on npm, PyPI and the GitHub Action now say what the
|
|
269
|
+
tool does rather than what category it belongs to.
|
|
270
|
+
|
|
271
|
+
### Fixed
|
|
272
|
+
- **A vanished scratch worktree no longer costs the whole probe** (#64). A
|
|
273
|
+
parallel session cleaning up its *own* leaked scratch worktrees deleted this
|
|
274
|
+
run's while it was mid-fault; `restore()` then tried to write the original
|
|
275
|
+
back into a directory that was gone, threw `ENOENT` out of a `finally`, and
|
|
276
|
+
the probe died with no evidence file at all — every verdict already decided
|
|
277
|
+
thrown away, and an exit code `GATE-SEMANTICS.md` does not define.
|
|
278
|
+
In worktree mode the restore is belt-and-braces: the mutation only ever
|
|
279
|
+
existed inside a scratch worktree that is discarded at the end of the run, so
|
|
280
|
+
a target that has vanished is already restored in every sense that matters.
|
|
281
|
+
It is now recorded as `detail.restoreSkipped: "target-missing"` — a reader
|
|
282
|
+
can still see that the tree moved underneath the run — and the probe
|
|
283
|
+
continues. Under `--in-place` the same condition is the user's own file gone
|
|
284
|
+
and is still raised, loudly; so is any other failure to write the original
|
|
285
|
+
back, in both modes, because a permission error can mean a mutated file left
|
|
286
|
+
on disk.
|
|
287
|
+
More generally, **one bad fault no longer costs the evidence for the other
|
|
288
|
+
forty**. An unexpected throw inside a fault's probe ends *that fault* as
|
|
289
|
+
`unverifiable` with reason `probe-error` and a `detail.message` naming what
|
|
290
|
+
threw; the run finishes and writes its document. The verdict gates, so the
|
|
291
|
+
failure is loud rather than absorbed into a green run, and a `probe-error`
|
|
292
|
+
record is never reused by a later run — it says something about the run, not
|
|
293
|
+
about the code. A **precondition** failure is the deliberate exception: the
|
|
294
|
+
interpreter loading the source from outside the probed tree, or a runner that
|
|
295
|
+
does not resolve, is a statement about every verdict in the run and still
|
|
296
|
+
refuses it outright.
|
|
297
|
+
- **A prior verdict is no longer reused across a fault edit.** `probe` decided
|
|
298
|
+
reuse from the source hash, the defender hashes and the defender set — never
|
|
299
|
+
from the fault itself. So editing a fault's `find` or `replace` kept the
|
|
300
|
+
verdict measured against the fault it replaced: a weakened fault held the
|
|
301
|
+
`survived` it earned before, and a repaired anchor held `unverifiable`, with
|
|
302
|
+
the evidence reporting a result nobody had measured against the file's
|
|
303
|
+
current contents. `status` flagged the changed hash afterwards, but by then
|
|
304
|
+
the evidence had already said something untrue — and for a tool whose
|
|
305
|
+
premise is that a fault edit can never be invisible, that was the wrong
|
|
306
|
+
default. Reuse is now also keyed on the fault's content hash; a prior record
|
|
307
|
+
that carries none is probed again, because the cost of re-measuring is a run
|
|
308
|
+
and the cost of the other direction is a verdict nobody took.
|
|
309
|
+
Found by using the tool on itself: repairing two rotted anchors left both
|
|
310
|
+
claims reporting `UNVERIFIABLE (reused)` on the next probe.
|
|
311
|
+
|
|
312
|
+
### Added
|
|
313
|
+
|
|
314
|
+
- **A survivor now has to prove the defenders execute the subject** (#74). A
|
|
315
|
+
green baseline makes one failure mode structurally impossible — a misread
|
|
316
|
+
runner cannot flatter the suite, because a kill requires the defenders to
|
|
317
|
+
pass on unmodified source first. It says nothing about the opposite
|
|
318
|
+
direction. If the fault is applied to code the test process never executes,
|
|
319
|
+
the baseline is green, every probe run is green, and **every claim is
|
|
320
|
+
reported SURVIVED**. That output reads as a devastating audit finding and is
|
|
321
|
+
entirely false, and nothing in the run contradicts it, because every
|
|
322
|
+
individual check passed.
|
|
323
|
+
This is not hypothetical: a strict editable Python install registers a meta
|
|
324
|
+
path finder that is consulted before `sys.path`, so the interpreter loads the
|
|
325
|
+
original module while TestGuard faults the copy in its scratch worktree.
|
|
326
|
+
Measured, not argued — a fault independently verified to fail 2 of 4 tests
|
|
327
|
+
reported `4 passed`.
|
|
328
|
+
A would-be `survived` is now charged one more run: the subject is replaced
|
|
329
|
+
with content its loader cannot parse, and the defenders run once. Going red
|
|
330
|
+
proves they execute it (`detail.negativeControl: "reached"`) and the verdict
|
|
331
|
+
stands. Staying green proves they do not, and the verdict becomes
|
|
332
|
+
`unverifiable` with reason `subject-not-executed`. Charged **only** on a
|
|
333
|
+
survivor — a kill already proves the defenders reached the code — and cached
|
|
334
|
+
per subject and defender set. Each runner supplies what "cannot compile"
|
|
335
|
+
means for its language; one that cannot say does not guess, and no control is
|
|
336
|
+
run.
|
|
337
|
+
The predicate is *"the run did not stay green"*, not *"a test failed"*: an
|
|
338
|
+
unparseable module usually fails to **load**, which the runner reports as an
|
|
339
|
+
error with no tests run at all.
|
|
340
|
+
|
|
341
|
+
- **A runner pinned to one engine keeps every capability the module has.**
|
|
342
|
+
`pinned()` rebuilt the runner as an explicit allow-list of properties, so
|
|
343
|
+
`--runner pytest` and `--runner unittest` silently lost the negative control
|
|
344
|
+
that `--runner python` has: a survivor probed through either flag was never
|
|
345
|
+
checked, and an unreachable subject reported `survived` exactly as before.
|
|
346
|
+
Found by the Python fixture, not by the unit test, which iterated the four
|
|
347
|
+
runner modules rather than the registry the tool actually resolves through —
|
|
348
|
+
it was checking the code that was written instead of the code that runs. The
|
|
349
|
+
test now iterates `RUNNERS`, which is what makes the next capability added to
|
|
350
|
+
a runner fail loudly instead of quietly.
|
|
351
|
+
|
|
352
|
+
- **`--progress auto|tty|plain|ndjson|none` on `probe`** (#68). A probe used
|
|
353
|
+
to print its stage line only when `stderr` was a terminal, so CI, a
|
|
354
|
+
redirected log and an agent harness saw nothing at all for the whole run —
|
|
355
|
+
indistinguishable from a hang, and a gate you cannot tell from a hang is a
|
|
356
|
+
gate people start killing.
|
|
357
|
+
The carriage-return rewriting is a *rendering* choice, not a reason to
|
|
358
|
+
withhold the information. `auto` (the default) keeps the rewritten line at a
|
|
359
|
+
terminal and prints append-only lines everywhere else. `ndjson` emits one
|
|
360
|
+
JSON object per line for stages **and** for verdicts as they land, so a
|
|
361
|
+
machine can watch a probe without waiting for the document at the end.
|
|
362
|
+
Progress always goes to **stderr**, so `--json` leaves stdout as one
|
|
363
|
+
parseable document. `--quiet` and `--json` imply `none`; an explicit
|
|
364
|
+
`--progress` overrides both, because an operator who asks for a stream of
|
|
365
|
+
events has said what they want.
|
|
366
|
+
Self-claims `TG-PROGRESS-NOT-ONLY-FOR-A-TTY`, `TG-PROGRESS-NEVER-ON-STDOUT`
|
|
367
|
+
and `TG-PROGRESS-EXPLICIT-WINS`, all probed and killed.
|
|
368
|
+
|
|
369
|
+
|
|
370
|
+
- **`--cost`, on `probe` and `claims`** — what the defenders actually cost,
|
|
371
|
+
per claim and per defender file, read back out of the `durationMs` the
|
|
372
|
+
probe already records. It re-measures nothing and spawns nothing.
|
|
373
|
+
A probe's wall clock is `(baseline runs + faults × --confirm) × the cost of
|
|
374
|
+
the claim's defender SET`, per claim, so one slow acceptance test named by
|
|
375
|
+
five claims is paid for thirty times — and until now nothing in the output
|
|
376
|
+
said which file that was. The report names it, under **shared defenders**,
|
|
377
|
+
with the claims that pay for it.
|
|
378
|
+
Per-file figures are an **upper bound, not a share**: a run executes the
|
|
379
|
+
whole defender set at once, so the runner never says how much belonged to
|
|
380
|
+
which file. They overlap by construction and do not sum to the total, which
|
|
381
|
+
the report says on its own face. Only `totalMs` is additive.
|
|
382
|
+
|
|
383
|
+
- **DocGuard governs this repository's documentation.** Six canonical
|
|
384
|
+
documents under `docs-canonical/` — architecture, data model, security,
|
|
385
|
+
test spec, environment and requirements — written against v0.6.0 rather
|
|
386
|
+
than generated as templates, plus `DRIFT-LOG.md` for deliberate,
|
|
387
|
+
recorded deviations. Every validator passes: structure, doc sections,
|
|
388
|
+
changelog, test spec, environment, security, freshness, traceability,
|
|
389
|
+
docs-coverage, document lifecycle and TODO tracking. Structural maturity
|
|
390
|
+
99/100.
|
|
391
|
+
- **Requirements are traceable to tests.** Fourteen functional and eight
|
|
392
|
+
non-functional requirements, each carrying an `@req` annotation in the
|
|
393
|
+
test that verifies it, and a traceability matrix that must agree with
|
|
394
|
+
those annotations. 27/27 traceability checks pass.
|
|
395
|
+
- **Doc drift is caught automatically, not reviewed by hand.** `ci.yml`
|
|
396
|
+
runs `docguard guard` on every pull request — authoritative, unbypassable
|
|
397
|
+
and visible in review — and reports the maturity score without gating on
|
|
398
|
+
it. `.pre-commit-config.yaml` gives contributors the same checks locally,
|
|
399
|
+
opt-in per clone, alongside the claims check, the change gate and the
|
|
400
|
+
probe on pre-push.
|
|
401
|
+
- Every configuration file in the repository is now documented in
|
|
402
|
+
`ARCHITECTURE.md`, including the distinction between
|
|
403
|
+
`.pre-commit-hooks.yaml` (what we publish for consumers) and
|
|
404
|
+
`.pre-commit-config.yaml` (what contributors to this repository use).
|
|
405
|
+
README gains the `Usage` section the Standard README spec expects.
|
|
406
|
+
- **A Python runner** (`--runner python|pytest|unittest`, and `auto` when no
|
|
407
|
+
JavaScript runner resolves). `pytest` when the project's interpreter can
|
|
408
|
+
import it, stdlib `unittest` when it cannot; the engine that actually ran is
|
|
409
|
+
what `run.runner.name` records, and `--runner pytest` / `--runner unittest`
|
|
410
|
+
pin the choice and fail rather than fall back. **Nothing is installed into
|
|
411
|
+
the project under test**: the reporters ship inside TestGuard and reach the
|
|
412
|
+
interpreter through `PYTHONPATH`, so a codebase whose test dependencies are
|
|
413
|
+
the standard library stays one. A `.py` defender runs under Python whatever
|
|
414
|
+
the project runner is, so one claim can be defended by a vitest test and a
|
|
415
|
+
pytest test at once.
|
|
416
|
+
- `--python <path>` (or `TESTGUARD_PYTHON`) names the interpreter. Otherwise:
|
|
417
|
+
the active `VIRTUAL_ENV`, then the project's `.venv`/`venv`/`.env`, then
|
|
418
|
+
`python3` on PATH. An interpreter you name is the only one tried — TestGuard
|
|
419
|
+
never quietly substitutes another, which would change what the tests ran
|
|
420
|
+
against without changing anything the evidence says.
|
|
421
|
+
- **An import-provenance precondition, and `detail.targetNotImported`.** A
|
|
422
|
+
green baseline proves the harness is not reporting everything as broken. It
|
|
423
|
+
proves nothing about the opposite direction, and Python can reach it: a
|
|
424
|
+
strict editable install puts an import hook ahead of `sys.path`, so the
|
|
425
|
+
interpreter loads the original file while TestGuard faults the copy. The
|
|
426
|
+
baseline stays green, every claim is reported `SURVIVED`, and the result
|
|
427
|
+
reads as a devastating finding while being entirely false. With the fault
|
|
428
|
+
applied, TestGuard now asks which file was actually imported: loaded from
|
|
429
|
+
outside the tree being probed **refuses the run**; never imported at all is
|
|
430
|
+
recorded as `detail.targetNotImported` and warned about, because an import
|
|
431
|
+
in a branch the fault does not reach is legitimate.
|
|
432
|
+
- **`target-attribute-patched`**, a new defender signal. `patch("pkg.mod.fn")`
|
|
433
|
+
is not `vi.mock("./mod")`: it replaces one attribute, so the file still
|
|
434
|
+
detects a fault anywhere else in that module. Applying the JavaScript rule
|
|
435
|
+
would have dropped such files and reported `nocover` for well-defended
|
|
436
|
+
claims. Only a patch of the module itself removes a defender; an attribute
|
|
437
|
+
patch keeps it and names the attributes.
|
|
438
|
+
- Python fault producers for `testguard scaffold`: `condition-forced`,
|
|
439
|
+
`statement-deleted`, `return-altered`, `literal-changed`, `call-removed`,
|
|
440
|
+
`field-dropped` and `argument-swapped`, in Python syntax. A statement is
|
|
441
|
+
removed by replacing it with `pass` rather than deleting the line, and a line
|
|
442
|
+
that leaves a bracket open is never removed — both because a fault that
|
|
443
|
+
cannot compile is a probe run that says nothing about the tests.
|
|
444
|
+
- Python defender discovery, mock-awareness and blast radius: matched by module
|
|
445
|
+
name, the way Python itself matches, so `import`, `from … import`, aliases,
|
|
446
|
+
relative imports and `importlib.import_module` all resolve.
|
|
447
|
+
- `fixtures/known-answer-python/`: sixteen faults covering every verdict, each
|
|
448
|
+
verified by hand before it became the oracle, and reproduced under **both**
|
|
449
|
+
engines.
|
|
450
|
+
|
|
451
|
+
### Changed
|
|
452
|
+
|
|
453
|
+
- **The self-probe gate no longer runs a fifty-second acceptance test thirty
|
|
454
|
+
times.** Five claims named `test/probe.fixture.test.mjs` as their only
|
|
455
|
+
defender, which is what made the release gate take ~24 minutes. The
|
|
456
|
+
decisions those claims are about — how escalation attributes an undeclared
|
|
457
|
+
killer, what counts as a measurable run for the flake rate, which binary a
|
|
458
|
+
runner invokes, what makes a fault edit visible — are rules you can state in
|
|
459
|
+
three lines, and none of them needs a fixture to falsify. They are now pure
|
|
460
|
+
functions in `src/probe/attribution.mjs` and exported `argvFor` builders in
|
|
461
|
+
the runners, unit-tested in `test/attribution.test.mjs` (~0.5s).
|
|
462
|
+
This mirrors why `classify()` has always been pure: the order of its checks
|
|
463
|
+
*is* the spec, and every branch should be falsifiable without a runner.
|
|
464
|
+
No claim was weakened to get there. Each re-pointed claim was re-probed and
|
|
465
|
+
still **kills** its faults; two of them SURVIVED on the first narrowing
|
|
466
|
+
attempt, which is exactly the failure mode `--cost` now makes visible before
|
|
467
|
+
someone ships a cheaper, weaker gate.
|
|
468
|
+
**Measured on the same machine, serially, from scratch: 24.1 min → 9.6 min**
|
|
469
|
+
(68 faults, 68 killed, 0 unproven). `--cost` names what is left: the jest
|
|
470
|
+
and replay suites now dominate, at 207s and 148s of defender time.
|
|
471
|
+
|
|
472
|
+
- Style exceptions in the canonical documents are declared with a reason
|
|
473
|
+
(`docguard:quality ... off — why`) rather than satisfied by rewording.
|
|
474
|
+
This tool is defined by what must not happen — no network, no telemetry,
|
|
475
|
+
never optimistic under uncertainty — and stating those positively would
|
|
476
|
+
misdescribe them. A declared, reasoned exception is auditable; contorted
|
|
477
|
+
prose is not.
|
|
478
|
+
- `--runner` accepts `python`, `pytest` and `unittest`; `--runner auto` tries
|
|
479
|
+
vitest, then jest, then python. A JavaScript project with neither vitest nor
|
|
480
|
+
jest installed now also reports why Python did not resolve.
|
|
481
|
+
- The GitHub Action and the GitLab template take `runner` and `python` inputs.
|
|
482
|
+
|
|
483
|
+
### Fixed
|
|
484
|
+
|
|
485
|
+
- `gate --changed` treated a changed `.py` file as `non-source` and excluded it,
|
|
486
|
+
so it could never be reported as uncovered. Python is source; Python test
|
|
487
|
+
files (`test_*.py`, `*_test.py`) are tests.
|
|
488
|
+
- `walk()` descended into `.venv`, `venv`, `site-packages`, `__pycache__`,
|
|
489
|
+
`.tox`, `.nox` and the pytest/mypy/ruff caches. Every `test_*.py` of every
|
|
490
|
+
installed package was collected as a test file of the project under probe.
|
|
491
|
+
- `testguard claims` printed a JavaScript comment marker (`// unasserted:`)
|
|
492
|
+
when suggesting how to annotate a mock in a Python file.
|
|
493
|
+
- A `--runner-cmd` whose reporter emits something other than the jest-compatible
|
|
494
|
+
JSON shape produced `unverifiable / defenders-failed-to-load` with no message
|
|
495
|
+
at all. The verdict was already safe — a report with no recognised tests is a
|
|
496
|
+
non-green baseline, never a pass — but nothing said why. It now names the
|
|
497
|
+
mismatch.
|
|
498
|
+
|
|
10
499
|
## [0.6.0] - 2026-09-17
|
|
11
500
|
|
|
12
501
|
Everything the field asked for. Two independent field reports on private
|