testguard-cli 0.6.0 → 0.8.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (45) hide show
  1. package/CHANGELOG.md +489 -0
  2. package/README.md +287 -34
  3. package/package.json +2 -2
  4. package/spec/GATE-SEMANTICS.md +210 -8
  5. package/spec/README.md +32 -11
  6. package/spec/lib/validate.mjs +125 -14
  7. package/spec/lib/wilson.mjs +112 -0
  8. package/spec/schemas/baseline.schema.json +7 -7
  9. package/spec/schemas/brief.schema.json +12 -12
  10. package/spec/schemas/calibration.schema.json +96 -26
  11. package/spec/schemas/claims.schema.json +129 -25
  12. package/spec/schemas/common.schema.json +127 -29
  13. package/spec/schemas/evidence.schema.json +175 -73
  14. package/spec/schemas/gate.schema.json +12 -12
  15. package/spec/schemas/ignore.schema.json +4 -4
  16. package/spec/schemas/replay.schema.json +10 -10
  17. package/spec/schemas/status.schema.json +25 -25
  18. package/src/cli.mjs +20 -5
  19. package/src/commands/claims.mjs +22 -2
  20. package/src/commands/probe.mjs +24 -4
  21. package/src/commands/replay.mjs +1 -1
  22. package/src/gate/changed.mjs +9 -2
  23. package/src/probe/attribution.mjs +123 -0
  24. package/src/probe/classify.mjs +18 -2
  25. package/src/probe/cost.mjs +205 -0
  26. package/src/probe/discover.mjs +14 -6
  27. package/src/probe/inject.mjs +41 -9
  28. package/src/probe/mocks.mjs +2 -0
  29. package/src/probe/probe.mjs +235 -52
  30. package/src/probe/progress.mjs +87 -0
  31. package/src/probe/pyimports.mjs +287 -0
  32. package/src/probe/rank.mjs +2 -0
  33. package/src/probe/runners/index.mjs +38 -11
  34. package/src/probe/runners/jest.mjs +13 -1
  35. package/src/probe/runners/playwright.mjs +9 -0
  36. package/src/probe/runners/python/_testguard_pytest_plugin.py +132 -0
  37. package/src/probe/runners/python/_testguard_report.py +141 -0
  38. package/src/probe/runners/python/_testguard_unittest_main.py +168 -0
  39. package/src/probe/runners/python.mjs +316 -0
  40. package/src/probe/runners/shared.mjs +10 -1
  41. package/src/probe/runners/vitest.mjs +13 -1
  42. package/src/replay/replay.mjs +50 -22
  43. package/src/scaffold/producers.python.mjs +209 -0
  44. package/src/scaffold/scaffold.mjs +86 -32
  45. package/src/util/glob.mjs +7 -1
package/CHANGELOG.md CHANGED
@@ -7,6 +7,495 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
7
7
 
8
8
  ## [Unreleased]
9
9
 
10
+ ## [0.8.0] - 2026-09-18
11
+
12
+ A calibration you can quote.
13
+
14
+ A calibration document now says what its number means and where its labels
15
+ came from, and it reproduces: `p` and every interval are recomputed by the
16
+ validator from `n`, `positives`, `method` and `confidence`, at the document's
17
+ own precision — the spec's own example had carried a truncated bound since
18
+ the day it was written, and nothing could notice. `nocover` counts as the
19
+ miss it is, so a project with no tests at all for a subsystem no longer
20
+ outscores one with weak tests. The format also carries what
21
+ websec-validator's shipped table already said — corpus, caveat, a producer
22
+ floor, backoff tiers, a labelled prior — so a second tool's honesty survives
23
+ translation; that table is now a conformance example, and "adoptable" means
24
+ the file validates rather than a sentence saying so. Underneath: the
25
+ self-probe gate is held to a cost budget (#89), and a reserved method has
26
+ somewhere to put its data (#91).
27
+
28
+ ### Added
29
+ - **A calibration can now say what it measures and where it came from** (#82).
30
+ websec-validator's shipped `calibration.json` already carried `corpus`,
31
+ `min_n`, `caveat`, `evidence_status` and `limitation` — the fields that say
32
+ "indicative, do not quote this number" — and claimspec had nowhere to put
33
+ them, so adopting it would have stripped that honesty in translation. It
34
+ also had nowhere to say what `p` *was*: testguard's is a miss rate where
35
+ high is bad, websec's is P(real) where high is good, and both validated
36
+ identically. The schema gains `measures` (a registry: `escape-missed`,
37
+ `finding-real`), provenance in `source` (`corpus`, `caveat`, `limitation`,
38
+ `evidenceStatus`, a `tool-oracle` kind, and an unconstrained `detail` on the
39
+ `methodDetail` pattern), a producer floor `minN` that consumers may raise
40
+ and never lower, ordered `backoff` tiers so a class→label→prior cascade is
41
+ expressible, a `fallback` whose shape says it is a guess, a per-cell
42
+ `breakdown` that sums to `n`, `p: null` at `n = 0`, and `|`-compound bucket
43
+ keys whose arity the validator checks. Every field is optional and absence
44
+ means *unattributed* — readable, not quotable — so no existing document
45
+ breaks; testguard's emitter always writes them and a self-claim guards it.
46
+ The proof of adoption is a file, not a sentence: websec-validator's table,
47
+ translated field for field, is now `spec/conformance/examples/calibration-websec.json`
48
+ and validates with every number reproducing. `GATE-SEMANTICS.md` records
49
+ the merge rule (sum counts, recompute, never average; only when `measures`
50
+ and `bucketBy` agree) that websec's shipped-plus-local overlay already
51
+ depends on.
52
+ - **The gate is now gated on its own cost** (#89). `--cost` has printed the
53
+ self-probe's wall clock into every CI log since #67, and the gate still went
54
+ from 9.6 to 23.6 minutes across one merged change without anything saying a
55
+ word. A measurement nothing gates on is not a check — which is the premise
56
+ this tool rests on, applied to itself.
57
+ `testguard.cost-budget.json` carries the ceiling, its measurement and why it
58
+ is set where it is. CI fails when the probe exceeds it and opens with the
59
+ most expensive claims, so the first thing a failure shows is where to look.
60
+ Raising it is a committed diff with a reason: growth is allowed, going
61
+ unnoticed is not.
62
+ The budget is on the **total**, never a per-fault average. The regression
63
+ that prompted this took the corpus from 59 faults to 91 while per-fault cost
64
+ barely moved — an average would have reported everything fine while the gate
65
+ tripled, and a test says so. A budget that is missing or not a positive
66
+ number is refused rather than treated as unlimited, so a typo cannot quietly
67
+ disable the gate.
68
+
69
+
70
+ ### Fixed
71
+ - **The cost budget is set against what CI measures.** 950 s was 18% over
72
+ the local 804 s figure and only 4.6% over the 906 s CI already measured on
73
+ `main`, so the first pull request to add any claim tripped it — #94 passed
74
+ its own leg only because the cache reused verdicts, and `main` went red at
75
+ 981 s from scratch the moment it merged. Raised to 1100 s (~14% over the
76
+ 962 s CI measures with #82's re-pointed defenders), with the CI figures
77
+ recorded beside the local one so the next person budgets against the
78
+ number the check actually sees. The heavy claims the failure now names are
79
+ the next #73-style cut.
80
+ - **Calibration's pure tests moved off the replay fixture** (the #73 pattern).
81
+ Three calibration claims were defended by `test/replay.test.mjs`, which
82
+ builds a scripted git corpus on every run; their kills come from pure
83
+ `calibrationFrom` tests that need none of it. The cost budget fired on the
84
+ first full run — 1022 s against 950 s, the top rows being exactly those
85
+ claims — so the tests now live in `test/calibration.test.mjs`, the claims
86
+ point there, and every fault was re-probed after the move, not assumed.
87
+ - **A cost-budget failure names its claims.** `checkCostBudget` read `id`
88
+ from a report whose claims carry `claimId`, so the one line meant to say
89
+ *which* claims to look at printed `undefined` five times — and its unit test
90
+ fed the shape the code wished for, so it passed. Found the first time the
91
+ budget actually fired. The test now goes through `costReport`, and
92
+ `TG-COST-BUDGET-NAMES-THE-CLAIM` guards it.
93
+ - **A calibration counts `nocover` as a miss** (#82). `nocover` — no test
94
+ even imports the broken file — is the worst replay outcome, and the ratio
95
+ excluded it: a project with no tests at all for a subsystem scored *better*
96
+ than one with weak tests, because its worst outcomes left the denominator
97
+ before the ratio was taken. It now enters both sides, as it does in Stryker
98
+ and PIT, and the replay summary says how many misses were blind and how many
99
+ had no test at all. The schema description, the emitter's docstring and
100
+ `GATE-SEMANTICS.md` had also disagreed about what `p` was — "P(finding is
101
+ real)" in one place, the miss rate in another, and the docstring's two
102
+ sentences conditioning on different denominators — and now say one thing:
103
+ for testguard, the share of real escaped bugs the suite failed to catch.
104
+ - **A calibration's arithmetic is checked, not trusted.** The validator
105
+ recomputed nothing: it checked that `p` sat inside `ci` and would pass a
106
+ document with every number invented. It now recomputes `p` from
107
+ `positives / n` and `ci` from `n`, `positives`, `method` and `confidence`
108
+ with `spec/lib/wilson.mjs` — one implementation, shared with the emitter —
109
+ at the document's own precision, accepting either the exact quantile or the
110
+ textbook 1.96. The spec's own example had carried a truncated lower bound
111
+ (6/30 → `0.09`; Wilson gives 0.0950 → `0.10`) since the day it was written,
112
+ and websec-validator's shipped table reproduces exactly.
113
+ - **A reserved method can now say something.** #81 reserved `assertion` and
114
+ `scan` without required fields, and `fault` / `detail` are both
115
+ `additionalProperties: false` — so a probe using a reserved method could
116
+ declare that it existed and **nothing whatsoever about it**: not which rule
117
+ was checked, not where. That is not a reservation, it is a dead end, and
118
+ DocGuard's writer would have hit it on its first line.
119
+ Both now carry `methodDetail`: an object the spec deliberately does not
120
+ constrain, on the probe and on the record. It is **forbidden under
121
+ `fault-injection`**, whose shape is specified and must not acquire a junk
122
+ drawer — a self-claim fails if that ever stops being enforced — and it is a
123
+ staging area rather than a permanent home. What the first real consumer puts
124
+ there is the evidence for what the specified shape should become.
125
+
126
+ ## [0.7.0] - 2026-09-18
127
+
128
+ Python, and a fault that cannot hide.
129
+
130
+ TestGuard now probes Python projects with nothing installed into them, and it
131
+ no longer takes a green suite at its word: a fault applied to code the tests
132
+ never execute used to be reported as SURVIVED — an audit finding that reads as
133
+ devastating and is entirely false — and is now `unverifiable`, because the
134
+ defenders are made to prove they can fail because of that file before a
135
+ survival about it is allowed to stand.
136
+
137
+ Alongside: the shared spec is named `claimspec` and no longer assumes fault
138
+ injection is the only way to falsify a claim, one bad fault can no longer cost
139
+ the evidence for the whole run, and the self-probe gate was measured rather
140
+ than guessed at.
141
+
142
+ ### Upgrading
143
+
144
+ **Some claims that were `survived` will become `unverifiable`, and your frozen
145
+ baseline will not suppress them.** This is the intended consequence of the
146
+ negative control, and it is worth reading before you re-freeze anything.
147
+
148
+ A fingerprint is derived from the claim, the subject, the file and **the
149
+ verdict**. A claim whose subject the defenders never execute was reported
150
+ `survived`; it is now `unverifiable` with reason `subject-not-executed`. That
151
+ is a different fingerprint, so a baseline frozen before this release does not
152
+ contain it, and it gates as new.
153
+
154
+ **The finding is real.** It says the defenders do not execute that file at all,
155
+ so every earlier verdict about it was a statement about their reach rather than
156
+ their assertions — including the `survived` your baseline was suppressing. A
157
+ `survived` you had accepted as known debt was, in these cases, not a measurement
158
+ of anything.
159
+
160
+ What to do, in order:
161
+
162
+ 1. `testguard probe` and read the records carrying
163
+ `detail.negativeControl: "not-reached"`. Each names a file no defender
164
+ loads.
165
+ 2. **Prefer fixing the defenders.** A claim whose subject is never imported is
166
+ the cheapest blind spot you will ever find — cheaper than a survivor,
167
+ because nothing was even tried.
168
+ 3. If you are accepting it as existing debt for now, `testguard baseline`
169
+ re-freezes today's findings so only newer ones gate. Do this **after**
170
+ looking at them, not instead of.
171
+
172
+ `baseline --restamp` is **not** the command for this: it moves a baseline's
173
+ `head` onto a clean identical probe, and it requires the same fingerprints.
174
+
175
+ Nothing else in this release requires migration. The `claimspec` rename changes
176
+ no emitted document, the `method` field defaults to `fault-injection` so every
177
+ existing claims and evidence file stays valid, and the gate work changed which
178
+ tests defend which claims, never a verdict.
179
+
180
+ ### Added
181
+
182
+ - **A five-page technical brief**, [`docs/testguard-explained.pdf`](docs/testguard-explained.pdf):
183
+ the whole idea on page one, the field measurements on page two, then
184
+ mechanics, architecture and prior art. Six diagrams, including a dot matrix
185
+ of the 39 injected faults from the second field report — 21 survived a fully
186
+ green suite, 39/39 killed after tests were written against the survivors. Built from
187
+ `docs/testguard-explained.html` with `node .github/scripts/build-one-pager.mjs`
188
+ (headless Chrome, so the one pinned runtime dependency stays one).
189
+
190
+ ### Changed
191
+ - **A probe says which method produced it, and fault injection is no longer
192
+ assumed** (#81). `claims` required `file`, `find` and `replace` on every
193
+ fault; `evidence` required `confirmRuns`, `mode`, `defenders`, `inputs` and
194
+ the baseline and probe runs on every record. All of those are what *fault
195
+ injection* means by showing its work — and DocGuard verifies claims by
196
+ reading code while websec-validator scans a surface, so neither could
197
+ conform without emitting empty arrays, which is lying, and the one thing
198
+ this format exists to make impossible.
199
+ A probe now carries `method`, and a run says which method produced it.
200
+ **It defaults to `fault-injection`, so every document written before the
201
+ field existed validates unchanged** — verified against this repository's own
202
+ 88-claim file and every conformance example, none of which carries the field.
203
+ The injection requirements did not disappear, they moved: the schema asks
204
+ for them only under `fault-injection`, and the validator enforces them there
205
+ in full. Relaxing the schema so a scanning tool can conform is not a licence
206
+ for an injecting tool to stop showing its work, and a self-claim now says so.
207
+ `assertion` and `scan` are **reserved**: their required fields are
208
+ deliberately unspecified, to be defined by their first real consumer with
209
+ conformance examples written from the shape that tool actually has.
210
+ Designing them for an absent tool is exactly the mistake that made this
211
+ change necessary — the spec was drafted against a single consumer and
212
+ hard-coded its assumptions as requirements.
213
+ - **The self-probe gate, re-measured and cut** (#73). `--cost` at `ffaaa81`
214
+ reported **1414 s across 546 defender runs for 91 faults** — back where #67
215
+ started, because #77 took the corpus from 59 faults to 91. The gate did not
216
+ decay; the workload grew, and nothing gated on that.
217
+ The issue's original table pointed at `test/replay.test.mjs`. Measurement
218
+ disagreed: that file is 138 s across *six* claims, about 23 s each, the
219
+ cheapest per-claim cost in the top ten — the table had read a cumulative
220
+ column as a per-claim one. The real cost was four claims paying an
221
+ end-to-end fixture probe, 33 to 37 s a run, to falsify decisions that are
222
+ pure functions.
223
+ Each was re-pointed at a test that does not need the expensive setup, and
224
+ **each was then re-probed rather than assumed**: 7 faults, 7 killed, 92 s
225
+ where they had cost about 860 s.
226
+ - `checkProvenance` is now exported and unit-tested. It was already a
227
+ standalone function, and only a 33 s Python fixture probe could falsify it.
228
+ - `argvFor` is pure, and its own comment says it is "exported so it is
229
+ falsifiable without spawning anything" — while the claim still paid two
230
+ fixture probes. `test/runner-argv.test.mjs` asserts the argv directly in
231
+ 126 ms, and covers `selectRunner`'s unknown-name guard too.
232
+ - The isolation fault needed new work before it could move: forcing
233
+ isolation to in-place left every fast test green, because they all probe
234
+ in place anyway, so only the 37 s fixture noticed. Caught by injecting the
235
+ fault by hand before trusting the re-point — the same trap that made two
236
+ of five claims survive in #67.
237
+ - **`TG-PIPELINE-REPRODUCES-THE-ORACLE` is restated as
238
+ `TG-PROBE-NEVER-EDITS-THE-TREE-IT-MEASURES`**, both faults carried over
239
+ unchanged, the old id retired through the ignore file with a reason.
240
+ The old statement promised that a probe "reproduces the known-answer
241
+ fixture's verdicts exactly", and **no fault in it ever falsified that
242
+ half**: one removes the restore, the other forces in-place isolation, and
243
+ both are about tree safety. A claim that is defended and still over-promises
244
+ is the one thing a probe cannot catch, so the statement was narrowed to what
245
+ its faults actually prove — swapping the defender alone would have kept the
246
+ over-promise and merely made it cheaper. The fixture oracle still runs on
247
+ every `npm test` and in CI; it is no longer the defender of a claim whose
248
+ cost it dominated.
249
+ - **The shared spec is named `claimspec`.** Schemas are now identified as
250
+ `urn:claimspec:v1:<kind>` rather than `urn:guard-spec:v1:<kind>`.
251
+ **No emitted document changes**: the identifier lives only in the schemas'
252
+ own `$id`/`$ref`, the validator's registry and the documentation, so every
253
+ evidence file, baseline, brief and status already written stays valid and
254
+ keeps validating. Nothing needs migrating.
255
+ The old name said which family of tools the formats came from. The name that
256
+ matters is what they are *for* — claims, the evidence that tried to falsify
257
+ them, and gating only the delta — and most of the tools that could adopt
258
+ them (a scanner with an existing-violations problem, a mutation tester, an
259
+ agent harness that needs `brief` and `status`) do no fault injection at all.
260
+ `spec/README.md` now says so, and says plainly which kinds a second tool can
261
+ adopt today and which cannot (#81).
262
+ - **The README leads with the plain explanation.** The research evidence is
263
+ still there and still matters, but it is not the first thing a newcomer
264
+ meets: an "In one minute" section now opens with what coverage cannot tell
265
+ you, the `objectContaining` exhibit, killed vs SURVIVED, the seven verdicts,
266
+ and — stated plainly — that mutation testing is from the 1970s and the novel
267
+ part is binding every fault to a claim.
268
+ - The one-line descriptions on npm, PyPI and the GitHub Action now say what the
269
+ tool does rather than what category it belongs to.
270
+
271
+ ### Fixed
272
+ - **A vanished scratch worktree no longer costs the whole probe** (#64). A
273
+ parallel session cleaning up its *own* leaked scratch worktrees deleted this
274
+ run's while it was mid-fault; `restore()` then tried to write the original
275
+ back into a directory that was gone, threw `ENOENT` out of a `finally`, and
276
+ the probe died with no evidence file at all — every verdict already decided
277
+ thrown away, and an exit code `GATE-SEMANTICS.md` does not define.
278
+ In worktree mode the restore is belt-and-braces: the mutation only ever
279
+ existed inside a scratch worktree that is discarded at the end of the run, so
280
+ a target that has vanished is already restored in every sense that matters.
281
+ It is now recorded as `detail.restoreSkipped: "target-missing"` — a reader
282
+ can still see that the tree moved underneath the run — and the probe
283
+ continues. Under `--in-place` the same condition is the user's own file gone
284
+ and is still raised, loudly; so is any other failure to write the original
285
+ back, in both modes, because a permission error can mean a mutated file left
286
+ on disk.
287
+ More generally, **one bad fault no longer costs the evidence for the other
288
+ forty**. An unexpected throw inside a fault's probe ends *that fault* as
289
+ `unverifiable` with reason `probe-error` and a `detail.message` naming what
290
+ threw; the run finishes and writes its document. The verdict gates, so the
291
+ failure is loud rather than absorbed into a green run, and a `probe-error`
292
+ record is never reused by a later run — it says something about the run, not
293
+ about the code. A **precondition** failure is the deliberate exception: the
294
+ interpreter loading the source from outside the probed tree, or a runner that
295
+ does not resolve, is a statement about every verdict in the run and still
296
+ refuses it outright.
297
+ - **A prior verdict is no longer reused across a fault edit.** `probe` decided
298
+ reuse from the source hash, the defender hashes and the defender set — never
299
+ from the fault itself. So editing a fault's `find` or `replace` kept the
300
+ verdict measured against the fault it replaced: a weakened fault held the
301
+ `survived` it earned before, and a repaired anchor held `unverifiable`, with
302
+ the evidence reporting a result nobody had measured against the file's
303
+ current contents. `status` flagged the changed hash afterwards, but by then
304
+ the evidence had already said something untrue — and for a tool whose
305
+ premise is that a fault edit can never be invisible, that was the wrong
306
+ default. Reuse is now also keyed on the fault's content hash; a prior record
307
+ that carries none is probed again, because the cost of re-measuring is a run
308
+ and the cost of the other direction is a verdict nobody took.
309
+ Found by using the tool on itself: repairing two rotted anchors left both
310
+ claims reporting `UNVERIFIABLE (reused)` on the next probe.
311
+
312
+ ### Added
313
+
314
+ - **A survivor now has to prove the defenders execute the subject** (#74). A
315
+ green baseline makes one failure mode structurally impossible — a misread
316
+ runner cannot flatter the suite, because a kill requires the defenders to
317
+ pass on unmodified source first. It says nothing about the opposite
318
+ direction. If the fault is applied to code the test process never executes,
319
+ the baseline is green, every probe run is green, and **every claim is
320
+ reported SURVIVED**. That output reads as a devastating audit finding and is
321
+ entirely false, and nothing in the run contradicts it, because every
322
+ individual check passed.
323
+ This is not hypothetical: a strict editable Python install registers a meta
324
+ path finder that is consulted before `sys.path`, so the interpreter loads the
325
+ original module while TestGuard faults the copy in its scratch worktree.
326
+ Measured, not argued — a fault independently verified to fail 2 of 4 tests
327
+ reported `4 passed`.
328
+ A would-be `survived` is now charged one more run: the subject is replaced
329
+ with content its loader cannot parse, and the defenders run once. Going red
330
+ proves they execute it (`detail.negativeControl: "reached"`) and the verdict
331
+ stands. Staying green proves they do not, and the verdict becomes
332
+ `unverifiable` with reason `subject-not-executed`. Charged **only** on a
333
+ survivor — a kill already proves the defenders reached the code — and cached
334
+ per subject and defender set. Each runner supplies what "cannot compile"
335
+ means for its language; one that cannot say does not guess, and no control is
336
+ run.
337
+ The predicate is *"the run did not stay green"*, not *"a test failed"*: an
338
+ unparseable module usually fails to **load**, which the runner reports as an
339
+ error with no tests run at all.
340
+
341
+ - **A runner pinned to one engine keeps every capability the module has.**
342
+ `pinned()` rebuilt the runner as an explicit allow-list of properties, so
343
+ `--runner pytest` and `--runner unittest` silently lost the negative control
344
+ that `--runner python` has: a survivor probed through either flag was never
345
+ checked, and an unreachable subject reported `survived` exactly as before.
346
+ Found by the Python fixture, not by the unit test, which iterated the four
347
+ runner modules rather than the registry the tool actually resolves through —
348
+ it was checking the code that was written instead of the code that runs. The
349
+ test now iterates `RUNNERS`, which is what makes the next capability added to
350
+ a runner fail loudly instead of quietly.
351
+
352
+ - **`--progress auto|tty|plain|ndjson|none` on `probe`** (#68). A probe used
353
+ to print its stage line only when `stderr` was a terminal, so CI, a
354
+ redirected log and an agent harness saw nothing at all for the whole run —
355
+ indistinguishable from a hang, and a gate you cannot tell from a hang is a
356
+ gate people start killing.
357
+ The carriage-return rewriting is a *rendering* choice, not a reason to
358
+ withhold the information. `auto` (the default) keeps the rewritten line at a
359
+ terminal and prints append-only lines everywhere else. `ndjson` emits one
360
+ JSON object per line for stages **and** for verdicts as they land, so a
361
+ machine can watch a probe without waiting for the document at the end.
362
+ Progress always goes to **stderr**, so `--json` leaves stdout as one
363
+ parseable document. `--quiet` and `--json` imply `none`; an explicit
364
+ `--progress` overrides both, because an operator who asks for a stream of
365
+ events has said what they want.
366
+ Self-claims `TG-PROGRESS-NOT-ONLY-FOR-A-TTY`, `TG-PROGRESS-NEVER-ON-STDOUT`
367
+ and `TG-PROGRESS-EXPLICIT-WINS`, all probed and killed.
368
+
369
+
370
+ - **`--cost`, on `probe` and `claims`** — what the defenders actually cost,
371
+ per claim and per defender file, read back out of the `durationMs` the
372
+ probe already records. It re-measures nothing and spawns nothing.
373
+ A probe's wall clock is `(baseline runs + faults × --confirm) × the cost of
374
+ the claim's defender SET`, per claim, so one slow acceptance test named by
375
+ five claims is paid for thirty times — and until now nothing in the output
376
+ said which file that was. The report names it, under **shared defenders**,
377
+ with the claims that pay for it.
378
+ Per-file figures are an **upper bound, not a share**: a run executes the
379
+ whole defender set at once, so the runner never says how much belonged to
380
+ which file. They overlap by construction and do not sum to the total, which
381
+ the report says on its own face. Only `totalMs` is additive.
382
+
383
+ - **DocGuard governs this repository's documentation.** Six canonical
384
+ documents under `docs-canonical/` — architecture, data model, security,
385
+ test spec, environment and requirements — written against v0.6.0 rather
386
+ than generated as templates, plus `DRIFT-LOG.md` for deliberate,
387
+ recorded deviations. Every validator passes: structure, doc sections,
388
+ changelog, test spec, environment, security, freshness, traceability,
389
+ docs-coverage, document lifecycle and TODO tracking. Structural maturity
390
+ 99/100.
391
+ - **Requirements are traceable to tests.** Fourteen functional and eight
392
+ non-functional requirements, each carrying an `@req` annotation in the
393
+ test that verifies it, and a traceability matrix that must agree with
394
+ those annotations. 27/27 traceability checks pass.
395
+ - **Doc drift is caught automatically, not reviewed by hand.** `ci.yml`
396
+ runs `docguard guard` on every pull request — authoritative, unbypassable
397
+ and visible in review — and reports the maturity score without gating on
398
+ it. `.pre-commit-config.yaml` gives contributors the same checks locally,
399
+ opt-in per clone, alongside the claims check, the change gate and the
400
+ probe on pre-push.
401
+ - Every configuration file in the repository is now documented in
402
+ `ARCHITECTURE.md`, including the distinction between
403
+ `.pre-commit-hooks.yaml` (what we publish for consumers) and
404
+ `.pre-commit-config.yaml` (what contributors to this repository use).
405
+ README gains the `Usage` section the Standard README spec expects.
406
+ - **A Python runner** (`--runner python|pytest|unittest`, and `auto` when no
407
+ JavaScript runner resolves). `pytest` when the project's interpreter can
408
+ import it, stdlib `unittest` when it cannot; the engine that actually ran is
409
+ what `run.runner.name` records, and `--runner pytest` / `--runner unittest`
410
+ pin the choice and fail rather than fall back. **Nothing is installed into
411
+ the project under test**: the reporters ship inside TestGuard and reach the
412
+ interpreter through `PYTHONPATH`, so a codebase whose test dependencies are
413
+ the standard library stays one. A `.py` defender runs under Python whatever
414
+ the project runner is, so one claim can be defended by a vitest test and a
415
+ pytest test at once.
416
+ - `--python <path>` (or `TESTGUARD_PYTHON`) names the interpreter. Otherwise:
417
+ the active `VIRTUAL_ENV`, then the project's `.venv`/`venv`/`.env`, then
418
+ `python3` on PATH. An interpreter you name is the only one tried — TestGuard
419
+ never quietly substitutes another, which would change what the tests ran
420
+ against without changing anything the evidence says.
421
+ - **An import-provenance precondition, and `detail.targetNotImported`.** A
422
+ green baseline proves the harness is not reporting everything as broken. It
423
+ proves nothing about the opposite direction, and Python can reach it: a
424
+ strict editable install puts an import hook ahead of `sys.path`, so the
425
+ interpreter loads the original file while TestGuard faults the copy. The
426
+ baseline stays green, every claim is reported `SURVIVED`, and the result
427
+ reads as a devastating finding while being entirely false. With the fault
428
+ applied, TestGuard now asks which file was actually imported: loaded from
429
+ outside the tree being probed **refuses the run**; never imported at all is
430
+ recorded as `detail.targetNotImported` and warned about, because an import
431
+ in a branch the fault does not reach is legitimate.
432
+ - **`target-attribute-patched`**, a new defender signal. `patch("pkg.mod.fn")`
433
+ is not `vi.mock("./mod")`: it replaces one attribute, so the file still
434
+ detects a fault anywhere else in that module. Applying the JavaScript rule
435
+ would have dropped such files and reported `nocover` for well-defended
436
+ claims. Only a patch of the module itself removes a defender; an attribute
437
+ patch keeps it and names the attributes.
438
+ - Python fault producers for `testguard scaffold`: `condition-forced`,
439
+ `statement-deleted`, `return-altered`, `literal-changed`, `call-removed`,
440
+ `field-dropped` and `argument-swapped`, in Python syntax. A statement is
441
+ removed by replacing it with `pass` rather than deleting the line, and a line
442
+ that leaves a bracket open is never removed — both because a fault that
443
+ cannot compile is a probe run that says nothing about the tests.
444
+ - Python defender discovery, mock-awareness and blast radius: matched by module
445
+ name, the way Python itself matches, so `import`, `from … import`, aliases,
446
+ relative imports and `importlib.import_module` all resolve.
447
+ - `fixtures/known-answer-python/`: sixteen faults covering every verdict, each
448
+ verified by hand before it became the oracle, and reproduced under **both**
449
+ engines.
450
+
451
+ ### Changed
452
+
453
+ - **The self-probe gate no longer runs a fifty-second acceptance test thirty
454
+ times.** Five claims named `test/probe.fixture.test.mjs` as their only
455
+ defender, which is what made the release gate take ~24 minutes. The
456
+ decisions those claims are about — how escalation attributes an undeclared
457
+ killer, what counts as a measurable run for the flake rate, which binary a
458
+ runner invokes, what makes a fault edit visible — are rules you can state in
459
+ three lines, and none of them needs a fixture to falsify. They are now pure
460
+ functions in `src/probe/attribution.mjs` and exported `argvFor` builders in
461
+ the runners, unit-tested in `test/attribution.test.mjs` (~0.5s).
462
+ This mirrors why `classify()` has always been pure: the order of its checks
463
+ *is* the spec, and every branch should be falsifiable without a runner.
464
+ No claim was weakened to get there. Each re-pointed claim was re-probed and
465
+ still **kills** its faults; two of them SURVIVED on the first narrowing
466
+ attempt, which is exactly the failure mode `--cost` now makes visible before
467
+ someone ships a cheaper, weaker gate.
468
+ **Measured on the same machine, serially, from scratch: 24.1 min → 9.6 min**
469
+ (68 faults, 68 killed, 0 unproven). `--cost` names what is left: the jest
470
+ and replay suites now dominate, at 207s and 148s of defender time.
471
+
472
+ - Style exceptions in the canonical documents are declared with a reason
473
+ (`docguard:quality ... off — why`) rather than satisfied by rewording.
474
+ This tool is defined by what must not happen — no network, no telemetry,
475
+ never optimistic under uncertainty — and stating those positively would
476
+ misdescribe them. A declared, reasoned exception is auditable; contorted
477
+ prose is not.
478
+ - `--runner` accepts `python`, `pytest` and `unittest`; `--runner auto` tries
479
+ vitest, then jest, then python. A JavaScript project with neither vitest nor
480
+ jest installed now also reports why Python did not resolve.
481
+ - The GitHub Action and the GitLab template take `runner` and `python` inputs.
482
+
483
+ ### Fixed
484
+
485
+ - `gate --changed` treated a changed `.py` file as `non-source` and excluded it,
486
+ so it could never be reported as uncovered. Python is source; Python test
487
+ files (`test_*.py`, `*_test.py`) are tests.
488
+ - `walk()` descended into `.venv`, `venv`, `site-packages`, `__pycache__`,
489
+ `.tox`, `.nox` and the pytest/mypy/ruff caches. Every `test_*.py` of every
490
+ installed package was collected as a test file of the project under probe.
491
+ - `testguard claims` printed a JavaScript comment marker (`// unasserted:`)
492
+ when suggesting how to annotate a mock in a Python file.
493
+ - A `--runner-cmd` whose reporter emits something other than the jest-compatible
494
+ JSON shape produced `unverifiable / defenders-failed-to-load` with no message
495
+ at all. The verdict was already safe — a report with no recognised tests is a
496
+ non-green baseline, never a pass — but nothing said why. It now names the
497
+ mismatch.
498
+
10
499
  ## [0.6.0] - 2026-09-17
11
500
 
12
501
  Everything the field asked for. Two independent field reports on private