testguard-cli 0.6.0 → 0.7.0

package/CHANGELOG.md

@@ -7,6 +7,379 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.7.0] - 2026-09-18
+
+Python, and a fault that cannot hide.
+
+TestGuard now probes Python projects with nothing installed into them, and it
+no longer takes a green suite at its word: a fault applied to code the tests
+never execute used to be reported as SURVIVED — an audit finding that reads as
+devastating and is entirely false — and is now `unverifiable`, because the
+defenders are made to prove they can fail because of that file before a
+survival about it is allowed to stand.
+
+Alongside: the shared spec is named `claimspec` and no longer assumes fault
+injection is the only way to falsify a claim, one bad fault can no longer cost
+the evidence for the whole run, and the self-probe gate was measured rather
+than guessed at.
+
+### Upgrading
+
+**Some claims that were `survived` will become `unverifiable`, and your frozen
+baseline will not suppress them.** This is the intended consequence of the
+negative control, and it is worth reading before you re-freeze anything.
+
+A fingerprint is derived from the claim, the subject, the file and **the
+verdict**. A claim whose subject the defenders never execute was reported
+`survived`; it is now `unverifiable` with reason `subject-not-executed`. That
+is a different fingerprint, so a baseline frozen before this release does not
+contain it, and it gates as new.
+
+**The finding is real.** It says the defenders do not execute that file at all,
+so every earlier verdict about it was a statement about their reach rather than
+their assertions — including the `survived` your baseline was suppressing. A
+`survived` you had accepted as known debt was, in these cases, not a measurement
+of anything.
+
+What to do, in order:
+
+1. `testguard probe` and read the records carrying
+   `detail.negativeControl: "not-reached"`. Each names a file no defender
+   loads.
+2. **Prefer fixing the defenders.** A claim whose subject is never imported is
+   the cheapest blind spot you will ever find — cheaper than a survivor,
+   because nothing was even tried.
+3. If you are accepting it as existing debt for now, `testguard baseline`
+   re-freezes today's findings so only newer ones gate. Do this **after**
+   looking at them, not instead of.
+
+`baseline --restamp` is **not** the command for this: it moves a baseline's
+`head` onto a clean identical probe, and it requires the same fingerprints.
+
+Nothing else in this release requires migration. The `claimspec` rename changes
+no emitted document, the `method` field defaults to `fault-injection` so every
+existing claims and evidence file stays valid, and the gate work changed which
+tests defend which claims, never a verdict.
+
+### Added
+
+- **A five-page technical brief**, [`docs/testguard-explained.pdf`](docs/testguard-explained.pdf):
+  the whole idea on page one, the field measurements on page two, then
+  mechanics, architecture and prior art. Six diagrams, including a dot matrix
+  of the 39 injected faults from the second field report — 21 survived a fully
+  green suite, 39/39 killed after tests were written against the survivors. Built from
+  `docs/testguard-explained.html` with `node .github/scripts/build-one-pager.mjs`
+  (headless Chrome, so the one pinned runtime dependency stays one).
+
+### Changed
+- **A probe says which method produced it, and fault injection is no longer
+  assumed** (#81). `claims` required `file`, `find` and `replace` on every
+  fault; `evidence` required `confirmRuns`, `mode`, `defenders`, `inputs` and
+  the baseline and probe runs on every record. All of those are what *fault
+  injection* means by showing its work — and DocGuard verifies claims by
+  reading code while websec-validator scans a surface, so neither could
+  conform without emitting empty arrays, which is lying, and the one thing
+  this format exists to make impossible.
+  A probe now carries `method`, and a run says which method produced it.
+  **It defaults to `fault-injection`, so every document written before the
+  field existed validates unchanged** — verified against this repository's own
+  88-claim file and every conformance example, none of which carries the field.
+  The injection requirements did not disappear, they moved: the schema asks
+  for them only under `fault-injection`, and the validator enforces them there
+  in full. Relaxing the schema so a scanning tool can conform is not a licence
+  for an injecting tool to stop showing its work, and a self-claim now says so.
+  `assertion` and `scan` are **reserved**: their required fields are
+  deliberately unspecified, to be defined by their first real consumer with
+  conformance examples written from the shape that tool actually has.
+  Designing them for an absent tool is exactly the mistake that made this
+  change necessary — the spec was drafted against a single consumer and
+  hard-coded its assumptions as requirements.
+- **The self-probe gate, re-measured and cut** (#73). `--cost` at `ffaaa81`
+  reported **1414 s across 546 defender runs for 91 faults** — back where #67
+  started, because #77 took the corpus from 59 faults to 91. The gate did not
+  decay; the workload grew, and nothing gated on that.
+  The issue's original table pointed at `test/replay.test.mjs`. Measurement
+  disagreed: that file is 138 s across *six* claims, about 23 s each, the
+  cheapest per-claim cost in the top ten — the table had read a cumulative
+  column as a per-claim one. The real cost was four claims paying an
+  end-to-end fixture probe, 33 to 37 s a run, to falsify decisions that are
+  pure functions.
+  Each was re-pointed at a test that does not need the expensive setup, and
+  **each was then re-probed rather than assumed**: 7 faults, 7 killed, 92 s
+  where they had cost about 860 s.
+  - `checkProvenance` is now exported and unit-tested. It was already a
+    standalone function, and only a 33 s Python fixture probe could falsify it.
+  - `argvFor` is pure, and its own comment says it is "exported so it is
+    falsifiable without spawning anything" — while the claim still paid two
+    fixture probes. `test/runner-argv.test.mjs` asserts the argv directly in
+    126 ms, and covers `selectRunner`'s unknown-name guard too.
+  - The isolation fault needed new work before it could move: forcing
+    isolation to in-place left every fast test green, because they all probe
+    in place anyway, so only the 37 s fixture noticed. Caught by injecting the
+    fault by hand before trusting the re-point — the same trap that made two
+    of five claims survive in #67.
+- **`TG-PIPELINE-REPRODUCES-THE-ORACLE` is restated as
+  `TG-PROBE-NEVER-EDITS-THE-TREE-IT-MEASURES`**, both faults carried over
+  unchanged, the old id retired through the ignore file with a reason.
+  The old statement promised that a probe "reproduces the known-answer
+  fixture's verdicts exactly", and **no fault in it ever falsified that
+  half**: one removes the restore, the other forces in-place isolation, and
+  both are about tree safety. A claim that is defended and still over-promises
+  is the one thing a probe cannot catch, so the statement was narrowed to what
+  its faults actually prove — swapping the defender alone would have kept the
+  over-promise and merely made it cheaper. The fixture oracle still runs on
+  every `npm test` and in CI; it is no longer the defender of a claim whose
+  cost it dominated.
+- **The shared spec is named `claimspec`.** Schemas are now identified as
+  `urn:claimspec:v1:<kind>` rather than `urn:guard-spec:v1:<kind>`.
+  **No emitted document changes**: the identifier lives only in the schemas'
+  own `$id`/`$ref`, the validator's registry and the documentation, so every
+  evidence file, baseline, brief and status already written stays valid and
+  keeps validating. Nothing needs migrating.
+  The old name said which family of tools the formats came from. The name that
+  matters is what they are *for* — claims, the evidence that tried to falsify
+  them, and gating only the delta — and most of the tools that could adopt
+  them (a scanner with an existing-violations problem, a mutation tester, an
+  agent harness that needs `brief` and `status`) do no fault injection at all.
+  `spec/README.md` now says so, and says plainly which kinds a second tool can
+  adopt today and which cannot (#81).
+- **The README leads with the plain explanation.** The research evidence is
+  still there and still matters, but it is not the first thing a newcomer
+  meets: an "In one minute" section now opens with what coverage cannot tell
+  you, the `objectContaining` exhibit, killed vs SURVIVED, the seven verdicts,
+  and — stated plainly — that mutation testing is from the 1970s and the novel
+  part is binding every fault to a claim.
+- The one-line descriptions on npm, PyPI and the GitHub Action now say what the
+  tool does rather than what category it belongs to.
+
+### Fixed
+- **A vanished scratch worktree no longer costs the whole probe** (#64). A
+  parallel session cleaning up its *own* leaked scratch worktrees deleted this
+  run's while it was mid-fault; `restore()` then tried to write the original
+  back into a directory that was gone, threw `ENOENT` out of a `finally`, and
+  the probe died with no evidence file at all — every verdict already decided
+  thrown away, and an exit code `GATE-SEMANTICS.md` does not define.
+  In worktree mode the restore is belt-and-braces: the mutation only ever
+  existed inside a scratch worktree that is discarded at the end of the run, so
+  a target that has vanished is already restored in every sense that matters.
+  It is now recorded as `detail.restoreSkipped: "target-missing"` — a reader
+  can still see that the tree moved underneath the run — and the probe
+  continues. Under `--in-place` the same condition is the user's own file gone
+  and is still raised, loudly; so is any other failure to write the original
+  back, in both modes, because a permission error can mean a mutated file left
+  on disk.
+  More generally, **one bad fault no longer costs the evidence for the other
+  forty**. An unexpected throw inside a fault's probe ends *that fault* as
+  `unverifiable` with reason `probe-error` and a `detail.message` naming what
+  threw; the run finishes and writes its document. The verdict gates, so the
+  failure is loud rather than absorbed into a green run, and a `probe-error`
+  record is never reused by a later run — it says something about the run, not
+  about the code. A **precondition** failure is the deliberate exception: the
+  interpreter loading the source from outside the probed tree, or a runner that
+  does not resolve, is a statement about every verdict in the run and still
+  refuses it outright.
+- **A prior verdict is no longer reused across a fault edit.** `probe` decided
+  reuse from the source hash, the defender hashes and the defender set — never
+  from the fault itself. So editing a fault's `find` or `replace` kept the
+  verdict measured against the fault it replaced: a weakened fault held the
+  `survived` it earned before, and a repaired anchor held `unverifiable`, with
+  the evidence reporting a result nobody had measured against the file's
+  current contents. `status` flagged the changed hash afterwards, but by then
+  the evidence had already said something untrue — and for a tool whose
+  premise is that a fault edit can never be invisible, that was the wrong
+  default. Reuse is now also keyed on the fault's content hash; a prior record
+  that carries none is probed again, because the cost of re-measuring is a run
+  and the cost of the other direction is a verdict nobody took.
+  Found by using the tool on itself: repairing two rotted anchors left both
+  claims reporting `UNVERIFIABLE (reused)` on the next probe.
+
+### Added
+
+- **A survivor now has to prove the defenders execute the subject** (#74). A
+  green baseline makes one failure mode structurally impossible — a misread
+  runner cannot flatter the suite, because a kill requires the defenders to
+  pass on unmodified source first. It says nothing about the opposite
+  direction. If the fault is applied to code the test process never executes,
+  the baseline is green, every probe run is green, and **every claim is
+  reported SURVIVED**. That output reads as a devastating audit finding and is
+  entirely false, and nothing in the run contradicts it, because every
+  individual check passed.
+  This is not hypothetical: a strict editable Python install registers a meta
+  path finder that is consulted before `sys.path`, so the interpreter loads the
+  original module while TestGuard faults the copy in its scratch worktree.
+  Measured, not argued — a fault independently verified to fail 2 of 4 tests
+  reported `4 passed`.
+  A would-be `survived` is now charged one more run: the subject is replaced
+  with content its loader cannot parse, and the defenders run once. Going red
+  proves they execute it (`detail.negativeControl: "reached"`) and the verdict
+  stands. Staying green proves they do not, and the verdict becomes
+  `unverifiable` with reason `subject-not-executed`. Charged **only** on a
+  survivor — a kill already proves the defenders reached the code — and cached
+  per subject and defender set. Each runner supplies what "cannot compile"
+  means for its language; one that cannot say does not guess, and no control is
+  run.
+  The predicate is *"the run did not stay green"*, not *"a test failed"*: an
+  unparseable module usually fails to **load**, which the runner reports as an
+  error with no tests run at all.
+
+- **A runner pinned to one engine keeps every capability the module has.**
+  `pinned()` rebuilt the runner as an explicit allow-list of properties, so
+  `--runner pytest` and `--runner unittest` silently lost the negative control
+  that `--runner python` has: a survivor probed through either flag was never
+  checked, and an unreachable subject reported `survived` exactly as before.
+  Found by the Python fixture, not by the unit test, which iterated the four
+  runner modules rather than the registry the tool actually resolves through —
+  it was checking the code that was written instead of the code that runs. The
+  test now iterates `RUNNERS`, which is what makes the next capability added to
+  a runner fail loudly instead of quietly.
+
+- **`--progress auto|tty|plain|ndjson|none` on `probe`** (#68). A probe used
+  to print its stage line only when `stderr` was a terminal, so CI, a
+  redirected log and an agent harness saw nothing at all for the whole run —
+  indistinguishable from a hang, and a gate you cannot tell from a hang is a
+  gate people start killing.
+  The carriage-return rewriting is a *rendering* choice, not a reason to
+  withhold the information. `auto` (the default) keeps the rewritten line at a
+  terminal and prints append-only lines everywhere else. `ndjson` emits one
+  JSON object per line for stages **and** for verdicts as they land, so a
+  machine can watch a probe without waiting for the document at the end.
+  Progress always goes to **stderr**, so `--json` leaves stdout as one
+  parseable document. `--quiet` and `--json` imply `none`; an explicit
+  `--progress` overrides both, because an operator who asks for a stream of
+  events has said what they want.
+  Self-claims `TG-PROGRESS-NOT-ONLY-FOR-A-TTY`, `TG-PROGRESS-NEVER-ON-STDOUT`
+  and `TG-PROGRESS-EXPLICIT-WINS`, all probed and killed.
+
+
+- **`--cost`, on `probe` and `claims`** — what the defenders actually cost,
+  per claim and per defender file, read back out of the `durationMs` the
+  probe already records. It re-measures nothing and spawns nothing.
+  A probe's wall clock is `(baseline runs + faults × --confirm) × the cost of
+  the claim's defender SET`, per claim, so one slow acceptance test named by
+  five claims is paid for thirty times — and until now nothing in the output
+  said which file that was. The report names it, under **shared defenders**,
+  with the claims that pay for it.
+  Per-file figures are an **upper bound, not a share**: a run executes the
+  whole defender set at once, so the runner never says how much belonged to
+  which file. They overlap by construction and do not sum to the total, which
+  the report says on its own face. Only `totalMs` is additive.
+
+- **DocGuard governs this repository's documentation.** Six canonical
+  documents under `docs-canonical/` — architecture, data model, security,
+  test spec, environment and requirements — written against v0.6.0 rather
+  than generated as templates, plus `DRIFT-LOG.md` for deliberate,
+  recorded deviations. Every validator passes: structure, doc sections,
+  changelog, test spec, environment, security, freshness, traceability,
+  docs-coverage, document lifecycle and TODO tracking. Structural maturity
+  99/100.
+- **Requirements are traceable to tests.** Fourteen functional and eight
+  non-functional requirements, each carrying an `@req` annotation in the
+  test that verifies it, and a traceability matrix that must agree with
+  those annotations. 27/27 traceability checks pass.
+- **Doc drift is caught automatically, not reviewed by hand.** `ci.yml`
+  runs `docguard guard` on every pull request — authoritative, unbypassable
+  and visible in review — and reports the maturity score without gating on
+  it. `.pre-commit-config.yaml` gives contributors the same checks locally,
+  opt-in per clone, alongside the claims check, the change gate and the
+  probe on pre-push.
+- Every configuration file in the repository is now documented in
+  `ARCHITECTURE.md`, including the distinction between
+  `.pre-commit-hooks.yaml` (what we publish for consumers) and
+  `.pre-commit-config.yaml` (what contributors to this repository use).
+  README gains the `Usage` section the Standard README spec expects.
+- **A Python runner** (`--runner python|pytest|unittest`, and `auto` when no
+  JavaScript runner resolves). `pytest` when the project's interpreter can
+  import it, stdlib `unittest` when it cannot; the engine that actually ran is
+  what `run.runner.name` records, and `--runner pytest` / `--runner unittest`
+  pin the choice and fail rather than fall back. **Nothing is installed into
+  the project under test**: the reporters ship inside TestGuard and reach the
+  interpreter through `PYTHONPATH`, so a codebase whose test dependencies are
+  the standard library stays one. A `.py` defender runs under Python whatever
+  the project runner is, so one claim can be defended by a vitest test and a
+  pytest test at once.
+- `--python <path>` (or `TESTGUARD_PYTHON`) names the interpreter. Otherwise:
+  the active `VIRTUAL_ENV`, then the project's `.venv`/`venv`/`.env`, then
+  `python3` on PATH. An interpreter you name is the only one tried — TestGuard
+  never quietly substitutes another, which would change what the tests ran
+  against without changing anything the evidence says.
+- **An import-provenance precondition, and `detail.targetNotImported`.** A
+  green baseline proves the harness is not reporting everything as broken. It
+  proves nothing about the opposite direction, and Python can reach it: a
+  strict editable install puts an import hook ahead of `sys.path`, so the
+  interpreter loads the original file while TestGuard faults the copy. The
+  baseline stays green, every claim is reported `SURVIVED`, and the result
+  reads as a devastating finding while being entirely false. With the fault
+  applied, TestGuard now asks which file was actually imported: loaded from
+  outside the tree being probed **refuses the run**; never imported at all is
+  recorded as `detail.targetNotImported` and warned about, because an import
+  in a branch the fault does not reach is legitimate.
+- **`target-attribute-patched`**, a new defender signal. `patch("pkg.mod.fn")`
+  is not `vi.mock("./mod")`: it replaces one attribute, so the file still
+  detects a fault anywhere else in that module. Applying the JavaScript rule
+  would have dropped such files and reported `nocover` for well-defended
+  claims. Only a patch of the module itself removes a defender; an attribute
+  patch keeps it and names the attributes.
+- Python fault producers for `testguard scaffold`: `condition-forced`,
+  `statement-deleted`, `return-altered`, `literal-changed`, `call-removed`,
+  `field-dropped` and `argument-swapped`, in Python syntax. A statement is
+  removed by replacing it with `pass` rather than deleting the line, and a line
+  that leaves a bracket open is never removed — both because a fault that
+  cannot compile is a probe run that says nothing about the tests.
+- Python defender discovery, mock-awareness and blast radius: matched by module
+  name, the way Python itself matches, so `import`, `from … import`, aliases,
+  relative imports and `importlib.import_module` all resolve.
+- `fixtures/known-answer-python/`: sixteen faults covering every verdict, each
+  verified by hand before it became the oracle, and reproduced under **both**
+  engines.
+
+### Changed
+
+- **The self-probe gate no longer runs a fifty-second acceptance test thirty
+  times.** Five claims named `test/probe.fixture.test.mjs` as their only
+  defender, which is what made the release gate take ~24 minutes. The
+  decisions those claims are about — how escalation attributes an undeclared
+  killer, what counts as a measurable run for the flake rate, which binary a
+  runner invokes, what makes a fault edit visible — are rules you can state in
+  three lines, and none of them needs a fixture to falsify. They are now pure
+  functions in `src/probe/attribution.mjs` and exported `argvFor` builders in
+  the runners, unit-tested in `test/attribution.test.mjs` (~0.5s).
+  This mirrors why `classify()` has always been pure: the order of its checks
+  *is* the spec, and every branch should be falsifiable without a runner.
+  No claim was weakened to get there. Each re-pointed claim was re-probed and
+  still **kills** its faults; two of them SURVIVED on the first narrowing
+  attempt, which is exactly the failure mode `--cost` now makes visible before
+  someone ships a cheaper, weaker gate.
+  **Measured on the same machine, serially, from scratch: 24.1 min → 9.6 min**
+  (68 faults, 68 killed, 0 unproven). `--cost` names what is left: the jest
+  and replay suites now dominate, at 207s and 148s of defender time.
+
+- Style exceptions in the canonical documents are declared with a reason
+  (`docguard:quality ... off — why`) rather than satisfied by rewording.
+  This tool is defined by what must not happen — no network, no telemetry,
+  never optimistic under uncertainty — and stating those positively would
+  misdescribe them. A declared, reasoned exception is auditable; contorted
+  prose is not.
+- `--runner` accepts `python`, `pytest` and `unittest`; `--runner auto` tries
+  vitest, then jest, then python. A JavaScript project with neither vitest nor
+  jest installed now also reports why Python did not resolve.
+- The GitHub Action and the GitLab template take `runner` and `python` inputs.
+
+### Fixed
+
+- `gate --changed` treated a changed `.py` file as `non-source` and excluded it,
+  so it could never be reported as uncovered. Python is source; Python test
+  files (`test_*.py`, `*_test.py`) are tests.
+- `walk()` descended into `.venv`, `venv`, `site-packages`, `__pycache__`,
+  `.tox`, `.nox` and the pytest/mypy/ruff caches. Every `test_*.py` of every
+  installed package was collected as a test file of the project under probe.
+- `testguard claims` printed a JavaScript comment marker (`// unasserted:`)
+  when suggesting how to annotate a mock in a Python file.
+- A `--runner-cmd` whose reporter emits something other than the jest-compatible
+  JSON shape produced `unverifiable / defenders-failed-to-load` with no message
+  at all. The verdict was already safe — a report with no recognised tests is a
+  non-green baseline, never a pass — but nothing said why. It now names the
+  mismatch.
+
 ## [0.6.0] - 2026-09-17
 
 Everything the field asked for. Two independent field reports on private