testguard-cli 0.6.0 → 0.8.0

package/README.md

@@ -1,5 +1,8 @@
 # TestGuard
 
+<!-- docguard:quality negation-load off — this README explains a tool defined by what must not happen; the negations are the product. -->
+<!-- docguard:quality passive-voice off — verdicts and artifacts are the subjects throughout ("a fault is applied", "evidence is written"); naming an actor would misdescribe a tool nobody operates interactively. -->
+
 [![CI](https://github.com/raccioly/testguard/actions/workflows/ci.yml/badge.svg)](https://github.com/raccioly/testguard/actions/workflows/ci.yml)
 [![npm](https://img.shields.io/npm/v/testguard-cli.svg)](https://www.npmjs.com/package/testguard-cli)
 [![PyPI](https://img.shields.io/pypi/v/testguard-cli.svg)](https://pypi.org/project/testguard-cli/)
@@ -7,9 +10,8 @@
 [![license](https://img.shields.io/badge/license-MIT-blue.svg)](./LICENSE)
 [![deps](https://img.shields.io/badge/runtime%20deps-1%20pinned-brightgreen.svg)](./package.json)
 
-> Proves that a test suite actually defends the claims a project makes — by
-> injecting the faults those claims say cannot happen, and reporting every
-> fault the tests fail to detect.
+> Breaks your code on purpose and reports every promise your tests did not
+> notice breaking.
 
 **Not a test generator. A claim verifier.** Test generation is what happens
 after a claim turns out to be unfalsifiable.
@@ -22,6 +24,65 @@ surface ↔ code). All three run one loop:
 > declare what must be true → try mechanically to falsify it → freeze a
 > baseline → gate only the delta → brief the agent before it writes code.
 
+## In one minute
+
+**Coverage tells you a line ran. It never tells you anyone checked the
+result.** So a codebase can be fully covered and completely undefended, and
+nothing in CI will say a word.
+
+Here is a real test, from this repository's own fixture:
+
+```js
+expect(store.writeAudit).toHaveBeenCalledWith(
+  expect.objectContaining({ action: 'MASK', scope: 'g1', ruleCount: 1 }),
+);  // `content` is never named — so nothing checks it
+```
+
+`objectContaining` ignores keys it does not list. Swap the redacted text for
+the **raw secret** and this test still passes. Coverage of that line: 100%.
+The audit log now leaks the very thing it exists to protect.
+
+**The method: break the code deliberately, then watch what the tests do.**
+
+- **killed** — you broke it, a test failed. Good. That behaviour is genuinely
+  defended.
+- **SURVIVED** — you broke it, everything stayed green. A blind spot.
+
+The word trips everyone once: it is the *fault* that survived, not the test.
+**SURVIVED is the bad news.** A healthy report is full of `killed`.
+
+But *"did the tests fail?"* is a sloppy question — a red suite is not proof of
+detection. So there are seven verdicts, and each means something different:
+
+| Verdict | What it means |
+|---|---|
+| `killed` | A test body ran and rejected the behaviour. The only good outcome. |
+| `SURVIVED` | Everything passed. A real blind spot in the tests. |
+| `NOCOVER` | No test even looks at this code. Not "weak tests" — *no* tests. |
+| `UNVERIFIABLE` | The fault could not be applied: its anchor moved, or matches twice. Nothing was learned. |
+| `FAULT-INVALID` | The break itself was broken — it did not compile. Our fault, not yours. |
+| `TIMEOUT` | The suite hung. A hang is not a detection. |
+| `FLAKY-DEFENDER` | The tests are not reliable enough on untouched code to be asked the question. |
+
+Every ambiguity rounds toward *unproven*: three runs rather than one, a green
+baseline required before any fault is injected, and a timeout, a load failure
+or a mixed result is never a kill. The reason is the constraint the whole
+design follows from — **a tool that reports everything as caught is worse than
+no tool at all, because nobody questions good news.**
+
+**What is not new:** breaking code to test your tests is *mutation testing*,
+and it dates to the 1970s. Stryker, PIT, mutmut and Cosmic Ray all do it.
+
+**What is different is the question.** Classic mutation testing mutates
+everything mechanically and hands you *"mutation score: 73%"* — a number that
+is not actionable, not auditable, and cannot tell you which promise is at
+risk. TestGuard binds every fault to a **stated claim**, so the output is not
+a score but a finding: *"Your project says a missing scope fails closed.
+Nothing checks that."* That is the difference between a metric and an audit.
+
+📄 **[Read the three-page technical brief (PDF)](docs/testguard-explained.pdf)**
+— the idea on page one, mechanics and prior art after it.
+
 ## Why
 
 Coverage cannot tell a test that pins *correct* behaviour from one that pins
@@ -65,14 +126,34 @@ raises coverage. TestGuard admits it because it fails when the claim is false.
 | npm | `npm i -D testguard-cli` then `npx testguard probe` |
 | pip | `pip install testguard-cli` then `testguard probe` (needs Node ≥ 20) |
 | Homebrew | `brew tap raccioly/tap && brew install testguard` |
-| GitHub Action | `uses: raccioly/testguard@v0.6.0` — see [`action.yml`](./action.yml) |
+| GitHub Action | `uses: raccioly/testguard@v0.8.0` — see [`action.yml`](./action.yml) |
 | pre-commit | `repo: https://github.com/raccioly/testguard`, hooks `testguard-claims`, `testguard-probe` |
-| GitLab CI | `include: - remote: https://raw.githubusercontent.com/raccioly/testguard/v0.5.0/packaging/gitlab/testguard.gitlab-ci.yml` with `inputs:` — see [`packaging/gitlab/`](./packaging/gitlab/testguard.gitlab-ci.yml) |
+| GitLab CI | `include: - remote: https://raw.githubusercontent.com/raccioly/testguard/v0.8.0/packaging/gitlab/testguard.gitlab-ci.yml` with `inputs:` — see [`packaging/gitlab/`](./packaging/gitlab/testguard.gitlab-ci.yml) |
 
 Projects that set `min-release-age` in `.npmrc` cannot see a version published
 less than that many days ago (`ENOVERSIONS`); install that one with
 `npm i -D testguard-cli --min-release-age=0`.
 
+## Usage
+
+```bash
+npx testguard-cli init                      # install the agent layer at the git root
+npx testguard-cli status --json             # where the project is, and the ONE next action
+npx testguard-cli claims                    # what does this project claim, and is each claim probeable?
+npx testguard-cli scaffold src/auth.ts      # propose faults for a file, as a draft to keep or drop
+npx testguard-cli probe                     # try to falsify each claim; report what the tests missed
+npx testguard-cli admit test/auth.test.ts --claim AUTH-ADMIN   # does this test satisfy the two-gate rule?
+npx testguard-cli baseline                  # freeze today's unproven findings; only new ones gate
+npx testguard-cli gate --changed origin/main # fail when a changed source file carries no claim
+npx testguard-cli brief --text              # tell the agent where the suite is blind, before it writes
+npx testguard-cli replay --since v1.0..HEAD # would this suite have caught the bugs that escaped?
+npx testguard-cli mcp                       # serve the read-only loop over MCP, on stdio
+```
+
+Exit codes are the contract: `0` nothing new to prove, `1` unproven claims or
+unclaimed changes, `2` a precondition failed and nothing was probed, `3` usage.
+Every command accepts `--json`.
+
 ## How it works
 
 ```bash
@@ -146,14 +227,39 @@ npx testguard-cli admit test/x.test.ts --claim X   # is this test green on HEAD
    there is, and the exact signature of one escaped bug in the field
    reports. Silence it, visibly, with `// unasserted: <why>` above the mock.
 
-   Runners: **vitest** and **jest** (`--runner auto` picks the first that
-   resolves; both read the same jest-compatible JSON report). A runner is
-   resolved from the **project's own package** first — its pinned version,
-   its own bin script — and only then from an executable on PATH, which the
-   evidence records as `runner.source: "path"`; `npx` is never asked,
-   because its cache answers for packages a project does not have. Anything
-   else goes through `--runner-cmd`. Each runner is proven against its own
-   copy of the known-answer fixture.
+   Runners: **vitest**, **jest**, **Playwright** and **Python** (`--runner
+   auto` picks the first of vitest, jest, python that resolves). A JavaScript
+   runner is resolved from the **project's own package** first — its pinned
+   version, its own bin script — and only then from an executable on PATH,
+   which the evidence records as `runner.source: "path"`; `npx` is never
+   asked, because its cache answers for packages a project does not have.
+   Anything else goes through `--runner-cmd`. Each runner is proven against
+   its own copy of the known-answer fixture.
+
+   **Python** runs under `pytest` when the project's interpreter can import
+   it and stdlib `unittest` when it cannot, and the evidence records the
+   engine that actually ran — `--runner pytest` or `--runner unittest` pins
+   the choice and fails rather than falling back. Nothing is installed into
+   the project: the reporters ship inside TestGuard and reach the interpreter
+   through `PYTHONPATH`, so a codebase whose test dependencies are the
+   standard library stays that way. The interpreter is the project's
+   `.venv`/`venv`, the active `VIRTUAL_ENV`, or `--python <path>`. A `.py`
+   defender runs under Python whatever the project runner is, so one claim
+   can be defended by a vitest test and a pytest test at once.
+
+   Two things Python forces that JavaScript does not. **`patch("pkg.mod.fn")`
+   is not `vi.mock`**: it replaces one attribute, so the file is still a
+   defender of every other fault in that module — it keeps its place and
+   carries a `target-attribute-patched` signal naming the attributes. And
+   **the fault must be the code that ran**: a strict editable install puts an
+   import hook ahead of `sys.path`, so Python can load the original file while
+   TestGuard faults the copy — a green baseline, every claim `SURVIVED`, and a
+   report that reads as a devastating finding while being entirely false. With
+   the fault applied, TestGuard asks which file was actually imported and
+   refuses the run when it is outside the tree being probed. When no defender
+   imported the subject at all, the run continues and the record says so
+   (`detail.targetNotImported`), because that is a fact about the defenders'
+   reach and not about their assertions.
 
    The two-gate rule, as one verb: `testguard admit <test-file> --claim <ID>`
    runs the claim's faults against your uncommitted test and answers
@@ -239,15 +345,18 @@ most resembles, which is the join key that turns an uninterpretable mutation
 score into a statement with a sample size:
 
 ```
-  guard-removed          blind   7/9    p=0.78  ci [0.45, 0.94]
-  field-dropped          blind   4/4    p=1.00  ci [0.51, 1.00]
+  guard-removed          missed  7/9    p=0.78  ci [0.45, 0.94]
+  field-dropped          missed  4/4    p=1.00  ci [0.51, 1.00]
 ```
 
-Read as *"when a fault of this class survives, how often does that
-correspond to a bug that really escaped"*. It is written as a
-`calibration` document beside the replay one. Only `caught` and `blind`
-carry information; the other verdicts are excluded from both sides of the
-ratio.
+Read as *"when a real bug of this class escapes, how often does the suite
+miss it"* — a miss rate, so a high `p` is bad. It is written as a
+`calibration` document beside the replay one. `caught`, `blind` and
+`nocover` are measurements and enter the ratio, `nocover` as a miss: leaving
+it out would score a project with no tests at all for a subsystem *better*
+than one with weak tests. `flaky` and `unverifiable` are failed measurements
+and enter neither side. Every `p` and `ci` is recomputable from `n` and the
+miss count, and the validator recomputes them.
 
 **The open question this exists to answer.** Does a calibration learned on a
 repository *with* history transfer to a greenfield one that has none? AI
@@ -318,12 +427,12 @@ detected base that does not resolve is a warning for `status` and `brief`
 # GitHub Actions
 - uses: actions/checkout@v4
   with: { fetch-depth: 0 }
-- uses: raccioly/testguard@v0.6.0
+- uses: raccioly/testguard@v0.8.0
   with: { command: gate }
 
 # GitLab CI — the component-shaped template: gate + probe, brief as an artifact and, opted in, as a merge-request note
 include:
-  - remote: 'https://raw.githubusercontent.com/raccioly/testguard/v0.5.0/packaging/gitlab/testguard.gitlab-ci.yml'
+  - remote: 'https://raw.githubusercontent.com/raccioly/testguard/v0.8.0/packaging/gitlab/testguard.gitlab-ci.yml'
     inputs: { dir: backend, post_note: true }   # post_note needs TESTGUARD_GITLAB_TOKEN (api scope); one note, updated in place
 ```
 
@@ -384,6 +493,45 @@ is the same one as everywhere else: a `claim` entry in
 Removal is allowed — claims can be wrong, superseded or split — and it is
 never silent.
 
+### Probing a Python project
+
+Nothing is installed into your project. TestGuard carries its own reporters and
+puts them on the interpreter's `PYTHONPATH`, so a codebase whose test
+dependencies are the standard library keeps none.
+
+```bash
+npx testguard-cli probe .                        # auto: pytest if importable, else stdlib unittest
+npx testguard-cli probe . --runner unittest      # pin the stdlib engine
+npx testguard-cli probe . --python .venv/bin/python
+```
+
+The evidence records the engine that actually ran, and `--runner pytest` fails
+rather than quietly using `unittest` — which engine ran changes what the
+evidence means.
+
+Three things behave differently from JavaScript, and each one is a place a
+naive port would have produced a confident wrong answer:
+
+- **Defenders are matched by module name**, the way Python itself matches, so
+  a test that reaches your package through `sys.path`, a `conftest.py` or an
+  installed distribution is found either way.
+- **`patch("pkg.mod.fn")` is not `vi.mock`.** It replaces one attribute, so the
+  file still detects a fault anywhere else in that module: it stays a defender
+  and carries a `target-attribute-patched` signal naming the attributes. Only a
+  patch of the module *itself* removes a defender.
+- **The fault has to be the code that ran.** A strict editable install puts an
+  import hook ahead of `sys.path`, so Python can load your original file while
+  TestGuard faults its copy — green baseline, every claim `SURVIVED`, and a
+  report that reads as a catastrophe while being entirely false. With the fault
+  applied TestGuard asks which file was really imported and **refuses the run**
+  if it came from outside the tree being probed. If no defender imported the
+  subject at all, the run continues and the record says so
+  (`detail.targetNotImported`): that is a fact about the tests' reach, not
+  about their assertions, and the two must not read alike.
+
+`testguard scaffold path/to/module.py` proposes Python faults the same way it
+does for JavaScript.
+
 ### Run it from any harness
 
 The operating loop above lives in a Claude Code skill and a session-start
@@ -452,6 +600,92 @@ things make that safe:
   previous verdict, and makes reviewing that edit the next action. Editing a
   claim is allowed — claims can be wrong — but it is never invisible.
 
+### Watching a probe that is still running
+
+A probe runs the defenders N times clean and N times per fault, which takes
+minutes. It used to print its stage line only when `stderr` was a terminal —
+so CI, a redirected log and an agent harness saw **nothing at all** for the
+whole run, which is indistinguishable from a hang. A gate you cannot tell
+from a hang is a gate people start killing.
+
+The rewriting is a rendering choice, not a reason to withhold the
+information:
+
+| `--progress` | What you get |
+|---|---|
+| `auto` (default) | `tty` at a terminal, `plain` everywhere else |
+| `tty` | one line, rewritten in place — for a human watching |
+| `plain` | one append-only line per stage — a log file, CI, an agent |
+| `ndjson` | one JSON object per line, stages **and** verdicts as they land |
+| `none` | silence |
+
+```bash
+npx testguard-cli probe . > probe.log 2>&1          # now reports; used to be silent
+npx testguard-cli probe . --progress ndjson 2>events.ndjson
+```
+
+```
+{"event":"stage","claim":"TG-COST-NAMES-SHARED-DEFENDERS","fault":"F1","stage":"baseline","run":1,"of":3,"at":"…"}
+{"event":"verdict","claim":"TG-COST-NAMES-SHARED-DEFENDERS","fault":"F1","verdict":"killed","severity":"medium","file":"src/probe/cost.mjs","at":"…"}
+```
+
+**Progress always goes to stderr**, so `--json` leaves stdout as one document
+and nothing else. `--quiet` and `--json` imply `none`, and an explicit
+`--progress` overrides both: an operator who asks for a stream of events has
+said what they want.
+
+### What the probe costs, and why a gate gets slow
+
+A probe's wall clock is not a function of how many claims you have. Per claim
+it is roughly
+
+```
+(baseline runs + faults × --confirm) × the cost of that claim's defender SET
+```
+
+so one slow acceptance test, named as a defender by five claims, is paid for
+thirty times. The symptom is a gate that takes twenty minutes; the cause is a
+single line in the claims file, and nothing in the output used to say which.
+
+```bash
+npx testguard-cli claims . --cost     # read back from the last probe's evidence
+npx testguard-cli probe . --cost      # what the run you just made spent
+```
+
+```
+63 fault records cost 24m across 378 defender runs.
+
+most expensive claims
+    4m 58s  TG-FAULT-EDIT-VISIBLE              1 fault  · 6 runs
+
+defender files, by the time of the runs that included them
+  (a claim runs its whole defender set at once, so these overlap and do not sum to the total)
+    24m 3s  test/probe.fixture.test.mjs                  5 claims
+
+shared defenders — each claim pays the file's full cost again
+    24m 3s  test/probe.fixture.test.mjs
+            named by TG-ESCALATION-N-RUNS, TG-FAULT-EDIT-VISIBLE, …
+```
+
+Every number is read back out of the `durationMs` the probe already records.
+`--cost` re-measures nothing, spawns nothing, and costs the price of reading
+one JSON file.
+
+**The per-file figures overlap.** A run executes a claim's whole defender set
+at once, so the runner never says how much of it belonged to which file. A
+file's `ms` is the time of every run that *included* it: an upper bound on
+what removing it could save, which is the decision you are making. Only
+`totalMs` is additive.
+
+**What to do about a shared defender.** Extract the decision it proves into a
+pure function, unit-test that, and point the claim at the unit test. This is
+not a testing trick — it is the same reason `classify()` in this codebase is
+pure. A rule you can state in three lines should not need a fifty-second
+acceptance test to falsify it. When the extraction is right the claim still
+fails on the fault; when it is wrong the claim SURVIVES, and the probe tells
+you so before you have shipped a weaker gate. Do not narrow `defendedBy` and
+assume — re-probe the claim and read the verdict.
+
 ### Authoring faults mechanically
 
 Writing faults by hand means reading the code to find exact anchors. Two
@@ -475,6 +709,18 @@ npx testguard-cli scaffold src/auth.ts --claim AUTH-ADMIN   # every proposal und
 | `element-removed` | a one-line JSX element — self-closing (`<Toggle … />`) or paired (`<button …>Save</button>`) — removed; the UI shape behind "the toggle is invisible", killable by a browser-layer defender |
 | `handler-dropped` | an `on<Event>={…}` prop removed, whether it is its own line or inline in the tag; the control renders and does nothing |
 
+The table above is JavaScript. `scaffold` reads Python too, and proposes the
+same fault classes in Python syntax: `if <guard>:` → `if False:`,
+`return <check>` → `return True`, `verify=True` → `verify=False`, a
+parameter-derived argument swapped for `None`, a key dropped from a payload
+`dict` or an allow-list. Two differences are deliberate. A statement is
+removed by replacing it with `pass`, never by deleting the line, because a
+block whose only statement is gone is an `IndentationError`; and a line that
+leaves a bracket open (`COLOURS = {`) is never removed at all. Both exist
+because a fault that cannot compile is a `fault-invalid` verdict — a probe run
+spent saying nothing about the tests. The two JSX shapes (`element-removed`,
+`handler-dropped`) have no Python meaning and are absent rather than faked.
+
 Every proposal's `find` is the exact line with `expectHits`/`occurrence`
 computed from the file, so it is verifiable by construction; provenance is
 `producer: derived`; `defendedBy` is prefilled from the tests that import
@@ -507,21 +753,28 @@ redacted text for the raw input and the test stays green. `probe` reports it
 as `SURVIVED`; the fixture's [README](fixtures/known-answer/README.md) walks
 through every verdict.
 
-## Status
+The same blind spot, in Python, is
+[`fixtures/known-answer-python/`](fixtures/known-answer-python/README.md) —
+run under **both** stdlib `unittest` and `pytest`, which have different report
+shapes and different notions of failure and must still agree on all sixteen
+verdicts.
 
-**v0.5.** Eleven commands (`status`, `init`, `claims`, `probe`, `admit`, `replay`, `baseline`, `brief`, `gate`, `scaffold`, `mcp`), vitest and jest runners, hand-authored faults plus
-**v0.5.** Nine commands (`status`, `init`, `claims`, `probe`, `admit`, `baseline`, `brief`, `gate`, `scaffold`), vitest, jest and Playwright runners, hand-authored faults plus
-a mechanical scaffold, an agent operating layer (`status`, `init`) and a
-change gate (`gate`). The contract
-spine — eight JSON Schemas shared with the other Guard tools — is under
-[`spec/`](spec/). One exact-pinned runtime dependency (`ajv`, for schema validation); Node ≥ 20.
+## Status
 
-Not yet: test generation (the acceptance half, `admit`, exists; the generating half stays the agent's), runners beyond
-vitest and jest, AST-aware producers, and the transfer of a calibration between repositories — `replay` measures it
-now; whether it carries to a repository with no history is unproven. Each is
+**v0.5.** Eleven commands (`status`, `init`, `claims`, `probe`, `admit`,
+`replay`, `baseline`, `brief`, `gate`, `scaffold`, `mcp`), vitest, jest,
+Playwright and Python (pytest / stdlib unittest) runners, hand-authored faults
+plus a mechanical scaffold for JavaScript and Python, an agent operating layer
+(`status`, `init`) and a change gate (`gate`). The contract spine — eight JSON
+Schemas shared with the other Guard tools — is under [`spec/`](spec/). One
+exact-pinned runtime dependency (`ajv`, for schema validation); Node ≥ 20, and
+Python ≥ 3.8 only when probing Python.
+
+Not yet: test generation (the acceptance half, `admit`, exists; the generating
+half stays the agent's), runners beyond those four, AST-aware producers, and
+the transfer of a calibration between repositories — `replay` measures it now;
+whether it carries to a repository with no history is unproven. Each is
 designed for; none is claimed.
-vitest, jest and Playwright, AST-aware producers, and calibration of fault classes
-against real escaped bugs. Each is designed for; none is claimed.
 
 ## Licence
 

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "testguard-cli",
-  "version": "0.6.0",
-  "description": "Proves a test suite defends the claims a project makes: injects the faults those claims forbid and reports every one the tests fail to detect.",
+  "version": "0.8.0",
+  "description": "Breaks your code on purpose and reports every promise your tests did not notice breaking. Mutation testing bound to the claims a project makes — vitest, jest, Playwright, pytest and unittest.",
   "type": "module",
   "license": "MIT",
   "engines": {