testguard-cli 0.5.0 → 0.6.0

package/README.md

@@ -44,6 +44,19 @@ re-implemented the authorization logic *inside the test* and asserted
 against the copy: fifteen green tests, zero detection. After wrapper-level
 tests were written against the survivors, 39/39 were killed.
 
+The peer-reviewed picture in 2026 says the same thing from the other side.
+Coverage and mutation score of LLM-generated suites track effectiveness only
+when the code under test is assumed correct; once it may be buggy they "no
+longer serve as reliable indicators" ([Zhao, Zhou and Cohen, ISSTA 2026](https://arxiv.org/abs/2607.22880)).
+Buggy code steers a model toward tests that assert the bug, and prompting
+with the specification is the mitigation that works ([arXiv 2607.22883](https://arxiv.org/abs/2607.22883)) —
+which is why a claim here comes from intent, a fault is bound to the claim,
+and the brief hands the agent the claim before it writes. And agents
+saturate whatever tests they can see, with the gap to held-out tests
+growing about 28 points per tenfold increase in code size ([SpecBench](https://arxiv.org/abs/2605.21384)).
+Every generator on the market admits a test because it compiles, passes and
+raises coverage. TestGuard admits it because it fails when the claim is false.
+
 ## Install
 
 | How | Command |
@@ -52,8 +65,9 @@ tests were written against the survivors, 39/39 were killed.
 | npm | `npm i -D testguard-cli` then `npx testguard probe` |
 | pip | `pip install testguard-cli` then `testguard probe` (needs Node ≥ 20) |
 | Homebrew | `brew tap raccioly/tap && brew install testguard` |
-| GitHub Action | `uses: raccioly/testguard@v0.5.0` — see [`action.yml`](./action.yml) |
+| GitHub Action | `uses: raccioly/testguard@v0.6.0` — see [`action.yml`](./action.yml) |
 | pre-commit | `repo: https://github.com/raccioly/testguard`, hooks `testguard-claims`, `testguard-probe` |
+| GitLab CI | `include: - remote: https://raw.githubusercontent.com/raccioly/testguard/v0.5.0/packaging/gitlab/testguard.gitlab-ci.yml` with `inputs:` — see [`packaging/gitlab/`](./packaging/gitlab/testguard.gitlab-ci.yml) |
 
 Projects that set `min-release-age` in `.npmrc` cannot see a version published
 less than that many days ago (`ENOVERSIONS`); install that one with
@@ -70,6 +84,7 @@ npx testguard-cli baseline    # freeze today's unproven findings; from now on on
 npx testguard-cli brief       # tell the agent where the suite is blind, before it writes
 npx testguard-cli gate --changed origin/main   # fail when a changed source file carries no claim at all
 npx testguard-cli scaffold src/x.ts   # propose faults for a file, as a draft to keep or drop
+npx testguard-cli admit test/x.test.ts --claim X   # is this test green on HEAD and does it fail on every fault of X? ADMITTED or NOT ADMITTED
 ```
 
 1. **Claims** live in `testguard.claims.json` (editors validate it against
@@ -83,7 +98,12 @@ npx testguard-cli scaffold src/x.ts   # propose faults for a file, as a draft to
 2. **Probe** confirms the defenders are green N times unmodified, applies
    each fault in a scratch git worktree (your tree is never touched), runs
    the defenders N times, re-runs survivors against the whole suite with
-   N-run attribution, restores, and classifies. Verdicts are a closed set:
+   N-run attribution (up to N: it stops as soon as no test has failed in
+   every run, because from there none can be named a killer), restores, and
+   classifies. It warns when another test runner is already running — a
+   contended machine turns a slow suite into a `TIMEOUT` verdict about the
+   load, not the claim — records that on the evidence, and offers `--serial`
+   to run one test file at a time. Verdicts are a closed set:
 
    | Verdict | Meaning |
    |---|---|
@@ -93,7 +113,7 @@ npx testguard-cli scaffold src/x.ts   # propose faults for a file, as a draft to
    | `UNVERIFIABLE` | the fault's anchor is missing or ambiguous — loud, never a skip |
    | `TIMEOUT` | the defenders hung; a hang is not a detection |
    | `FAULT-INVALID` | the replacement does not load — a bad fault, not a finding |
-   | `FLAKY-DEFENDER` | the defenders are not reliably green, or disagreed across runs |
+   | `FLAKY-DEFENDER` | the defenders are not reliably green, or disagreed across runs — `detail.flakeRate` says how often (`failures` of `runs` on unmodified source) |
 
    Never a single score. Findings are ranked by severity, claim provenance
    and blast radius (relative imports, `tsconfig` path aliases and
@@ -102,20 +122,48 @@ npx testguard-cli scaffold src/x.ts   # propose faults for a file, as a draft to
    it is written.
 
    Worktree mode probes a **commit**. If a defender or target file has
-   uncommitted changes, `probe` refuses and says so — otherwise your new
-   tests would be silently absent and the same survivors would come back
-   with no hint why. `--include-dirty` snapshots the working tree (tracked
-   edits and new files) into a throwaway commit and probes that; your tree,
-   HEAD and index are never touched. Every summary names the commit probed.
+   uncommitted changes and you asked for the implicit HEAD, `probe` refuses
+   and says so — otherwise your new tests would be silently absent and the
+   same survivors would come back with no hint why. `--include-dirty`
+   snapshots the working tree (tracked edits and new files) into a throwaway
+   commit and probes that; your tree, HEAD and index are never touched. An
+   **explicit `--ref`** (a pre-fix commit in a post-mortem, say) is honoured
+   over a dirty tree, as is `--ignore-dirty`: a warning names the files and
+   the evidence records them as `repo.ignoredDirty`, so the run says what it
+   did not look at. Every summary names the commit probed.
 
    A claim with no `defendedBy` has its defenders **discovered**: the test
-   files that import the fault's target, by relative path or resolved alias.
-   `NOCOVER` then means exactly "no test file imports this source".
+   files that import the fault's target, by relative path or resolved alias
+   (tsconfig `paths`, through `extends` and `references`, vite/vitest
+   `resolve.alias`, package.json `imports`) — **minus the files that mock
+   it**. A test that `vi.mock`s / `jest.mock`s the target cannot detect any
+   fault in it; counting it would make `NOCOVER` under-report and waste runs.
+   `NOCOVER` therefore means exactly "no test file imports this source
+   without mocking it". `claims` prints the split (`18 import · 16 mock · 2
+   can detect`), the evidence lists the mocking files, and a mocking file
+   that never `expect(...)`s anything imported from the target carries the
+   static signal `mocked-never-asserted` — the cheapest blind-spot signal
+   there is, and the exact signature of one escaped bug in the field
+   reports. Silence it, visibly, with `// unasserted: <why>` above the mock.
 
    Runners: **vitest** and **jest** (`--runner auto` picks the first that
-   resolves; both read the same jest-compatible JSON report). Anything else
-   goes through `--runner-cmd`. Each runner is proven against its own copy
-   of the known-answer fixture.
+   resolves; both read the same jest-compatible JSON report). A runner is
+   resolved from the **project's own package** first — its pinned version,
+   its own bin script — and only then from an executable on PATH, which the
+   evidence records as `runner.source: "path"`; `npx` is never asked,
+   because its cache answers for packages a project does not have. Anything
+   else goes through `--runner-cmd`. Each runner is proven against its own
+   copy of the known-answer fixture.
+
+   The two-gate rule, as one verb: `testguard admit <test-file> --claim <ID>`
+   runs the claim's faults against your uncommitted test and answers
+   `ADMITTED` (exit 0: the test is green on unmodified HEAD and fails on
+   every fault, N/N) or `NOT ADMITTED` (exit 1: the first blocking fault and
+   what to do about it). The test must be a declared or discovered defender
+   of the claim. It is sugar over `probe --claim <ID> --include-dirty`, so
+   it can never disagree with the gate. Every generator on the market admits
+   a test because it compiles, passes and raises coverage; `admit` admits it
+   because it fails when the claim is false.
 
    Practical loop: first pass `--no-escalate` (escalation re-runs the whole
    suite N times per survivor); iterate on one claim with `--claim <ID>` and
@@ -132,20 +180,105 @@ npx testguard-cli scaffold src/x.ts   # propose faults for a file, as a draft to
 3. **Baseline** freezes every non-passing fingerprint. Later probes suppress
    what was already known and exit non-zero only on what is new. Claims whose
    source and defenders are unchanged reuse their prior verdict, so a probe
-   in CI costs only what changed.
+   in CI costs only what changed. A baseline frozen from `--include-dirty`
+   evidence records the snapshot and points at the *parent* of the commit
+   that will carry your tests; after you commit, a clean `probe` plus
+   `baseline --restamp` moves it to that commit — only when the fingerprints
+   are identical, never otherwise. `status` notes a baseline that predates
+   HEAD without making it a state.
 4. **Brief** turns evidence plus baseline into a ranked, capped
    `## TEST BLINDSPOT CONTEXT` block, printed and also written to
-   `.testguard/brief.json` (`--text` prints only). Wire it into an agent's session start
+   `.testguard/brief.json` (`--text` prints only; `--markdown` prints the same brief as a
+   merge-request note for the human reviewer, unclaimed changes first). Wire it into an agent's session start
    — for Claude Code, in `.claude/settings.json`:
 
    ```json
    { "hooks": { "SessionStart": [ { "hooks": [
-     { "type": "command", "command": "npx testguard-cli brief --text" }
+     { "type": "command", "command": "node_modules/.bin/testguard brief --text 2>/dev/null || { command -v testguard >/dev/null 2>&1 && testguard brief --text 2>/dev/null; } || true" }
    ] } ] } }
    ```
 
-   `--text` prints only, and exits 0 silently when there is no evidence yet,
-   so the hook can never break a session.
+   `--text` prints only, and exits 0 silently when there is no evidence yet;
+   the command resolves the project's own install, then a `testguard` on
+   `PATH`, and ends in `true` — so the hook can never break a session. There
+   is no `npx` in it, in any form: `--no-install` resolves the package from
+   the registry before declining to install it, so it is quiet rather than
+   offline. The brief's first line says which install answered
+   (`local install` or `global`), so a stale one is visible.
+
+### Would this suite have caught the bugs that already escaped?
+
+An injected fault is a fault somebody thought of. A bug that actually shipped
+is ground truth: a human already confirmed it was a defect, and there is no
+equivalent-mutant argument to have about it.
+
+```bash
+npx testguard-cli replay --since HEAD~50..HEAD --max 10
+```
+
+For each fix commit in the range — one that changes source **and** a test
+together — `replay` checks it out in a scratch worktree, reverts only its
+source files to the parent, **deletes the test the fix shipped** (that test
+proves nothing about what the suite knew before it existed), and runs the
+tests that import the reverted code:
+
+| Verdict | Meaning |
+|---|---|
+| `caught` | a remaining test failed by assertion, every run. The suite knew. |
+| `blind` | the suite stayed green on known-broken code. |
+| `nocover` | no test imports the reverted files — worse than blind. |
+| `flaky` | the runs disagreed, so nothing can be concluded. A flaky failure reads as detection, which biases this metric *optimistically*; mixed runs are never `caught`. |
+| `unverifiable` | the revert did not apply, or the suite could not load. |
+
+One patch counts once (`git patch-id`), because a dual-branch topology
+carries the same fix under two or three shas. **`replay` reports and never
+gates**: a bug that escaped is history, not a regression in this change.
+
+Each replayed bug is also labelled with the injected-fault class its diff
+most resembles, which is the join key that turns an uninterpretable mutation
+score into a statement with a sample size:
+
+```
+  guard-removed          blind   7/9    p=0.78  ci [0.45, 0.94]
+  field-dropped          blind   4/4    p=1.00  ci [0.51, 1.00]
+```
+
+Read as *"when a fault of this class survives, how often does that
+correspond to a bug that really escaped"*. It is written as a
+`calibration` document beside the replay one. Only `caught` and `blind`
+carry information; the other verdicts are excluded from both sides of the
+ratio.
+
+**The open question this exists to answer.** Does a calibration learned on a
+repository *with* history transfer to a greenfield one that has none? AI
+authored code has no history, so replay cannot help it directly and
+calibration is the only bridge. That transfer is unproven, it is the core
+product bet, and this is the instrument for testing it — not the answer.
+
+### Read CI's evidence locally
+
+Verdict reuse makes a probe cheap, but the evidence lives where `probe` ran
+and is gitignored. On a fresh clone, or on a laptop where CI does the
+probing, the session-start brief is empty and `status` says `unprobed` while
+the default branch has full evidence. Point either command at CI's document:
+
+```bash
+testguard status . --evidence .testguard/ci/ci-self-evidence.json
+testguard brief . --text --evidence .testguard/ci/ci-self-evidence.json
+```
+
+A foreign document is not trusted blindly. `status` marks it
+`evidenceSource: provided`, prints the commit it describes next to the
+commit in your tree, and still computes staleness from the recorded input
+hashes — so a file you have edited since CI probed it goes `evidence-stale`
+for exactly those claims.
+
+`testguard init --ci-evidence github` (or `gitlab`) writes
+`.testguard/fetch-ci-evidence.sh`, which downloads the branch-named artifact
+with the platform CLI you already have and then briefs from it. **It is an
+on-demand helper, not a hook**: the session-start hook never touches the
+network, and the helper exits 0 with a message when the CLI or the artifact
+is missing. TestGuard itself still makes no network calls.
 
 ### Every change needs a claim
 
@@ -185,16 +318,100 @@ detected base that does not resolve is a warning for `status` and `brief`
 # GitHub Actions
 - uses: actions/checkout@v4
   with: { fetch-depth: 0 }
-- uses: raccioly/testguard@v0.5.0
+- uses: raccioly/testguard@v0.6.0
   with: { command: gate }
 
-# GitLab CI — or include: remote: the template in packaging/gitlab/
-testguard:gate:
-  image: node:22
-  rules: [{ if: $CI_PIPELINE_SOURCE == "merge_request_event" }]
-  script: [npx -y testguard-cli gate .]
+# GitLab CI — the component-shaped template: gate + probe, brief as an artifact and, opted in, as a merge-request note
+include:
+  - remote: 'https://raw.githubusercontent.com/raccioly/testguard/v0.5.0/packaging/gitlab/testguard.gitlab-ci.yml'
+    inputs: { dir: backend, post_note: true }   # post_note needs TESTGUARD_GITLAB_TOKEN (api scope); one note, updated in place
+```
+
+The template carries `spec: inputs:` (`version`, `dir`, `image`, `severity`,
+`confirm`, `budget`, `no_escalate`, `strict`, `post_note`, `stage`), so
+mirrored into a GitLab project it is a catalog component that a compliance
+framework can require on every project in a group. The CLI itself never
+talks to the network; only the job posts, and only when told to.
+
+### Properties
+
+- **Deterministic measurement.** No model decides a verdict. Faults are
+  string substitutions from a reviewed file; verdicts come from the test
+  runner's structured report, confirmed N times. The same inputs give the
+  same evidence, which is what makes it replayable for an auditor.
+- **No network, no telemetry.** The CLI never opens a socket. The
+  session-start hook resolves an installed binary and never fetches one.
+  Nothing leaves the machine unless a CI job you configured posts it.
+- **Artifacts are data in the repository.** Claims, evidence, baseline and
+  brief are JSON validated against a published schema before they are
+  written. Nothing is healed, regenerated or re-targeted at run time; a
+  claims file is code, and is reviewed like code.
+- **Never optimistic.** A timeout, a load failure, a mixed N-run result, a
+  flaky defender or a missing anchor is reported as unproven, never rounded
+  toward green. The one pass is `killed`; everything else gates.
+- **One exact-pinned runtime dependency** (`ajv`), Node ≥ 20, MIT.
+
+### What TestGuard is not
+
+- **Not a test generator.** It judges a test the agent wrote (`admit`); the
+  generating half stays with the agent, where the market is putting it.
+- **Not a mutation-score dashboard.** No blanket mutants, no single score, no
+  threshold. Faults are few and bound to stated claims; findings are ranked,
+  never summed.
+- **Not a self-healing runner.** A defender that adapts itself to a change is
+  a defender that did not observe it; TestGuard reports that, it does not do
+  it.
+- **Not a coverage tool.** A line executed says nothing about whether an
+  assertion would notice. `NOCOVER` here means no test even imports the
+  file; everything above that is measured by injecting the fault.
+
+A claim that **disappeared** is invisible to all of the above: `probe`
+verifies the claims that exist, `gate` sees the source file covered by some
+other claim, and `status` says clean. Deleting a claim is therefore cheaper
+than weakening its fault — and `changedFaults` exists precisely because
+weakening was the cheap escape. So:
+
+```bash
+npx testguard-cli claims --since origin/main    # what existed there and does not now
+```
+
+`removed-claim` and `removed-fault` gate; a rename that keeps the statement
+verbatim is reported as `renamed-claim` and does not. Where evidence is
+present, the line names the verdict the claim last had, because *"it was
+SURVIVED when it was removed"* is the sentence that matters. The escape hatch
+is the same one as everywhere else: a `claim` entry in
+`testguard.ignore.json` with a reason, expiring if you give it an `expires`.
+Removal is allowed — claims can be wrong, superseded or split — and it is
+never silent.
+
+### Run it from any harness
+
+The operating loop above lives in a Claude Code skill and a session-start
+hook. Both vanish the moment the harness is Cursor, Codex, Devin or whatever
+comes next — and the harness is exactly the layer most likely to change. So
+the same documents are also served over the Model Context Protocol:
+
+```bash
+npx testguard-cli mcp            # JSON-RPC 2.0 on stdio
+npx testguard-cli init --mcp     # prints the config for Claude Code, Cursor and Codex
 ```
 
+Five tools, all **read-only**: `testguard_status`, `testguard_brief`,
+`testguard_claims`, `testguard_evidence`, `testguard_next_command`.
+
+**No tool runs a probe.** A probe is long-running, budgeted, and the person
+should see it happen, so `next_command` hands back the exact shell line for
+the agent to run in its own terminal. Nothing here writes a file either —
+editing a claims file through a connector would defeat the point of recording
+every fault edit. The server declares only a `tools` capability: no prompts,
+no resources, no sampling.
+
+It is hand-written against a pinned protocol version rather than built on the
+official SDK, because this tool has one exact-pinned runtime dependency and
+keeps it that way. The surface is three methods and five tools and will not
+grow; tests speak the wire format to a real child process and check each
+tool against the CLI's own `--json` output, so the two cannot drift.
+
 ### Built for agents to run
 
 TestGuard is meant to be driven by an AI agent, not typed by a person. Three
@@ -205,11 +422,30 @@ things make that safe:
   working tree. Every human rendering — the CLI text, the session-start
   brief, the skill — derives from it, so they cannot disagree. Every command
   accepts `--json`.
-- **An installable operating loop.** `testguard init` writes
+- **An installable operating loop.** `testguard init [dir]` writes the
+  **agent layer at the git root**, where agent sessions run —
   `.claude/skills/testguard/SKILL.md` (state → action, verdict → the only
   acceptable fix, the two-gate rule for any test the agent writes), the
-  `brief --text` session-start hook, an `AGENTS.md` section and the
-  `.gitignore` lines. Idempotent.
+  `brief --text` session-start hook, an `AGENTS.md` section — and the
+  **project layer** (`.gitignore` lines) beside the claims file. A second
+  project in the same repository adds a hook line and an `AGENTS.md`
+  bullet; `--here` keeps everything in the subdirectory. The hook
+  resolves the project's `node_modules/.bin/testguard`, then the git root's,
+  then a `testguard` on `PATH` via `command -v`, then does nothing — **no
+  `npx` in any form, so no path to the network**; a pre-0.6 `npx -y` or
+  `npx --no-install` hook is replaced. A written file that
+  `.gitignore` swallows is reported, not offered for commit. Idempotent.
+- **Independence is recorded (L3).** `probe` measures *power* — would this
+  test notice if the code were wrong. It also records, on every kill, whether
+  the killing test was last touched by the same change (or the same author) as
+  the code it guards: `detail.independence` is `co-authored`,
+  `separate-change` or `unknown`. A co-authored kill is legitimate — a fix
+  *should* ship with its regression test — but a repository where every kill
+  is co-authored has no independent verification, whatever its claim
+  verification rate says. It is a **signal**: ranking reads it, verdicts never
+  do, and the brief says it in one line. Nothing else in this category
+  measures it, and agents saturate the tests they can see
+  ([SpecBench](https://arxiv.org/abs/2605.21384)).
 - **Gaming is visible.** The cheapest way to make a survivor disappear is to
   weaken its fault, not to write a test. Evidence records every fault's
   content hash; `status` lists any fault edited after it survived, with its
@@ -219,7 +455,7 @@ things make that safe:
 ### Authoring faults mechanically
 
 Writing faults by hand means reading the code to find exact anchors. Two
-field reports found that ~80% of hand-written faults are one of five shapes,
+field reports found that ~80% of hand-written faults are one of nine shapes,
 so `scaffold` proposes them for you:
 
 ```bash
@@ -234,6 +470,10 @@ npx testguard-cli scaffold src/auth.ts --claim AUTH-ADMIN   # every proposal und
 | `return-altered` | `return <check>;` (`===`, `.includes(`, `&&`, …) → `return true;` |
 | `literal-changed` | `httpOnly`/`secure` flipped, `sameSite` → `none`, a cost/rounds → `1`, a ttl/tolerance/window/limit ×1000 |
 | `call-removed` | a bare `verify…()` / `validate…()` / `check…()` / `authorize…()` call removed |
+| `field-dropped` | a field removed from an object that is returned, built by an arrow, assigned to a payload-ish name, passed to a `save`/`update`/`send`/`write`/… call, or is a `z.object({…})`-style schema; a string entry removed from an allow-list array; a `...base,` line or inline `{ ...base, … }` merge dropped. Only a line that can go on its own (balanced, comma-terminated or followed by the closer); never inside tests, fixtures or migrations |
+| `argument-swapped` | a call kept, its first argument swapped for `undefined` (and `{}` when the argument is itself a call) — only when that argument is derived from a parameter of the enclosing function or a request-like value (`req`, `ctx`, `event`, …). The seam fault: `decide(deriveFrom(input), now)` → `decide(undefined, now)` |
+| `element-removed` | a one-line JSX element — self-closing (`<Toggle … />`) or paired (`<button …>Save</button>`) — removed; the UI shape behind "the toggle is invisible", killable by a browser-layer defender |
+| `handler-dropped` | an `on<Event>={…}` prop removed, whether it is its own line or inline in the tag; the control renders and does nothing |
 
 Every proposal's `find` is the exact line with `expectHits`/`occurrence`
 computed from the file, so it is verifiable by construction; provenance is
@@ -269,14 +509,18 @@ through every verdict.
 
 ## Status
 
-**v0.4.** Eight commands, vitest and jest runners, hand-authored faults plus
+**v0.5.** Eleven commands (`status`, `init`, `claims`, `probe`, `admit`, `replay`, `baseline`, `brief`, `gate`, `scaffold`, `mcp`), vitest and jest runners, hand-authored faults plus
+**v0.5.** Nine commands (`status`, `init`, `claims`, `probe`, `admit`, `baseline`, `brief`, `gate`, `scaffold`), vitest, jest and Playwright runners, hand-authored faults plus
 a mechanical scaffold, an agent operating layer (`status`, `init`) and a
 change gate (`gate`). The contract
 spine — eight JSON Schemas shared with the other Guard tools — is under
 [`spec/`](spec/). One exact-pinned runtime dependency (`ajv`, for schema validation); Node ≥ 20.
 
-Not yet: test generation (the two-gate acceptance loop), runners beyond
-vitest and jest, AST-aware producers, and calibration of fault classes
+Not yet: test generation (the acceptance half, `admit`, exists; the generating half stays the agent's), runners beyond
+vitest and jest, AST-aware producers, and the transfer of a calibration between repositories — `replay` measures it
+now; whether it carries to a repository with no history is unproven. Each is
+designed for; none is claimed.
+vitest, jest and Playwright, AST-aware producers, and calibration of fault classes
 against real escaped bugs. Each is designed for; none is claimed.
 
 ## Licence

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "testguard-cli",
-  "version": "0.5.0",
+  "version": "0.6.0",
   "description": "Proves a test suite defends the claims a project makes: injects the faults those claims forbid and reports every one the tests fail to detect.",
   "type": "module",
   "license": "MIT",

package/spec/GATE-SEMANTICS.md

@@ -32,7 +32,14 @@ Rules that follow from the table:
    timeout does not, and a suite that fails to load does not: neither is
    evidence that the suite defends the claim. Tools parse the runner's
    structured report, never its exit code, because the exit code cannot tell
-   these apart.
+   these apart. A runner's own retry mechanism does not change this: a test
+   that failed and then passed on retry (Playwright's `flaky`) is a
+   **non-green run** even when the runner exits 0, and a test the runner
+   reports as timed out (`timedOut`) is a timeout, never an assertion
+   failure. When one claim's defenders run under several runners, their
+   runs merge pessimistically — any load error is an error, any timeout is
+   a timeout, any failure is a failure — and the evidence names which file
+   ran under which runner.
 4. **A mixed result is not a kill.** If some of the N probe runs fail and
    others pass, the defender's response to the fault is nondeterministic.
    Reporting `killed` would be the optimistic bias flakiness introduces;
@@ -43,6 +50,11 @@ Rules that follow from the table:
    and `detail.undeclaredKillers` names the tests, so the author can fix
    `defendedBy`. The claim's stated evidence chain is broken even though the
    suite is not blind. It gates, and ranks below a true survivor.
+   Attribution runs **up to N times**, and stops as soon as no test has failed
+   in every run so far: from there no further run can name a killer. The
+   evidence records the early stop (`detail.escalationStoppedEarly`). Stopping
+   can only fail to *name* a killer, never upgrade a verdict, which is the
+   pessimistic side.
 6. **A verdict names the commit it is about.** Evidence records `repo.head`;
    when the working tree was probed instead, `repo.snapshot` holds the
    throwaway commit that captured it. A tool must refuse to probe a commit
@@ -57,6 +69,35 @@ Rules that follow from the table:
    changed since it was probed, with its previous verdict. The change is
    allowed; it is never invisible.
 
+## Replay reports; it never gates
+
+A replayed bug is history. It escaped, by definition, which means the suite
+missed it; a gate on that is a gate nobody can pass on a first run, and it
+would only teach teams to stop looking. `replay` therefore always exits `0`
+unless it could not run at all.
+
+Its verdicts mirror the probe's, for the same reasons:
+
+| Verdict | Meaning |
+|---|---|
+| `caught` | a remaining test failed **by assertion** on the reverted source, every run. The suite knew. |
+| `blind` | the suite stayed green on known-broken code. |
+| `nocover` | no test imports the reverted files. Worse than `blind`: nothing was even tried. |
+| `unverifiable` | the revert did not apply, the suite failed to load, or it timed out. Carries a `reason`. |
+| `flaky` | the runs disagreed. A flaky failure reads as "the suite caught it", so flakiness biases this metric **optimistically** — mixed runs are never `caught`. |
+
+Two rules that follow:
+
+1. **One patch, one row.** De-duplicate by `git patch-id`; a dual-branch
+   topology carries the same fix under two or three shas, and counting it
+   twice corrupts the corpus a calibration is computed from.
+2. **The fix's own test is removed before the run.** It proves nothing about
+   what the suite knew before the fix existed.
+
+Only `caught` and `blind` carry information. `nocover`, `flaky` and
+`unverifiable` are excluded from both sides of a calibration ratio, because a
+number computed over them would mean nothing.
+
 ## Baseline and delta
 
 A baseline freezes the fingerprints of every non-passing finding at a point
@@ -72,6 +113,15 @@ derived from identities and outcome, never from the fault's `find`/`replace`
 text, so repairing a rotted anchor does not churn the baseline — while a
 change of verdict on the same claim+subject does surface as new.
 
+A baseline records the commit its evidence was taken at. When that evidence
+came from a working-tree snapshot, the baseline says so (`snapshot`) and its
+`head` is the **parent** of the commit that will carry the tests. A tool may
+offer to re-stamp such a baseline onto a later commit, but only when a clean
+probe of that commit reproduced exactly the same fingerprints; otherwise the
+user re-probes and freezes a new baseline. A frozen contract is never
+silently rewritten, and a status document notes a baseline that predates
+HEAD without turning that into a state.
+
 Adopting tools may reconcile an existing baseline format (for example a
 `{version, fingerprints:{hash:count}}` file) by mapping it onto this shape;
 the suppress-up-to-count semantics are identical.
@@ -84,6 +134,13 @@ the suppress-up-to-count semantics are identical.
   for reasonless `path` entries.
 - An annotation is **strictly additive**. It never changes, suppresses, or
   drops a finding. Ranking may read annotations; verdicts never do.
+- A **signal** is a static, annotation-grade fact about a defender, recorded
+  on the evidence and never a verdict. `mocked-never-asserted`: a test file
+  mocks the subject's module and never asserts on anything imported from it.
+  An author may silence it with `unasserted: <reason>` above the mock; the
+  tool then records `unasserted-annotated` with the reason — silenced, never
+  hidden. A file that mocks the subject is never a discovered defender: a
+  mock cannot detect a fault in what it replaced.
 
 ## Claim coverage of a change
 

package/spec/README.md

@@ -22,6 +22,7 @@ The pattern all such tools run:
 | `calibration` | [`schemas/calibration.schema.json`](schemas/calibration.schema.json) | P(finding is real) per bucket, with a Wilson interval and the sample size behind it. |
 | `brief` | [`schemas/brief.schema.json`](schemas/brief.schema.json) | What to tell an agent before it writes code — ranked, capped, never a single score, and carrying the one next action. |
 | `status` | [`schemas/status.schema.json`](schemas/status.schema.json) | Where the project is and what happens next — the single machine-readable truth every human rendering derives from. Surfaces faults whose content changed since they were probed, and changed files that carry no claim. |
+| `replay` | [`schemas/replay.schema.json`](schemas/replay.schema.json) | Would this suite have caught the bugs that already escaped? Replays real fix commits: revert the source, remove the test the fix shipped, run what remains. A replayed bug is ground truth — a human already confirmed it was a defect — which is what a fault model is calibrated against. |
 | `gate` | [`schemas/gate.schema.json`](schemas/gate.schema.json) | Claim coverage of one change: which changed files carry a claim, which are excused (and by which ignore entry), which are unclaimed. The delta gate for code that has no claim yet. |
 
 Shared definitions (verdicts, fault classes, provenance, annotations) live in

package/spec/lib/validate.mjs

@@ -6,7 +6,7 @@ import { fingerprint } from './fingerprint.mjs';
 
 const schemaDir = join(dirname(fileURLToPath(import.meta.url)), '..', 'schemas');
 
-export const KINDS = Object.freeze(['claims', 'evidence', 'baseline', 'ignore', 'calibration', 'brief', 'status', 'gate']);
+export const KINDS = Object.freeze(['claims', 'evidence', 'baseline', 'ignore', 'calibration', 'brief', 'status', 'gate', 'replay']);
 export const PASSING_VERDICTS = Object.freeze(new Set(['killed']));
 
 const ajv = new Ajv2020({ strict: true, allErrors: true });
@@ -47,6 +47,8 @@ const semantic = {
     const n = doc.run.confirmRuns;
     if (n < 3 && doc.run.provisional !== true) errors.push({ path: '/run/provisional', message: `confirmRuns ${n} is below 3; the run must declare provisional: true` });
     if (n >= 3 && doc.run.provisional === true) errors.push({ path: '/run/provisional', message: `confirmRuns ${n} is confirmed; provisional must be absent or false` });
+    if (doc.run.runners && doc.run.runner && !doc.run.runners.some((x) => x.name === doc.run.runner.name)) errors.push({ path: '/run/runners', message: 'runners must include the project runner named in run.runner' });
+    if (doc.run.repo.ignoredDirty && doc.run.repo.snapshot) errors.push({ path: '/run/repo/ignoredDirty', message: 'a working-tree snapshot has no ignored dirty files: the tree was probed as it is' });
     doc.records.forEach((r, i) => {
       const p = `/records/${i}`;
       const expected = fingerprint({ claimId: r.claim.id, subjectId: r.subject.id, file: r.subject.file ?? '', verdict: r.verdict });
@@ -74,9 +76,31 @@ const semantic = {
       if (r.verdict === 'nocover' && !r.defenders.nocover) {
         errors.push({ path: `${p}/defenders/nocover`, message: 'nocover verdict requires defenders.nocover = true' });
       }
+      if (r.detail.flakeRate) {
+        const { runs, failures } = r.detail.flakeRate;
+        if (failures > runs) errors.push({ path: `${p}/detail/flakeRate`, message: `flakeRate failures (${failures}) exceed runs (${runs})` });
+        if (failures > 0 && r.verdict !== 'flaky-defender') errors.push({ path: `${p}/detail/flakeRate`, message: 'a defender that failed on unmodified source makes the verdict flaky-defender' });
+      }
+      if (r.detail.independence) {
+        if (r.verdict !== 'killed') errors.push({ path: `${p}/detail/independence`, message: 'independence is a property of a kill; only killed records carry it' });
+        const { class: cls, defenderCommit, targetCommit } = r.detail.independence;
+        if (cls !== 'unknown' && !(defenderCommit && targetCommit)) errors.push({ path: `${p}/detail/independence`, message: `independence class ${cls} requires both defenderCommit and targetCommit` });
+      }
       if (r.detail.undeclaredKillers && r.detail.reason !== 'killed-by-undeclared-tests') {
         errors.push({ path: `${p}/detail/undeclaredKillers`, message: 'undeclaredKillers is only meaningful with reason killed-by-undeclared-tests' });
       }
+      if (r.defenders.byRunner) {
+        const resolved = new Set(r.defenders.resolved);
+        for (const [runner, files] of Object.entries(r.defenders.byRunner)) for (const f of files) if (!resolved.has(f)) errors.push({ path: `${p}/defenders/byRunner/${runner}`, message: `${f} ran under ${runner} but is not a resolved defender` });
+        if (Object.keys(r.defenders.byRunner).length < 2) errors.push({ path: `${p}/defenders/byRunner`, message: 'byRunner is only meaningful when more than one runner ran the defenders' });
+      }
+      if (r.defenders.mocking && r.defenders.discovered) {
+        for (const f of r.defenders.mocking) if (r.defenders.resolved.includes(f)) errors.push({ path: `${p}/defenders/mocking`, message: `${f} mocks the subject and was discovered; it cannot also be a resolved defender` });
+      }
+      for (const s of r.defenders.signals ?? []) {
+        if (s.signal === 'unasserted-annotated' && !s.reason) errors.push({ path: `${p}/defenders/signals`, message: `unasserted-annotated on ${s.file} requires the annotation's reason` });
+        if (!(r.defenders.mocking ?? []).includes(s.file)) errors.push({ path: `${p}/defenders/signals`, message: `${s.file} carries a mock signal but is not listed in defenders.mocking` });
+      }
       if (r.verdict === 'unverifiable' && !r.detail.reason) {
         errors.push({ path: `${p}/detail/reason`, message: 'unverifiable requires a reason (e.g. anchor-missing, anchor-ambiguous)' });
       }
@@ -104,10 +128,42 @@ const semantic = {
     if (doc.state === 'clean' && (doc.counts.new ?? 0) > 0) errors.push({ path: '/state', message: 'clean with new findings' });
     if (doc.state === 'unclaimed-changes' && !(doc.changes && doc.changes.uncovered.length > 0)) errors.push({ path: '/changes/uncovered', message: 'unclaimed-changes requires at least one uncovered changed file' });
     if (doc.state === 'unclaimed-changes' && doc.next.action !== 'claim') errors.push({ path: '/next/action', message: 'unclaimed-changes requires next.action = claim' });
+    if (doc.evidenceSource && !doc.evidenceHead) errors.push({ path: '/evidenceHead', message: 'evidence was read, so the commit it describes must be recorded' });
     if (doc.changes && doc.changes.uncovered.length > 0 && !['no-claims', 'unclaimed-changes'].includes(doc.state)) errors.push({ path: '/state', message: 'uncovered changed files are hidden behind a later state; unclaimed-changes precedes every evidence state' });
     return errors;
   },
 
+  replay(doc) {
+    const errors = [];
+    const n = doc.run.confirmRuns;
+    const seen = new Set();
+    doc.records.forEach((r, i) => {
+      const p = `/records/${i}`;
+      // One patch, one row: a dual-branch topology carries the same fix under
+      // two or three shas, and counting it twice corrupts the corpus the
+      // calibration is computed from.
+      if (seen.has(r.patchId)) errors.push({ path: `${p}/patchId`, message: `duplicate patch-id ${r.patchId.slice(0, 12)}: the same fix counted twice` });
+      seen.add(r.patchId);
+      if (['caught', 'blind'].includes(r.verdict) && r.runs.length !== n) {
+        errors.push({ path: `${p}/runs`, message: `${r.verdict} requires exactly confirmRuns (${n}) runs, got ${r.runs.length}` });
+      }
+      // Only a test body rejecting the reverted source counts as caught —
+      // the same rule as `killed`, for the same reason.
+      if (r.verdict === 'caught' && !r.runs.every((x) => x.outcome === 'fail' && (x.assertionFailures ?? 0) > 0)) {
+        errors.push({ path: `${p}/runs`, message: 'caught requires every run to fail with at least one assertion failure' });
+      }
+      if (r.verdict === 'blind' && !r.runs.every((x) => x.outcome === 'pass')) {
+        errors.push({ path: `${p}/runs`, message: 'blind requires every run to pass: the suite stayed green on known-broken code' });
+      }
+      if (r.verdict === 'flaky' && !(r.runs.some((x) => x.outcome === 'pass') && r.runs.some((x) => x.outcome === 'fail'))) {
+        errors.push({ path: `${p}/runs`, message: 'flaky requires runs that disagree' });
+      }
+      if (r.verdict === 'nocover' && r.ranTests) errors.push({ path: `${p}/ranTests`, message: 'nocover means no test exercises the reverted files; it cannot have run tests' });
+      if (r.verdict === 'unverifiable' && !r.reason) errors.push({ path: `${p}/reason`, message: 'unverifiable requires a reason (e.g. revert-did-not-apply, suite-failed-to-load)' });
+    });
+    return errors;
+  },
+
   gate(doc) {
     const errors = [];
     if (doc.evaluated !== doc.covered.length + doc.uncovered.length) errors.push({ path: '/evaluated', message: `evaluated (${doc.evaluated}) must equal covered (${doc.covered.length}) + uncovered (${doc.uncovered.length})` });

package/spec/schemas/baseline.schema.json

@@ -15,6 +15,14 @@
       "description": "Whether the working tree had uncommitted changes when the evidence behind this baseline was taken.",
       "type": "boolean"
     },
+    "snapshot": {
+      "description": "When the evidence behind this baseline was a working-tree snapshot: that throwaway commit. `head` is then the parent of the commit that will carry the tests; `baseline --restamp` moves `head` once a clean probe reproduces the same fingerprints.",
+      "$ref": "urn:guard-spec:v1:common#/$defs/gitSha"
+    },
+    "restampedAt": {
+      "description": "Set when `head` was moved by `baseline --restamp` (same fingerprints, clean tree). `createdAt` keeps the original freeze time.",
+      "$ref": "urn:guard-spec:v1:common#/$defs/isoDateTime"
+    },
     "fingerprints": {
       "type": "object",
       "propertyNames": { "pattern": "^[a-f0-9]{64}$" },