testguard-cli 0.4.0 → 0.6.0

package/README.md

@@ -44,6 +44,19 @@ re-implemented the authorization logic *inside the test* and asserted
 against the copy: fifteen green tests, zero detection. After wrapper-level
 tests were written against the survivors, 39/39 were killed.
 
+The peer-reviewed picture in 2026 says the same thing from the other side.
+Coverage and mutation score of LLM-generated suites track effectiveness only
+when the code under test is assumed correct; once it may be buggy they "no
+longer serve as reliable indicators" ([Zhao, Zhou and Cohen, ISSTA 2026](https://arxiv.org/abs/2607.22880)).
+Buggy code steers a model toward tests that assert the bug, and prompting
+with the specification is the mitigation that works ([arXiv 2607.22883](https://arxiv.org/abs/2607.22883)) —
+which is why a claim here comes from intent, a fault is bound to the claim,
+and the brief hands the agent the claim before it writes. And agents
+saturate whatever tests they can see, with the gap to held-out tests
+growing about 28 points per tenfold increase in code size ([SpecBench](https://arxiv.org/abs/2605.21384)).
+Every generator on the market admits a test because it compiles, passes and
+raises coverage. TestGuard admits it because it fails when the claim is false.
+
 ## Install
 
 | How | Command |
@@ -52,8 +65,9 @@ tests were written against the survivors, 39/39 were killed.
 | npm | `npm i -D testguard-cli` then `npx testguard probe` |
 | pip | `pip install testguard-cli` then `testguard probe` (needs Node ≥ 20) |
 | Homebrew | `brew tap raccioly/tap && brew install testguard` |
-| GitHub Action | `uses: raccioly/testguard@v0.4.0` — see [`action.yml`](./action.yml) |
+| GitHub Action | `uses: raccioly/testguard@v0.6.0` — see [`action.yml`](./action.yml) |
 | pre-commit | `repo: https://github.com/raccioly/testguard`, hooks `testguard-claims`, `testguard-probe` |
+| GitLab CI | `include: - remote: https://raw.githubusercontent.com/raccioly/testguard/v0.5.0/packaging/gitlab/testguard.gitlab-ci.yml` with `inputs:` — see [`packaging/gitlab/`](./packaging/gitlab/testguard.gitlab-ci.yml) |
 
 Projects that set `min-release-age` in `.npmrc` cannot see a version published
 less than that many days ago (`ENOVERSIONS`); install that one with
@@ -68,7 +82,9 @@ npx testguard-cli claims      # what does this project claim, and is every claim
 npx testguard-cli probe       # try to falsify each claim; report what the tests missed
 npx testguard-cli baseline    # freeze today's unproven findings; from now on only new ones gate
 npx testguard-cli brief       # tell the agent where the suite is blind, before it writes
+npx testguard-cli gate --changed origin/main   # fail when a changed source file carries no claim at all
 npx testguard-cli scaffold src/x.ts   # propose faults for a file, as a draft to keep or drop
+npx testguard-cli admit test/x.test.ts --claim X   # is this test green on HEAD and does it fail on every fault of X? ADMITTED or NOT ADMITTED
 ```
 
 1. **Claims** live in `testguard.claims.json` (editors validate it against
@@ -82,7 +98,12 @@ npx testguard-cli scaffold src/x.ts   # propose faults for a file, as a draft to
 2. **Probe** confirms the defenders are green N times unmodified, applies
    each fault in a scratch git worktree (your tree is never touched), runs
    the defenders N times, re-runs survivors against the whole suite with
-   N-run attribution, restores, and classifies. Verdicts are a closed set:
+   N-run attribution (up to N: it stops as soon as no test has failed in
+   every run, because from there none can be named a killer), restores, and
+   classifies. It warns when another test runner is already running — a
+   contended machine turns a slow suite into a `TIMEOUT` verdict about the
+   load, not the claim — records that on the evidence, and offers `--serial`
+   to run one test file at a time. Verdicts are a closed set:
 
    | Verdict | Meaning |
    |---|---|
@@ -92,7 +113,7 @@ npx testguard-cli scaffold src/x.ts   # propose faults for a file, as a draft to
    | `UNVERIFIABLE` | the fault's anchor is missing or ambiguous — loud, never a skip |
    | `TIMEOUT` | the defenders hung; a hang is not a detection |
    | `FAULT-INVALID` | the replacement does not load — a bad fault, not a finding |
-   | `FLAKY-DEFENDER` | the defenders are not reliably green, or disagreed across runs |
+   | `FLAKY-DEFENDER` | the defenders are not reliably green, or disagreed across runs — `detail.flakeRate` says how often (`failures` of `runs` on unmodified source) |
 
    Never a single score. Findings are ranked by severity, claim provenance
    and blast radius (relative imports, `tsconfig` path aliases and
@@ -101,20 +122,48 @@ npx testguard-cli scaffold src/x.ts   # propose faults for a file, as a draft to
    it is written.
 
    Worktree mode probes a **commit**. If a defender or target file has
-   uncommitted changes, `probe` refuses and says so — otherwise your new
-   tests would be silently absent and the same survivors would come back
-   with no hint why. `--include-dirty` snapshots the working tree (tracked
-   edits and new files) into a throwaway commit and probes that; your tree,
-   HEAD and index are never touched. Every summary names the commit probed.
+   uncommitted changes and you asked for the implicit HEAD, `probe` refuses
+   and says so — otherwise your new tests would be silently absent and the
+   same survivors would come back with no hint why. `--include-dirty`
+   snapshots the working tree (tracked edits and new files) into a throwaway
+   commit and probes that; your tree, HEAD and index are never touched. An
+   **explicit `--ref`** (a pre-fix commit in a post-mortem, say) is honoured
+   over a dirty tree, as is `--ignore-dirty`: a warning names the files and
+   the evidence records them as `repo.ignoredDirty`, so the run says what it
+   did not look at. Every summary names the commit probed.
 
    A claim with no `defendedBy` has its defenders **discovered**: the test
-   files that import the fault's target, by relative path or resolved alias.
-   `NOCOVER` then means exactly "no test file imports this source".
+   files that import the fault's target, by relative path or resolved alias
+   (tsconfig `paths`, through `extends` and `references`, vite/vitest
+   `resolve.alias`, package.json `imports`) — **minus the files that mock
+   it**. A test that `vi.mock`s / `jest.mock`s the target cannot detect any
+   fault in it; counting it would make `NOCOVER` under-report and waste runs.
+   `NOCOVER` therefore means exactly "no test file imports this source
+   without mocking it". `claims` prints the split (`18 import · 16 mock · 2
+   can detect`), the evidence lists the mocking files, and a mocking file
+   that never `expect(...)`s anything imported from the target carries the
+   static signal `mocked-never-asserted` — the cheapest blind-spot signal
+   there is, and the exact signature of one escaped bug in the field
+   reports. Silence it, visibly, with `// unasserted: <why>` above the mock.
 
    Runners: **vitest** and **jest** (`--runner auto` picks the first that
-   resolves; both read the same jest-compatible JSON report). Anything else
-   goes through `--runner-cmd`. Each runner is proven against its own copy
-   of the known-answer fixture.
+   resolves; both read the same jest-compatible JSON report). A runner is
+   resolved from the **project's own package** first — its pinned version,
+   its own bin script — and only then from an executable on PATH, which the
+   evidence records as `runner.source: "path"`; `npx` is never asked,
+   because its cache answers for packages a project does not have. Anything
+   else goes through `--runner-cmd`. Each runner is proven against its own
+   copy of the known-answer fixture.
+
+   The two-gate rule, as one verb: `testguard admit <test-file> --claim <ID>`
+   runs the claim's faults against your uncommitted test and answers
+   `ADMITTED` (exit 0: the test is green on unmodified HEAD and fails on
+   every fault, N/N) or `NOT ADMITTED` (exit 1: the first blocking fault and
+   what to do about it). The test must be a declared or discovered defender
+   of the claim. It is sugar over `probe --claim <ID> --include-dirty`, so
+   it can never disagree with the gate. Every generator on the market admits
+   a test because it compiles, passes and raises coverage; `admit` admits it
+   because it fails when the claim is false.
 
    Practical loop: first pass `--no-escalate` (escalation re-runs the whole
    suite N times per survivor); iterate on one claim with `--claim <ID>` and
@@ -131,20 +180,237 @@ npx testguard-cli scaffold src/x.ts   # propose faults for a file, as a draft to
 3. **Baseline** freezes every non-passing fingerprint. Later probes suppress
    what was already known and exit non-zero only on what is new. Claims whose
    source and defenders are unchanged reuse their prior verdict, so a probe
-   in CI costs only what changed.
+   in CI costs only what changed. A baseline frozen from `--include-dirty`
+   evidence records the snapshot and points at the *parent* of the commit
+   that will carry your tests; after you commit, a clean `probe` plus
+   `baseline --restamp` moves it to that commit — only when the fingerprints
+   are identical, never otherwise. `status` notes a baseline that predates
+   HEAD without making it a state.
 4. **Brief** turns evidence plus baseline into a ranked, capped
    `## TEST BLINDSPOT CONTEXT` block, printed and also written to
-   `.testguard/brief.json` (`--text` prints only). Wire it into an agent's session start
+   `.testguard/brief.json` (`--text` prints only; `--markdown` prints the same brief as a
+   merge-request note for the human reviewer, unclaimed changes first). Wire it into an agent's session start
    — for Claude Code, in `.claude/settings.json`:
 
    ```json
    { "hooks": { "SessionStart": [ { "hooks": [
-     { "type": "command", "command": "npx testguard-cli brief --text" }
+     { "type": "command", "command": "node_modules/.bin/testguard brief --text 2>/dev/null || { command -v testguard >/dev/null 2>&1 && testguard brief --text 2>/dev/null; } || true" }
    ] } ] } }
    ```
 
-   `--text` prints only, and exits 0 silently when there is no evidence yet,
-   so the hook can never break a session.
+   `--text` prints only, and exits 0 silently when there is no evidence yet;
+   the command resolves the project's own install, then a `testguard` on
+   `PATH`, and ends in `true` — so the hook can never break a session. There
+   is no `npx` in it, in any form: `--no-install` resolves the package from
+   the registry before declining to install it, so it is quiet rather than
+   offline. The brief's first line says which install answered
+   (`local install` or `global`), so a stale one is visible.
+
+### Would this suite have caught the bugs that already escaped?
+
+An injected fault is a fault somebody thought of. A bug that actually shipped
+is ground truth: a human already confirmed it was a defect, and there is no
+equivalent-mutant argument to have about it.
+
+```bash
+npx testguard-cli replay --since HEAD~50..HEAD --max 10
+```
+
+For each fix commit in the range — one that changes source **and** a test
+together — `replay` checks it out in a scratch worktree, reverts only its
+source files to the parent, **deletes the test the fix shipped** (that test
+proves nothing about what the suite knew before it existed), and runs the
+tests that import the reverted code:
+
+| Verdict | Meaning |
+|---|---|
+| `caught` | a remaining test failed by assertion, every run. The suite knew. |
+| `blind` | the suite stayed green on known-broken code. |
+| `nocover` | no test imports the reverted files — worse than blind. |
+| `flaky` | the runs disagreed, so nothing can be concluded. A flaky failure reads as detection, which biases this metric *optimistically*; mixed runs are never `caught`. |
+| `unverifiable` | the revert did not apply, or the suite could not load. |
+
+One patch counts once (`git patch-id`), because a dual-branch topology
+carries the same fix under two or three shas. **`replay` reports and never
+gates**: a bug that escaped is history, not a regression in this change.
+
+Each replayed bug is also labelled with the injected-fault class its diff
+most resembles, which is the join key that turns an uninterpretable mutation
+score into a statement with a sample size:
+
+```
+  guard-removed          blind   7/9    p=0.78  ci [0.45, 0.94]
+  field-dropped          blind   4/4    p=1.00  ci [0.51, 1.00]
+```
+
+Read as *"when a fault of this class survives, how often does that
+correspond to a bug that really escaped"*. It is written as a
+`calibration` document beside the replay one. Only `caught` and `blind`
+carry information; the other verdicts are excluded from both sides of the
+ratio.
+
+**The open question this exists to answer.** Does a calibration learned on a
+repository *with* history transfer to a greenfield one that has none? AI
+authored code has no history, so replay cannot help it directly and
+calibration is the only bridge. That transfer is unproven, it is the core
+product bet, and this is the instrument for testing it — not the answer.
+
+### Read CI's evidence locally
+
+Verdict reuse makes a probe cheap, but the evidence lives where `probe` ran
+and is gitignored. On a fresh clone, or on a laptop where CI does the
+probing, the session-start brief is empty and `status` says `unprobed` while
+the default branch has full evidence. Point either command at CI's document:
+
+```bash
+testguard status . --evidence .testguard/ci/ci-self-evidence.json
+testguard brief . --text --evidence .testguard/ci/ci-self-evidence.json
+```
+
+A foreign document is not trusted blindly. `status` marks it
+`evidenceSource: provided`, prints the commit it describes next to the
+commit in your tree, and still computes staleness from the recorded input
+hashes — so a file you have edited since CI probed it goes `evidence-stale`
+for exactly those claims.
+
+`testguard init --ci-evidence github` (or `gitlab`) writes
+`.testguard/fetch-ci-evidence.sh`, which downloads the branch-named artifact
+with the platform CLI you already have and then briefs from it. **It is an
+on-demand helper, not a hook**: the session-start hook never touches the
+network, and the helper exits 0 with a message when the CLI or the artifact
+is missing. TestGuard itself still makes no network calls.
+
+### Every change needs a claim
+
+`probe` can only verify claims that exist. Every escaped defect in the field
+reports so far was a **claim gap**: the feature shipped green with zero
+claims, and a verifier with no claim about a feature is silent about it by
+construction. `gate` closes that hole on the delta:
+
+```bash
+npx testguard-cli gate --changed origin/main            # in a PR: the files changed since the base branch
+npx testguard-cli gate --changed HEAD --include-dirty   # before a commit: the working tree, staged or not
+```
+
+Every changed source file must carry a fault, resolve as a defender of a
+claim (test files), or be excused by an unexpired `path` entry in
+`testguard.ignore.json` — with a reason a reviewer will accept. One
+unclaimed file exits `1`; there is no percentage. Every reliance on an ignore
+entry is printed, so a reviewer sees *why* the gate passed; an expired entry
+excuses nothing. Non-source files and documented never-claimed patterns
+(`*.d.ts`, `*.config.*`, fixtures, mocks; `--explain` lists them) are
+excluded and said so; `--strict` fails a change that evaluated nothing.
+
+With a reference known, `status --changed <ref>` reports `unclaimed-changes`
+**before** any evidence state and makes the claim the next action; the brief
+lists the unclaimed files first. The claim is written before more code.
+
+**In CI the base is detected** — GitHub Actions (`GITHUB_BASE_REF`) and GitLab
+merge request pipelines (`CI_MERGE_REQUEST_DIFF_BASE_SHA`, then
+`CI_MERGE_REQUEST_TARGET_BRANCH_NAME`); `TESTGUARD_CHANGED_REF` overrides both.
+The base must exist locally: GitHub — `actions/checkout` with `fetch-depth: 0`;
+GitLab — the diff base sha needs nothing extra on a merge request pipeline,
+the branch name needs `GIT_DEPTH: 0` or a `git fetch origin <target>`. A
+detected base that does not resolve is a warning for `status` and `brief`
+(they keep working) and an error for `gate` (its whole job is the measurement).
+
+```yaml
+# GitHub Actions
+- uses: actions/checkout@v4
+  with: { fetch-depth: 0 }
+- uses: raccioly/testguard@v0.6.0
+  with: { command: gate }
+
+# GitLab CI — the component-shaped template: gate + probe, brief as an artifact and, opted in, as a merge-request note
+include:
+  - remote: 'https://raw.githubusercontent.com/raccioly/testguard/v0.5.0/packaging/gitlab/testguard.gitlab-ci.yml'
+    inputs: { dir: backend, post_note: true }   # post_note needs TESTGUARD_GITLAB_TOKEN (api scope); one note, updated in place
+```
+
+The template carries `spec: inputs:` (`version`, `dir`, `image`, `severity`,
+`confirm`, `budget`, `no_escalate`, `strict`, `post_note`, `stage`), so
+mirrored into a GitLab project it is a catalog component that a compliance
+framework can require on every project in a group. The CLI itself never
+talks to the network; only the job posts, and only when told to.
+
+### Properties
+
+- **Deterministic measurement.** No model decides a verdict. Faults are
+  string substitutions from a reviewed file; verdicts come from the test
+  runner's structured report, confirmed N times. The same inputs give the
+  same evidence, which is what makes it replayable for an auditor.
+- **No network, no telemetry.** The CLI never opens a socket. The
+  session-start hook resolves an installed binary and never fetches one.
+  Nothing leaves the machine unless a CI job you configured posts it.
+- **Artifacts are data in the repository.** Claims, evidence, baseline and
+  brief are JSON validated against a published schema before they are
+  written. Nothing is healed, regenerated or re-targeted at run time; a
+  claims file is code, and is reviewed like code.
+- **Never optimistic.** A timeout, a load failure, a mixed N-run result, a
+  flaky defender or a missing anchor is reported as unproven, never rounded
+  toward green. The one pass is `killed`; everything else gates.
+- **One exact-pinned runtime dependency** (`ajv`), Node ≥ 20, MIT.
+
+### What TestGuard is not
+
+- **Not a test generator.** It judges a test the agent wrote (`admit`); the
+  generating half stays with the agent, where the market is putting it.
+- **Not a mutation-score dashboard.** No blanket mutants, no single score, no
+  threshold. Faults are few and bound to stated claims; findings are ranked,
+  never summed.
+- **Not a self-healing runner.** A defender that adapts itself to a change is
+  a defender that did not observe it; TestGuard reports that, it does not do
+  it.
+- **Not a coverage tool.** A line executed says nothing about whether an
+  assertion would notice. `NOCOVER` here means no test even imports the
+  file; everything above that is measured by injecting the fault.
+
+A claim that **disappeared** is invisible to all of the above: `probe`
+verifies the claims that exist, `gate` sees the source file covered by some
+other claim, and `status` says clean. Deleting a claim is therefore cheaper
+than weakening its fault — and `changedFaults` exists precisely because
+weakening was the cheap escape. So:
+
+```bash
+npx testguard-cli claims --since origin/main    # what existed there and does not now
+```
+
+`removed-claim` and `removed-fault` gate; a rename that keeps the statement
+verbatim is reported as `renamed-claim` and does not. Where evidence is
+present, the line names the verdict the claim last had, because *"it was
+SURVIVED when it was removed"* is the sentence that matters. The escape hatch
+is the same one as everywhere else: a `claim` entry in
+`testguard.ignore.json` with a reason, expiring if you give it an `expires`.
+Removal is allowed — claims can be wrong, superseded or split — and it is
+never silent.
+
+### Run it from any harness
+
+The operating loop above lives in a Claude Code skill and a session-start
+hook. Both vanish the moment the harness is Cursor, Codex, Devin or whatever
+comes next — and the harness is exactly the layer most likely to change. So
+the same documents are also served over the Model Context Protocol:
+
+```bash
+npx testguard-cli mcp            # JSON-RPC 2.0 on stdio
+npx testguard-cli init --mcp     # prints the config for Claude Code, Cursor and Codex
+```
+
+Five tools, all **read-only**: `testguard_status`, `testguard_brief`,
+`testguard_claims`, `testguard_evidence`, `testguard_next_command`.
+
+**No tool runs a probe.** A probe is long-running, budgeted, and the person
+should see it happen, so `next_command` hands back the exact shell line for
+the agent to run in its own terminal. Nothing here writes a file either —
+editing a claims file through a connector would defeat the point of recording
+every fault edit. The server declares only a `tools` capability: no prompts,
+no resources, no sampling.
+
+It is hand-written against a pinned protocol version rather than built on the
+official SDK, because this tool has one exact-pinned runtime dependency and
+keeps it that way. The surface is three methods and five tools and will not
+grow; tests speak the wire format to a real child process and check each
+tool against the CLI's own `--json` output, so the two cannot drift.
 
 ### Built for agents to run
 
@@ -156,11 +422,30 @@ things make that safe:
   working tree. Every human rendering — the CLI text, the session-start
   brief, the skill — derives from it, so they cannot disagree. Every command
   accepts `--json`.
-- **An installable operating loop.** `testguard init` writes
+- **An installable operating loop.** `testguard init [dir]` writes the
+  **agent layer at the git root**, where agent sessions run —
   `.claude/skills/testguard/SKILL.md` (state → action, verdict → the only
   acceptable fix, the two-gate rule for any test the agent writes), the
-  `brief --text` session-start hook, an `AGENTS.md` section and the
-  `.gitignore` lines. Idempotent.
+  `brief --text` session-start hook, an `AGENTS.md` section — and the
+  **project layer** (`.gitignore` lines) beside the claims file. A second
+  project in the same repository adds a hook line and an `AGENTS.md`
+  bullet; `--here` keeps everything in the subdirectory. The hook
+  resolves the project's `node_modules/.bin/testguard`, then the git root's,
+  then a `testguard` on `PATH` via `command -v`, then does nothing — **no
+  `npx` in any form, so no path to the network**; a pre-0.6 `npx -y` or
+  `npx --no-install` hook is replaced. A written file that
+  `.gitignore` swallows is reported, not offered for commit. Idempotent.
+- **Independence is recorded (L3).** `probe` measures *power* — would this
+  test notice if the code were wrong. It also records, on every kill, whether
+  the killing test was last touched by the same change (or the same author) as
+  the code it guards: `detail.independence` is `co-authored`,
+  `separate-change` or `unknown`. A co-authored kill is legitimate — a fix
+  *should* ship with its regression test — but a repository where every kill
+  is co-authored has no independent verification, whatever its claim
+  verification rate says. It is a **signal**: ranking reads it, verdicts never
+  do, and the brief says it in one line. Nothing else in this category
+  measures it, and agents saturate the tests they can see
+  ([SpecBench](https://arxiv.org/abs/2605.21384)).
 - **Gaming is visible.** The cheapest way to make a survivor disappear is to
   weaken its fault, not to write a test. Evidence records every fault's
   content hash; `status` lists any fault edited after it survived, with its
@@ -170,7 +455,7 @@ things make that safe:
 ### Authoring faults mechanically
 
 Writing faults by hand means reading the code to find exact anchors. Two
-field reports found that ~80% of hand-written faults are one of five shapes,
+field reports found that ~80% of hand-written faults are one of nine shapes,
 so `scaffold` proposes them for you:
 
 ```bash
@@ -185,6 +470,10 @@ npx testguard-cli scaffold src/auth.ts --claim AUTH-ADMIN   # every proposal und
 | `return-altered` | `return <check>;` (`===`, `.includes(`, `&&`, …) → `return true;` |
 | `literal-changed` | `httpOnly`/`secure` flipped, `sameSite` → `none`, a cost/rounds → `1`, a ttl/tolerance/window/limit ×1000 |
 | `call-removed` | a bare `verify…()` / `validate…()` / `check…()` / `authorize…()` call removed |
+| `field-dropped` | a field removed from an object that is returned, built by an arrow, assigned to a payload-ish name, passed to a `save`/`update`/`send`/`write`/… call, or is a `z.object({…})`-style schema; a string entry removed from an allow-list array; a `...base,` line or inline `{ ...base, … }` merge dropped. Only a line that can go on its own (balanced, comma-terminated or followed by the closer); never inside tests, fixtures or migrations |
+| `argument-swapped` | a call kept, its first argument swapped for `undefined` (and `{}` when the argument is itself a call) — only when that argument is derived from a parameter of the enclosing function or a request-like value (`req`, `ctx`, `event`, …). The seam fault: `decide(deriveFrom(input), now)` → `decide(undefined, now)` |
+| `element-removed` | a one-line JSX element — self-closing (`<Toggle … />`) or paired (`<button …>Save</button>`) — removed; the UI shape behind "the toggle is invisible", killable by a browser-layer defender |
+| `handler-dropped` | an `on<Event>={…}` prop removed, whether it is its own line or inline in the tag; the control renders and does nothing |
 
 Every proposal's `find` is the exact line with `expectHits`/`occurrence`
 computed from the file, so it is verifiable by construction; provenance is
@@ -220,13 +509,18 @@ through every verdict.
 
 ## Status
 
-**v0.4.** Seven commands, vitest and jest runners, hand-authored faults plus
-a mechanical scaffold, and an agent operating layer (`status`, `init`). The contract
-spine — six JSON Schemas shared with the other Guard tools — is under
+**v0.5.** Eleven commands (`status`, `init`, `claims`, `probe`, `admit`, `replay`, `baseline`, `brief`, `gate`, `scaffold`, `mcp`), vitest and jest runners, hand-authored faults plus
+**v0.5.** Nine commands (`status`, `init`, `claims`, `probe`, `admit`, `baseline`, `brief`, `gate`, `scaffold`), vitest, jest and Playwright runners, hand-authored faults plus
+a mechanical scaffold, an agent operating layer (`status`, `init`) and a
+change gate (`gate`). The contract
+spine — eight JSON Schemas shared with the other Guard tools — is under
 [`spec/`](spec/). One exact-pinned runtime dependency (`ajv`, for schema validation); Node ≥ 20.
 
-Not yet: test generation (the two-gate acceptance loop), runners beyond
-vitest and jest, AST-aware producers, and calibration of fault classes
+Not yet: test generation (the acceptance half, `admit`, exists; the generating half stays the agent's), runners beyond
+vitest and jest, AST-aware producers, and the transfer of a calibration between repositories — `replay` measures it
+now; whether it carries to a repository with no history is unproven. Each is
+designed for; none is claimed.
+vitest, jest and Playwright, AST-aware producers, and calibration of fault classes
 against real escaped bugs. Each is designed for; none is claimed.
 
 ## Licence

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "testguard-cli",
-  "version": "0.4.0",
+  "version": "0.6.0",
   "description": "Proves a test suite defends the claims a project makes: injects the faults those claims forbid and reports every one the tests fail to detect.",
   "type": "module",
   "license": "MIT",

package/spec/GATE-SEMANTICS.md

@@ -32,7 +32,14 @@ Rules that follow from the table:
    timeout does not, and a suite that fails to load does not: neither is
    evidence that the suite defends the claim. Tools parse the runner's
    structured report, never its exit code, because the exit code cannot tell
-   these apart.
+   these apart. A runner's own retry mechanism does not change this: a test
+   that failed and then passed on retry (Playwright's `flaky`) is a
+   **non-green run** even when the runner exits 0, and a test the runner
+   reports as timed out (`timedOut`) is a timeout, never an assertion
+   failure. When one claim's defenders run under several runners, their
+   runs merge pessimistically — any load error is an error, any timeout is
+   a timeout, any failure is a failure — and the evidence names which file
+   ran under which runner.
 4. **A mixed result is not a kill.** If some of the N probe runs fail and
    others pass, the defender's response to the fault is nondeterministic.
    Reporting `killed` would be the optimistic bias flakiness introduces;
@@ -43,6 +50,11 @@ Rules that follow from the table:
    and `detail.undeclaredKillers` names the tests, so the author can fix
    `defendedBy`. The claim's stated evidence chain is broken even though the
    suite is not blind. It gates, and ranks below a true survivor.
+   Attribution runs **up to N times**, and stops as soon as no test has failed
+   in every run so far: from there no further run can name a killer. The
+   evidence records the early stop (`detail.escalationStoppedEarly`). Stopping
+   can only fail to *name* a killer, never upgrade a verdict, which is the
+   pessimistic side.
 6. **A verdict names the commit it is about.** Evidence records `repo.head`;
    when the working tree was probed instead, `repo.snapshot` holds the
    throwaway commit that captured it. A tool must refuse to probe a commit
@@ -57,6 +69,35 @@ Rules that follow from the table:
    changed since it was probed, with its previous verdict. The change is
    allowed; it is never invisible.
 
+## Replay reports; it never gates
+
+A replayed bug is history. It escaped, by definition, which means the suite
+missed it; a gate on that is a gate nobody can pass on a first run, and it
+would only teach teams to stop looking. `replay` therefore always exits `0`
+unless it could not run at all.
+
+Its verdicts mirror the probe's, for the same reasons:
+
+| Verdict | Meaning |
+|---|---|
+| `caught` | a remaining test failed **by assertion** on the reverted source, every run. The suite knew. |
+| `blind` | the suite stayed green on known-broken code. |
+| `nocover` | no test imports the reverted files. Worse than `blind`: nothing was even tried. |
+| `unverifiable` | the revert did not apply, the suite failed to load, or it timed out. Carries a `reason`. |
+| `flaky` | the runs disagreed. A flaky failure reads as "the suite caught it", so flakiness biases this metric **optimistically** — mixed runs are never `caught`. |
+
+Two rules that follow:
+
+1. **One patch, one row.** De-duplicate by `git patch-id`; a dual-branch
+   topology carries the same fix under two or three shas, and counting it
+   twice corrupts the corpus a calibration is computed from.
+2. **The fix's own test is removed before the run.** It proves nothing about
+   what the suite knew before the fix existed.
+
+Only `caught` and `blind` carry information. `nocover`, `flaky` and
+`unverifiable` are excluded from both sides of a calibration ratio, because a
+number computed over them would mean nothing.
+
 ## Baseline and delta
 
 A baseline freezes the fingerprints of every non-passing finding at a point
@@ -72,6 +113,15 @@ derived from identities and outcome, never from the fault's `find`/`replace`
 text, so repairing a rotted anchor does not churn the baseline — while a
 change of verdict on the same claim+subject does surface as new.
 
+A baseline records the commit its evidence was taken at. When that evidence
+came from a working-tree snapshot, the baseline says so (`snapshot`) and its
+`head` is the **parent** of the commit that will carry the tests. A tool may
+offer to re-stamp such a baseline onto a later commit, but only when a clean
+probe of that commit reproduced exactly the same fingerprints; otherwise the
+user re-probes and freezes a new baseline. A frozen contract is never
+silently rewritten, and a status document notes a baseline that predates
+HEAD without turning that into a state.
+
 Adopting tools may reconcile an existing baseline format (for example a
 `{version, fingerprints:{hash:count}}` file) by mapping it onto this shape;
 the suppress-up-to-count semantics are identical.
@@ -84,6 +134,64 @@ the suppress-up-to-count semantics are identical.
   for reasonless `path` entries.
 - An annotation is **strictly additive**. It never changes, suppresses, or
   drops a finding. Ranking may read annotations; verdicts never do.
+- A **signal** is a static, annotation-grade fact about a defender, recorded
+  on the evidence and never a verdict. `mocked-never-asserted`: a test file
+  mocks the subject's module and never asserts on anything imported from it.
+  An author may silence it with `unasserted: <reason>` above the mock; the
+  tool then records `unasserted-annotated` with the reason — silenced, never
+  hidden. A file that mocks the subject is never a discovered defender: a
+  mock cannot detect a fault in what it replaced.
+
+## Claim coverage of a change
+
+`probe` asks whether the tests defend the claims that exist. It says nothing
+about code that has no claim, by construction — and every escaped defect the
+field reports share was a *claim gap*, not a defender gap: the feature shipped
+green with zero claims. A repository with ten old claims and one new,
+unclaimed module exits 0 under `probe` forever.
+
+The change gate closes that hole. Given a reference, a tool measures the
+**delta** — the files changed since `merge-base(ref, HEAD)`, or since that
+merge-base in the working tree — file by file:
+
+- A changed file is **excluded** when it is not source (by extension) or
+  matches a documented never-claimed pattern (`*.d.ts`, `*.config.*`,
+  fixtures, mocks, snapshots, the tool's own directory). Exclusions are
+  listed, never silent.
+- A changed source file is **covered** when at least one fault anchors to it,
+  when it is a test file that resolves as a defender of some claim, or when an
+  **unexpired** `path` ignore entry excuses it.
+- Anything else is **uncovered**. The unit is the file, on purpose: a fault
+  anchors to a file, so the file is the smallest unit the rest of the
+  contract already understands.
+
+Rules:
+
+1. **One uncovered file gates.** Exit `1`. There is no threshold and no
+   percentage; a percentage is how the gap hid before.
+2. **Every reliance on an ignore entry is reported.** The gate may pass
+   *because of* an excuse; a reviewer reads which entries carried it, with
+   their reasons and expiry. An expired entry excuses nothing and is reported
+   as expired.
+3. **A test file needs a claim too.** A test that defends no claim is the
+   authorship trap the pattern exists for; it is uncovered until a claim
+   names it in `defendedBy` (or discovery resolves it as a defender).
+4. **Never a silent pass on nothing.** A non-empty change whose files were
+   all excluded passes with an explicit "0 evaluated" line; a tool offers a
+   strict mode that fails it instead. An empty change is an honest `0`.
+5. **The reference is never guessed.** An explicit flag, or a CI-provided base
+   branch, or an error. An upstream that already contains the change has an
+   empty diff and would pass trivially.
+6. **Unclaimed changes precede every evidence state.** When the status
+   document knows a reference and finds uncovered files, its state is
+   `unclaimed-changes` and its next action is to write the claim, before any
+   `unproven` finding is surfaced. The brief renders them first. The claim is
+   written before more code.
+
+Exit codes: `0` every changed source file is claimed or excused (or nothing
+changed); `1` at least one uncovered file (or strict mode over an
+all-excluded change); `2` the change cannot be evaluated (unresolvable
+reference, invalid claims or ignore file, no repository); `3` no reference.
 
 ## Severity floor
 

package/spec/README.md

@@ -1,6 +1,6 @@
 # Guard spec — the contract spine
 
-Six formats that tools following the Guard pattern share. They share
+Eight formats that tools following the Guard pattern share. They share
 *formats*, not code: [`docguard-cli`](https://www.npmjs.com/package/docguard-cli)
 is Node ESM, [`websec-validator`](https://pypi.org/project/websec-validator/)
 is Python, and porting one runtime into the other is not worth it. A tool
@@ -21,7 +21,9 @@ The pattern all such tools run:
 | `ignore` | [`schemas/ignore.schema.json`](schemas/ignore.schema.json) | Reviewable scoping. Every entry has a reason. |
 | `calibration` | [`schemas/calibration.schema.json`](schemas/calibration.schema.json) | P(finding is real) per bucket, with a Wilson interval and the sample size behind it. |
 | `brief` | [`schemas/brief.schema.json`](schemas/brief.schema.json) | What to tell an agent before it writes code — ranked, capped, never a single score, and carrying the one next action. |
-| `status` | [`schemas/status.schema.json`](schemas/status.schema.json) | Where the project is and what happens next — the single machine-readable truth every human rendering derives from. Surfaces faults whose content changed since they were probed. |
+| `status` | [`schemas/status.schema.json`](schemas/status.schema.json) | Where the project is and what happens next — the single machine-readable truth every human rendering derives from. Surfaces faults whose content changed since they were probed, and changed files that carry no claim. |
+| `replay` | [`schemas/replay.schema.json`](schemas/replay.schema.json) | Would this suite have caught the bugs that already escaped? Replays real fix commits: revert the source, remove the test the fix shipped, run what remains. A replayed bug is ground truth — a human already confirmed it was a defect — which is what a fault model is calibrated against. |
+| `gate` | [`schemas/gate.schema.json`](schemas/gate.schema.json) | Claim coverage of one change: which changed files carry a claim, which are excused (and by which ignore entry), which are unclaimed. The delta gate for code that has no claim yet. |
 
 Shared definitions (verdicts, fault classes, provenance, annotations) live in
 [`schemas/common.schema.json`](schemas/common.schema.json). Gate behaviour —