testguard-cli 0.14.0__tar.gz → 0.14.1__tar.gz

{testguard_cli-0.14.0 → testguard_cli-0.14.1}/.gitignore

@@ -14,6 +14,7 @@ coverage/
 # Local agent tooling — not part of the public project
 .wolf/
 .claude/
+.codex/
 .agent/
 CLAUDE.local.md
 graphify-out/

{testguard_cli-0.14.0 → testguard_cli-0.14.1}/.npmignore

@@ -3,6 +3,7 @@
 .github/
 .wolf/
 .claude/
+.codex/
 graphify-out/
 test/
 fixtures/

{testguard_cli-0.14.0 → testguard_cli-0.14.1}/AGENTS.md

@@ -1,6 +1,6 @@
 # AI Agent Instructions — TestGuard
 
-<!-- docguard:last-reviewed 2026-09-19 -->
+<!-- docguard:last-reviewed 2026-09-22 -->
 
 Read this before changing anything. It is short on purpose.
 
@@ -17,6 +17,7 @@ their first consumer.
 
 ```bash
 npm test               # unit + fixture acceptance (~15s)
+npm ci --prefix fixtures/known-answer-playwright   # once; without it the Playwright fixture acceptance silently skips
 npm run test:spec      # conformance suite only
 npm run self:probe     # TestGuard probes its own claims; must exit 0
 npm run test:install   # the packed tarball must run with production deps only

{testguard_cli-0.14.0 → testguard_cli-0.14.1}/CHANGELOG.md

@@ -7,6 +7,78 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.14.1] - 2026-09-27
+
+Automated weekly release — everything merged since `v0.14.0`.
+
+### Changed
+
+- fix: four defects that made `nocover` report discovery failures as findings (#169)
+- chore: untrack local agent tooling and a generated fixture artifact (#168)
+- chore(homebrew): sha256 for v0.14.0 (#167)
+- docs(brief): page 6 carries what 0.14.0 changed there (#166)
+
+
+### Fixed
+
+- **`sweep` no longer refuses to write after a sweep that learned nothing.**
+  `learnedProductivity` named every evidence document it *read* as an ordering
+  source, while `observed` counts only `killed` and `survived` records. A
+  document whose records are all `nocover` — the ordinary shape of a first
+  sweep — therefore produced `sources: [...]` with `observed: 0`, which the
+  sweep validator correctly refuses. Because that document is written to
+  `.testguard/sweep-evidence.json`, **every later sweep failed the same way,
+  permanently**, until someone deleted a gitignored file nothing mentioned. An
+  ordering now names only the documents that contributed an observation, which
+  is what "the documents it was learned from" always meant. Measured on an
+  external corpus: this fired on 10 of 34 sweep invocations.
+- **`replay` resolves Python importers with the Python resolver.**
+  `replay` matched test files against a reverted source file with the
+  JavaScript importer for every language. `from pkg.mod import x` carries no
+  quoted specifier, so nothing ever matched, the defender set was always empty
+  for Python, and **every Python verdict was `nocover`** — "no test imports the
+  reverted source" — however complete the suite. `discoverDefendersDetailed`
+  has always branched on the target's language; `replay` now does the same.
+- **A runner that matches no test file is no longer selected silently.**
+  `--runner auto` tries vitest, jest, then python. A TypeScript project on
+  Playwright resolves neither JavaScript runner, so `auto` reached python
+  wherever a `python3` existed, globbed for `.py`, found none, and reported
+  every fault as `nocover`. `auto` now prefers a resolvable runner that can see
+  test files, and when none can it says so (`no runner matched a test file …
+  every verdict will be nocover`) instead of letting discovery's failure read
+  as a finding about the project. An explicitly named `--runner` is still
+  honoured; every selection now reports `testFiles`.
+
+- **`replay` no longer reports `nocover` when it removed the only test file.**
+  Removing the fix's own test removes the whole FILE, and on a project with
+  few, large test files that also removes tests which pre-dated the fix and
+  might have caught the bug. Nothing is left to run, so nothing can be
+  concluded: the verdict is now `unverifiable` with reason
+  `the-fix-shipped-the-only-test-file`. `nocover` is a statement about the
+  PROJECT — "no test exercises this" — and it enters the calibration as a
+  miss, so the old behaviour charged a project for evidence the measurement
+  itself destroyed. The validator now refuses the mislabelled form, and
+  `GATE-SEMANTICS.md` carries it as a third rule beside de-duplication and
+  test removal.
+
+### Changed
+
+- **Python: a module-level dunder assignment is no longer proposed as a
+  fault.** `__version__`, `__all__` and `__author__` at module level are
+  metadata nobody writes a test for, so a survivor on one teaches the
+  survival-learned ordering to prefer a barren class — the same reasoning that
+  already sets a presentational JSX element aside. Narrow on purpose: a
+  module-level *constant* such as `DEFAULT_MAX_SIZE = 100` is still proposed,
+  because removing a real default changes a real default.
+
+### Changed
+
+- The brief's page 6 says what 0.14.0 changed there: the evidence now names
+  the mocked layer and the assertion shape beside a save-path survivor's
+  verdict, and the fourth rule of the lower rungs — whatever a sweep does not
+  probe, it counts, presentational elements included. Re-rendered; still eight
+  sheets from eight pages.
+
 ## [0.14.0] - 2026-09-22
 
 Automated weekly release — everything merged since `v0.13.0`.

{testguard_cli-0.14.0 → testguard_cli-0.14.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: testguard-cli
-Version: 0.14.0
+Version: 0.14.1
 Summary: Breaks your code on purpose and reports every promise your tests did not notice breaking. Mutation testing bound to the claims a project makes — vitest, jest, Playwright, pytest and unittest. Ships the Node.js CLI (requires Node.js 20+).
 Project-URL: Homepage, https://github.com/raccioly/testguard
 Project-URL: Documentation, https://github.com/raccioly/testguard#readme
@@ -151,9 +151,9 @@ raises coverage. TestGuard admits it because it fails when the claim is false.
 | npm | `npm i -D testguard-cli` then `npx testguard probe` |
 | pip | `pip install testguard-cli` then `testguard probe` (needs Node ≥ 20) |
 | Homebrew | `brew tap raccioly/tap && brew install testguard` |
-| GitHub Action | `uses: raccioly/testguard@v0.14.0` — see [`action.yml`](./action.yml) |
+| GitHub Action | `uses: raccioly/testguard@v0.14.1` — see [`action.yml`](./action.yml) |
 | pre-commit | `repo: https://github.com/raccioly/testguard`, hooks `testguard-claims`, `testguard-probe` |
-| GitLab CI | `include: - remote: https://raw.githubusercontent.com/raccioly/testguard/v0.14.0/packaging/gitlab/testguard.gitlab-ci.yml` with `inputs:` — see [`packaging/gitlab/`](./packaging/gitlab/testguard.gitlab-ci.yml) |
+| GitLab CI | `include: - remote: https://raw.githubusercontent.com/raccioly/testguard/v0.14.1/packaging/gitlab/testguard.gitlab-ci.yml` with `inputs:` — see [`packaging/gitlab/`](./packaging/gitlab/testguard.gitlab-ci.yml) |
 
 Projects that set `min-release-age` in `.npmrc` cannot see a version published
 less than that many days ago (`ENOVERSIONS`); install that one with
@@ -687,12 +687,12 @@ detected base that does not resolve is a warning for `status` and `brief`
 # GitHub Actions
 - uses: actions/checkout@v4
   with: { fetch-depth: 0 }
-- uses: raccioly/testguard@v0.14.0
+- uses: raccioly/testguard@v0.14.1
   with: { command: gate }
 
 # GitLab CI — the component-shaped template: gate + probe, brief as an artifact and, opted in, as a merge-request note
 include:
-  - remote: 'https://raw.githubusercontent.com/raccioly/testguard/v0.14.0/packaging/gitlab/testguard.gitlab-ci.yml'
+  - remote: 'https://raw.githubusercontent.com/raccioly/testguard/v0.14.1/packaging/gitlab/testguard.gitlab-ci.yml'
     inputs: { dir: backend, post_note: true }   # post_note needs TESTGUARD_GITLAB_TOKEN (api scope); one note, updated in place
 ```
 
@@ -1007,13 +1007,20 @@ The table above is JavaScript. `scaffold` reads Python too, and proposes the
 same fault classes in Python syntax: `if <guard>:` → `if False:`,
 `return <check>` → `return True`, `verify=True` → `verify=False`, a
 parameter-derived argument swapped for `None`, a key dropped from a payload
-`dict` or an allow-list. Two differences are deliberate. A statement is
+`dict` or an allow-list. Three differences are deliberate. A statement is
 removed by replacing it with `pass`, never by deleting the line, because a
 block whose only statement is gone is an `IndentationError`; and a line that
 leaves a bracket open (`COLOURS = {`), or sits inside one opened on an earlier
 line (`help="…"` in an `add_argument(`), is never removed at all. Both exist
 because a fault that cannot compile is a `fault-invalid` verdict — a probe run
-spent saying nothing about the tests. The two JSX shapes (`element-removed`,
+spent saying nothing about the tests. Third, a **module-level dunder
+assignment** (`__version__`, `__all__`, `__author__`) is not proposed. Deleting
+one does change behaviour, so it is not an equivalent mutant; it is something
+nobody writes a test for, which is the disqualification an icon already gets on
+the JSX side — a survivor there teaches the survival-learned ordering to prefer
+a barren class. The rule is narrow on purpose: a module-level *constant*
+(`DEFAULT_MAX_SIZE = 100`) is still proposed, because removing a real default
+changes a real default. The two JSX shapes (`element-removed`,
 `handler-dropped`) have no Python meaning and are absent rather than faked.
 
 Every proposal's `find` is the exact line with `expectHits`/`occurrence`

{testguard_cli-0.14.0 → testguard_cli-0.14.1}/README.md

@@ -127,9 +127,9 @@ raises coverage. TestGuard admits it because it fails when the claim is false.
 | npm | `npm i -D testguard-cli` then `npx testguard probe` |
 | pip | `pip install testguard-cli` then `testguard probe` (needs Node ≥ 20) |
 | Homebrew | `brew tap raccioly/tap && brew install testguard` |
-| GitHub Action | `uses: raccioly/testguard@v0.14.0` — see [`action.yml`](./action.yml) |
+| GitHub Action | `uses: raccioly/testguard@v0.14.1` — see [`action.yml`](./action.yml) |
 | pre-commit | `repo: https://github.com/raccioly/testguard`, hooks `testguard-claims`, `testguard-probe` |
-| GitLab CI | `include: - remote: https://raw.githubusercontent.com/raccioly/testguard/v0.14.0/packaging/gitlab/testguard.gitlab-ci.yml` with `inputs:` — see [`packaging/gitlab/`](./packaging/gitlab/testguard.gitlab-ci.yml) |
+| GitLab CI | `include: - remote: https://raw.githubusercontent.com/raccioly/testguard/v0.14.1/packaging/gitlab/testguard.gitlab-ci.yml` with `inputs:` — see [`packaging/gitlab/`](./packaging/gitlab/testguard.gitlab-ci.yml) |
 
 Projects that set `min-release-age` in `.npmrc` cannot see a version published
 less than that many days ago (`ENOVERSIONS`); install that one with
@@ -663,12 +663,12 @@ detected base that does not resolve is a warning for `status` and `brief`
 # GitHub Actions
 - uses: actions/checkout@v4
   with: { fetch-depth: 0 }
-- uses: raccioly/testguard@v0.14.0
+- uses: raccioly/testguard@v0.14.1
   with: { command: gate }
 
 # GitLab CI — the component-shaped template: gate + probe, brief as an artifact and, opted in, as a merge-request note
 include:
-  - remote: 'https://raw.githubusercontent.com/raccioly/testguard/v0.14.0/packaging/gitlab/testguard.gitlab-ci.yml'
+  - remote: 'https://raw.githubusercontent.com/raccioly/testguard/v0.14.1/packaging/gitlab/testguard.gitlab-ci.yml'
     inputs: { dir: backend, post_note: true }   # post_note needs TESTGUARD_GITLAB_TOKEN (api scope); one note, updated in place
 ```
 
@@ -983,13 +983,20 @@ The table above is JavaScript. `scaffold` reads Python too, and proposes the
 same fault classes in Python syntax: `if <guard>:` → `if False:`,
 `return <check>` → `return True`, `verify=True` → `verify=False`, a
 parameter-derived argument swapped for `None`, a key dropped from a payload
-`dict` or an allow-list. Two differences are deliberate. A statement is
+`dict` or an allow-list. Three differences are deliberate. A statement is
 removed by replacing it with `pass`, never by deleting the line, because a
 block whose only statement is gone is an `IndentationError`; and a line that
 leaves a bracket open (`COLOURS = {`), or sits inside one opened on an earlier
 line (`help="…"` in an `add_argument(`), is never removed at all. Both exist
 because a fault that cannot compile is a `fault-invalid` verdict — a probe run
-spent saying nothing about the tests. The two JSX shapes (`element-removed`,
+spent saying nothing about the tests. Third, a **module-level dunder
+assignment** (`__version__`, `__all__`, `__author__`) is not proposed. Deleting
+one does change behaviour, so it is not an equivalent mutant; it is something
+nobody writes a test for, which is the disqualification an icon already gets on
+the JSX side — a survivor there teaches the survival-learned ordering to prefer
+a barren class. The rule is narrow on purpose: a module-level *constant*
+(`DEFAULT_MAX_SIZE = 100`) is still proposed, because removing a real default
+changes a real default. The two JSX shapes (`element-removed`,
 `handler-dropped`) have no Python meaning and are absent rather than faked.
 
 Every proposal's `find` is the exact line with `expectHits`/`occurrence`

{testguard_cli-0.14.0 → testguard_cli-0.14.1}/action.yml

@@ -46,7 +46,7 @@ inputs:
   version:
     description: 'testguard-cli version to run'
     required: false
-    default: '0.14.0'
+    default: '0.14.1'
   node-version:
     description: 'Node.js version (20+)'
     required: false

{testguard_cli-0.14.0 → testguard_cli-0.14.1}/docs/testguard-explained.html

@@ -144,7 +144,7 @@
 <div class="page">
   <div class="masthead">
     <div class="wordmark">Test<span class="mark">Guard</span></div>
-    <div class="meta">Technical Brief · v0.14.0 · MIT</div>
+    <div class="meta">Technical Brief · v0.14.1 · MIT</div>
   </div>
 
   <h1>Your tests pass. That is not the same as your tests working.</h1>
@@ -752,8 +752,8 @@ field-dropped  missed 4/4  p=1.00  ci [0.51, 1.00]</pre>
 
   <section class="rule-top">
     <h2>Field report C — where a save button actually fails</h2>
-    <p>Twenty-six faults injected on the persistence path of seven server actions in a real
-    AI-authored application: the database write itself, and the fields of the payload it carries.</p>
+    <p>Twenty-six faults on the persistence path of seven server actions in a real AI-authored
+    application: the write itself, and the fields it carries.</p>
     <figure style="margin-top:11px">
       <svg viewBox="0 0 760 150" role="img" aria-label="Save-path measurement: 12 of 26 faults survived, all of them a dropped payload field; across the whole write surface, 32 of 37 files are defended only by tests that mock the database">
         <rect x="0" y="0" width="243" height="74" rx="4" fill="#ffffff" stroke="#e6e2d8"/>
@@ -781,7 +781,9 @@ field-dropped  missed 4/4  p=1.00  ci [0.51, 1.00]</pre>
     <p class="note">The third number is the read-back answer. A test that replaced the database proves the
     call shape and nothing beyond it — a <code>where</code> matching no rows, a rolled-back transaction, a
     rejected constraint all pass against a mock. On this surface a clean probe is not evidence of
-    persistence; it is evidence that nothing could have measured it, and the tool says so before any verdict.</p>
+    persistence; it is evidence that nothing could have measured it, and the tool says so before any verdict.
+    The evidence now names the mocked layer and the assertion shape — exact, partial or argument-free —
+    beside the survivor’s verdict.</p>
   </section>
 
   <section class="rule-top">
@@ -792,12 +794,11 @@ field-dropped  missed 4/4  p=1.00  ci [0.51, 1.00]</pre>
         <p class="note"><code>sweep</code> takes the files the gate just called uncovered, proposes
         faults mechanically, probes a bounded selection, and reports what nothing noticed — ordering
         candidates by what this project’s own runs have shown survives, the feedback that took
-        Google’s productive rate from 15% to 89%, read from evidence already on disk.</p>
+        Google’s productive rate from 15% to 89%.</p>
       </div>
       <div>
         <h3>A sentence or ten</h3>
-        <p class="note">A <em>concern</em> names a kind of promise and where to look. Five lines of JSON
-        on the same application — “every admin route refuses a caller without the admin role” — found a
+        <p class="note">A <em>concern</em> names a kind of promise and where to look. Five lines of JSON — “every admin route refuses a caller without the admin role” — found a
         route whose owner/manager check can be forced to <code>false</code> with no test failing. One
         sentence, twenty-four routes, no model.</p>
       </div>
@@ -807,15 +808,16 @@ field-dropped  missed 4/4  p=1.00  ci [0.51, 1.00]</pre>
         and a reproducer already in hand.</p>
       </div>
     </div>
-    <p class="note" style="margin-top:9px">Three rules keep the lower rungs safe. A sweep <strong>never
+    <p class="note" style="margin-top:9px">Four rules keep the lower rungs safe. A sweep <strong>never
     writes the claims file</strong> — a sentence nobody wrote is not a claim. Its evidence <strong>never
     replaces</strong> the canonical record. And it <strong>does not fail on its own bad guess</strong>:
-    only a survivor or an untested file exits non-zero.</p>
+    only a survivor or an untested file exits non-zero. And whatever it does not probe, it counts —
+    past the cap, outside the concern, or presentational: an icon or a static wrapper nobody would test.</p>
   </section>
 
   <div class="pull">
-    <p>A sweep is weaker evidence than a probe, and says so. Nobody stated that the behaviour mattered.
-    It is stronger than nothing, which is what a repository with no claims has.</p>
+    <p>A sweep is weaker evidence than a probe, and says so. It is stronger than nothing,
+    which is what a repository with no claims has.</p>
     <div class="attrib">Why the ladder has a bottom rung</div>
   </div>
 

{testguard_cli-0.14.0 → testguard_cli-0.14.1}/docs-canonical/TEST-SPEC.md

@@ -2,7 +2,7 @@
 
 <!-- docguard:version 1.0.0 -->
 <!-- docguard:status approved -->
-<!-- docguard:last-reviewed 2026-09-19 -->
+<!-- docguard:last-reviewed 2026-09-22 -->
 <!-- docguard:owner @raccioly -->
 <!-- docguard:quality negation-load off — the governing rule is a prohibition (a test that would pass with the behaviour broken is not a test), and the verdict rules are defined by what does not count as detection. -->
 
@@ -20,7 +20,7 @@ the test that defends it, and CI fails if any injected fault survives.
 | Unit | `test/*.test.mjs` | One pure function's branches, without a runner | milliseconds |
 | Conformance | `spec/conformance/` | Every valid example validates; every must-reject document is rejected for the defect its filename names | ~1 s |
 | Anchor preflight | `test/claims-anchors.test.mjs` | Exact anchors still locate, supported replacements parse, the check stays below one second, and CI reaches it before self-probe | <1 s |
-| Fixture acceptance | `test/probe.fixture.test.mjs`, `test/probe.jest.test.mjs`, `test/probe.playwright.test.mjs` | The whole pipeline reproduces a known-answer oracle, one per runner | 10–50 s each |
+| Fixture acceptance | `test/probe.fixture.test.mjs`, `test/probe.jest.test.mjs`, `test/probe.playwright.test.mjs` | The whole pipeline reproduces a known-answer oracle, one per runner. The Playwright fixture carries its own dependencies: without `npm ci --prefix fixtures/known-answer-playwright` that file skips, and a skipped acceptance is not a passing one | 10–50 s each |
 | Self-verification | `testguard.claims.json`, run by `npm run self:probe` | The tool's own invariants survive fault injection | **20.2 min** cold (measured by the v0.10.1 release job on 2026-09-19 at `70e9d61`: 154 faults, 924 defender runs, 1210 s); seconds when verdicts are reused, which is why PR CI restores previous evidence from cache. Budgeted at 1440 s in `testguard.cost-budget.json` for the additional four-fault live-tap claim and **gated** in CI — cost growth is reviewed instead of disappearing into a passing log |
 | Install smoke | `.github/scripts/install-smoke.mjs` | The **packed tarball** runs with production dependencies only | ~20 s |
 | Runtime budget | `ci.yml` | The suite has not silently started walking the wrong tree | gate, not a test |

{testguard_cli-0.14.0 → testguard_cli-0.14.1}/package-lock.json

@@ -1,12 +1,12 @@
 {
   "name": "testguard-cli",
-  "version": "0.14.0",
+  "version": "0.14.1",
   "lockfileVersion": 3,
   "requires": true,
   "packages": {
     "": {
       "name": "testguard-cli",
-      "version": "0.14.0",
+      "version": "0.14.1",
       "license": "MIT",
       "dependencies": {
         "ajv": "8.20.0"

{testguard_cli-0.14.0 → testguard_cli-0.14.1}/package.json

@@ -1,6 +1,6 @@
 {
   "name": "testguard-cli",
-  "version": "0.14.0",
+  "version": "0.14.1",
   "description": "Breaks your code on purpose and reports every promise your tests did not notice breaking. Mutation testing bound to the claims a project makes — vitest, jest, Playwright, pytest and unittest.",
   "type": "module",
   "license": "MIT",

{testguard_cli-0.14.0 → testguard_cli-0.14.1}/packaging/gitlab/testguard.gitlab-ci.yml

@@ -3,7 +3,7 @@
 # Include it from a project's .gitlab-ci.yml, pinned to a release:
 #
 #   include:
-#     - remote: 'https://raw.githubusercontent.com/raccioly/testguard/v0.14.0/packaging/gitlab/testguard.gitlab-ci.yml'
+#     - remote: 'https://raw.githubusercontent.com/raccioly/testguard/v0.14.1/packaging/gitlab/testguard.gitlab-ci.yml'
 #       inputs:
 #         dir: backend            # where testguard.claims.json lives (default ".")
 #         post_note: true         # post the brief as a merge-request note (needs TESTGUARD_GITLAB_TOKEN, api scope)
@@ -29,7 +29,7 @@ spec:
   inputs:
     version:
       description: testguard-cli version to run
-      default: "0.14.0"
+      default: "0.14.1"
     dir:
       description: directory whose test runner is invoked (where testguard.claims.json lives)
       default: "."

{testguard_cli-0.14.0 → testguard_cli-0.14.1}/packaging/homebrew/testguard.rb

@@ -15,8 +15,8 @@
 class Testguard < Formula
   desc "Proves a test suite defends the claims a project makes"
   homepage "https://github.com/raccioly/testguard"
-  url "https://registry.npmjs.org/testguard-cli/-/testguard-cli-0.14.0.tgz"
-  sha256 "7931292332345eee9a6f5deb118e3931f06396b06e98c41c227b0de99c78df1d"
+  url "https://registry.npmjs.org/testguard-cli/-/testguard-cli-0.14.1.tgz"
+  sha256 "eb0ca2d7808338bc12db057dda87259e357d0db47a62917dbabcff08bd8c1df4"
   license "MIT"
 
   depends_on "node"

{testguard_cli-0.14.0 → testguard_cli-0.14.1}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "testguard-cli"
-version = "0.14.0"
+version = "0.14.1"
 description = "Breaks your code on purpose and reports every promise your tests did not notice breaking. Mutation testing bound to the claims a project makes — vitest, jest, Playwright, pytest and unittest. Ships the Node.js CLI (requires Node.js 20+)."
 readme = "README.md"
 license = { text = "MIT" }

{testguard_cli-0.14.0 → testguard_cli-0.14.1}/spec/GATE-SEMANTICS.md

@@ -197,8 +197,8 @@ Its verdicts mirror the probe's, for the same reasons:
 |---|---|
 | `caught` | a remaining test failed **by assertion** on the reverted source, every run. The suite knew. |
 | `blind` | the suite stayed green on known-broken code. |
-| `nocover` | no test imports the reverted files. Worse than `blind`: nothing was even tried. |
-| `unverifiable` | the revert did not apply, the suite failed to load, or it timed out. Carries a `reason`. |
+| `nocover` | no test imports the reverted files, **and at least one test file remained to ask**. Worse than `blind`: nothing was even tried. |
+| `unverifiable` | the revert did not apply, the suite failed to load, it timed out, or removing the fix's own test left **no test file at all** (`the-fix-shipped-the-only-test-file`). Carries a `reason`. |
 | `flaky` | the runs disagreed. A flaky failure reads as "the suite caught it", so flakiness biases this metric **optimistically** — mixed runs are never `caught`. |
 
 Two rules that follow:
@@ -208,6 +208,13 @@ Two rules that follow:
    twice corrupts the corpus a calibration is computed from.
 2. **The fix's own test is removed before the run.** It proves nothing about
    what the suite knew before the fix existed.
+3. **If that leaves no test file, the run is `unverifiable`, not `nocover`.**
+   Removal takes the whole FILE, so on a project with few, large test files it
+   also removes tests that pre-dated the fix and might have caught the bug. The
+   suite the measurement needed no longer exists, so nothing can be concluded.
+   `nocover` is a statement about the PROJECT — "no test exercises this" — and
+   using it here would charge the project, in the calibration, for evidence the
+   method itself destroyed.
 
 `caught`, `blind` and `nocover` are measurements and enter the calibration
 ratio: `caught` as a hit, the other two as misses. `nocover` counts as a miss
@@ -216,7 +223,9 @@ because it is one — the worst kind. Leaving it out would let a project with
 since its worst outcomes would leave the denominator before the ratio is
 taken; Stryker and PIT count no-coverage mutants against the headline score
 for the same reason. `flaky` and `unverifiable` are failed measurements and
-enter neither side.
+enter neither side — which is why rule 3 above matters: a run with no test
+file left is a failed measurement wearing `nocover`'s clothes, and left
+unclassified it would inflate every miss rate computed from the corpus.
 
 ### Calibration arithmetic is reproducible
 

testguard_cli-0.14.1/spec/conformance/invalid/replay.nocover-when-no-test-file-remained.json

@@ -0,0 +1,44 @@
+{
+  "$schema": "../../schemas/replay.schema.json",
+  "schemaVersion": 1,
+  "tool": {
+    "name": "testguard",
+    "version": "0.5.0"
+  },
+  "run": {
+    "id": "replay-20260917T220000",
+    "startedAt": "2026-09-17T22:00:00Z",
+    "finishedAt": "2026-09-17T22:14:00Z",
+    "range": "HEAD~50..HEAD",
+    "runner": {
+      "name": "vitest",
+      "version": "5.0.1"
+    },
+    "confirmRuns": 3,
+    "candidates": 12,
+    "duplicates": 4
+  },
+  "records": [
+    {
+      "commit": "4a1c9f2e77b0d3a8c5e61f04b9d27a3e8c6f01b2",
+      "patchId": "9f2e77b0d3a8c5e61f04b9d27a3e8c6f01b24a1c",
+      "subject": "fix(redact): keep the raw input out of the audit row",
+      "verdict": "nocover",
+      "source": [
+        "src/redact.mjs"
+      ],
+      "removedTests": [
+        "test/audit-fix.test.mjs"
+      ],
+      "ranTests": 0,
+      "faultClass": "field-dropped",
+      "runs": [
+        {
+          "outcome": "error",
+          "durationMs": 0
+        }
+      ],
+      "reason": "the-fix-shipped-the-only-test-file"
+    }
+  ]
+}

{testguard_cli-0.14.0 → testguard_cli-0.14.1}/spec/lib/validate.mjs

@@ -297,6 +297,14 @@ const semantic = {
       }
       if (r.verdict === 'nocover' && r.ranTests) errors.push({ path: `${p}/ranTests`, message: 'nocover means no test exercises the reverted files; it cannot have run tests' });
       if (r.verdict === 'unverifiable' && !r.reason) errors.push({ path: `${p}/reason`, message: 'unverifiable requires a reason (e.g. revert-did-not-apply, suite-failed-to-load)' });
+      // A run that had no test file left is a FAILED MEASUREMENT, never a
+      // finding about the project: removing the fix's test removed the file it
+      // lived in, and with it any pre-existing test that might have caught the
+      // bug. Recording that as `nocover` would put it in the calibration as a
+      // miss and charge the project for evidence the method destroyed.
+      if (r.reason === 'the-fix-shipped-the-only-test-file' && r.verdict !== 'unverifiable') {
+        errors.push({ path: `${p}/verdict`, message: 'no test file remained after removing the fix\'s own, so nothing could be concluded: that is unverifiable, not a finding about the project' });
+      }
     });
     return errors;
   },

testguard_cli-0.14.1/spec/schemas/replay.schema.json

@@ -0,0 +1,157 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "urn:claimspec:v1:replay",
+  "title": "claimspec v1 \u2014 bug replay",
+  "description": "Would this suite have caught the bugs that actually escaped? For each fix commit in a range: revert only the source to its parent, remove the test the fix shipped, run what remains. `blind` means the suite was blind to a bug that really shipped. Unlike an injected fault, a replayed bug is ground truth \u2014 a human already confirmed it was a defect \u2014 which is what makes this the oracle a fault model is calibrated against.",
+  "type": "object",
+  "required": [
+    "schemaVersion",
+    "tool",
+    "run",
+    "records"
+  ],
+  "properties": {
+    "$schema": {
+      "type": "string"
+    },
+    "schemaVersion": {
+      "const": 1
+    },
+    "tool": {
+      "$ref": "urn:claimspec:v1:common#/$defs/tool"
+    },
+    "run": {
+      "type": "object",
+      "required": [
+        "id",
+        "startedAt",
+        "range",
+        "confirmRuns"
+      ],
+      "properties": {
+        "id": {
+          "type": "string",
+          "minLength": 1,
+          "maxLength": 128
+        },
+        "startedAt": {
+          "$ref": "urn:claimspec:v1:common#/$defs/isoDateTime"
+        },
+        "finishedAt": {
+          "$ref": "urn:claimspec:v1:common#/$defs/isoDateTime"
+        },
+        "range": {
+          "description": "The commit range searched for fix commits, as given.",
+          "type": "string",
+          "minLength": 1,
+          "maxLength": 256
+        },
+        "runner": {
+          "type": "object",
+          "required": [
+            "name"
+          ],
+          "properties": {
+            "name": {
+              "type": "string"
+            },
+            "version": {
+              "type": "string"
+            }
+          },
+          "additionalProperties": false
+        },
+        "confirmRuns": {
+          "type": "integer",
+          "minimum": 1
+        },
+        "candidates": {
+          "description": "Fix commits found in the range before de-duplication by patch-id.",
+          "type": "integer",
+          "minimum": 0
+        },
+        "duplicates": {
+          "description": "Commits dropped as the same patch under another sha. A dual-branch topology carries the same fix two or three times; counting it twice corrupts the corpus.",
+          "type": "integer",
+          "minimum": 0
+        }
+      },
+      "additionalProperties": false
+    },
+    "records": {
+      "type": "array",
+      "items": {
+        "type": "object",
+        "required": [
+          "commit",
+          "patchId",
+          "verdict",
+          "source",
+          "removedTests",
+          "runs"
+        ],
+        "properties": {
+          "commit": {
+            "$ref": "urn:claimspec:v1:common#/$defs/gitSha"
+          },
+          "patchId": {
+            "description": "`git patch-id --stable` of the fix: the identity of the CHANGE rather than of the commit. A dual-branch topology carries the same fix under several shas, and counting it twice corrupts the corpus. Hex, 40 characters for a SHA-1 repository and 64 for a SHA-256 one; the commit sha when git could not compute one.",
+            "type": "string",
+            "pattern": "^[a-f0-9]{40}(?:[a-f0-9]{24})?$"
+          },
+          "subject": {
+            "description": "The fix commit's subject line, for a human reading the corpus.",
+            "type": "string",
+            "maxLength": 512
+          },
+          "verdict": {
+            "description": "`caught` = a remaining test failed by assertion on the reverted source, every run. `blind` = the suite stayed green on known-broken code. `nocover` = no test exercises the reverted files at all, which is worse than blind; it requires that at least one test file REMAINED after the fix's own were removed, otherwise nothing could be concluded and the verdict is `unverifiable` with reason `the-fix-shipped-the-only-test-file`. `unverifiable` = nothing could be concluded: the revert did not apply, the suite could not load, it timed out, or the commit only ADDS source (`no-prior-version` \u2014 there is no earlier behaviour to have been blind to, so it is not a bug the suite could have caught and it never enters a calibration). `flaky` = the runs disagreed, so nothing can be concluded.",
+            "enum": [
+              "caught",
+              "blind",
+              "nocover",
+              "unverifiable",
+              "flaky"
+            ]
+          },
+          "reason": {
+            "type": "string",
+            "maxLength": 256
+          },
+          "source": {
+            "description": "Source files reverted to the fix's parent.",
+            "type": "array",
+            "minItems": 1,
+            "items": {
+              "$ref": "urn:claimspec:v1:common#/$defs/repoPath"
+            }
+          },
+          "removedTests": {
+            "description": "Test files the fix shipped, removed before the run: the fix's own test proves nothing about what the suite knew before it.",
+            "type": "array",
+            "items": {
+              "$ref": "urn:claimspec:v1:common#/$defs/repoPath"
+            }
+          },
+          "ranTests": {
+            "type": "integer",
+            "minimum": 0
+          },
+          "faultClass": {
+            "description": "The injected-fault class this real diff most resembles, derived mechanically from the diff shape. This is the join key between ground truth and a fault model: it is what lets P(this fault class is a real bug) be computed.",
+            "$ref": "urn:claimspec:v1:common#/$defs/faultClass"
+          },
+          "runs": {
+            "type": "array",
+            "minItems": 1,
+            "items": {
+              "$ref": "urn:claimspec:v1:evidence#/$defs/testRun"
+            }
+          }
+        },
+        "additionalProperties": false
+      }
+    }
+  },
+  "additionalProperties": false
+}