testguard-cli 0.7.0 → 0.8.0

package/CHANGELOG.md

@@ -7,6 +7,122 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.8.0] - 2026-09-18
+
+A calibration you can quote.
+
+A calibration document now says what its number means and where its labels
+came from, and it reproduces: `p` and every interval are recomputed by the
+validator from `n`, `positives`, `method` and `confidence`, at the document's
+own precision — the spec's own example had carried a truncated bound since
+the day it was written, and nothing could notice. `nocover` counts as the
+miss it is, so a project with no tests at all for a subsystem no longer
+outscores one with weak tests. The format also carries what
+websec-validator's shipped table already said — corpus, caveat, a producer
+floor, backoff tiers, a labelled prior — so a second tool's honesty survives
+translation; that table is now a conformance example, and "adoptable" means
+the file validates rather than a sentence saying so. Underneath: the
+self-probe gate is held to a cost budget (#89), and a reserved method has
+somewhere to put its data (#91).
+
+### Added
+- **A calibration can now say what it measures and where it came from** (#82).
+  websec-validator's shipped `calibration.json` already carried `corpus`,
+  `min_n`, `caveat`, `evidence_status` and `limitation` — the fields that say
+  "indicative, do not quote this number" — and claimspec had nowhere to put
+  them, so adopting it would have stripped that honesty in translation. It
+  also had nowhere to say what `p` *was*: testguard's is a miss rate where
+  high is bad, websec's is P(real) where high is good, and both validated
+  identically. The schema gains `measures` (a registry: `escape-missed`,
+  `finding-real`), provenance in `source` (`corpus`, `caveat`, `limitation`,
+  `evidenceStatus`, a `tool-oracle` kind, and an unconstrained `detail` on the
+  `methodDetail` pattern), a producer floor `minN` that consumers may raise
+  and never lower, ordered `backoff` tiers so a class→label→prior cascade is
+  expressible, a `fallback` whose shape says it is a guess, a per-cell
+  `breakdown` that sums to `n`, `p: null` at `n = 0`, and `|`-compound bucket
+  keys whose arity the validator checks. Every field is optional and absence
+  means *unattributed* — readable, not quotable — so no existing document
+  breaks; testguard's emitter always writes them and a self-claim guards it.
+  The proof of adoption is a file, not a sentence: websec-validator's table,
+  translated field for field, is now `spec/conformance/examples/calibration-websec.json`
+  and validates with every number reproducing. `GATE-SEMANTICS.md` records
+  the merge rule (sum counts, recompute, never average; only when `measures`
+  and `bucketBy` agree) that websec's shipped-plus-local overlay already
+  depends on.
+- **The gate is now gated on its own cost** (#89). `--cost` has printed the
+  self-probe's wall clock into every CI log since #67, and the gate still went
+  from 9.6 to 23.6 minutes across one merged change without anything saying a
+  word. A measurement nothing gates on is not a check — which is the premise
+  this tool rests on, applied to itself.
+  `testguard.cost-budget.json` carries the ceiling, its measurement and why it
+  is set where it is. CI fails when the probe exceeds it and opens with the
+  most expensive claims, so the first thing a failure shows is where to look.
+  Raising it is a committed diff with a reason: growth is allowed, going
+  unnoticed is not.
+  The budget is on the **total**, never a per-fault average. The regression
+  that prompted this took the corpus from 59 faults to 91 while per-fault cost
+  barely moved — an average would have reported everything fine while the gate
+  tripled, and a test says so. A budget that is missing or not a positive
+  number is refused rather than treated as unlimited, so a typo cannot quietly
+  disable the gate.
+
+
+### Fixed
+- **The cost budget is set against what CI measures.** 950 s was 18% over
+  the local 804 s figure and only 4.6% over the 906 s CI already measured on
+  `main`, so the first pull request to add any claim tripped it — #94 passed
+  its own leg only because the cache reused verdicts, and `main` went red at
+  981 s from scratch the moment it merged. Raised to 1100 s (~14% over the
+  962 s CI measures with #82's re-pointed defenders), with the CI figures
+  recorded beside the local one so the next person budgets against the
+  number the check actually sees. The heavy claims the failure now names are
+  the next #73-style cut.
+- **Calibration's pure tests moved off the replay fixture** (the #73 pattern).
+  Three calibration claims were defended by `test/replay.test.mjs`, which
+  builds a scripted git corpus on every run; their kills come from pure
+  `calibrationFrom` tests that need none of it. The cost budget fired on the
+  first full run — 1022 s against 950 s, the top rows being exactly those
+  claims — so the tests now live in `test/calibration.test.mjs`, the claims
+  point there, and every fault was re-probed after the move, not assumed.
+- **A cost-budget failure names its claims.** `checkCostBudget` read `id`
+  from a report whose claims carry `claimId`, so the one line meant to say
+  *which* claims to look at printed `undefined` five times — and its unit test
+  fed the shape the code wished for, so it passed. Found the first time the
+  budget actually fired. The test now goes through `costReport`, and
+  `TG-COST-BUDGET-NAMES-THE-CLAIM` guards it.
+- **A calibration counts `nocover` as a miss** (#82). `nocover` — no test
+  even imports the broken file — is the worst replay outcome, and the ratio
+  excluded it: a project with no tests at all for a subsystem scored *better*
+  than one with weak tests, because its worst outcomes left the denominator
+  before the ratio was taken. It now enters both sides, as it does in Stryker
+  and PIT, and the replay summary says how many misses were blind and how many
+  had no test at all. The schema description, the emitter's docstring and
+  `GATE-SEMANTICS.md` had also disagreed about what `p` was — "P(finding is
+  real)" in one place, the miss rate in another, and the docstring's two
+  sentences conditioning on different denominators — and now say one thing:
+  for testguard, the share of real escaped bugs the suite failed to catch.
+- **A calibration's arithmetic is checked, not trusted.** The validator
+  recomputed nothing: it checked that `p` sat inside `ci` and would pass a
+  document with every number invented. It now recomputes `p` from
+  `positives / n` and `ci` from `n`, `positives`, `method` and `confidence`
+  with `spec/lib/wilson.mjs` — one implementation, shared with the emitter —
+  at the document's own precision, accepting either the exact quantile or the
+  textbook 1.96. The spec's own example had carried a truncated lower bound
+  (6/30 → `0.09`; Wilson gives 0.0950 → `0.10`) since the day it was written,
+  and websec-validator's shipped table reproduces exactly.
+- **A reserved method can now say something.** #81 reserved `assertion` and
+  `scan` without required fields, and `fault` / `detail` are both
+  `additionalProperties: false` — so a probe using a reserved method could
+  declare that it existed and **nothing whatsoever about it**: not which rule
+  was checked, not where. That is not a reservation, it is a dead end, and
+  DocGuard's writer would have hit it on its first line.
+  Both now carry `methodDetail`: an object the spec deliberately does not
+  constrain, on the probe and on the record. It is **forbidden under
+  `fault-injection`**, whose shape is specified and must not acquire a junk
+  drawer — a self-claim fails if that ever stops being enforced — and it is a
+  staging area rather than a permanent home. What the first real consumer puts
+  there is the evidence for what the specified shape should become.
+
 ## [0.7.0] - 2026-09-18
 
 Python, and a fault that cannot hide.

package/README.md

@@ -126,9 +126,9 @@ raises coverage. TestGuard admits it because it fails when the claim is false.
 | npm | `npm i -D testguard-cli` then `npx testguard probe` |
 | pip | `pip install testguard-cli` then `testguard probe` (needs Node ≥ 20) |
 | Homebrew | `brew tap raccioly/tap && brew install testguard` |
-| GitHub Action | `uses: raccioly/testguard@v0.7.0` — see [`action.yml`](./action.yml) |
+| GitHub Action | `uses: raccioly/testguard@v0.8.0` — see [`action.yml`](./action.yml) |
 | pre-commit | `repo: https://github.com/raccioly/testguard`, hooks `testguard-claims`, `testguard-probe` |
-| GitLab CI | `include: - remote: https://raw.githubusercontent.com/raccioly/testguard/v0.7.0/packaging/gitlab/testguard.gitlab-ci.yml` with `inputs:` — see [`packaging/gitlab/`](./packaging/gitlab/testguard.gitlab-ci.yml) |
+| GitLab CI | `include: - remote: https://raw.githubusercontent.com/raccioly/testguard/v0.8.0/packaging/gitlab/testguard.gitlab-ci.yml` with `inputs:` — see [`packaging/gitlab/`](./packaging/gitlab/testguard.gitlab-ci.yml) |
 
 Projects that set `min-release-age` in `.npmrc` cannot see a version published
 less than that many days ago (`ENOVERSIONS`); install that one with
@@ -345,15 +345,18 @@ most resembles, which is the join key that turns an uninterpretable mutation
 score into a statement with a sample size:
 
 ```
-  guard-removed          blind   7/9    p=0.78  ci [0.45, 0.94]
-  field-dropped          blind   4/4    p=1.00  ci [0.51, 1.00]
+  guard-removed          missed  7/9    p=0.78  ci [0.45, 0.94]
+  field-dropped          missed  4/4    p=1.00  ci [0.51, 1.00]
 ```
 
-Read as *"when a fault of this class survives, how often does that
-correspond to a bug that really escaped"*. It is written as a
-`calibration` document beside the replay one. Only `caught` and `blind`
-carry information; the other verdicts are excluded from both sides of the
-ratio.
+Read as *"when a real bug of this class escapes, how often does the suite
+miss it"* — a miss rate, so a high `p` is bad. It is written as a
+`calibration` document beside the replay one. `caught`, `blind` and
+`nocover` are measurements and enter the ratio, `nocover` as a miss: leaving
+it out would score a project with no tests at all for a subsystem *better*
+than one with weak tests. `flaky` and `unverifiable` are failed measurements
+and enter neither side. Every `p` and `ci` is recomputable from `n` and the
+miss count, and the validator recomputes them.
 
 **The open question this exists to answer.** Does a calibration learned on a
 repository *with* history transfer to a greenfield one that has none? AI
@@ -424,12 +427,12 @@ detected base that does not resolve is a warning for `status` and `brief`
 # GitHub Actions
 - uses: actions/checkout@v4
   with: { fetch-depth: 0 }
-- uses: raccioly/testguard@v0.7.0
+- uses: raccioly/testguard@v0.8.0
   with: { command: gate }
 
 # GitLab CI — the component-shaped template: gate + probe, brief as an artifact and, opted in, as a merge-request note
 include:
-  - remote: 'https://raw.githubusercontent.com/raccioly/testguard/v0.7.0/packaging/gitlab/testguard.gitlab-ci.yml'
+  - remote: 'https://raw.githubusercontent.com/raccioly/testguard/v0.8.0/packaging/gitlab/testguard.gitlab-ci.yml'
     inputs: { dir: backend, post_note: true }   # post_note needs TESTGUARD_GITLAB_TOKEN (api scope); one note, updated in place
 ```
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "testguard-cli",
-  "version": "0.7.0",
+  "version": "0.8.0",
   "description": "Breaks your code on purpose and reports every promise your tests did not notice breaking. Mutation testing bound to the claims a project makes — vitest, jest, Playwright, pytest and unittest.",
   "type": "module",
   "license": "MIT",

package/spec/GATE-SEMANTICS.md

@@ -167,6 +167,14 @@ Rules that follow from the table:
    under `fault-injection`, and the validator enforces them there in full: the
    relaxation is for a tool that injects nothing, never a licence for an
    injecting tool to stop showing its work.
+   A reserved method carries its own data in `methodDetail`, on the probe and
+   on the record: an object the spec deliberately does not constrain, so a tool
+   can say which rule it checked and where. Without it, "reserved" would mean a
+   probe that can declare it exists and nothing about it, which is not a
+   reservation but a dead end. It is **forbidden under `fault-injection`**,
+   whose shape is specified and must not acquire a junk drawer, and it is a
+   staging area rather than a permanent home: what the first consumer puts
+   there is the evidence for what the specified shape should become.
    `assertion` and `scan` are **reserved**. Their required fields are
    deliberately unspecified and will be defined by their first real consumer,
    with its own conformance examples written from the shape that tool actually
@@ -201,9 +209,96 @@ Two rules that follow:
 2. **The fix's own test is removed before the run.** It proves nothing about
    what the suite knew before the fix existed.
 
-Only `caught` and `blind` carry information. `nocover`, `flaky` and
-`unverifiable` are excluded from both sides of a calibration ratio, because a
-number computed over them would mean nothing.
+`caught`, `blind` and `nocover` are measurements and enter the calibration
+ratio: `caught` as a hit, the other two as misses. `nocover` counts as a miss
+because it is one — the worst kind. Leaving it out would let a project with
+**no tests at all** for a subsystem score better than one with weak tests,
+since its worst outcomes would leave the denominator before the ratio is
+taken; Stryker and PIT count no-coverage mutants against the headline score
+for the same reason. `flaky` and `unverifiable` are failed measurements and
+enter neither side.
+
+### Calibration arithmetic is reproducible
+
+A calibration cell declares `n` and `positives`, and with them has declared
+`p`; a document declares `method: wilson` at `confidence`, and with them has
+declared every `ci`. The validator recomputes both with `lib/wilson.mjs` —
+the single implementation, as `lib/fingerprint.mjs` is for fingerprints — and
+a cell that does not reproduce does not conform. Before this rule the
+validator checked only that `p` lay inside `ci`, which a document with every
+number invented satisfies, and the spec's own example shipped a truncated
+lower bound (6/30 → `0.09`, where Wilson gives 0.0950 → `0.10`) that nothing
+could notice.
+
+- **Precision is the document's own**: the most decimal places any `p` or
+  `ci` value shows, capped at 6. Values are compared after rounding to that
+  precision, so a producer at 3 dp and one at 4 dp both reproduce and a
+  truncated bound does not.
+- **z may be the exact quantile or the textbook rounding** (1.96, 2.576,
+  1.645). The two differ by up to 9.2e-6 in the interval — enough to flip a
+  rounding boundary at 4 dp — so either reproduces, and a document never fails
+  for having used the number in the textbook.
+- **`p` is `positives / n`.** A smoothed or shrunk point estimate is a
+  different method and must say so; under `wilson` the point estimate is the
+  observed proportion.
+
+### Calibration provenance, backoff and merging
+
+Everything beyond a document's primary buckets is optional and additive, in
+the way `method` is on claims and evidence: absence has a defined meaning, and
+no document written before a field existed becomes invalid. The meaning of
+absence is **unattributed** — readable, auditable, and not a source of
+numbers. A tool that emits calibrations always emits these fields; a
+self-claim guards that it does.
+
+- **`measures`** says what `p` is the probability of. It is the one field
+  that lets two calibrations be told apart: testguard's `escape-missed` is
+  P(a real escaped bug of this class was not caught), and high is bad;
+  websec-validator's `finding-real` is P(a reported finding of this class is a
+  real vulnerability), and high is good. A document without it cannot be
+  quoted and cannot be merged. Registry: `escape-missed`, `finding-real`. A
+  new value is added here by the tool that first emits it.
+- **`source`** carries provenance. `corpus` names external ground-truth
+  sources (a bug-replay's is the repository's own history, which `ref` already
+  names); `caveat` is the one sentence a consumer shows beside any quoted
+  number; `limitation` is the longer story; `evidenceStatus` says whether the
+  labels were reviewed; `kind: tool-oracle` is a tool confirming its own
+  findings, as websec-validator's dynamic probes do. `detail` is the tool's
+  own provenance, unconstrained, on the `methodDetail` pattern. A number
+  quoted without its `caveat` is a misquote.
+- **`minN`** is the producer's floor. A cell below it is not quoted on its
+  own; the consumer backs off a tier. A consumer may raise it — a gate that
+  blocks a merge wants more certainty than a report — and may never lower
+  it: lowering it resurrects a cell the producer suppressed. Absent, the
+  producer sets no floor and the consumer applies its own. A spec-level
+  constant would be wrong in both directions: websec-validator's 5 would
+  suppress every cell of a corpus the size of DocGuard's benchmark.
+- **`backoff`** is the ordered list of coarser tiers consulted below `minN`.
+  Each tier is held to the same arithmetic as the primary. Keys may be
+  compound (`attackClass|confidence` backs off to `confidence`), and every key
+  in a tier has as many `|`-parts as its `bucketBy` — a key with fewer was
+  translated from another table and lost a dimension on the way.
+- **`fallback`** is what remains when no tier has a cell above `minN`: a
+  labelled guess, not a measurement, and anything quoted from it carries
+  `n = 0`, `ci = [0, 1]` and its `basis`. It lives in the document so that
+  "what did it assume when it had no data?" has one auditable answer instead
+  of one per consumer, each unlabelled.
+- **`breakdown`** splits a cell's `n` by outcome when the producer can say
+  (testguard: `caught` / `blind` / `nocover`), and sums to `n`. It lets a
+  consumer recover a narrower rate — misses among covered code, say —
+  without the producer publishing a second, misleading headline.
+
+**Merging.** Two calibrations merge by summing `n` and `positives` per cell
+and recomputing `p` and `ci` from the sums — never by averaging `p` or `ci`,
+which have no meaning combined. They merge only when `measures`, `bucketBy`
+and `confidence` agree, and the merged document's `source.kind` is `mixed`.
+This is how websec-validator folds an operator's own confirmed samples over
+its shipped table, and it works only because cells carry raw counts.
+
+**The second producer is a conformance example.** websec-validator's shipped
+table, translated field for field, is `conformance/examples/calibration-websec.json`
+— corpus, caveat, limitation, floor, backoff tier and labelled fallback all
+present. That file, not a sentence in a README, is what "adoptable" means.
 
 ## Baseline and delta
 

package/spec/README.md

@@ -28,7 +28,7 @@ that carry claims, baselines, scoping and briefs do not care.
 | `evidence` | [`schemas/evidence.schema.json`](schemas/evidence.schema.json) | One run's findings on disk, readable without the tool. Every verdict carries every run that produced it. |
 | `baseline` | [`schemas/baseline.schema.json`](schemas/baseline.schema.json) | Frozen fingerprints of existing debt. Gate only what is new. |
 | `ignore` | [`schemas/ignore.schema.json`](schemas/ignore.schema.json) | Reviewable scoping. Every entry has a reason. |
-| `calibration` | [`schemas/calibration.schema.json`](schemas/calibration.schema.json) | P(finding is real) per bucket, with a Wilson interval and the sample size behind it. |
+| `calibration` | [`schemas/calibration.schema.json`](schemas/calibration.schema.json) | The observed rate of a bucket's outcome — what it measures is in the document — with a Wilson interval and the sample size behind it. Every number is recomputable and the validator recomputes it; provenance, a producer floor, backoff tiers and a labelled fallback ride along so a second tool's honesty survives translation. |
 | `brief` | [`schemas/brief.schema.json`](schemas/brief.schema.json) | What to tell an agent before it writes code — ranked, capped, never a single score, and carrying the one next action. |
 | `status` | [`schemas/status.schema.json`](schemas/status.schema.json) | Where the project is and what happens next — the single machine-readable truth every human rendering derives from. Surfaces faults whose content changed since they were probed, and changed files that carry no claim. |
 | `replay` | [`schemas/replay.schema.json`](schemas/replay.schema.json) | Would this suite have caught the bugs that already escaped? Replays real fix commits: revert the source, remove the test the fix shipped, run what remains. A replayed bug is ground truth — a human already confirmed it was a defect — which is what a fault model is calibrated against. |
@@ -65,7 +65,10 @@ consumer. It moves to its own repository only when a second tool adopts it.
 
 Adoption is additive and optional: an existing tool gains an output writer, and
 nothing it already detects or already writes changes. That holds today for
-`baseline`, `ignore`, `brief` and `status`. It does **not** yet hold for
+`baseline`, `ignore`, `brief`, `status` and `calibration` — websec-validator's
+shipped calibration table is `conformance/examples/calibration-websec.json`,
+translated field for field with its corpus, caveat, floor, backoff and
+fallback intact. It does **not** yet hold for
 `claims` and `evidence`, whose required fields assume fault injection
 (`find`/`replace` anchors, `confirmRuns`, `baselineRuns`/`probeRuns`,
 `defenders`) — a tool that scans or reads would have to emit empty arrays to

package/spec/lib/validate.mjs

@@ -3,6 +3,7 @@ import { readFileSync, readdirSync } from 'node:fs';
 import { fileURLToPath } from 'node:url';
 import { dirname, join } from 'node:path';
 import { fingerprint } from './fingerprint.mjs';
+import { reproduces, decimals, MAX_PLACES } from './wilson.mjs';
 
 const schemaDir = join(dirname(fileURLToPath(import.meta.url)), '..', 'schemas');
 
@@ -77,6 +78,9 @@ const semantic = {
       // runs is not a leaner document, it is a verdict with nothing behind it;
       // relaxing the schema was to let a scanner conform, never to let an
       // injecting tool stop showing its work.
+      if (injection && r.detail.methodDetail) {
+        errors.push({ path: `${p}/detail/methodDetail`, message: 'methodDetail is for a method the spec has not specified; fault-injection has a specified shape and must not acquire a junk drawer' });
+      }
       if (injection) {
         for (const k of ['defenders', 'inputs']) {
           if (!r[k]) errors.push({ path: `${p}/${k}`, message: `fault-injection requires ${k} on every record` });
@@ -186,12 +190,43 @@ const semantic = {
 
   calibration(doc) {
     const errors = [];
-    for (const [key, b] of Object.entries(doc.buckets)) {
-      const p = `/buckets/${key}`;
-      if (b.positives > b.n) errors.push({ path: `${p}/positives`, message: `positives (${b.positives}) exceed n (${b.n})` });
-      const [lo, hi] = b.ci;
-      if (lo > hi) errors.push({ path: `${p}/ci`, message: `ci lower bound ${lo} exceeds upper bound ${hi}` });
-      if (b.p < lo || b.p > hi) errors.push({ path: `${p}/p`, message: `p (${b.p}) lies outside ci [${lo}, ${hi}]` });
+    // The primary buckets and every backoff tier are held to the same rules:
+    // a coarser tier a consumer falls back to is exactly where a wrong number
+    // would go unnoticed.
+    const tiers = [{ bucketBy: doc.bucketBy, buckets: doc.buckets, path: '' }, ...(doc.backoff ?? []).map((t, i) => ({ ...t, path: `/backoff/${i}` }))];
+    // The precision the document wrote at: the most places any value shows.
+    // Parsing drops trailing zeros (0.10 → 0.1), so a single value can
+    // under-report; the maximum cannot unless every value ended in zero.
+    const places = tiers.flatMap((t) => Object.values(t.buckets)).reduce((m, b) => Math.max(m, decimals(b.p ?? 0), decimals(b.ci[0]), decimals(b.ci[1])), 0);
+    for (const t of tiers) {
+      // A compound bucketBy (`attackClass|confidence`) means compound keys,
+      // each with the same number of parts — a key with fewer was translated
+      // from another tool's table and lost a dimension on the way.
+      const arity = t.bucketBy.split('|').length;
+      for (const [key, b] of Object.entries(t.buckets)) {
+        const p = `${t.path}/buckets/${key}`;
+        const parts = key.split('|').length;
+        if (parts !== arity) errors.push({ path: p, message: `bucket key "${key}" has ${parts} part(s); bucketBy "${t.bucketBy}" has ${arity}` });
+        if (b.positives > b.n) errors.push({ path: `${p}/positives`, message: `positives (${b.positives}) exceed n (${b.n})` });
+        const [lo, hi] = b.ci;
+        if (lo > hi) errors.push({ path: `${p}/ci`, message: `ci lower bound ${lo} exceeds upper bound ${hi}` });
+        if (b.p !== null && (b.p < lo || b.p > hi)) errors.push({ path: `${p}/p`, message: `p (${b.p}) lies outside ci [${lo}, ${hi}]` });
+        if (b.breakdown) {
+          const sum = Object.values(b.breakdown).reduce((s, v) => s + v, 0);
+          if (sum !== b.n) errors.push({ path: `${p}/breakdown`, message: `breakdown sums to ${sum}; n is ${b.n}` });
+        }
+        // A cell that declares n and positives has declared p; a document that
+        // declares `wilson` at `confidence` has declared every ci. Recompute
+        // both with the spec's own implementation: "conforms" means
+        // "reproducible", not "internally plausible". Before this the validator
+        // would pass a document with every number invented, and the spec's own
+        // example carried a truncated bound nothing could notice.
+        // A cell with more positives than trials has no proportion to reproduce.
+        if (b.positives > b.n) continue;
+        const r = reproduces(b, { confidence: doc.confidence, places });
+        if (!r.p) errors.push({ path: `${p}/p`, message: `p (${b.p}) is not positives/n: ${b.positives}/${b.n} = ${r.expected.p} at ${Math.min(places, MAX_PLACES)} dp` });
+        if (!r.ci) errors.push({ path: `${p}/ci`, message: `ci [${lo}, ${hi}] does not reproduce: ${doc.method} at ${doc.confidence} for ${b.positives}/${b.n} is [${r.expected.ci[0]}, ${r.expected.ci[1]}]` });
+      }
     }
     return errors;
   },

package/spec/lib/wilson.mjs

@@ -0,0 +1,112 @@
+/**
+ * The single Wilson interval implementation, as `fingerprint.mjs` is the
+ * single fingerprint: a producer emits with it, the validator recomputes with
+ * it, and "conforms" means the two agree. Before this file existed the emitter
+ * and the validator had no shared arithmetic at all — the validator checked
+ * that `p` sat inside `ci` and nothing else, and the spec's own example
+ * carried a truncated lower bound (6/30 → 0.09, where Wilson gives 0.0950 →
+ * 0.10) that nothing could notice.
+ */
+
+// Inverse normal CDF: Acklam's rational approximation, |relative error| < 1.2e-9.
+// Enough to reproduce an interval to 6 decimal places, which is the most any
+// producer writes (websec-validator 3, testguard 4, docguard 6).
+const A = [-3.969683028665376e+01, 2.209460984245205e+02, -2.759285104469687e+02, 1.383577518672690e+02, -3.066479806614716e+01, 2.506628277459239e+00];
+const B = [-5.447609879822406e+01, 1.615858368580409e+02, -1.556989798598866e+02, 6.680131188771972e+01, -1.328068155288572e+01];
+const C = [-7.784894002430293e-03, -3.223964580411365e-01, -2.400758277161838e+00, -2.549732539343734e+00, 4.374664141464968e+00, 2.938163982698783e+00];
+const D = [7.784695709041462e-03, 3.224671290700398e-01, 2.445134137142996e+00, 3.754408661907416e+00];
+const P_LOW = 0.02425;
+
+/** The standard normal quantile: z such that Φ(z) = p. */
+export function probit(p) {
+  if (!(p > 0 && p < 1)) throw new RangeError(`probit is defined on (0, 1), got ${p}`);
+  let q;
+  if (p < P_LOW) {
+    q = Math.sqrt(-2 * Math.log(p));
+    return (((((C[0] * q + C[1]) * q + C[2]) * q + C[3]) * q + C[4]) * q + C[5]) / ((((D[0] * q + D[1]) * q + D[2]) * q + D[3]) * q + 1);
+  }
+  if (p <= 1 - P_LOW) {
+    q = p - 0.5;
+    const r = q * q;
+    return (((((A[0] * r + A[1]) * r + A[2]) * r + A[3]) * r + A[4]) * r + A[5]) * q / (((((B[0] * r + B[1]) * r + B[2]) * r + B[3]) * r + B[4]) * r + 1);
+  }
+  q = Math.sqrt(-2 * Math.log(1 - p));
+  return -(((((C[0] * q + C[1]) * q + C[2]) * q + C[3]) * q + C[4]) * q + C[5]) / ((((D[0] * q + D[1]) * q + D[2]) * q + D[3]) * q + 1);
+}
+
+/**
+ * The textbook roundings of z that producers actually type. testguard and
+ * docguard use 1.96; websec-validator uses the exact quantile. Across n ≤ 200
+ * the two differ by at most 9.2e-6 in the interval, which can still flip a
+ * rounding boundary at 4 dp — so the validator accepts either, and a document
+ * never fails for having used the number in the textbook.
+ */
+export const CONVENTIONAL_Z = Object.freeze({ 0.9: 1.645, 0.95: 1.96, 0.99: 2.576 });
+
+/** Every z a producer may legitimately have used for a two-sided interval at `confidence`. */
+export function zCandidates(confidence) {
+  const exact = probit(1 - (1 - confidence) / 2);
+  const conventional = CONVENTIONAL_Z[confidence];
+  return conventional === undefined ? [exact] : [exact, conventional];
+}
+
+/**
+ * Wilson score interval for `positives` in `n`, unrounded, clamped to [0, 1].
+ * Wilson rather than the normal approximation because it stays sane at small
+ * n and extreme p — exactly the regime a young corpus is in. n = 0 is maximal
+ * ignorance: [0, 1].
+ */
+export function wilsonInterval(positives, n, z) {
+  if (n === 0) return [0, 1];
+  const p = positives / n;
+  const d = 1 + (z * z) / n;
+  const centre = (p + (z * z) / (2 * n)) / d;
+  const half = (z * Math.sqrt((p * (1 - p)) / n + (z * z) / (4 * n * n))) / d;
+  return [Math.max(0, centre - half), Math.min(1, centre + half)];
+}
+
+/** Round half away from zero to `places` decimals — what `toFixed` does on the value's binary form. */
+export const roundTo = (x, places) => Number(x.toFixed(places));
+
+/**
+ * How many decimal places a parsed JSON number still shows. Parsing drops
+ * trailing zeros (0.10 → 0.1), so a single value under-reports; the validator
+ * takes the maximum across a document, which is the precision the producer
+ * wrote at as long as at least one value did not end in zero.
+ */
+export function decimals(x) {
+  if (!Number.isFinite(x)) return 0;
+  const s = String(x);
+  const e = s.indexOf('e-');
+  if (e !== -1) {
+    const mantissa = s.slice(0, e);
+    const dot = mantissa.indexOf('.');
+    return Number(s.slice(e + 2)) + (dot === -1 ? 0 : mantissa.length - dot - 1);
+  }
+  const dot = s.indexOf('.');
+  return dot === -1 ? 0 : s.length - dot - 1;
+}
+
+/** The most decimal places the validator will hold a document to; beyond it the quantile approximation is the noise. */
+export const MAX_PLACES = 6;
+
+const near = (a, b) => Math.abs(a - b) < 1e-9;
+
+/**
+ * Does a cell's declared `p` and `ci` reproduce from its own `n` and
+ * `positives`? Everything is compared after rounding to `places` — the
+ * precision the document wrote at — so a producer at 3 dp and one at 4 dp
+ * both reproduce, and a truncated bound does not. A `p` of `null` is a cell
+ * that declines to give a point estimate; only its interval is checked.
+ */
+export function reproduces(cell, { confidence, places }) {
+  const { n, positives, p, ci } = cell;
+  const pl = Math.min(places, MAX_PLACES);
+  const expectedP = n === 0 ? null : roundTo(positives / n, pl);
+  const candidates = zCandidates(confidence).map((z) => wilsonInterval(positives, n, z).map((v) => roundTo(v, pl)));
+  return {
+    p: p === null || n === 0 || near(expectedP, p),
+    ci: candidates.some(([lo, hi]) => near(lo, ci[0]) && near(hi, ci[1])),
+    expected: { p: expectedP, ci: candidates[0] },
+  };
+}

package/spec/schemas/calibration.schema.json

@@ -2,7 +2,7 @@
   "$schema": "https://json-schema.org/draft/2020-12/schema",
   "$id": "urn:claimspec:v1:calibration",
   "title": "claimspec v1 — calibration",
-  "description": "P(finding is real) per bucket, with a confidence interval and the sample size behind it. A label without n is a vibe; this is the format that stops it being one.",
+  "description": "The observed rate of a bucket's outcome — for testguard, the share of real escaped bugs of a fault class that the suite failed to catch, so a high p is bad — with a confidence interval and the sample size behind it. A label without n is a vibe; this is the format that stops it being one. p and ci are recomputable from n, positives, method and confidence, and the validator recomputes them. Everything beyond the primary buckets is optional and additive, as `method` is on claims and evidence: a document that omits `measures` or `source.caveat` stays valid and is UNATTRIBUTED — a consumer may read it and audit it, merge nothing into it, and quote no number from it.",
   "type": "object",
   "required": ["schemaVersion", "tool", "computedAt", "method", "confidence", "bucketBy", "source", "buckets"],
   "properties": {
@@ -11,44 +11,114 @@
     "tool": { "$ref": "urn:claimspec:v1:common#/$defs/tool" },
     "computedAt": { "$ref": "urn:claimspec:v1:common#/$defs/isoDateTime" },
     "method": {
-      "description": "Interval method. `wilson` is the only method defined in v1.",
+      "description": "Interval method. `wilson` is the only method defined in v1; under it the point estimate is the observed proportion positives / n, and a smoothed or shrunk estimate is a different method.",
       "enum": ["wilson"]
     },
     "confidence": { "type": "number", "exclusiveMinimum": 0, "exclusiveMaximum": 1 },
+    "measures": {
+      "description": "What `p` is the probability of — the one field that lets two calibrations be told apart. Registry in GATE-SEMANTICS.md: `escape-missed` (testguard: a real escaped bug of this class was not caught; high is bad) and `finding-real` (websec-validator: a reported finding of this class is a real vulnerability; high is good). Two documents merge only when this, `bucketBy` and `confidence` agree. Absent = unattributed: nothing may be quoted from the document.",
+      "type": "string",
+      "pattern": "^[a-z][a-z0-9-]*$",
+      "maxLength": 64
+    },
     "bucketBy": {
-      "description": "The field findings are bucketed on, e.g. `faultClass` or `severity`.",
-      "$ref": "urn:claimspec:v1:common#/$defs/identifier"
+      "description": "The field findings are bucketed on, e.g. `faultClass` or `severity`. A compound key joins fields with `|` (`attackClass|confidence`), and every bucket key then has the same number of parts.",
+      "$ref": "#/$defs/bucketKey"
+    },
+    "minN": {
+      "description": "The producer's floor. A cell with fewer samples is not quoted on its own; the consumer backs off a tier, then to `fallback`, then to nothing. A consumer may raise this — a gate that blocks a merge wants more certainty than a report — and may never lower it: lowering it resurrects a cell the producer suppressed. Absent = the producer sets no floor and the consumer applies its own.",
+      "type": "integer",
+      "minimum": 0
     },
     "source": {
-      "description": "Where the ground-truth labels came from.",
+      "description": "Where the ground-truth labels came from. `kind` is the mechanism (`tool-oracle`: a tool confirming its own findings, as websec-validator's dynamic probes do). `corpus` names external ground-truth sources — a bug-replay's ground truth is the repository's own history, already named by `ref`, so it is usually absent there. `caveat` is the one sentence a consumer must show beside any quoted number; `limitation` is the longer story; `evidenceStatus` says whether the labels were reviewed. `detail` is the tool's own provenance, deliberately unconstrained, on the `methodDetail` pattern: what the first consumers put there is the evidence for what should become specified.",
       "type": "object",
       "required": ["kind"],
       "properties": {
-        "kind": { "enum": ["bug-replay", "human-label", "mixed"] },
-        "ref": { "type": "string", "maxLength": 512 }
+        "kind": { "enum": ["bug-replay", "human-label", "tool-oracle", "mixed"] },
+        "ref": { "type": "string", "maxLength": 512 },
+        "corpus": {
+          "type": "array",
+          "minItems": 1,
+          "items": { "type": "string", "minLength": 1, "maxLength": 128 }
+        },
+        "caveat": { "type": "string", "minLength": 1, "maxLength": 512 },
+        "limitation": { "type": "string", "minLength": 1, "maxLength": 1024 },
+        "evidenceStatus": { "enum": ["verified", "unverified", "quarantined"] },
+        "detail": { "type": "object", "minProperties": 1, "additionalProperties": true }
       },
       "additionalProperties": false
     },
-    "buckets": {
-      "type": "object",
-      "propertyNames": { "pattern": "^[A-Za-z0-9][A-Za-z0-9._-]*$" },
-      "additionalProperties": {
+    "buckets": { "$ref": "#/$defs/buckets" },
+    "backoff": {
+      "description": "Ordered coarser tiers consulted when a primary cell is below `minN`: websec-validator's `attackClass|confidence` backs off to `confidence`. Each tier is held to the same arithmetic as the primary. Absent = no backoff.",
+      "type": "array",
+      "minItems": 1,
+      "items": {
         "type": "object",
-        "required": ["n", "positives", "p", "ci"],
+        "required": ["bucketBy", "buckets"],
         "properties": {
-          "n": { "type": "integer", "minimum": 0 },
-          "positives": { "type": "integer", "minimum": 0 },
-          "p": { "type": "number", "minimum": 0, "maximum": 1 },
-          "ci": {
-            "type": "array",
-            "minItems": 2,
-            "maxItems": 2,
-            "items": { "type": "number", "minimum": 0, "maximum": 1 }
-          }
+          "bucketBy": { "$ref": "#/$defs/bucketKey" },
+          "buckets": { "$ref": "#/$defs/buckets" }
         },
         "additionalProperties": false
       }
+    },
+    "fallback": {
+      "description": "What remains when no tier has a cell above `minN`: a labelled guess, not a measurement, and the shape says so — it has a `basis` and no `n`, and anything quoted from it carries n = 0, ci = [0, 1] and the basis. It lives in the document so that \"what did it assume when it had no data?\" has one auditable answer instead of one per consumer.",
+      "type": "object",
+      "required": ["basis", "values"],
+      "properties": {
+        "basis": { "const": "uncalibrated-prior" },
+        "values": {
+          "type": "object",
+          "minProperties": 1,
+          "propertyNames": { "pattern": "^[A-Za-z0-9][A-Za-z0-9._-]*(\\|[A-Za-z0-9][A-Za-z0-9._-]*)*$" },
+          "additionalProperties": { "type": "number", "minimum": 0, "maximum": 1 }
+        }
+      },
+      "additionalProperties": false
     }
   },
-  "additionalProperties": false
+  "additionalProperties": false,
+  "$defs": {
+    "bucketKey": {
+      "type": "string",
+      "pattern": "^[A-Za-z0-9][A-Za-z0-9._-]*(\\|[A-Za-z0-9][A-Za-z0-9._-]*)*$",
+      "maxLength": 128
+    },
+    "cell": {
+      "type": "object",
+      "required": ["n", "positives", "p", "ci"],
+      "properties": {
+        "n": { "type": "integer", "minimum": 0 },
+        "positives": { "type": "integer", "minimum": 0 },
+        "p": {
+          "description": "positives / n at the document's precision, or null when n = 0: a cell with no trials has no proportion, and 0 would be a fabricated one.",
+          "type": ["number", "null"],
+          "minimum": 0,
+          "maximum": 1
+        },
+        "ci": {
+          "type": "array",
+          "minItems": 2,
+          "maxItems": 2,
+          "items": { "type": "number", "minimum": 0, "maximum": 1 }
+        },
+        "breakdown": {
+          "description": "How n splits by outcome, when the producer can say (testguard: caught / blind / nocover). Values sum to n. Lets a consumer recover a narrower rate — misses among covered code, say — without the producer publishing a second, misleading headline.",
+          "type": "object",
+          "minProperties": 1,
+          "propertyNames": { "pattern": "^[A-Za-z0-9][A-Za-z0-9._-]*$" },
+          "additionalProperties": { "type": "integer", "minimum": 0 }
+        }
+      },
+      "additionalProperties": false
+    },
+    "buckets": {
+      "type": "object",
+      "propertyNames": { "pattern": "^[A-Za-z0-9][A-Za-z0-9._-]*(\\|[A-Za-z0-9][A-Za-z0-9._-]*)*$" },
+      "additionalProperties": { "$ref": "#/$defs/cell" }
+    }
+  }
 }

package/spec/schemas/claims.schema.json

@@ -152,6 +152,12 @@
         },
         "method": {
           "$ref": "urn:claimspec:v1:common#/$defs/method"
+        },
+        "methodDetail": {
+          "description": "Data belonging to a method the spec has not specified yet. `assertion` and `scan` are reserved names with no required fields, so a tool adopting one needs somewhere to put the rule it checked and the place it checked \u2014 without which it can declare that a probe exists and nothing about it. Deliberately unconstrained, deliberately namespaced, and **forbidden under `fault-injection`**, whose shape IS specified and must not acquire a junk drawer. What the first real consumer puts here is the evidence for what the specified shape should become; it is a staging area, not a permanent home.",
+          "type": "object",
+          "minProperties": 1,
+          "additionalProperties": true
         }
       },
       "additionalProperties": false,
@@ -184,7 +190,8 @@
               "faultClass": true,
               "file": true,
               "find": true,
-              "replace": true
+              "replace": true,
+              "methodDetail": false
             },
             "required": [
               "faultClass",

package/spec/schemas/evidence.schema.json

@@ -501,6 +501,12 @@
                 }
               },
               "additionalProperties": false
+            },
+            "methodDetail": {
+              "description": "Data belonging to a method the spec has not specified yet. `assertion` and `scan` are reserved names with no required fields, so a tool adopting one needs somewhere to put the rule it checked and the place it checked \u2014 without which it can declare that a probe exists and nothing about it. Deliberately unconstrained, deliberately namespaced, and **forbidden under `fault-injection`**, whose shape IS specified and must not acquire a junk drawer. What the first real consumer puts here is the evidence for what the specified shape should become; it is a staging area, not a permanent home.",
+              "type": "object",
+              "minProperties": 1,
+              "additionalProperties": true
             }
           },
           "additionalProperties": false,

package/src/commands/replay.mjs

@@ -50,7 +50,7 @@ export async function replayCommand({ projectDir, values, version }, io) {
     io.out(renderReplay(doc));
     io.out('');
     for (const [cls, b] of Object.entries(calibration.buckets).sort((a, b2) => b2[1].n - a[1].n)) {
-      io.out(`  ${cls.padEnd(22)} blind ${String(b.positives).padStart(3)}/${String(b.n).padEnd(3)}  p=${b.p.toFixed(2)}  ci [${b.ci[0].toFixed(2)}, ${b.ci[1].toFixed(2)}]`);
+      io.out(`  ${cls.padEnd(22)} missed ${String(b.positives).padStart(3)}/${String(b.n).padEnd(3)}  p=${b.p === null ? ' n/a' : b.p.toFixed(2)}  ci [${b.ci[0].toFixed(2)}, ${b.ci[1].toFixed(2)}]`);
     }
     if (Object.keys(calibration.buckets).length === 0) io.out('  no measurable bug in this range: nothing to calibrate from yet.');
     io.out('');

package/src/probe/cost.mjs

@@ -143,3 +143,63 @@ export function renderCost(report, { limit = 10 } = {}) {
   }
   return out.join('\n');
 }
+
+/**
+ * Is this probe within its cost budget?
+ *
+ * The gate went from 9.6 minutes to 23.6 across one merged pull request, and
+ * nothing said a word — `--cost` had been printing the number into every CI log
+ * since the previous optimisation. A measurement nothing gates on is not a
+ * check, which is the premise this whole tool rests on, applied to itself.
+ *
+ * Budget the TOTAL, never the per-fault average. The regression that prompted
+ * this raised the fault count from 59 to 91 while per-fault cost barely moved,
+ * so an average would have reported everything fine while the gate tripled.
+ * Growth is legitimate; the point is that someone signs for it in a diff.
+ *
+ * `previousMs` is optional and only ever informational: "804s, budget 900s" is
+ * a much weaker signal than "804s, was 780s", and the prior total is already in
+ * the evidence CI restores for verdict reuse.
+ *
+ * Pure. Reads a report, returns a decision; the caller owns exit codes and I/O.
+ */
+export function checkCostBudget(report, { budgetSeconds, previousMs, worst = 5 } = {}) {
+  if (typeof budgetSeconds !== 'number' || !Number.isFinite(budgetSeconds) || budgetSeconds <= 0) {
+    throw new TypeError('checkCostBudget: budgetSeconds must be a positive number');
+  }
+  const budgetMs = budgetSeconds * 1000;
+  const totalMs = report.totalMs ?? 0;
+  return {
+    ok: totalMs <= budgetMs,
+    totalMs,
+    budgetMs,
+    overByMs: Math.max(0, totalMs - budgetMs),
+    headroomMs: Math.max(0, budgetMs - totalMs),
+    ...(typeof previousMs === 'number' ? { previousMs, deltaMs: totalMs - previousMs } : {}),
+    // Named so a failure opens with which claims to look at, not just a number.
+    // claimCosts emits `claimId`; reading `id` here printed "undefined" for every
+    // claim the first time the budget fired, and the test had fed the wished-for shape.
+    worst: (report.claims ?? []).slice(0, worst).map((c) => ({ id: c.claimId, ms: c.ms, runs: c.runs })),
+  };
+}
+
+/** The budget decision as text, for a CI log that someone reads only when it fails. */
+export function renderCostBudget(d) {
+  const out = [];
+  const delta = typeof d.deltaMs === 'number'
+    ? ` (${d.deltaMs >= 0 ? '+' : ''}${secs(Math.abs(d.deltaMs))} against the previous run's ${secs(d.previousMs)})`
+    : '';
+  out.push(d.ok
+    ? `cost ${secs(d.totalMs)} of a ${secs(d.budgetMs)} budget — ${secs(d.headroomMs)} to spare${delta}.`
+    : `COST BUDGET EXCEEDED: ${secs(d.totalMs)} against a ${secs(d.budgetMs)} budget, over by ${secs(d.overByMs)}${delta}.`);
+  if (!d.ok && d.worst.length) {
+    out.push('');
+    out.push('most expensive claims');
+    for (const c of d.worst) out.push(`  ${secs(c.ms).padStart(7)}  ${c.id}  (${c.runs} runs)`);
+    out.push('');
+    out.push('Either move a claim onto a defender that does not need the expensive setup —');
+    out.push('re-probing afterwards, never assuming — or raise the budget in a commit that');
+    out.push('says why. Growth is allowed; going unnoticed is not.');
+  }
+  return out.join('\n');
+}

package/src/replay/replay.mjs

@@ -8,6 +8,7 @@ import { parseCommandTemplate } from '../probe/runners/shared.mjs';
 import { fileImports } from '../probe/rank.mjs';
 import { IS_PY_TEST, pyFileImports } from '../probe/pyimports.mjs';
 import { labelDiff } from './label.mjs';
+import { wilsonInterval, roundTo } from '../../spec/lib/wilson.mjs';
 
 const TEST_RE = /(^|\/)(__tests__|tests?)\//;
 const IS_TEST = (f) => /\.(test|spec)\.[cm]?[jt]sx?$/.test(f) || IS_PY_TEST.test(f) || TEST_RE.test(f);
@@ -247,35 +248,50 @@ export function classifyReplay(runs) {
   return { verdict: 'flaky', reason: 'runs-disagreed' };
 }
 
-/** Wilson score interval: the reason a label carries n rather than a vibe. */
+/**
+ * Wilson score interval at 4 dp — the reason a label carries n rather than a
+ * vibe. Computed by the spec's single implementation, which the validator
+ * recomputes with: an emitter and a validator with separate arithmetic can
+ * drift, and did (see `spec/lib/wilson.mjs`).
+ */
 export function wilson(positives, n, z = 1.96) {
-  if (n === 0) return { p: 0, ci: [0, 1] };
-  const p = positives / n;
-  const d = 1 + (z * z) / n;
-  const centre = (p + (z * z) / (2 * n)) / d;
-  const half = (z * Math.sqrt((p * (1 - p)) / n + (z * z) / (4 * n * n))) / d;
-  const round = (x) => Number(Math.min(1, Math.max(0, x)).toFixed(4));
-  return { p: Number(p.toFixed(4)), ci: [round(centre - half), round(centre + half)] };
+  if (n === 0) return { p: null, ci: [0, 1] };
+  const [lo, hi] = wilsonInterval(positives, n, z);
+  return { p: roundTo(positives / n, 4), ci: [roundTo(lo, 4), roundTo(hi, 4)] };
 }
 
+/**
+ * Replay verdicts that are measurements of the suite. `caught` is a hit.
+ * `blind` is a miss. `nocover` is also a miss — the worst kind, no test even
+ * imported the broken file — and it counts as one: excluding it would score a
+ * project with no tests for a subsystem BETTER than one with weak tests,
+ * because its worst outcomes would leave the denominator before the ratio is
+ * taken. Stryker and PIT count no-coverage mutants against the headline score
+ * for the same reason. `flaky` and `unverifiable` are failed measurements
+ * and enter neither side.
+ */
+export const MEASURED_VERDICTS = Object.freeze(['caught', 'blind', 'nocover']);
+
 /**
  * Calibration from a replay document: per fault class, the share of real
- * escaped bugs of that shape the suite did NOT catch.
+ * escaped bugs of that shape the suite failed to catch.
  *
- * Read it as "when a fault of this class survives, how often does that
- * correspond to a bug that really escaped" — the number that turns an
- * uninterpretable mutation score into a statement with a sample size.
- * `caught` and `blind` are the only outcomes that carry information;
- * `nocover`, `flaky` and `unverifiable` are excluded from both sides.
+ * `p` is P(the suite misses it | a bug of this class escapes) — a miss rate,
+ * so a HIGH number is bad. It is the number that turns an uninterpretable
+ * mutation score into a statement with a sample size. `positives` counts the
+ * misses; see MEASURED_VERDICTS for which verdicts are measurements at all
+ * and why `nocover` is a miss rather than an exclusion.
  */
 export function calibrationFrom(replayDoc, { confidence = 0.95, computedAt = new Date().toISOString(), toolVersion } = {}) {
   const buckets = {};
   for (const r of replayDoc.records) {
-    if (!['caught', 'blind'].includes(r.verdict)) continue;
+    if (!MEASURED_VERDICTS.includes(r.verdict)) continue;
     const key = r.faultClass ?? 'other';
-    buckets[key] ??= { n: 0, positives: 0 };
+    buckets[key] ??= { n: 0, positives: 0, breakdown: {} };
     buckets[key].n += 1;
-    if (r.verdict === 'blind') buckets[key].positives += 1;
+    buckets[key].breakdown[r.verdict] = (buckets[key].breakdown[r.verdict] ?? 0) + 1;
+    // Anything measured that was not caught is a miss — blind and nocover alike.
+    if (r.verdict !== 'caught') buckets[key].positives += 1;
   }
   for (const [k, b] of Object.entries(buckets)) buckets[k] = { ...b, ...wilson(b.positives, b.n) };
   return {
@@ -284,8 +300,18 @@ export function calibrationFrom(replayDoc, { confidence = 0.95, computedAt = new
     computedAt,
     method: 'wilson',
     confidence,
+    // What p is the probability of. Without it the document is unattributed
+    // and a consumer may quote nothing from it — see GATE-SEMANTICS.
+    measures: 'escape-missed',
     bucketBy: 'faultClass',
-    source: { kind: 'bug-replay', ref: replayDoc.run.range },
+    source: {
+      kind: 'bug-replay',
+      ref: replayDoc.run.range,
+      // The ground truth is this repository's own history, which `ref` names,
+      // so there is no external corpus to list. The caveat is the sentence a
+      // consumer shows beside any number it quotes from here.
+      caveat: 'measured on this repository\'s own fix history: every replayed bug is one that escaped, so a first run is expected to be high — the number to move is this one, over time',
+    },
     buckets,
   };
 }
@@ -303,8 +329,9 @@ export function renderReplay(doc) {
   lines.push(`${doc.records.length} fix commit${doc.records.length === 1 ? '' : 's'} replayed from ${doc.run.candidates} candidate${doc.run.candidates === 1 ? '' : 's'}` +
     (doc.run.duplicates ? ` (${doc.run.duplicates} dropped as the same patch)` : '') + ': ' +
     order.filter((v) => by[v]).map((v) => `${by[v]} ${v}`).join(', ') + '.');
-  const measurable = (by.caught ?? 0) + (by.blind ?? 0);
-  if (measurable) lines.push(`${by.blind ?? 0} of ${measurable} measurable bugs were invisible to the suite — the escaped-bug replay rate is ${(((by.blind ?? 0) / measurable) * 100).toFixed(0)}%.`);
+  const measurable = MEASURED_VERDICTS.reduce((s, v) => s + (by[v] ?? 0), 0);
+  const missed = measurable - (by.caught ?? 0);
+  if (measurable) lines.push(`${missed} of ${measurable} measurable bugs were invisible to the suite (${by.blind ?? 0} blind, ${by.nocover ?? 0} with no test at all) — the escaped-bug miss rate is ${((missed / measurable) * 100).toFixed(0)}%.`);
   lines.push('A bug that reached production is by construction one the suite missed, so a high rate is expected on a first run. The number to move is this one, over time.');
   return lines.join('\n');
 }