testguard-cli 0.6.0 → 0.8.0

package/spec/GATE-SEMANTICS.md

@@ -12,7 +12,7 @@ The verdict set is closed. Only one value is a pass.
 | `killed` | Fault applied; every one of N probe runs failed with a genuine assertion failure, on a defender set that was green N/N unmodified. | no |
 | `survived` | Fault applied; every one of N probe runs passed. The claim is **unproven**. | **yes** |
 | `nocover` | No defending test exists: the declared globs resolve to nothing, or no test imports the subject. Worse than `survived` — nothing was even tried. | **yes** |
-| `unverifiable` | The claim could not be probed: the fault's anchor is missing or ambiguous, or its defenders failed to load (`defenders-failed-to-load`). Carries a `reason`. A loud, gating verdict — a claim that cannot be probed is not "skipped", it is undefended until someone fixes the fault or the defenders. Never confused with `flaky-defender`, which requires tests that *ran*. | **yes** |
+| `unverifiable` | The claim could not be probed: the fault's anchor is missing or ambiguous, its defenders failed to load (`defenders-failed-to-load`), probing it threw (`probe-error`), or the defenders do not execute the subject at all (`subject-not-executed`). Carries a `reason`. A loud, gating verdict — a claim that cannot be probed is not "skipped", it is undefended until someone fixes the fault or the defenders. Never confused with `flaky-defender`, which requires tests that *ran*. | **yes** |
 | `timeout` | Probe run exceeded its budget. Not counted as a kill; the pessimistic reading is the safe one because flakiness biases the metric optimistically. | **yes** |
 | `fault-invalid` | The replacement does not load or compile. A bad fault, not a detection. | **yes** |
 | `flaky-defender` | Defenders were not green N/N on unmodified source (`defenders-not-green`), or the N probe runs disagreed with each other (`inconsistent-probe`). Either way no verdict about the fault can be trusted; fix the defenders first. | **yes** |
@@ -36,10 +36,16 @@ Rules that follow from the table:
    that failed and then passed on retry (Playwright's `flaky`) is a
    **non-green run** even when the runner exits 0, and a test the runner
    reports as timed out (`timedOut`) is a timeout, never an assertion
-   failure. When one claim's defenders run under several runners, their
-   runs merge pessimistically — any load error is an error, any timeout is
-   a timeout, any failure is a failure — and the evidence names which file
-   ran under which runner.
+   failure. Python is read the same way: an `AssertionError` and any other
+   exception raised by the test body both count, because both are the suite
+   rejecting the behaviour; a collection error — a module that will not
+   import, which is what a replacement that does not parse looks like from
+   Python — is a load failure and never a kill; and a test the suite expected
+   to fail that passed (`xpass`) makes the run non-green without ever being an
+   assertion failure, exactly as Playwright's `flaky` does. When one claim's
+   defenders run under several runners, their runs merge pessimistically —
+   any load error is an error, any timeout is a timeout, any failure is a
+   failure — and the evidence names which file ran under which runner.
 4. **A mixed result is not a kill.** If some of the N probe runs fail and
    others pass, the defender's response to the fault is nondeterministic.
    Reporting `killed` would be the optimistic bias flakiness introduces;
@@ -68,6 +74,115 @@ Rules that follow from the table:
    content hash; the status document lists every fault whose content
    changed since it was probed, with its previous verdict. The change is
    allowed; it is never invisible.
+   The same hash governs **reuse**: a tool that carries a prior verdict
+   forward to save a run may only do so for the fault that produced it. A
+   verdict is an answer about one exact `find`/`replace`, so once those
+   change the old answer is not an answer to the new question — a weakened
+   fault would keep the verdict it earned before, and a repaired anchor would
+   keep `unverifiable`. Where the prior record has no content hash to compare,
+   the fault is probed again: the cost of re-measuring is a run, the cost of
+   the other direction is a verdict nobody measured.
+
+9. **The fault must be the code that ran.** A green baseline proves the
+   harness is not reporting everything as broken. It proves nothing about the
+   opposite direction: a fault the interpreter never executes leaves the
+   baseline green, every verdict `survived`, and a report that reads as a
+   devastating finding while being entirely false. A tool that can tell which
+   file was actually loaded must check it while the fault is applied, and must
+   **refuse the run** when the subject was loaded from outside the tree being
+   probed — in Python an editable install (PEP 660) registers an import hook
+   consulted ahead of every path entry, and a plain install leaves a copy in
+   `site-packages`, so this is the normal case, not an exotic one. When no
+   defender imported the subject at all, the run continues: an import inside a
+   branch the fault does not reach is legitimate. It is recorded as
+   `detail.targetNotImported`, because a `survived` there is a statement about
+   the defenders' reach and not about their assertions, and the two must not
+   be read as the same finding.
+10. **A signal describes a file, and says which kind of blindness it is.**
+   Replacing a whole module (`vi.mock`, `jest.mock`, a Python `patch` of the
+   module itself) removes a file from the defenders: it cannot detect anything
+   in what it replaced. Replacing **one attribute** of a module —
+   `patch("pkg.mod.fn")`, which is what Python code almost always does — does
+   not: a fault anywhere else in that module is still fully detectable, and
+   dropping the file would report `nocover` for a claim that is well defended.
+   The file stays a defender and carries `target-attribute-patched`, naming
+   the attributes. Signals never change a verdict.
+
+11. **One fault's failure costs that fault, not the run.** A fault whose probe
+   throws is recorded as `unverifiable` with reason `probe-error` and a
+   `detail.message` naming what threw; the run continues and still writes its
+   evidence. Losing the document would throw away every verdict already
+   decided and leave a caller with an exit code this document does not define,
+   which is strictly worse than one loud unverifiable claim. A `probe-error`
+   record is never reused by a later run: it says something about the run, not
+   about the code. The one exception is a **precondition** failure — the
+   interpreter loading the source from outside the probed tree, a runner that
+   does not resolve — which is a statement about every verdict in the run and
+   still refuses it outright.
+   Restoring the source is held to the same reading. The restore is
+   belt-and-braces in worktree mode, where the mutation only ever existed
+   inside a scratch worktree that is discarded anyway, so a target that has
+   vanished is already restored and is recorded as
+   `detail.restoreSkipped: "target-missing"` rather than raised — a reader can
+   still see that the tree moved underneath the run. Under `--in-place` the
+   same condition is the user's own file gone and must be raised. Any other
+   failure to write the original back is raised in both modes: it can mean a
+   mutated file left on disk.
+
+12. **A survivor must prove the defenders execute the subject.** A green
+   baseline proves the defenders can *pass*. It does not prove they can fail
+   *because of this file* — and if the fault is applied to code the test
+   process never executes, the baseline is green, every probe run is green,
+   and every claim is reported `survived`. That output reads as a devastating
+   audit finding and is entirely false, and no individual check in the run
+   contradicts it.
+   So a would-be `survived` is charged one more run: the subject is replaced
+   with content its loader cannot parse, and the defenders run once. Going red
+   proves they execute it (`detail.negativeControl: "reached"`), and the
+   `survived` stands. Staying green proves they do not, and the verdict is
+   `unverifiable` with reason `subject-not-executed` — the pessimistic reading,
+   and the only honest one, because nothing was measured about their
+   assertions.
+   Charged **only** on a would-be survivor, and cached per subject and defender
+   set. A kill already proves the defenders reached the code, so paying there
+   would be waste. A runner that cannot say what "cannot compile" means for its
+   language does not guess: no control is run, no signal is recorded, and the
+   verdict is unchanged. `detail.targetNotImported` (reported by runners that
+   can name the file the interpreter loaded) answers the same question more
+   precisely for one language; where both are present they must agree.
+
+13. **Fault injection is one method, not the definition.** A claim declares
+   what must be true; a probe declares how a tool would try to make it false,
+   and `method` says which way. `fault-injection` — mutate the source, run the
+   claim's defenders, confirm over N runs — is the only method specified in
+   v1, and it is the default when the field is absent, so every document
+   written before the field existed is held to exactly the rules it was
+   written against.
+   Everything in this document above this rule is `fault-injection`'s
+   semantics. `confirmRuns`, `mode`, `defenders`, `inputs` and the baseline
+   and probe runs are what *it* means by showing its work; a tool that reads
+   code or scans a surface has none of them, and requiring them would force it
+   to emit empty arrays to conform — which is lying, and the one thing this
+   format exists to make impossible. The schema therefore requires them only
+   under `fault-injection`, and the validator enforces them there in full: the
+   relaxation is for a tool that injects nothing, never a licence for an
+   injecting tool to stop showing its work.
+   A reserved method carries its own data in `methodDetail`, on the probe and
+   on the record: an object the spec deliberately does not constrain, so a tool
+   can say which rule it checked and where. Without it, "reserved" would mean a
+   probe that can declare it exists and nothing about it, which is not a
+   reservation but a dead end. It is **forbidden under `fault-injection`**,
+   whose shape is specified and must not acquire a junk drawer, and it is a
+   staging area rather than a permanent home: what the first consumer puts
+   there is the evidence for what the specified shape should become.
+   `assertion` and `scan` are **reserved**. Their required fields are
+   deliberately unspecified and will be defined by their first real consumer,
+   with its own conformance examples written from the shape that tool actually
+   has. Designing them for an absent tool is precisely the mistake that made
+   this rule necessary: the spec was drafted against a single consumer and
+   hard-coded its assumptions as requirements.
+   The verdict set stays closed and shared. What each method may legitimately
+   report is part of defining it.
 
 ## Replay reports; it never gates
 
@@ -94,9 +209,96 @@ Two rules that follow:
 2. **The fix's own test is removed before the run.** It proves nothing about
    what the suite knew before the fix existed.
 
-Only `caught` and `blind` carry information. `nocover`, `flaky` and
-`unverifiable` are excluded from both sides of a calibration ratio, because a
-number computed over them would mean nothing.
+`caught`, `blind` and `nocover` are measurements and enter the calibration
+ratio: `caught` as a hit, the other two as misses. `nocover` counts as a miss
+because it is one — the worst kind. Leaving it out would let a project with
+**no tests at all** for a subsystem score better than one with weak tests,
+since its worst outcomes would leave the denominator before the ratio is
+taken; Stryker and PIT count no-coverage mutants against the headline score
+for the same reason. `flaky` and `unverifiable` are failed measurements and
+enter neither side.
+
+### Calibration arithmetic is reproducible
+
+A calibration cell declares `n` and `positives`, and with them has declared
+`p`; a document declares `method: wilson` at `confidence`, and with them has
+declared every `ci`. The validator recomputes both with `lib/wilson.mjs` —
+the single implementation, as `lib/fingerprint.mjs` is for fingerprints — and
+a cell that does not reproduce does not conform. Before this rule the
+validator checked only that `p` lay inside `ci`, which a document with every
+number invented satisfies, and the spec's own example shipped a truncated
+lower bound (6/30 → `0.09`, where Wilson gives 0.0950 → `0.10`) that nothing
+could notice.
+
+- **Precision is the document's own**: the most decimal places any `p` or
+  `ci` value shows, capped at 6. Values are compared after rounding to that
+  precision, so a producer at 3 dp and one at 4 dp both reproduce and a
+  truncated bound does not.
+- **z may be the exact quantile or the textbook rounding** (1.96, 2.576,
+  1.645). The two differ by up to 9.2e-6 in the interval — enough to flip a
+  rounding boundary at 4 dp — so either reproduces, and a document never fails
+  for having used the number in the textbook.
+- **`p` is `positives / n`.** A smoothed or shrunk point estimate is a
+  different method and must say so; under `wilson` the point estimate is the
+  observed proportion.
+
+### Calibration provenance, backoff and merging
+
+Everything beyond a document's primary buckets is optional and additive, in
+the way `method` is on claims and evidence: absence has a defined meaning, and
+no document written before a field existed becomes invalid. The meaning of
+absence is **unattributed** — readable, auditable, and not a source of
+numbers. A tool that emits calibrations always emits these fields; a
+self-claim guards that it does.
+
+- **`measures`** says what `p` is the probability of. It is the one field
+  that lets two calibrations be told apart: testguard's `escape-missed` is
+  P(a real escaped bug of this class was not caught), and high is bad;
+  websec-validator's `finding-real` is P(a reported finding of this class is a
+  real vulnerability), and high is good. A document without it cannot be
+  quoted and cannot be merged. Registry: `escape-missed`, `finding-real`. A
+  new value is added here by the tool that first emits it.
+- **`source`** carries provenance. `corpus` names external ground-truth
+  sources (a bug-replay's is the repository's own history, which `ref` already
+  names); `caveat` is the one sentence a consumer shows beside any quoted
+  number; `limitation` is the longer story; `evidenceStatus` says whether the
+  labels were reviewed; `kind: tool-oracle` is a tool confirming its own
+  findings, as websec-validator's dynamic probes do. `detail` is the tool's
+  own provenance, unconstrained, on the `methodDetail` pattern. A number
+  quoted without its `caveat` is a misquote.
+- **`minN`** is the producer's floor. A cell below it is not quoted on its
+  own; the consumer backs off a tier. A consumer may raise it — a gate that
+  blocks a merge wants more certainty than a report — and may never lower
+  it: lowering it resurrects a cell the producer suppressed. Absent, the
+  producer sets no floor and the consumer applies its own. A spec-level
+  constant would be wrong in both directions: websec-validator's 5 would
+  suppress every cell of a corpus the size of DocGuard's benchmark.
+- **`backoff`** is the ordered list of coarser tiers consulted below `minN`.
+  Each tier is held to the same arithmetic as the primary. Keys may be
+  compound (`attackClass|confidence` backs off to `confidence`), and every key
+  in a tier has as many `|`-parts as its `bucketBy` — a key with fewer was
+  translated from another table and lost a dimension on the way.
+- **`fallback`** is what remains when no tier has a cell above `minN`: a
+  labelled guess, not a measurement, and anything quoted from it carries
+  `n = 0`, `ci = [0, 1]` and its `basis`. It lives in the document so that
+  "what did it assume when it had no data?" has one auditable answer instead
+  of one per consumer, each unlabelled.
+- **`breakdown`** splits a cell's `n` by outcome when the producer can say
+  (testguard: `caught` / `blind` / `nocover`), and sums to `n`. It lets a
+  consumer recover a narrower rate — misses among covered code, say —
+  without the producer publishing a second, misleading headline.
+
+**Merging.** Two calibrations merge by summing `n` and `positives` per cell
+and recomputing `p` and `ci` from the sums — never by averaging `p` or `ci`,
+which have no meaning combined. They merge only when `measures`, `bucketBy`
+and `confidence` agree, and the merged document's `source.kind` is `mixed`.
+This is how websec-validator folds an operator's own confirmed samples over
+its shipped table, and it works only because cells carry raw counts.
+
+**The second producer is a conformance example.** websec-validator's shipped
+table, translated field for field, is `conformance/examples/calibration-websec.json`
+— corpus, caveat, limitation, floor, backoff tier and labelled fallback all
+present. That file, not a sentence in a README, is what "adoptable" means.
 
 ## Baseline and delta
 

package/spec/README.md

@@ -1,16 +1,25 @@
-# Guard spec — the contract spine
+# claimspec — the contract spine
 
-Eight formats that tools following the Guard pattern share. They share
-*formats*, not code: [`docguard-cli`](https://www.npmjs.com/package/docguard-cli)
-is Node ESM, [`websec-validator`](https://pypi.org/project/websec-validator/)
-is Python, and porting one runtime into the other is not worth it. A tool
-conforms by emitting these shapes; it keeps its own language, CLI and UX.
+Nine formats for declaring what must be true about a codebase, recording the
+evidence that tried to falsify it, and gating only what is new.
 
-The pattern all such tools run:
+They are shared as *formats*, not as code. The first tools to adopt them are
+[`docguard-cli`](https://www.npmjs.com/package/docguard-cli) (Node ESM) and
+[`websec-validator`](https://pypi.org/project/websec-validator/) (Python) —
+porting one runtime into the other was never worth it, and never the point. A
+tool conforms by emitting these shapes; it keeps its own language, CLI and UX.
+
+The method they have in common:
 
 > declare what must be true → try mechanically to falsify it → freeze a
 > baseline → gate only the delta → brief the agent before it writes code.
 
+**Falsifying by fault injection is one method, not the definition.** A tool
+that verifies a claim by reading code, by scanning a surface, or by replaying
+history is as much a consumer as one that mutates source. The formats that
+carry *how* a claim was tested say which method produced them; the formats
+that carry claims, baselines, scoping and briefs do not care.
+
 ## Formats
 
 | Kind | Schema | Purpose |
@@ -19,7 +28,7 @@ The pattern all such tools run:
 | `evidence` | [`schemas/evidence.schema.json`](schemas/evidence.schema.json) | One run's findings on disk, readable without the tool. Every verdict carries every run that produced it. |
 | `baseline` | [`schemas/baseline.schema.json`](schemas/baseline.schema.json) | Frozen fingerprints of existing debt. Gate only what is new. |
 | `ignore` | [`schemas/ignore.schema.json`](schemas/ignore.schema.json) | Reviewable scoping. Every entry has a reason. |
-| `calibration` | [`schemas/calibration.schema.json`](schemas/calibration.schema.json) | P(finding is real) per bucket, with a Wilson interval and the sample size behind it. |
+| `calibration` | [`schemas/calibration.schema.json`](schemas/calibration.schema.json) | The observed rate of a bucket's outcome — what it measures is in the document — with a Wilson interval and the sample size behind it. Every number is recomputable and the validator recomputes it; provenance, a producer floor, backoff tiers and a labelled fallback ride along so a second tool's honesty survives translation. |
 | `brief` | [`schemas/brief.schema.json`](schemas/brief.schema.json) | What to tell an agent before it writes code — ranked, capped, never a single score, and carrying the one next action. |
 | `status` | [`schemas/status.schema.json`](schemas/status.schema.json) | Where the project is and what happens next — the single machine-readable truth every human rendering derives from. Surfaces faults whose content changed since they were probed, and changed files that carry no claim. |
 | `replay` | [`schemas/replay.schema.json`](schemas/replay.schema.json) | Would this suite have caught the bugs that already escaped? Replays real fix commits: revert the source, remove the test the fix shipped, run what remains. A replayed bug is ground truth — a human already confirmed it was a defect — which is what a fault model is calibrated against. |
@@ -30,7 +39,7 @@ Shared definitions (verdicts, fault classes, provenance, annotations) live in
 which verdicts turn CI red, baseline and delta rules, exit codes — is in
 [`GATE-SEMANTICS.md`](GATE-SEMANTICS.md).
 
-Schemas are JSON Schema 2020-12 and identified as `urn:guard-spec:v1:<kind>`.
+Schemas are JSON Schema 2020-12 and identified as `urn:claimspec:v1:<kind>`.
 
 ## Conformance
 
@@ -53,5 +62,17 @@ npm run test:spec
 
 The spec is drafted here and validated against `testguard` as its first
 consumer. It moves to its own repository only when a second tool adopts it.
-Adoption by an existing tool is one additive output writer — no rewrite, no
-behaviour change, and optional until that tool wants it.
+
+Adoption is additive and optional: an existing tool gains an output writer, and
+nothing it already detects or already writes changes. That holds today for
+`baseline`, `ignore`, `brief`, `status` and `calibration` — websec-validator's
+shipped calibration table is `conformance/examples/calibration-websec.json`,
+translated field for field with its corpus, caveat, floor, backoff and
+fallback intact. It does **not** yet hold for
+`claims` and `evidence`, whose required fields assume fault injection
+(`find`/`replace` anchors, `confirmRuns`, `baselineRuns`/`probeRuns`,
+`defenders`) — a tool that scans or reads would have to emit empty arrays to
+satisfy the schema, which is conforming by lying. Making those two kinds
+method-agnostic is tracked as
+[#81](https://github.com/raccioly/testguard/issues/81) and blocks adoption of
+either.

package/spec/lib/validate.mjs

@@ -3,10 +3,18 @@ import { readFileSync, readdirSync } from 'node:fs';
 import { fileURLToPath } from 'node:url';
 import { dirname, join } from 'node:path';
 import { fingerprint } from './fingerprint.mjs';
+import { reproduces, decimals, MAX_PLACES } from './wilson.mjs';
 
 const schemaDir = join(dirname(fileURLToPath(import.meta.url)), '..', 'schemas');
 
 export const KINDS = Object.freeze(['claims', 'evidence', 'baseline', 'ignore', 'calibration', 'brief', 'status', 'gate', 'replay']);
+/**
+ * How a document says it tried to falsify its claims. Absent means
+ * `fault-injection`, so every document written before the field existed is
+ * held to exactly the rules it was written against.
+ */
+export const methodOf = (x) => x?.method ?? 'fault-injection';
+const isInjection = (x) => methodOf(x) === 'fault-injection';
 export const PASSING_VERDICTS = Object.freeze(new Set(['killed']));
 
 const ajv = new Ajv2020({ strict: true, allErrors: true });
@@ -15,7 +23,7 @@ for (const f of readdirSync(schemaDir).filter((n) => n.endsWith('.schema.json'))
 }
 
 const schemaFor = (kind) => {
-  const v = ajv.getSchema(`urn:guard-spec:v1:${kind}`);
+  const v = ajv.getSchema(`urn:claimspec:v1:${kind}`);
   if (!v) throw new RangeError(`unknown Guard-spec kind: ${kind}`);
   return v;
 };
@@ -33,10 +41,15 @@ const semantic = {
         const p = `/claims/${ci}/faults/${fi}`;
         if (seenFault.has(f.id)) errors.push({ path: `${p}/id`, message: `duplicate fault id "${f.id}" in claim "${c.id}"` });
         seenFault.add(f.id);
-        const occ = f.occurrence ?? 1;
-        const hits = f.expectHits ?? 1;
-        if (occ > hits) errors.push({ path: `${p}/occurrence`, message: `occurrence ${occ} exceeds expectHits ${hits}` });
-        if (f.find === f.replace) errors.push({ path: `${p}/replace`, message: 'replace is identical to find; the fault is a no-op' });
+        // Anchor arithmetic is a property of fault injection. A probe that does
+        // not substitute text has no `find` to compare, and `undefined ===
+        // undefined` would report every one of them as a no-op.
+        if (isInjection(f)) {
+          const occ = f.occurrence ?? 1;
+          const hits = f.expectHits ?? 1;
+          if (occ > hits) errors.push({ path: `${p}/occurrence`, message: `occurrence ${occ} exceeds expectHits ${hits}` });
+          if (f.find === f.replace) errors.push({ path: `${p}/replace`, message: 'replace is identical to find; the fault is a no-op' });
+        }
       });
     });
     return errors;
@@ -44,9 +57,15 @@ const semantic = {
 
   evidence(doc) {
     const errors = [];
+    const injection = isInjection(doc.run);
     const n = doc.run.confirmRuns;
-    if (n < 3 && doc.run.provisional !== true) errors.push({ path: '/run/provisional', message: `confirmRuns ${n} is below 3; the run must declare provisional: true` });
-    if (n >= 3 && doc.run.provisional === true) errors.push({ path: '/run/provisional', message: `confirmRuns ${n} is confirmed; provisional must be absent or false` });
+    // N-run agreement is what fault injection means by "confirmed". A method
+    // that does not re-run anything has no N, and a rule about one would be a
+    // rule about a number that is not there.
+    if (injection) {
+      if (n < 3 && doc.run.provisional !== true) errors.push({ path: '/run/provisional', message: `confirmRuns ${n} is below 3; the run must declare provisional: true` });
+      if (n >= 3 && doc.run.provisional === true) errors.push({ path: '/run/provisional', message: `confirmRuns ${n} is confirmed; provisional must be absent or false` });
+    }
     if (doc.run.runners && doc.run.runner && !doc.run.runners.some((x) => x.name === doc.run.runner.name)) errors.push({ path: '/run/runners', message: 'runners must include the project runner named in run.runner' });
     if (doc.run.repo.ignoredDirty && doc.run.repo.snapshot) errors.push({ path: '/run/repo/ignoredDirty', message: 'a working-tree snapshot has no ignored dirty files: the tree was probed as it is' });
     doc.records.forEach((r, i) => {
@@ -54,6 +73,26 @@ const semantic = {
       const expected = fingerprint({ claimId: r.claim.id, subjectId: r.subject.id, file: r.subject.file ?? '', verdict: r.verdict });
       if (r.fingerprint !== expected) errors.push({ path: `${p}/fingerprint`, message: `fingerprint does not match spec derivation (expected ${expected})` });
 
+      // What the schema no longer requires, the validator still does — for the
+      // one method that means it. Fault injection without defenders, inputs or
+      // runs is not a leaner document, it is a verdict with nothing behind it;
+      // relaxing the schema was to let a scanner conform, never to let an
+      // injecting tool stop showing its work.
+      if (injection && r.detail.methodDetail) {
+        errors.push({ path: `${p}/detail/methodDetail`, message: 'methodDetail is for a method the spec has not specified; fault-injection has a specified shape and must not acquire a junk drawer' });
+      }
+      if (injection) {
+        for (const k of ['defenders', 'inputs']) {
+          if (!r[k]) errors.push({ path: `${p}/${k}`, message: `fault-injection requires ${k} on every record` });
+        }
+        for (const k of ['baselineRuns', 'probeRuns']) {
+          if (!Array.isArray(r.detail[k])) errors.push({ path: `${p}/detail/${k}`, message: `fault-injection requires detail.${k} on every record` });
+        }
+      }
+      // Every rule below reads those fields, so a record that is already
+      // malformed must not also throw on the way to being reported.
+      if (!injection || !r.defenders || !r.inputs || !Array.isArray(r.detail.baselineRuns) || !Array.isArray(r.detail.probeRuns)) return;
+
       const { baselineRuns, probeRuns } = r.detail;
       if (r.verdict === 'killed' || r.verdict === 'survived') {
         if (baselineRuns.length !== n) errors.push({ path: `${p}/detail/baselineRuns`, message: `${r.verdict} requires exactly confirmRuns (${n}) baseline runs, got ${baselineRuns.length}` });
@@ -99,23 +138,95 @@ const semantic = {
       }
       for (const s of r.defenders.signals ?? []) {
         if (s.signal === 'unasserted-annotated' && !s.reason) errors.push({ path: `${p}/defenders/signals`, message: `unasserted-annotated on ${s.file} requires the annotation's reason` });
-        if (!(r.defenders.mocking ?? []).includes(s.file)) errors.push({ path: `${p}/defenders/signals`, message: `${s.file} carries a mock signal but is not listed in defenders.mocking` });
+        // An attribute patch (Python) leaves the file a defender, so it belongs
+        // to `resolved` and not to `mocking`; every other signal is about a file
+        // that replaced the module and must be listed as mocking it.
+        if (s.signal === 'target-attribute-patched') {
+          if (!s.reason) errors.push({ path: `${p}/defenders/signals`, message: `target-attribute-patched on ${s.file} requires the patched attributes as its reason` });
+          if (!r.defenders.resolved.includes(s.file)) errors.push({ path: `${p}/defenders/signals`, message: `${s.file} patches attributes of the subject but is not a resolved defender` });
+        } else if (!(r.defenders.mocking ?? []).includes(s.file)) {
+          errors.push({ path: `${p}/defenders/signals`, message: `${s.file} carries a mock signal but is not listed in defenders.mocking` });
+        }
+      }
+      // A module the defenders never imported cannot have been killed by them.
+      if (r.detail.targetNotImported && r.verdict === 'killed') {
+        errors.push({ path: `${p}/detail/targetNotImported`, message: 'the subject was never imported, so the defenders cannot have killed the fault' });
+      }
+      // The control is only ever charged on a would-be survivor, so a `reached`
+      // record must be one, and `not-reached` must have become unverifiable.
+      if (r.detail.negativeControl === 'reached' && r.verdict !== 'survived') {
+        errors.push({ path: `${p}/detail/negativeControl`, message: `negativeControl "reached" belongs to a survivor, not to ${r.verdict}` });
+      }
+      if (r.detail.negativeControl === 'not-reached' && !(r.verdict === 'unverifiable' && r.detail.reason === 'subject-not-executed')) {
+        errors.push({ path: `${p}/detail/negativeControl`, message: 'negativeControl "not-reached" requires verdict unverifiable with reason subject-not-executed' });
+      }
+      if (r.detail.reason === 'subject-not-executed' && r.detail.negativeControl !== 'not-reached') {
+        errors.push({ path: `${p}/detail/reason`, message: 'subject-not-executed requires the negativeControl that established it' });
+      }
+      // Python's import provenance and the general control answer the same
+      // question. A document where they disagree describes a run that cannot
+      // have happened.
+      if (r.detail.targetNotImported && r.detail.negativeControl === 'reached') {
+        errors.push({ path: `${p}/detail/negativeControl`, message: 'the subject was never imported, so the negative control cannot have reached it' });
       }
       if (r.verdict === 'unverifiable' && !r.detail.reason) {
         errors.push({ path: `${p}/detail/reason`, message: 'unverifiable requires a reason (e.g. anchor-missing, anchor-ambiguous)' });
       }
+      // `probe-error` is the one reason that names nothing on its own: it says
+      // the run threw, not what threw. Without the message the record is a
+      // dead end for anyone reading the document instead of rerunning it.
+      if (r.detail.reason === 'probe-error' && !r.detail.message) {
+        errors.push({ path: `${p}/detail/message`, message: 'probe-error requires a message naming what threw' });
+      }
+      // In-place is the mode where a vanished target is the user's own file
+      // gone; it raises rather than records. A document claiming otherwise
+      // describes a run that cannot have happened.
+      if (r.detail.restoreSkipped && doc.run.mode === 'in-place') {
+        errors.push({ path: `${p}/detail/restoreSkipped`, message: 'restoreSkipped cannot occur in in-place mode, where a missing target is raised' });
+      }
     });
     return errors;
   },
 
   calibration(doc) {
     const errors = [];
-    for (const [key, b] of Object.entries(doc.buckets)) {
-      const p = `/buckets/${key}`;
-      if (b.positives > b.n) errors.push({ path: `${p}/positives`, message: `positives (${b.positives}) exceed n (${b.n})` });
-      const [lo, hi] = b.ci;
-      if (lo > hi) errors.push({ path: `${p}/ci`, message: `ci lower bound ${lo} exceeds upper bound ${hi}` });
-      if (b.p < lo || b.p > hi) errors.push({ path: `${p}/p`, message: `p (${b.p}) lies outside ci [${lo}, ${hi}]` });
+    // The primary buckets and every backoff tier are held to the same rules:
+    // a coarser tier a consumer falls back to is exactly where a wrong number
+    // would go unnoticed.
+    const tiers = [{ bucketBy: doc.bucketBy, buckets: doc.buckets, path: '' }, ...(doc.backoff ?? []).map((t, i) => ({ ...t, path: `/backoff/${i}` }))];
+    // The precision the document wrote at: the most places any value shows.
+    // Parsing drops trailing zeros (0.10 → 0.1), so a single value can
+    // under-report; the maximum cannot unless every value ended in zero.
+    const places = tiers.flatMap((t) => Object.values(t.buckets)).reduce((m, b) => Math.max(m, decimals(b.p ?? 0), decimals(b.ci[0]), decimals(b.ci[1])), 0);
+    for (const t of tiers) {
+      // A compound bucketBy (`attackClass|confidence`) means compound keys,
+      // each with the same number of parts — a key with fewer was translated
+      // from another tool's table and lost a dimension on the way.
+      const arity = t.bucketBy.split('|').length;
+      for (const [key, b] of Object.entries(t.buckets)) {
+        const p = `${t.path}/buckets/${key}`;
+        const parts = key.split('|').length;
+        if (parts !== arity) errors.push({ path: p, message: `bucket key "${key}" has ${parts} part(s); bucketBy "${t.bucketBy}" has ${arity}` });
+        if (b.positives > b.n) errors.push({ path: `${p}/positives`, message: `positives (${b.positives}) exceed n (${b.n})` });
+        const [lo, hi] = b.ci;
+        if (lo > hi) errors.push({ path: `${p}/ci`, message: `ci lower bound ${lo} exceeds upper bound ${hi}` });
+        if (b.p !== null && (b.p < lo || b.p > hi)) errors.push({ path: `${p}/p`, message: `p (${b.p}) lies outside ci [${lo}, ${hi}]` });
+        if (b.breakdown) {
+          const sum = Object.values(b.breakdown).reduce((s, v) => s + v, 0);
+          if (sum !== b.n) errors.push({ path: `${p}/breakdown`, message: `breakdown sums to ${sum}; n is ${b.n}` });
+        }
+        // A cell that declares n and positives has declared p; a document that
+        // declares `wilson` at `confidence` has declared every ci. Recompute
+        // both with the spec's own implementation: "conforms" means
+        // "reproducible", not "internally plausible". Before this the validator
+        // would pass a document with every number invented, and the spec's own
+        // example carried a truncated bound nothing could notice.
+        // A cell with more positives than trials has no proportion to reproduce.
+        if (b.positives > b.n) continue;
+        const r = reproduces(b, { confidence: doc.confidence, places });
+        if (!r.p) errors.push({ path: `${p}/p`, message: `p (${b.p}) is not positives/n: ${b.positives}/${b.n} = ${r.expected.p} at ${Math.min(places, MAX_PLACES)} dp` });
+        if (!r.ci) errors.push({ path: `${p}/ci`, message: `ci [${lo}, ${hi}] does not reproduce: ${doc.method} at ${doc.confidence} for ${b.positives}/${b.n} is [${r.expected.ci[0]}, ${r.expected.ci[1]}]` });
+      }
     }
     return errors;
   },

package/spec/lib/wilson.mjs

@@ -0,0 +1,112 @@
+/**
+ * The single Wilson interval implementation, as `fingerprint.mjs` is the
+ * single fingerprint: a producer emits with it, the validator recomputes with
+ * it, and "conforms" means the two agree. Before this file existed the emitter
+ * and the validator had no shared arithmetic at all — the validator checked
+ * that `p` sat inside `ci` and nothing else, and the spec's own example
+ * carried a truncated lower bound (6/30 → 0.09, where Wilson gives 0.0950 →
+ * 0.10) that nothing could notice.
+ */
+
+// Inverse normal CDF: Acklam's rational approximation, |relative error| < 1.2e-9.
+// Enough to reproduce an interval to 6 decimal places, which is the most any
+// producer writes (websec-validator 3, testguard 4, docguard 6).
+const A = [-3.969683028665376e+01, 2.209460984245205e+02, -2.759285104469687e+02, 1.383577518672690e+02, -3.066479806614716e+01, 2.506628277459239e+00];
+const B = [-5.447609879822406e+01, 1.615858368580409e+02, -1.556989798598866e+02, 6.680131188771972e+01, -1.328068155288572e+01];
+const C = [-7.784894002430293e-03, -3.223964580411365e-01, -2.400758277161838e+00, -2.549732539343734e+00, 4.374664141464968e+00, 2.938163982698783e+00];
+const D = [7.784695709041462e-03, 3.224671290700398e-01, 2.445134137142996e+00, 3.754408661907416e+00];
+const P_LOW = 0.02425;
+
+/** The standard normal quantile: z such that Φ(z) = p. */
+export function probit(p) {
+  if (!(p > 0 && p < 1)) throw new RangeError(`probit is defined on (0, 1), got ${p}`);
+  let q;
+  if (p < P_LOW) {
+    q = Math.sqrt(-2 * Math.log(p));
+    return (((((C[0] * q + C[1]) * q + C[2]) * q + C[3]) * q + C[4]) * q + C[5]) / ((((D[0] * q + D[1]) * q + D[2]) * q + D[3]) * q + 1);
+  }
+  if (p <= 1 - P_LOW) {
+    q = p - 0.5;
+    const r = q * q;
+    return (((((A[0] * r + A[1]) * r + A[2]) * r + A[3]) * r + A[4]) * r + A[5]) * q / (((((B[0] * r + B[1]) * r + B[2]) * r + B[3]) * r + B[4]) * r + 1);
+  }
+  q = Math.sqrt(-2 * Math.log(1 - p));
+  return -(((((C[0] * q + C[1]) * q + C[2]) * q + C[3]) * q + C[4]) * q + C[5]) / ((((D[0] * q + D[1]) * q + D[2]) * q + D[3]) * q + 1);
+}
+
+/**
+ * The textbook roundings of z that producers actually type. testguard and
+ * docguard use 1.96; websec-validator uses the exact quantile. Across n ≤ 200
+ * the two differ by at most 9.2e-6 in the interval, which can still flip a
+ * rounding boundary at 4 dp — so the validator accepts either, and a document
+ * never fails for having used the number in the textbook.
+ */
+export const CONVENTIONAL_Z = Object.freeze({ 0.9: 1.645, 0.95: 1.96, 0.99: 2.576 });
+
+/** Every z a producer may legitimately have used for a two-sided interval at `confidence`. */
+export function zCandidates(confidence) {
+  const exact = probit(1 - (1 - confidence) / 2);
+  const conventional = CONVENTIONAL_Z[confidence];
+  return conventional === undefined ? [exact] : [exact, conventional];
+}
+
+/**
+ * Wilson score interval for `positives` in `n`, unrounded, clamped to [0, 1].
+ * Wilson rather than the normal approximation because it stays sane at small
+ * n and extreme p — exactly the regime a young corpus is in. n = 0 is maximal
+ * ignorance: [0, 1].
+ */
+export function wilsonInterval(positives, n, z) {
+  if (n === 0) return [0, 1];
+  const p = positives / n;
+  const d = 1 + (z * z) / n;
+  const centre = (p + (z * z) / (2 * n)) / d;
+  const half = (z * Math.sqrt((p * (1 - p)) / n + (z * z) / (4 * n * n))) / d;
+  return [Math.max(0, centre - half), Math.min(1, centre + half)];
+}
+
+/** Round half away from zero to `places` decimals — what `toFixed` does on the value's binary form. */
+export const roundTo = (x, places) => Number(x.toFixed(places));
+
+/**
+ * How many decimal places a parsed JSON number still shows. Parsing drops
+ * trailing zeros (0.10 → 0.1), so a single value under-reports; the validator
+ * takes the maximum across a document, which is the precision the producer
+ * wrote at as long as at least one value did not end in zero.
+ */
+export function decimals(x) {
+  if (!Number.isFinite(x)) return 0;
+  const s = String(x);
+  const e = s.indexOf('e-');
+  if (e !== -1) {
+    const mantissa = s.slice(0, e);
+    const dot = mantissa.indexOf('.');
+    return Number(s.slice(e + 2)) + (dot === -1 ? 0 : mantissa.length - dot - 1);
+  }
+  const dot = s.indexOf('.');
+  return dot === -1 ? 0 : s.length - dot - 1;
+}
+
+/** The most decimal places the validator will hold a document to; beyond it the quantile approximation is the noise. */
+export const MAX_PLACES = 6;
+
+const near = (a, b) => Math.abs(a - b) < 1e-9;
+
+/**
+ * Does a cell's declared `p` and `ci` reproduce from its own `n` and
+ * `positives`? Everything is compared after rounding to `places` — the
+ * precision the document wrote at — so a producer at 3 dp and one at 4 dp
+ * both reproduce, and a truncated bound does not. A `p` of `null` is a cell
+ * that declines to give a point estimate; only its interval is checked.
+ */
+export function reproduces(cell, { confidence, places }) {
+  const { n, positives, p, ci } = cell;
+  const pl = Math.min(places, MAX_PLACES);
+  const expectedP = n === 0 ? null : roundTo(positives / n, pl);
+  const candidates = zCandidates(confidence).map((z) => wilsonInterval(positives, n, z).map((v) => roundTo(v, pl)));
+  return {
+    p: p === null || n === 0 || near(expectedP, p),
+    ci: candidates.some(([lo, hi]) => near(lo, ci[0]) && near(hi, ci[1])),
+    expected: { p: expectedP, ci: candidates[0] },
+  };
+}