driftproof 0.6.0 → 0.7.1

package/README.md

@@ -1,7 +1,7 @@
 <!-- SPDX-License-Identifier: Apache-2.0 -->
 # Driftproof
 
-**Continuous, model-version-bound verification of agent skills.**
+**A dated proof that this skill, this hash, this model, still helps.**
 
 [![driftproof](https://img.shields.io/endpoint?url=https://driftproofhq.com/badges/commit-message-conventions.json)](https://driftproofhq.com)
 &nbsp;— live badge for the bundled `commit-message-conventions` example, generated from its own receipt.
@@ -15,11 +15,11 @@ across model releases and you get a **drift report**.
 Driftproof **consumes** the [`agentskills.io/evals`](https://agentskills.io) eval
 format; it does not invent its own.
 
-📊 **Six published reports** (each re-derived from committed receipts, nothing
-hand-entered), spanning five published report types, the fifth being revision
-drift. All six share one band-based, floor-gated verdict rule and differ in what
-moves underneath the skill — or, in the value report, in which axes are
-measured:
+📊 **Seven published reports** (each re-derived from committed receipts, nothing
+hand-entered), spanning six published report types, the newest being instrument
+re-measurement. All seven share one band-based, floor-gated verdict rule and
+differ in what moves underneath the skill — or, in the value report, in which
+axes are measured; or, in the newest, in the instrument itself:
 
 - **[Report #001](https://driftproofhq.com/reports/001/)** — *release drift*: ten
   public agent skills across a current-vs-previous Sonnet release; 9 of 10 moved
@@ -51,11 +51,23 @@ measured:
   established — and the receipt spec gains generation sampling as a result. **No
   cause is asserted**; the control proves non-reproduction and cannot say why.
   *The tally a refusal carries: 3 cells, 0 measured, 3 refused.*
+- **[Report #007](https://driftproofhq.com/reports/007/)** — *instrument
+  re-measurement*: the three cells Report #005 published for these skills, run
+  again with the generation sampled adaptively instead of once and with the call
+  timeout the surface policy declares. **No cell separates**: every lift is
+  smaller than its own band, and the 21 cases read 3 improved, 0 regressed,
+  18 no effect, 0 not measured. Five of six comparisons against the archive were
+  **refused** on their baseline control. The instrument defect it reports is its
+  own: a declared 300 s timeout had been shadowed by a `120000` literal since
+  2026-07-27, and the truncated run it caused measured *less* variance than the
+  clean re-run, which is the direction that flatters an instrument. Both runs are
+  published, the broken one as evidence. Amends #005 to v1.2 and #006 to v1.1.
 
 ✍️ The launch essay, **[Three model releases later: what actually happens to agent
-skills](https://driftproofhq.com/writing/three-releases/)**, reads all six reports
-together: what moves underneath a skill, and what the skill costs to run. Revised
-2026-08-29; every figure in it is gate-checked against the report page it cites.
+skills](https://driftproofhq.com/writing/three-releases/)**, reads all seven reports
+together: what moves underneath a skill, what the skill costs to run, and what a
+corrected instrument did to three published results. Revised 2026-09-01; every
+figure in it is gate-checked against the report page it cites.
 
 ## Why
 
@@ -67,7 +79,7 @@ re-checks.
 
 **Verdicts age because the substrate moves.** Driftproof exists to keep the
 verdict current: cheap, repeatable, hash-stamped measurements bound to a specific
-model version, so "does this skill still help?" has a dated, reproducible answer
+model version, so "does this skill still help?" has a dated, verifiable answer
 instead of a stale one.
 
 The hard part isn't running an eval once — it's making the number **credible
@@ -100,8 +112,10 @@ npx driftproof init my-skill
 export CLAUDE_PROVIDER=api
 read -rsp "Anthropic API key: " ANTHROPIC_API_KEY && export ANTHROPIC_API_KEY
 
-# 4. Run the suite with a hard $2 budget (the run is refused up front if it would exceed it)
-npx driftproof run my-skill --models claude-haiku-4-5 --max-usd 2
+# 4. Run the suite. The shipped defaults (DEV_MAX_USD / DEV_MAX_CALLS in config.js)
+#    refuse the run up front if the projection exceeds either; --max-usd and
+#    --max-calls override them.
+npx driftproof run my-skill --models claude-haiku-4-5
 
 # 5. Read the receipt + human summary written to ./receipts/
 cat receipts/*.summary.md
@@ -187,11 +201,16 @@ The surface and judge settings used are recorded in every receipt.
 
 ### Cost guard
 
-Sampling multiplies calls. Each case costs `2 + 2 × samples` model calls
-(two generations, each judged `samples` times) — 12 calls per case at the default
-`--samples 5`. Driftproof **projects the whole run up front, prints the count, and
-refuses before spending anything** if it would exceed `--max-calls` (default 200).
-The default model list is `haiku` only.
+Sampling multiplies calls on two axes, and the second one is easy to miss. Each
+case costs `SAMPLING.max × (2 + 2 × samples)` model calls: two arms, each **drawn
+up to `SAMPLING.max` times** (generation sampling, receipt spec v0.5), and every
+draw judged `samples` times. Read at a single draw, that formula understates a run
+by the whole draw factor — which is how the shipped caps came to sit a release
+behind the estimator. Driftproof **projects the whole run up front, prints the
+count, and refuses before spending anything** if the projection exceeds the
+per-model call cap (`DEV_MAX_CALLS`) or the dollar budget (`DEV_MAX_USD`) — both
+declared with their derivation in [`config.js`](config.js), both overridable with
+`--max-calls` / `--max-usd`. The default model list is `haiku` only.
 
 ## Receipt anatomy
 
@@ -201,7 +220,7 @@ A receipt is the unit of evidence — one JSON document conforming to
 
 ```jsonc
 {
-  "schema_version": "0.4",
+  "schema_version": "0.5",
   "skill":  { "name": "commit-message-conventions", "version": "0.2.0",
               "content_hash": "…sha256 over SKILL.md + bundled files…" },
   "suite":  { "format": "agentskills.io/evals", "suite_hash": "…", "case_count": 10 },
@@ -210,7 +229,7 @@ A receipt is the unit of evidence — one JSON document conforming to
     "model_release_date": "2025-10-01",
     "provider": "anthropic",
     "surface": "claude-cli",
-    "runner_version": "0.6.0",
+    "runner_version": "0.7.1",
     "date_utc": "2026-07-27T…Z",
     "registry": "registered",
     "transcripts": "hashes-only",
@@ -270,7 +289,7 @@ receipts. The rule that keeps it honest: a **regression** (or improvement) is
 claimed **only when the two bands do not overlap**. Overlapping bands are reported
 as **within noise** and never counted as a regression.
 
-## Continuous verification in CI (GitHub Action + badge)
+## Verification in CI (GitHub Action + badge)
 
 Wire drift detection into a repo so the skill is re-checked on every push and when
 the model underneath it changes.
@@ -284,12 +303,13 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
-      - uses: driftproofhq/driftproof@v0.6.0
+      - uses: driftproofhq/driftproof@v0.7.1
         with:
           skill-dir: skills/my-skill
           models: claude-haiku-4-5
-          max-usd: '2'
           api-key: ${{ secrets.ANTHROPIC_API_KEY }}
+          # max-usd:   <n>    # override the dollar budget (default: DEV_MAX_USD in config.js)
+          # max-calls: <n>    # override the per-model call cap (default: DEV_MAX_CALLS)
           # fail-on-regression: 'true'   # (default) fail the job if the skill REGRESSED
 ```
 
@@ -307,7 +327,7 @@ runs).
 JSON object. Commit it somewhere public and point a shields endpoint URL at it:
 
 ```bash
-npx driftproof run skills/my-skill --models claude-haiku-4-5 --max-usd 2 --out receipts
+npx driftproof run skills/my-skill --models claude-haiku-4-5 --out receipts
 npx driftproof badge receipts/*.json --out badges/my-skill.json
 git add badges/my-skill.json && git commit -m "chore: driftproof badge"
 ```
@@ -323,7 +343,7 @@ site, so it reflects a real dated run, not a hand-set color.
 
 ## Reports
 
-Six reports are published, spanning five report types. A report page lives at a
+Seven reports are published, spanning six report types. A report page lives at a
 draft path — `docs/reports/NNN-draft/` — until the publish sequence renames it, and
 `scripts/build-public.sh` excludes every `*-draft/` path from the published tree
 (see the roll at the top of this README, and

package/bin/driftproof

@@ -4,9 +4,10 @@
 
 const fs = require('fs');
 const path = require('path');
-const { PROJECT_NAME, RUNNER_VERSION, DEFAULT_JUDGE_SAMPLES, DEV_MAX_USD } = require('../config');
+const { PROJECT_NAME, RUNNER_VERSION, DEFAULT_JUDGE_SAMPLES, DEV_MAX_USD, DEV_MAX_CALLS } = require('../config');
 const { loadSkill } = require('../lib/skill');
 const { runSkillOnModel, summarizeReceipt, projectCalls } = require('../lib/run');
+const { SAMPLING } = require('../lib/sampling');
 const { validateReceipt, verifyReceiptHash } = require('../lib/receipt');
 const { buildDriftReport, revisionPairProblem } = require('../lib/diff');
 const { surfaceForModel, isSubscriptionSurface, resolveModel } = require('../lib/provider');
@@ -91,8 +92,12 @@ ENV
 NOTES
   - Default models list is 'haiku' only (cheap dev default).
   - Sampled judging: each case does 2 generations + 2×samples judge calls.
-    Default --samples 5 → 12 calls/case; --samples 1 disables bands.
-  - --max-calls is a hard per-run call cap (default 200). The whole run is
+    Default --samples 5 → 12 calls/case; --samples 1 disables bands AND the
+    variance ratio: with one judge sample the per-draw judge spread is zero by
+    construction, so the receipt records variance_ratio: null with
+    variance_ratio_unavailable: "single_judge_sample". Legal, but a run whose
+    ratio you intend to publish needs --samples 2 or more.
+  - --max-calls is a hard per-run call cap (default ${DEV_MAX_CALLS}). The whole run is
     projected up front and REFUSED before any call if it would exceed the cap.
   - --max-usd is a hard dollar budget (default ${DEV_MAX_USD} for dev runs). The projected
     cost is printed up front and the run is REFUSED on BOTH surfaces if it would
@@ -126,7 +131,7 @@ async function cmdRun(positional, flags) {
   const rc = loadRc(skillDir);
   const models = String(flags.models || rc.models || 'haiku').split(',').map((s) => s.trim()).filter(Boolean);
   const maxCases = flags['max-cases'] ? parseInt(flags['max-cases'], 10) : (rc.max_cases != null ? parseInt(rc.max_cases, 10) : null);
-  const maxCalls = flags['max-calls'] ? parseInt(flags['max-calls'], 10) : (rc.max_calls != null ? parseInt(rc.max_calls, 10) : 200);
+  const maxCalls = flags['max-calls'] ? parseInt(flags['max-calls'], 10) : (rc.max_calls != null ? parseInt(rc.max_calls, 10) : DEV_MAX_CALLS);
   const samples = flags.samples ? parseInt(flags.samples, 10) : (rc.samples != null ? parseInt(rc.samples, 10) : DEFAULT_JUDGE_SAMPLES);
   const judgeModel = flags['judge-model'] || rc.judge_model || null;
   const concurrency = flags.concurrency ? parseInt(flags.concurrency, 10) : (rc.concurrency != null ? parseInt(rc.concurrency, 10) : 1);
@@ -137,14 +142,20 @@ async function cmdRun(positional, flags) {
 
   const skill = loadSkill(skillDir);
   const nCases = maxCases ? Math.min(maxCases, skill.suite.caseCount) : skill.suite.caseCount;
-  const perModelCalls = projectCalls(nCases, samples);
+  // v0.5 draws the generation up to SAMPLING.max times per arm, so BOTH the
+  // printed projection and the dollar guard below must be scaled by it. Left at
+  // draws=1 the guard would admit a run costing up to ten times its own
+  // projection — the display would say one number and the runner would refuse
+  // quoting another. Conservative on purpose: refusing a run that would have fit
+  // is recoverable, overspending is not.
+  const perModelCalls = projectCalls(nCases, samples, SAMPLING.max);
   const totalProjected = perModelCalls * models.length;
 
   // Dollar cost guard (Week 3): project the METERED USD cost up front. On the
   // `api` surface this is real spend, so we refuse if it would exceed --max-usd.
   // On `claude-cli` the metered spend is $0 (subscription); the figure is the
   // hypothetical "if run on the metered API" cost — printed, never blocks.
-  const cost = estimateRunCostUSD({ caseCount: nCases, samples, models: models.map((m) => require('../lib/provider').resolveModel(m)), judgeModel: judgeModel || 'haiku' });
+  const cost = estimateRunCostUSD({ caseCount: nCases, draws: SAMPLING.max, samples, models: models.map((m) => require('../lib/provider').resolveModel(m)), judgeModel: judgeModel || 'haiku' });
   // Surface is per-model now (a run may mix an Anthropic and an OpenAI target).
   const surfaces = [...new Set(models.map((m) => surfaceForModel(m)))];
   const surface = surfaces.join(', ');
@@ -315,7 +326,7 @@ function cmdInit(positional) {
   console.log(`\nNext:
   1. Edit ${rel(path.join(dir, 'SKILL.md'))} with your skill's instructions.
   2. Edit ${rel(path.join(dir, 'evals', 'evals.json'))} — replace the 3 example cases (rubrics anchored at 0.80).
-  3. driftproof run ${rel(dir)} --models claude-haiku-4-5 --max-usd 2
+  3. driftproof run ${rel(dir)} --models claude-haiku-4-5
 See AUTHORING.md for how to write a fair suite.`);
 }
 

package/config.js

@@ -9,7 +9,7 @@ const PROJECT_NAME = 'driftproof';
 // Bumped whenever the runner's behaviour or receipt-generation semantics change
 // in a way that could affect results. Recorded into every receipt as
 // run.runner_version so a receipt is reproducible against a known engine.
-const RUNNER_VERSION = '0.6.0';
+const RUNNER_VERSION = '0.7.1';
 
 // The eval format we CONSUME (we deliberately do not invent our own).
 const SUITE_FORMAT = 'agentskills.io/evals';
@@ -29,7 +29,7 @@ const SUITE_FORMAT = 'agentskills.io/evals';
 //        at run time so derived dollars stay reproducible), and the derived
 //        `economics` block. v0.3.1 is frozen as receipt.v0.3.1.schema.json;
 //        v0.1/v0.2/v0.3/v0.3.1 receipts all still load.
-const RECEIPT_SCHEMA_VERSION = '0.4';
+const RECEIPT_SCHEMA_VERSION = '0.5';
 
 // Hard USD budget defaults per entry point (Week 4). --max-usd overrides any of
 // these. The projection is refused before any call if it exceeds the cap, on
@@ -37,8 +37,33 @@ const RECEIPT_SCHEMA_VERSION = '0.4';
 //   dev     — an interactive `driftproof run`
 //   report  — a full Report-#001-style suite run (scripts/run-report-001.js)
 //   trigger — a release-trigger-initiated prepare-report run
-const DEV_MAX_USD = 2;
-const REPORT_MAX_USD = 40;
+// Recalibrated 2026-09-01 (spec 018). BOTH literals below were set against a
+// draws=1 projection and were never rescaled when spec 014 (5e08ba2) made the
+// runner draw the generation up to SAMPLING.max times per arm and spec 016 made
+// estimateRunCostUSD require the draw factor. The projection grew 10x; these did
+// not, so `npx driftproof run` refused every suite of two or more cases and the
+// v0.7.0 Action self-test aborted with exit 3. REPORT_MAX_USD was raised 40->300
+// on 2026-08-31 for exactly this reason; that pass missed the dev path, which is
+// the one the Action and every npx user take.
+//
+// DERIVATION — the headroom the pre-sampling defaults carried is preserved, not
+// widened. Bundled 10-case example, 5 judge samples, per-case factor 2+2*5 = 12:
+//   draws=1  ->  120 calls, $0.3525   headroom 200/120 = 1.67x,  $2/$0.3525 = 5.67x
+//   draws=10 -> 1200 calls, $3.5250   1200 * 1.67 = 2000,  $3.5250 * 5.67 = $20.00
+// A 2000-call cap under draws=10 is exactly as tight as 200 was under draws=1.
+// LITERALS ON PURPOSE (spec 018 AC-2): a default derived from the suite in hand
+// can never fire, which retires the guard instead of recalibrating it.
+const DEV_MAX_USD = 20;
+const DEV_MAX_CALLS = 2000;
+// Raised from $40 on 2026-08-31 by the repository owner, in daylight, recorded in
+// DECISIONS. The run did not get more expensive — the projection got honest:
+// spec 016 made estimateRunCostUSD require a draw factor, and v0.5 has drawn the
+// generation up to SAMPLING.max times per arm since spec 014, so the old cap had
+// been passing on a figure up to ten times too small. Set above the ceiling this
+// instrument can currently reach (#007's measured basis at the ceiling is
+// $268.83) rather than just above today's staging figure, because a cap tripped
+// by the next honest run teaches everyone to nudge it.
+const REPORT_MAX_USD = 300;
 const TRIGGER_MAX_USD = 25;
 
 // Default number of judge samples per case in sampled mode. Sampling is what
@@ -56,6 +81,24 @@ const DEFAULT_JUDGE_SAMPLES = 5;
 // spec/RECEIPT.md § "Drift verdict rule" and the report methodology.
 const EFFECT_FLOOR = 0.05;
 
+// ── generation sampling (receipt spec v0.5) ─────────────────────────────────
+//
+// Report #006 measured across-draw spread at sd 0.186 and 0.183 while the
+// instrument sampled only the judge. These are the policy that samples the
+// other axis. Constants, not literals in the runner: lib/sampling.js reads
+// them, so moving one here moves the policy, and the gate asserts that by
+// value rather than by grepping for a name.
+//
+// MIN is 3 because two draws give an sd that is barely a measurement and one
+// gives none at all. MAX is 10: #006's probe used 20 by hand and found the
+// shape at well under half of that, and a per-case ceiling bounds the spend.
+// The SD threshold is the effect floor — a spread wider than the smallest move
+// the verdict rule will call real is exactly when more draws are owed.
+const GENERATION_SAMPLES_MIN = 3;
+const GENERATION_SAMPLES_MAX = 10;
+const GENERATION_SD_THRESHOLD = EFFECT_FLOOR;
+const GENERATION_STABILITY_EPS = 0.01;
+
 // Report #002 is the first CROSS-PROVIDER report: the same suites, the same fixed
 // Haiku judge, run on two substrates — a Claude flagship and a GPT flagship. The
 // GPT flagship is a config constant (not hard-coded across scripts) so a future
@@ -90,7 +133,8 @@ const REPORT_005_JUDGE_MODEL = 'claude-haiku-4-5';
 
 module.exports = {
   PROJECT_NAME, RUNNER_VERSION, SUITE_FORMAT, RECEIPT_SCHEMA_VERSION, DEFAULT_JUDGE_SAMPLES,
-  EFFECT_FLOOR, DEV_MAX_USD, REPORT_MAX_USD, TRIGGER_MAX_USD,
+  EFFECT_FLOOR, DEV_MAX_USD, DEV_MAX_CALLS, REPORT_MAX_USD, TRIGGER_MAX_USD,
+  GENERATION_SAMPLES_MIN, GENERATION_SAMPLES_MAX, GENERATION_SD_THRESHOLD, GENERATION_STABILITY_EPS,
   REPORT_002_CLAUDE_MODEL, REPORT_002_GPT_MODEL, REPORT_002_JUDGE_MODEL,
   REPORT_003_NEW_MODEL, REPORT_003_OLD_MODEL, REPORT_003_JUDGE_MODEL,
   REPORT_004_BASE_MODEL, REPORT_004_FRONTIER_MODEL, REPORT_004_JUDGE_MODEL,

package/lib/canary.js

@@ -0,0 +1,27 @@
+// SPDX-License-Identifier: Apache-2.0
+'use strict';
+
+// Suite canary — receipt spec v0.5.
+//
+// Terminal-Bench 4.0 records a canary string so that a benchmark appearing in
+// training data is detectable. Adopted, translated: our unit of publication is
+// the SUITE, so the canary is per suite, derived from the suite's identity and
+// its case ids rather than randomly assigned, so the same suite yields the same
+// canary on every machine and no registry has to be kept.
+//
+// It is a detection aid, not a control: a canary tells you a suite leaked, it
+// does not stop the leak, and it cannot prove the absence of one.
+
+const crypto = require('crypto');
+
+const NAMESPACE = 'driftproof/suite-canary/v1';
+
+function suiteCanary(suite) {
+  const id = String((suite && (suite.id || suite.name)) || '');
+  const caseIds = ((suite && suite.cases) || []).map((c) => String((c && c.id) || '')).sort();
+  const h = crypto.createHash('sha256').update(`${NAMESPACE}\n${id}\n${caseIds.join('\n')}`).digest('hex');
+  // Formatted as a GUID so it is recognisable as one in a corpus scan.
+  return [h.slice(0, 8), h.slice(8, 12), h.slice(12, 16), h.slice(16, 20), h.slice(20, 32)].join('-');
+}
+
+module.exports = { suiteCanary, NAMESPACE };

package/lib/cost.js

@@ -58,7 +58,25 @@ function perCallCostUSD(modelId, kind) {
 // generated on each target model, judged `samples` times each on `judgeModel`.
 //
 //   returns { totalUSD, perModel: [{ model, usd }], judgeUSD, genUSD, assumptions }
-function estimateRunCostUSD({ caseCount, samples, models, judgeModel }) {
+// THE DRAW FACTOR IS REQUIRED AND HAS NO DEFAULT (spec 016 AC-7).
+//
+// v0.5 draws the generation up to `SAMPLING.max` times per arm, so a run costs
+// its one-draw estimate times the number of draws. `projectCalls` learned this in
+// spec 014 with a `draws` argument DEFAULTING TO 1 "so every existing caller
+// projects exactly what it projected before" — and every existing caller then
+// went on projecting a tenth of the run, silently, for two more loops. Spec 015
+// corrected the CALL projection in three report scripts and left the DOLLAR
+// projection at one draw everywhere, which is the figure a human actually reads
+// before authorising a paid run.
+//
+// A default is what made that invisible, so there is none. Omitting the factor
+// throws, which turns a silent understatement into a loud stop — and every call
+// site has to say what it means, including the ones that legitimately mean 1.
+function estimateRunCostUSD({ caseCount, samples, models, judgeModel, draws }) {
+  if (!Number.isFinite(draws) || draws < 1) {
+    throw new Error('estimateRunCostUSD: `draws` is required and must be >= 1 — pass SAMPLING.max to project a v0.5 run, or 1 to price a single draw deliberately. It is not defaulted, because a default is how the dollar projection stayed at one draw through two loops.');
+  }
+  caseCount = caseCount * draws;
   const judgePrice = priceFor(judgeModel);
   const perModel = [];
   let judgeUSD = 0;
@@ -80,7 +98,7 @@ function estimateRunCostUSD({ caseCount, samples, models, judgeModel }) {
     perModel,
     judgeUSD: round4(judgeUSD),
     genUSD: round4(genUSD),
-    assumptions: { tokens: TOKENS, judgeModel, note: 'rough upper-bound estimate; registry per-MTok pricing; not measured with count_tokens' },
+    assumptions: { tokens: TOKENS, judgeModel, draws, note: 'rough upper-bound estimate; registry per-MTok pricing; not measured with count_tokens' },
   };
 }
 

package/lib/diff.js

@@ -4,6 +4,7 @@
 const { bandVerdict, round } = require('./stats');
 const { EFFECT_FLOOR } = require('../config');
 const { revisionHeadline } = require('./revision');
+const { baselineReproduces, REFUSAL_REASONS, bandOf } = require('./reuse');
 
 // Practical-significance gate applied ON TOP of band separation. bandVerdict()
 // stays a pure geometry test (kept that way so its unit checks are unambiguous);
@@ -32,11 +33,23 @@ function isWithinNoise(v) { return v === 'within noise' || v === WITHIN_NOISE_FL
 
 // Map case-id → { mean, stddev } for a receipt's with_skill cases. Falls back to
 // score/0 for v0.1 receipts that have no per-case band.
+// NO RENDERING SWITCHES. An earlier revision carried two const-true flags so each
+// half of F-015-B's fix could be removed in a probe — and the dead branch of one
+// of them contained, verbatim, the false headline AC-4 exists to forbid. Nothing
+// could fire it, and it was still a defect-restoring branch living in the tree
+// that gets published. The mutation probes patch this source in a disposable copy.
+
 function withSkillBands(receipt) {
   const out = {};
   for (const c of receipt.results.cases) {
     if (c.mode !== 'with_skill') continue;
-    out[c.id] = { mean: c.mean != null ? c.mean : c.score, stddev: c.stddev || 0 };
+  // ONE BAND DEFINITION (spec 016 AC-1). This built its own from the v0.4-shaped
+  // fields, which worked — and that is the point: three copies existed, two
+  // reading the legacy shape and one reading only v0.5, and the one that read
+  // only v0.5 was the one a cross-version control depended on (F-015-C). Routing
+  // every comparison path through `bandOf` means a future shape is added once.
+    const b = bandOf(c);
+    if (b) out[c.id] = { mean: b.mean, stddev: b.sd, source: b.source };
   }
   return out;
 }
@@ -51,7 +64,26 @@ function aggWithBand(receipt) {
 
 function fmt(n) { return n == null ? 'n/a' : (n >= 0 ? '+' : '') + n.toFixed(3); }
 function pct(n) { return n == null ? 'n/a' : n.toFixed(3); }
-function bandStr(x) { return `${x.mean.toFixed(3)} ± ${x.stddev.toFixed(3)}`; }
+// THE BAND SAYS WHICH BAND IT IS (spec 017 AC-7).
+//
+// `bandOf` computes `source: 'legacy' | 'generation'` and carries it onto every
+// band; nothing rendered it, so a reader comparing an archived receipt with a
+// v0.5 one was comparing a JUDGE-SAMPLE spread against an ACROSS-DRAW spread
+// with nothing on the page saying so. They are different statistics over
+// different things, and the comparison is still the only one v0.4 admits — which
+// is exactly why the page has to name them rather than leave them to look alike.
+//
+// THE MARKER IS THE RECEIPT'S OWN WORD — `legacy` or `generation`, exactly as
+// `bandOf` records it — rather than a prettier synonym. A reader who greps the
+// page for what a receipt says should find the same token; a rendering that
+// renames the thing it is disclosing has disclosed a different thing. Omitted
+// when a band carries no source (an aggregate band is computed from case means,
+// not from one case's draws).
+
+function bandStr(x) {
+  const label = x && x.source;
+  return `${x.mean.toFixed(3)} ± ${x.stddev.toFixed(3)}${label ? ` (${label})` : ''}`;
+}
 function short(h) { return h ? String(h).slice(0, 12) : 'n/a'; }
 
 // The headline is a SUMMARY of the per-case band-overlap verdicts — the credibility
@@ -85,6 +117,25 @@ function revisionPairProblem(a, b) {
 
 function buildDriftReport(a, b, { labelA = 'A', labelB = 'B', mode = 'release' } = {}) {
   const revision = mode === 'revision';
+
+  // AC-6 (spec 014) — THE PRECONDITION, IN THE PATH THAT EMITS THE VERDICT.
+  //
+  // A comparison across schema versions assumes the two runs measured the same
+  // thing. The assumption is checkable: the no-skill arm contains no skill text,
+  // so nothing about a skill or a spec revision can move it. If it fails to
+  // reproduce, the comparison has no ground, and Report #006 is what that looks
+  // like when it is checked — 3 cells, 0 measured, 3 refused.
+  //
+  // SCOPED TO CROSS-VERSION PAIRS, which is what AC-6 names. Revision-mode pairs
+  // already carry their own register through revisionPairProblem(); widening
+  // this to every same-version comparison is a live question, recorded in
+  // tasks.md as OPEN-QUESTION-3 rather than decided here.
+  const crossVersion = String(a.schema_version || '') !== String(b.schema_version || '');
+  let refusal = null;
+  if (crossVersion) {
+    const pre = baselineReproduces(a, b);
+    if (!pre.ok) refusal = { key: pre.key, reason: REFUSAL_REASONS[pre.key](pre) };
+  }
   const aB = withSkillBands(a);
   const bB = withSkillBands(b);
   const ids = [...new Set([...Object.keys(aB), ...Object.keys(bB)])];
@@ -95,18 +146,21 @@ function buildDriftReport(a, b, { labelA = 'A', labelB = 'B', mode = 'release' }
   // improvement verdicts are never computed from evidence we did not run.
   const levelOf = (r) => r.verification_level || 'TESTED';
   const belowTested = [[labelA, a], [labelB, b]].filter(([, r]) => levelOf(r) !== 'TESTED');
-  const measured = belowTested.length === 0;
+  // A REFUSAL IS A RESULT, and it suppresses the verdict exactly as an
+  // untested input does: no delta is asserted, and the reason travels with it.
+  const measured = belowTested.length === 0 && !refusal;
 
   const perCase = ids.map((id) => {
     const before = aB[id] || null;
     const after = bB[id] || null;
     const delta = (before && after) ? round(after.mean - before.mean) : null;
-    const verdict = !measured ? 'not measured'
+    const verdict = refusal ? 'refused'
+      : !measured ? 'not measured'
       : (before && after) ? verdictWithFloor(before, after, delta) : 'n/a';
     return { id, before, after, delta, verdict };
   });
   // Sort worst-first: regressions, then by delta.
-  const order = { regression: 0, 'within noise': 1, [WITHIN_NOISE_FLOOR]: 1, improvement: 2, 'n/a': 3, 'not measured': 3 };
+  const order = { regression: 0, 'within noise': 1, [WITHIN_NOISE_FLOOR]: 1, improvement: 2, 'n/a': 3, 'not measured': 3, refused: 3 };
   perCase.sort((x, y) => (order[x.verdict] - order[y.verdict]) || ((x.delta || 0) - (y.delta || 0)));
 
   const aAgg = aggWithBand(a);
@@ -172,13 +226,47 @@ function buildDriftReport(a, b, { labelA = 'A', labelB = 'B', mode = 'release' }
   if (!revision && a.suite.suite_hash !== b.suite.suite_hash) warnings.push('suite_hash differs — the eval suite changed; per-case comparison may be misleading.');
   if (a.skill.name !== b.skill.name) warnings.push(`different skills (${a.skill.name} vs ${b.skill.name}) — comparison is not meaningful.`);
   if ((a.run.judge || {}).samples <= 1 || (b.run.judge || {}).samples <= 1) warnings.push('one or both receipts are single-sample (no bands) — non-overlap can only be trusted when both sides are sampled.');
-  if (!measured) warnings.push(`verdicts NOT computed — ${belowTested.map(([l, r]) => `${l} is ${levelOf(r)}${r.run && r.run.source ? ` (${r.run.source})` : ''}`).join('; ')}. Drift verdicts require TESTED receipts on both sides; declared numbers are shown as context only (see /interop.html).`);
+  // WHAT SUPPRESSED THE VERDICT IS SAID, AND IT IS SAID CORRECTLY (spec 016 AC-3
+  // and AC-4, closing F-015-B).
+  //
+  // TWO DIFFERENT THINGS can suppress a verdict, and this line used to describe
+  // only one of them. A REFUSAL — the baseline-reproduction precondition — set
+  // `measured` false and then the caveat rendered `belowTested`, which on a
+  // refused pair is EMPTY: the page read `verdicts NOT computed — .` under a
+  // headline claiming `0 receipt(s) below TESTED`, on a pair where both receipts
+  // were TESTED. An empty reason and a false statement, in the artifact a report's
+  // verdicts come from. The refusal's own cause-honest reason was computed into
+  // `refusal.reason` and thrown away by the renderer, so nothing a reader could
+  // see said why the comparison had stopped.
+  //
+  // Each half is proved load-bearing by a mutation that patches THIS source in a
+  // disposable copy. An earlier revision guarded them with const-true flags and
+  // this sentence described those; the flags were removed because a
+  // defect-restoring branch resident in the published tree is a hazard, and the
+  // sentence outlived them by one commit.
+  if (refusal) {
+    warnings.push(`verdicts NOT computed — the comparison was REFUSED before any verdict was formed: ${refusal.reason}`);
+  }
+  if (belowTested.length) {
+    warnings.push(`verdicts NOT computed — ${belowTested.map(([l, r]) => `${l} is ${levelOf(r)}${r.run && r.run.source ? ` (${r.run.source})` : ''}`).join('; ')}. Drift verdicts require TESTED receipts on both sides; declared numbers are shown as context only (see /interop.html).`);
+  }
   // Cross-provider / cross-surface disclosure (Phase 6). A comparison across
   // providers is a skill-DURABILITY comparison across substrates, not model drift
   // over time; across surfaces, sampling control differs. Both are flagged so a
   // reader never mistakes one for the other (see docs/neutrality.html).
   if (!revision && (a.run.provider || 'anthropic') !== (b.run.provider || 'anthropic')) warnings.push(`different providers (${a.run.provider || 'anthropic'} vs ${b.run.provider || 'anthropic'}) — this is a cross-substrate durability comparison, not model drift over time; read the delta, not absolute scores (see the neutrality policy).`);
   if (!revision && a.run.surface !== b.run.surface) warnings.push(`different surfaces (${a.run.surface} vs ${b.run.surface}) — sampling control differs between surfaces; compare with care.`);
+  // The legend for the band markers, printed whenever either side carries one.
+  // A two-letter marker a reader cannot decode is worse than no marker.
+  // GATED ON WHETHER A LABEL IS ACTUALLY RENDERED, not on whether the data
+  // carries a source. A legend explains markers on the page; if `bandStr` emits
+  // none, the legend is describing something the reader cannot see. Asked of
+  // `bandStr` itself rather than recomputed, so the two cannot disagree.
+  const anyLabelRendered = [...Object.values(aB), ...Object.values(bB)]
+    .some((x) => x && /\([a-z]+\)\s*$/.test(bandStr(x)));
+  if (anyLabelRendered) {
+    warnings.push('band provenance: `(generation)` is an ACROSS-DRAW spread — receipt spec v0.5, n generation draws per arm. `(legacy)` is a JUDGE-SAMPLE spread over a single generation, which is what v0.4 and earlier recorded. They are different statistics. The comparison is the only one the older receipt admits, and it is not like for like.');
+  }
   if (warnings.length) {
     L.push('> **⚠ Caveats**');
     for (const w of warnings) L.push(`> - ${w}`);
@@ -190,7 +278,20 @@ function buildDriftReport(a, b, { labelA = 'A', labelB = 'B', mode = 'release' }
   L.push(`## Headline`);
   L.push('');
   if (!measured) {
-    L.push(`**NOT MEASURED — ${belowTested.length} receipt(s) below TESTED. Drift verdicts require TESTED receipts on both sides; the declared numbers above are context, not band-verified evidence.**`);
+    // THE HEADLINE NAMES THE CAUSE THAT ACTUALLY APPLIES. `belowTested.length`
+    // was printed unconditionally, so a refused pair got "0 receipt(s) below
+    // TESTED" — a statement measurably false of the receipts it was given.
+    if (refusal) {
+      // The reason is a complete sentence and already ends by saying no verdict
+      // is asserted; prefixing that again produced "REFUSED — no verdict is
+      // asserted. the baseline arm did not reproduce…" — a duplicated clause and
+      // a lower-case sentence start, in the artifact a published report quotes.
+      L.push(`**REFUSED — ${refusal.reason}**`);
+    } else if (belowTested.length) {
+      L.push(`**NOT MEASURED — ${belowTested.length} receipt(s) below TESTED. Drift verdicts require TESTED receipts on both sides; the declared numbers above are context, not band-verified evidence.**`);
+    } else {
+      L.push('**NOT MEASURED — no verdict is asserted.**');
+    }
     L.push('');
   } else {
     L.push(`**${revision ? revisionHeadline(perCase) : headlineVerdict(perCase)}**`);
@@ -218,7 +319,12 @@ function buildDriftReport(a, b, { labelA = 'A', labelB = 'B', mode = 'release' }
     L.push('');
   }
 
-  return { markdown: L.join('\n'), perCase, headlineDelta, regressions };
+  return {
+    markdown: L.join('\n'), perCase, headlineDelta, regressions,
+    refused: !!refusal,
+    refusal_reason: refusal ? refusal.reason : null,
+    refusal_key: refusal ? refusal.key : null,
+  };
 }
 
 module.exports = { buildDriftReport, withSkillBands, revisionPairProblem };

package/lib/judge.js

@@ -99,7 +99,12 @@ async function gradeOnce({ task, response, rubric, model, timeoutMs, temperature
 //   { samples:[scores], mean, stddev, reason, judge_settings, model_id, rubric_hash }
 // `mean` ± `stddev` is the per-case confidence band (raw spread of the N scores)
 // used by the borderline-outcome rule and per-case drift band-overlap logic.
-async function gradeSamples({ task, response, rubric, model, samples = 5, timeoutMs = 120000 }) {
+// NO DEFAULT TIMEOUT HERE (spec 017 AC-2). This defaulted to 120000, which
+// outranked the per-surface policy exactly as lib/run.js's literal did — so the
+// JUDGE calls timed out on the api policy while running on a CLI surface, which
+// is the second shadowing site and the one #007's prep session had not found.
+// Passing `undefined` through lets lib/provider.js resolve the declared policy.
+async function gradeSamples({ task, response, rubric, model, samples = 5, timeoutMs }) {
   const settings = judgeSettings(samples, model);
   const scores = [];
   const reasons = [];