driftproof 0.4.0 → 0.6.0

package/README.md

@@ -9,15 +9,17 @@
 Driftproof is an open **receipt spec** plus a **runner** that measures whether an
 agent skill actually helps — by running the skill's eval suite **with** and
 **without** the skill on a named model version, judging each case several times to
-get a confidence band, and emitting a signed, dated **receipt**. Diff two receipts
+get a confidence band, and emitting a **hash-verified**, dated **receipt**. Diff two receipts
 across model releases and you get a **drift report**.
 
 Driftproof **consumes** the [`agentskills.io/evals`](https://agentskills.io) eval
 format; it does not invent its own.
 
-📊 **Four published reports** (each re-derived from committed receipts, nothing
-hand-entered), spanning three report types that share one band-based, floor-gated
-verdict rule and differ only in what moves underneath the skill:
+📊 **Six published reports** (each re-derived from committed receipts, nothing
+hand-entered), spanning five published report types, the fifth being revision
+drift. All six share one band-based, floor-gated verdict rule and differ in what
+moves underneath the skill — or, in the value report, in which axes are
+measured:
 
 - **[Report #001](https://driftproofhq.com/reports/001/)** — *release drift*: ten
   public agent skills across a current-vs-previous Sonnet release; 9 of 10 moved
@@ -31,10 +33,29 @@ verdict rule and differ only in what moves underneath the skill:
   `claude-opus-5` (flagship) vs `claude-fable-5` (frontier tier);
   3 durable, 5 tier-dependent, 0 regressions, 2 no effect — encoded expertise
   survives the frontier tier.
+- **[Report #005](https://driftproofhq.com/reports/005/)** — *value*: what a skill
+  *costs* to run, on three axes (accuracy, cost, latency); the same ten suites on
+  three substrates (`claude-sonnet-5`, `claude-fable-5`, `gpt-5.6-sol`) —
+  14 of 30 cells cleared the floor on aggregate: 10 carry a price and 4 report a
+  saving instead, having improved quality while reducing cost.
+  *Three of those cells carry an amendment (v1.1, applied when Report #006
+  published): their lifts rest on single-draw baselines since shown
+  unstable. Cause-agnostic, no corrected figures offered, and the cost-driver and
+  substrate-disagreement findings below are unaffected.*
+- **[Report #006](https://driftproofhq.com/reports/006/)** — *revision drift*: the pinned skill revision against the one
+  upstream ships today, on a held substrate. **The reuse premise was tested and
+  refused: 3 of 3 cells returned no verdict**, each blocked by its own baseline
+  control. A 120-call probe found generation-level sampling noise 3.2× and 7.5×
+  larger than the judge-level noise this instrument actually samples — enough to
+  account for every gap the controls saw without any other cause being
+  established — and the receipt spec gains generation sampling as a result. **No
+  cause is asserted**; the control proves non-reproduction and cannot say why.
+  *The tally a refusal carries: 3 cells, 0 measured, 3 refused.*
 
 ✍️ The launch essay, **[Three model releases later: what actually happens to agent
-skills](https://driftproofhq.com/writing/three-releases/)**, reads the first three
-reports together.
+skills](https://driftproofhq.com/writing/three-releases/)**, reads all six reports
+together: what moves underneath a skill, and what the skill costs to run. Revised
+2026-08-29; every figure in it is gate-checked against the report page it cites.
 
 ## Why
 
@@ -55,6 +76,15 @@ by more than the drift you're trying to detect. Driftproof's answer is to **samp
 the judge and report confidence bands**, and to **only claim a regression when the
 bands don't overlap**. A tool that cries wolf is worse than no tool.
 
+**A verdict without a price is half an answer.** The same receipts price the
+marginal cost of a skill firing, and Report #005 found the dominant cost driver is
+not the skill's own text but the input it causes the model to pull in: across those
+30 cells the input delta tracks cost at `r = +0.92` while the skill's own length
+tracks it at only `r = +0.33`, and one 738-token skill drew 34× its own size in
+extra input. Identical token deltas also price very differently across substrates —
+the same skill at near-identical deltas costs 3.3× more on `claude-fable-5` than on
+`claude-sonnet-5`, which is exactly their input-rate ratio in the frozen snapshot.
+
 ## Quickstart — receipt for your own skill in ~10 minutes
 
 You need Node ≥ 22 and an `ANTHROPIC_API_KEY`.
@@ -86,6 +116,15 @@ bands don't overlap. The **effect floor** (0.05, one judge quantization step) is
 minimum real move required before a change counts as more than noise — band
 separation *plus* a floor-sized delta, never either alone.
 
+**What the band does not cover.** A verdict rests on **one generation draw per
+arm**: the band is the spread of the *judge* re-scoring that single response, not
+the spread of the model writing a different one. Report #006 measured the second
+directly and found it larger — draw-to-draw spread up to **sd 0.186** on the 0–1
+scale, against judge-level noise several times smaller. So treat a surprising
+single-run verdict as **provisional and worth re-running** before you act on it.
+Generation sampling lands in the next receipt spec; until it does, this is a
+limit of the instrument, stated rather than implied.
+
 ### Install
 
 ```bash
@@ -162,7 +201,7 @@ A receipt is the unit of evidence — one JSON document conforming to
 
 ```jsonc
 {
-  "schema_version": "0.3.1",
+  "schema_version": "0.4",
   "skill":  { "name": "commit-message-conventions", "version": "0.2.0",
               "content_hash": "…sha256 over SKILL.md + bundled files…" },
   "suite":  { "format": "agentskills.io/evals", "suite_hash": "…", "case_count": 10 },
@@ -171,7 +210,7 @@ A receipt is the unit of evidence — one JSON document conforming to
     "model_release_date": "2025-10-01",
     "provider": "anthropic",
     "surface": "claude-cli",
-    "runner_version": "0.4.0",
+    "runner_version": "0.6.0",
     "date_utc": "2026-07-27T…Z",
     "registry": "registered",
     "transcripts": "hashes-only",
@@ -194,6 +233,12 @@ A receipt is the unit of evidence — one JSON document conforming to
   },
   "comparison": { "with_skill_score": 0.81, "baseline_score": 0.42,
                   "delta": 0.39, "delta_uncertainty": 0.036 },
+  // v0.4 economics, all derived and never composited into one score: "run.pricing_snapshot"
+  // freezes the rates; each case carries "usage" and a separate "judge_usage"; "economics"
+  // holds basis, surface, with_skill/baseline (call_count, mean_input_tokens,
+  // mean_output_tokens, mean_cost_usd_per_call, median_wall_ms + p25/p75/IQR),
+  // skill_incremental_cost_usd_per_call, skill_incremental_cost_usd_per_1k_calls,
+  // output_tokens_delta, median_wall_ms_delta, judge_excluded (const true), judge_overhead.
   "verification_level": "TESTED",
   "receipt_hash": "…sha256 of the canonical receipt with this field removed…"
 }
@@ -209,6 +254,12 @@ Key ideas:
 - **`delta_uncertainty`** is the combined band on the with-skill-vs-baseline lift.
 - **`verification_level`** uses the community lattice: `UNVERIFIED` / `DECLARED` /
   `TESTED` (Driftproof emits `TESTED`). `FORMAL` is reserved.
+- **`economics`** is *derived*, never a second measurement: the token delta is the
+  durable fact, and the dollars are exactly those tokens at the rates frozen into
+  `run.pricing_snapshot`, so a receipt keeps its meaning after a vendor reprices.
+  Judge cost is recorded apart as `judge_usage` and excluded from every skill-value
+  figure (`judge_excluded` is `const true`) — measuring the skill is our cost, not
+  the skill's.
 - **`receipt_hash`** is a self-hash for tamper-evidence (integrity, not yet a key
   signature — see the spec's open questions).
 
@@ -233,7 +284,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
-      - uses: driftproofhq/driftproof@v0.4.0
+      - uses: driftproofhq/driftproof@v0.6.0
         with:
           skill-dir: skills/my-skill
           models: claude-haiku-4-5
@@ -272,11 +323,14 @@ site, so it reflects a real dated run, not a hand-set color.
 
 ## Reports
 
-Four reports are published, spanning three report types (see the roll at the top
-of this README, and [REPORT-STYLE.md](REPORT-STYLE.md) for the shared chrome
-every report inherits). Each report and every verdict in it are **re-derived from
-the receipts** committed under [`receipts/`](receipts/)
-(`receipts/report-001/` … `receipts/report-004/`) — nothing is hand-entered.
+Six reports are published, spanning five report types. A report page lives at a
+draft path — `docs/reports/NNN-draft/` — until the publish sequence renames it, and
+`scripts/build-public.sh` excludes every `*-draft/` path from the published tree
+(see the roll at the top of this README, and
+[REPORT-STYLE.md](REPORT-STYLE.md) for the shared chrome every report inherits).
+Each report and every verdict in it are **re-derived from the receipts** committed
+under [`receipts/`](receipts/)
+(`receipts/report-001/` … `receipts/report-006/`) — nothing is hand-entered.
 
 Driftproof does **not** commit third-party skill content. Each `SKILL.md` is
 fetched at run time from a pinned commit and verified by sha256 against
@@ -290,9 +344,10 @@ node scripts/run-report-001.js --concurrency 5   # run both models × with/basel
 node scripts/build-report-001.js                 # re-derive the report from the receipts
 ```
 
-Reports #002–#004 have their own runners
-(`scripts/prepare-report-00N.js`) following the same
-fetch → run → re-derive shape.
+Reports #002–#005 have their own runners
+(`scripts/prepare-report-00N.js` — Report #005's is
+[`scripts/prepare-report-005.js`](scripts/prepare-report-005.js)) following the
+same fetch → run → re-derive shape.
 
 **Model-release triggers are live**: `scripts/release-watch.js` (keyless — it
 reads the public models registry) notices a new model release, re-runs the

package/bin/driftproof

@@ -8,7 +8,7 @@ const { PROJECT_NAME, RUNNER_VERSION, DEFAULT_JUDGE_SAMPLES, DEV_MAX_USD } = req
 const { loadSkill } = require('../lib/skill');
 const { runSkillOnModel, summarizeReceipt, projectCalls } = require('../lib/run');
 const { validateReceipt, verifyReceiptHash } = require('../lib/receipt');
-const { buildDriftReport } = require('../lib/diff');
+const { buildDriftReport, revisionPairProblem } = require('../lib/diff');
 const { surfaceForModel, isSubscriptionSurface, resolveModel } = require('../lib/provider');
 const { estimateRunCostUSD, BudgetTracker } = require('../lib/cost');
 const { registryStatus } = require('../lib/models');
@@ -76,7 +76,7 @@ USAGE
   ${PROJECT_NAME} run <skill-dir> [--models a,b] [--samples N] [--max-cases N] [--max-calls N]
                         [--judge-model M] [--concurrency N] [--max-usd N]
                         [--keep-transcripts] [--out DIR]
-  ${PROJECT_NAME} diff <receiptA.json> <receiptB.json> [--out FILE]
+  ${PROJECT_NAME} diff <receiptA.json> <receiptB.json> [--out FILE] [--mode release|revision]
   ${PROJECT_NAME} validate <receipt.json>
   ${PROJECT_NAME} badge <receipt.json> [--out FILE] [--github-output]
   ${PROJECT_NAME} import <results.json> --from agent-skills-eval|skillgrade [--out DIR]
@@ -250,9 +250,32 @@ function cmdDiff(positional, flags) {
     if (!verifyReceiptHash(r)) console.error(`  ⚠ ${path.basename(p)}: receipt_hash does not verify (tampered or hand-edited)`);
   }
 
-  const labelA = dateStamp(a.run.date_utc);
-  const labelB = dateStamp(b.run.date_utc);
-  const { markdown } = buildDriftReport(a, b, { labelA, labelB });
+  // --mode revision inverts the axis: the skill text is the variable under test
+  // and the substrate is the control. The fields release drift merely warns about
+  // are preconditions here, so a pair that is not a revision pair is REFUSED
+  // (exit 6) rather than rendered with a caveat nobody reads. A differing model
+  // would be release drift wearing a revision label — the one confound this mode
+  // exists to exclude — and an EQUAL content_hash has no revision to measure.
+  const mode = flags.mode || 'release';
+  if (mode !== 'release' && mode !== 'revision') {
+    console.error(`unknown --mode "${mode}" — supported: release, revision`);
+    process.exit(2);
+  }
+  if (mode === 'revision') {
+    const problem = revisionPairProblem(a, b);
+    if (problem) {
+      const why = problem === 'skill.content_hash'
+        ? 'the two receipts carry the SAME skill.content_hash — there is no revision between them to measure'
+        : `${problem} differs between the two receipts — revision drift requires the substrate to be held fixed, and a differing ${problem} would confound the revision with release drift`;
+      console.error(`  ✗ REFUSED (--mode revision): ${why}.`);
+      console.error(`    Compare these two with the default release mode, or supply a pair that differs only in skill.content_hash.`);
+      process.exit(6);
+    }
+  }
+
+  const labelA = mode === 'revision' ? `pinned (${dateStamp(a.run.date_utc)})` : dateStamp(a.run.date_utc);
+  const labelB = mode === 'revision' ? `current (${dateStamp(b.run.date_utc)})` : dateStamp(b.run.date_utc);
+  const { markdown } = buildDriftReport(a, b, { labelA, labelB, mode });
 
   if (flags.out) {
     fs.writeFileSync(path.resolve(flags.out), markdown);

package/config.js

@@ -9,7 +9,7 @@ const PROJECT_NAME = 'driftproof';
 // Bumped whenever the runner's behaviour or receipt-generation semantics change
 // in a way that could affect results. Recorded into every receipt as
 // run.runner_version so a receipt is reproducible against a known engine.
-const RUNNER_VERSION = '0.4.0';
+const RUNNER_VERSION = '0.6.0';
 
 // The eval format we CONSUME (we deliberately do not invent our own).
 const SUITE_FORMAT = 'agentskills.io/evals';
@@ -22,7 +22,14 @@ const SUITE_FORMAT = 'agentskills.io/evals';
 //        surface enums (openai-api/openai-cli), optional run.surface_overhead_note,
 //        optional per-case checks[] (deterministic post-checks), and optional
 //        skill.tokens (value-per-token axis). v0.1/v0.2/v0.3 receipts still load.
-const RECEIPT_SCHEMA_VERSION = '0.3.1';
+// v0.4   (additive over v0.3.1) adds the ECONOMICS axis: per-case, per-arm
+//        generation `usage` (input/output/cached tokens + measured wall_ms), a
+//        separate per-case `judge_usage` (measurement overhead, excluded from
+//        every skill-value figure), run.pricing_snapshot (registry prices frozen
+//        at run time so derived dollars stay reproducible), and the derived
+//        `economics` block. v0.3.1 is frozen as receipt.v0.3.1.schema.json;
+//        v0.1/v0.2/v0.3/v0.3.1 receipts all still load.
+const RECEIPT_SCHEMA_VERSION = '0.4';
 
 // Hard USD budget defaults per entry point (Week 4). --max-usd overrides any of
 // these. The projection is refused before any call if it exceeds the cap, on
@@ -73,10 +80,19 @@ const REPORT_004_BASE_MODEL = 'claude-opus-5';     // flagship tier
 const REPORT_004_FRONTIER_MODEL = 'claude-fable-5'; // frontier tier (full id — no alias)
 const REPORT_004_JUDGE_MODEL = 'claude-haiku-4-5';
 
+// Report #005 is a VALUE report — the fourth report type. It asks what a skill
+// COSTS to run alongside whether it helps, over three substrates, and shows the
+// three axes (accuracy lift / Δcost / Δlatency) side by side and never combined.
+// Same suites, same fixed judge; the substrate list spans two providers so the
+// economics are read across surfaces, not within one vendor's pricing.
+const REPORT_005_MODELS = ['claude-sonnet-5', 'claude-fable-5', 'gpt-5.6-sol'];
+const REPORT_005_JUDGE_MODEL = 'claude-haiku-4-5';
+
 module.exports = {
   PROJECT_NAME, RUNNER_VERSION, SUITE_FORMAT, RECEIPT_SCHEMA_VERSION, DEFAULT_JUDGE_SAMPLES,
   EFFECT_FLOOR, DEV_MAX_USD, REPORT_MAX_USD, TRIGGER_MAX_USD,
   REPORT_002_CLAUDE_MODEL, REPORT_002_GPT_MODEL, REPORT_002_JUDGE_MODEL,
   REPORT_003_NEW_MODEL, REPORT_003_OLD_MODEL, REPORT_003_JUDGE_MODEL,
   REPORT_004_BASE_MODEL, REPORT_004_FRONTIER_MODEL, REPORT_004_JUDGE_MODEL,
+  REPORT_005_MODELS, REPORT_005_JUDGE_MODEL,
 };

package/lib/diff.js

@@ -3,6 +3,7 @@
 
 const { bandVerdict, round } = require('./stats');
 const { EFFECT_FLOOR } = require('../config');
+const { revisionHeadline } = require('./revision');
 
 // Practical-significance gate applied ON TOP of band separation. bandVerdict()
 // stays a pure geometry test (kept that way so its unit checks are unambiguous);
@@ -68,7 +69,22 @@ function headlineVerdict(perCase) {
   return 'WITHIN NOISE — no case moved beyond its confidence band; the skill holds up.';
 }
 
-function buildDriftReport(a, b, { labelA = 'A', labelB = 'B' } = {}) {
+// A revision pair is a pair in which the SKILL TEXT is the only thing that
+// moved. `diff` was built for release drift, where the model varies and the text
+// is fixed; this inverts it, so the fields that release drift merely warns about
+// become the preconditions of the comparison. Returns the offending field name,
+// or null when the pair is a valid revision pair.
+function revisionPairProblem(a, b) {
+  if ((a.run.model_id || '') !== (b.run.model_id || '')) return 'run.model_id';
+  if ((a.run.provider || 'anthropic') !== (b.run.provider || 'anthropic')) return 'run.provider';
+  if ((a.run.surface || '') !== (b.run.surface || '')) return 'run.surface';
+  if ((a.suite.suite_hash || '') !== (b.suite.suite_hash || '')) return 'suite.suite_hash';
+  if ((a.skill.content_hash || '') === (b.skill.content_hash || '')) return 'skill.content_hash';
+  return null;
+}
+
+function buildDriftReport(a, b, { labelA = 'A', labelB = 'B', mode = 'release' } = {}) {
+  const revision = mode === 'revision';
   const aB = withSkillBands(a);
   const bB = withSkillBands(b);
   const ids = [...new Set([...Object.keys(aB), ...Object.keys(bB)])];
@@ -99,6 +115,32 @@ function buildDriftReport(a, b, { labelA = 'A', labelB = 'B' } = {}) {
   const regressions = perCase.filter((r) => r.verdict === 'regression');
 
   const L = [];
+  if (revision) {
+    // The axis leads. In a release-drift report the model is what moved and it
+    // belongs at the top; here the model is the control and the skill's own text
+    // is the finding, so content_hash is the first row a reader meets.
+    L.push(`# Revision drift report`);
+    L.push('');
+    L.push(`**Skill:** ${a.skill.name} \`${a.skill.version}\``);
+    L.push('');
+    L.push(`**Report type:** revision drift — the skill's own text is the variable under test; the substrate is held fixed.`);
+    L.push('');
+    L.push(`Left column is the **pinned** revision; right column is the **current** upstream revision.`);
+    L.push('');
+    L.push(`| | ${labelA} | ${labelB} |`);
+    L.push(`|---|---|---|`);
+    L.push(`| skill content_hash | \`${short(a.skill.content_hash)}\` | \`${short(b.skill.content_hash)}\` |`);
+    L.push(`| model (held) | \`${a.run.model_id}\` | \`${b.run.model_id}\` |`);
+    L.push(`| provider (held) | ${a.run.provider || 'anthropic'} | ${b.run.provider || 'anthropic'} |`);
+    L.push(`| surface (held) | ${a.run.surface} | ${b.run.surface} |`);
+    L.push(`| suite_hash (held) | \`${short(a.suite.suite_hash)}\` | \`${short(b.suite.suite_hash)}\` |`);
+    L.push(`| run date (UTC) | ${a.run.date_utc} | ${b.run.date_utc} |`);
+    L.push(`| judge samples/case | ${(a.run.judge || {}).samples || 1} | ${(b.run.judge || {}).samples || 1} |`);
+    L.push(`| with_skill (mean ± band) | ${bandStr(aAgg)} | ${bandStr(bAgg)} |`);
+    L.push(`| baseline score | ${pct(a.comparison.baseline_score)} | ${pct(b.comparison.baseline_score)} |`);
+    L.push(`| skill lift (Δ) | ${fmt(a.comparison.delta)} | ${fmt(b.comparison.delta)} |`);
+    L.push('');
+  } else {
   L.push(`# Drift report`);
   L.push('');
   L.push(`**Skill:** ${a.skill.name} \`${a.skill.version}\``);
@@ -115,10 +157,19 @@ function buildDriftReport(a, b, { labelA = 'A', labelB = 'B' } = {}) {
   L.push(`| baseline score | ${pct(a.comparison.baseline_score)} | ${pct(b.comparison.baseline_score)} |`);
   L.push(`| skill lift (Δ) | ${fmt(a.comparison.delta)} | ${fmt(b.comparison.delta)} |`);
   L.push('');
+  }
 
   const warnings = [];
-  if (a.skill.content_hash !== b.skill.content_hash) warnings.push('skill content_hash differs — the skill itself changed between receipts, so drift mixes skill edits with model drift.');
-  if (a.suite.suite_hash !== b.suite.suite_hash) warnings.push('suite_hash differs — the eval suite changed; per-case comparison may be misleading.');
+  // In revision mode the changed skill text is the INDEPENDENT VARIABLE, not
+  // contamination, and the held-constant substrate is what makes the comparison
+  // valid. The release-drift caveat below says the opposite of both, so it is
+  // replaced rather than suppressed: a reader is told what is held, and why a
+  // separated band is attributable to the revision.
+  if (revision) {
+    warnings.push(`model \`${a.run.model_id}\`, provider ${a.run.provider || 'anthropic'}, surface ${a.run.surface} and suite_hash \`${short(a.suite.suite_hash)}\` are held constant across both receipts — the skill text is the only variable under test, so a band-separated move above the ${EFFECT_FLOOR} floor is attributable to the revision.`);
+  }
+  if (!revision && a.skill.content_hash !== b.skill.content_hash) warnings.push('skill content_hash differs — the skill itself changed between receipts, so drift mixes skill edits with model drift.');
+  if (!revision && a.suite.suite_hash !== b.suite.suite_hash) warnings.push('suite_hash differs — the eval suite changed; per-case comparison may be misleading.');
   if (a.skill.name !== b.skill.name) warnings.push(`different skills (${a.skill.name} vs ${b.skill.name}) — comparison is not meaningful.`);
   if ((a.run.judge || {}).samples <= 1 || (b.run.judge || {}).samples <= 1) warnings.push('one or both receipts are single-sample (no bands) — non-overlap can only be trusted when both sides are sampled.');
   if (!measured) warnings.push(`verdicts NOT computed — ${belowTested.map(([l, r]) => `${l} is ${levelOf(r)}${r.run && r.run.source ? ` (${r.run.source})` : ''}`).join('; ')}. Drift verdicts require TESTED receipts on both sides; declared numbers are shown as context only (see /interop.html).`);
@@ -126,8 +177,8 @@ function buildDriftReport(a, b, { labelA = 'A', labelB = 'B' } = {}) {
   // providers is a skill-DURABILITY comparison across substrates, not model drift
   // over time; across surfaces, sampling control differs. Both are flagged so a
   // reader never mistakes one for the other (see docs/neutrality.html).
-  if ((a.run.provider || 'anthropic') !== (b.run.provider || 'anthropic')) warnings.push(`different providers (${a.run.provider || 'anthropic'} vs ${b.run.provider || 'anthropic'}) — this is a cross-substrate durability comparison, not model drift over time; read the delta, not absolute scores (see the neutrality policy).`);
-  if (a.run.surface !== b.run.surface) warnings.push(`different surfaces (${a.run.surface} vs ${b.run.surface}) — sampling control differs between surfaces; compare with care.`);
+  if (!revision && (a.run.provider || 'anthropic') !== (b.run.provider || 'anthropic')) warnings.push(`different providers (${a.run.provider || 'anthropic'} vs ${b.run.provider || 'anthropic'}) — this is a cross-substrate durability comparison, not model drift over time; read the delta, not absolute scores (see the neutrality policy).`);
+  if (!revision && a.run.surface !== b.run.surface) warnings.push(`different surfaces (${a.run.surface} vs ${b.run.surface}) — sampling control differs between surfaces; compare with care.`);
   if (warnings.length) {
     L.push('> **⚠ Caveats**');
     for (const w of warnings) L.push(`> - ${w}`);
@@ -142,7 +193,7 @@ function buildDriftReport(a, b, { labelA = 'A', labelB = 'B' } = {}) {
     L.push(`**NOT MEASURED — ${belowTested.length} receipt(s) below TESTED. Drift verdicts require TESTED receipts on both sides; the declared numbers above are context, not band-verified evidence.**`);
     L.push('');
   } else {
-    L.push(`**${headlineVerdict(perCase)}**`);
+    L.push(`**${revision ? revisionHeadline(perCase) : headlineVerdict(perCase)}**`);
     L.push('');
     L.push(`with_skill mean moved ${fmt(headlineDelta)} (${bandStr(aAgg)} → ${bandStr(bAgg)}; band = suite dispersion). Per-case band-overlap verdicts: ${regressions.length} regression(s), ${perCase.filter((r) => r.verdict === 'improvement').length} improvement(s), ${nWithin} within noise${nFloor ? ` (${nFloor} of them band-separated but below the ${EFFECT_FLOOR} effect floor)` : ''}.`);
     L.push('');
@@ -170,4 +221,4 @@ function buildDriftReport(a, b, { labelA = 'A', labelB = 'B' } = {}) {
   return { markdown: L.join('\n'), perCase, headlineDelta, regressions };
 }
 
-module.exports = { buildDriftReport, withSkillBands };
+module.exports = { buildDriftReport, withSkillBands, revisionPairProblem };

package/lib/hygiene.js

@@ -0,0 +1,113 @@
+// SPDX-License-Identifier: Apache-2.0
+//
+// The single definition of what counts as a leak.
+//
+// Two callers scan for the same classes of secret: `tests/gate.js`, which walks
+// the working tree at gate time, and `scripts/merge-check.js`, which walks the
+// candidate tree read out of the commit before permitting a merge. They must
+// agree. Two copies of a deny-list drift, and the copy that quietly stops
+// matching still reads as protection — the same argument the acknowledgment
+// deny-list bite test already makes, applied to the scanner itself.
+//
+// Backlog B-8 records four approval records that reached `dev` carrying a host
+// path. The recorded cause — "the scan walks tracked files" — is false, and this
+// module's existence does not depend on it: `listFiles()` in the repo gate has
+// always enumerated untracked files too, and a planted path in an untracked
+// evidence record does take the repo gate red. The real window is VANTAGE. An
+// approval session measures in a disposable copy made before it writes its
+// record into the shared checkout, so the record is simply not in the tree that
+// was scanned. Scanning the working tree harder cannot close that. Scanning the
+// commit that is about to merge does, and that is what merge-check now uses this
+// for.
+//
+// Patterns are written so a regex literal cannot match its own text: a
+// metacharacter or a character class follows each fixed prefix.
+
+const dec = (b64) => JSON.parse(Buffer.from(b64, 'base64').toString('utf8'));
+
+// Build-box host + private IP, base64 so the plaintext never lands in a file.
+const HOSTIP = dec('WyJpcC0xNzItMzEtNDAtMTU5LmFwLXNvdXRoZWFzdC0xLmNvbXB1dGUuaW50ZXJuYWwiLCIxNzIuMzEuNDAuMTU5Il0=');
+
+const EMAIL_ALLOW = new Set(['example.com', 'example.org', 'driftproofhq.com']);
+
+const PATTERNS = [
+  { name: 'home-path', re: /\/home\/[a-z0-9_-]+\/|\/Users\/[A-Za-z0-9_-]+\// },
+  { name: 'ec2-internal-host', re: /ip-\d+-\d+-\d+-\d+\.[a-z0-9.-]*compute\.(internal|amazonaws\.com)/i },
+  { name: 'private-ip', re: /\b(10\.\d{1,3}\.\d{1,3}\.\d{1,3}|172\.(1[6-9]|2\d|3[01])\.\d{1,3}\.\d{1,3}|192\.168\.\d{1,3}\.\d{1,3})\b/ },
+  { name: 'anthropic-key', re: /sk-ant-[A-Za-z0-9_-]{8,}/ },
+  { name: 'github-token', re: /gh[pousr]_[A-Za-z0-9]{20,}/ },
+  { name: 'aws-key', re: /\bAKIA[0-9A-Z]{16}\b/ },
+  { name: 'private-key-block', re: /-----BEGIN [A-Z ]*PRIVATE KEY-----/ },
+  { name: 'env-secret-assignment', re: /\b(ANTHROPIC_API_KEY|AWS_SECRET_ACCESS_KEY|OPENAI_API_KEY)\s*=\s*\S+/ },
+  // Codex subscription auth material (~/.codex/auth.json contents) must NEVER
+  // land in a committed file: the id_token/access_token are JWTs, and an OpenAI
+  // secret key is sk-proj-/sk-svcacct-/sk-admin-. We ban the CONTENTS (tokens),
+  // not the documented path string (`~/.codex/auth.json` is referenced in help
+  // text and docs by design).
+  { name: 'jwt-token', re: /\beyJ[A-Za-z0-9_=-]{10,}\.eyJ[A-Za-z0-9_=-]{10,}\.[A-Za-z0-9_=-]{6,}/ },
+  { name: 'openai-secret-key', re: /\bsk-(proj|svcacct|admin)-[A-Za-z0-9_-]{20,}/ },
+];
+
+// A path is a hit on its own name, with no content read: a committed .env file
+// is a leak whatever it happens to contain.
+function scanPath(rel) {
+  return /(^|\/)\.env(\.|$)/.test(rel) ? [{ file: rel, kind: 'env-file' }] : [];
+}
+
+function scanContent(rel, content) {
+  const hits = [];
+  for (const p of PATTERNS) {
+    const m = content.match(p.re);
+    if (m) hits.push({ file: rel, kind: p.name, sample: m[0].slice(0, 24) });
+  }
+  for (const hv of HOSTIP) if (content.includes(hv)) hits.push({ file: rel, kind: 'build-host-or-ip' });
+  const emails = content.match(/\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b/g) || [];
+  for (const e of emails) {
+    const dom = e.split('@')[1].toLowerCase();
+    if (!EMAIL_ALLOW.has(dom) && !dom.endsWith('.example')) hits.push({ file: rel, kind: 'email', sample: e });
+  }
+  return hits;
+}
+
+// scanFiles(files, read, readLink) — `read(rel)` returns the file's text, or
+// throws/returns null for anything unreadable (binary, deleted, a symlink to
+// nowhere), which is skipped exactly as the gate has always skipped it.
+//
+// `readLink(rel)` is optional and returns a symbolic link's TARGET PATH as a
+// string, or null for an entry that is not a link. Where it is supplied, that
+// target is scanned as the entry's content through `scanContent`, so every
+// pattern above applies to it with no second deny-list to drift (spec 011,
+// AC-2). A link's target is a path, and a path is exactly the class of leak the
+// `home-path` pattern exists to catch: P-1 shipped `node_modules ->
+// /home/<user>/... ` into a public tree past a scan that read the link as an
+// unreadable file and skipped it.
+//
+// The argument is optional so a caller supplying nothing behaves exactly as
+// before. That compatibility is deliberate, and it is also the standing risk: a
+// caller added later is blind by default. The repo gate asserts that every call
+// site supplies a reader, with `scripts/merge-check.js` the one exception — its
+// reader is `git show <ref>:<path>`, and git stores a link's blob as its target
+// string, so the target already arrives as content there.
+function scanFiles(files, read, readLink) {
+  const hits = [];
+  for (const rel of files) {
+    hits.push(...scanPath(rel));
+    // The link branch is a single condition on purpose: it is the mutation seam
+    // the spec-011 gate neutralises to prove the scanner goes blind without it.
+    if (typeof readLink === 'function') {
+      let target;
+      try { target = readLink(rel); } catch (_e) { target = null; }
+      if (typeof target === 'string') {
+        hits.push(...scanContent(rel, target));
+        continue;
+      }
+    }
+    let c;
+    try { c = read(rel); } catch (_e) { continue; }
+    if (typeof c !== 'string') continue;
+    hits.push(...scanContent(rel, c));
+  }
+  return hits;
+}
+
+module.exports = { PATTERNS, EMAIL_ALLOW, HOSTIP, scanPath, scanContent, scanFiles };

package/lib/judge.js

@@ -5,6 +5,7 @@ const { complete, surfaceForModel } = require('./provider');
 const { extractJsonObject } = require('./json');
 const { sha256 } = require('./canonical');
 const { mean, stddev } = require('./stats');
+const { sumUsage } = require('./usage');
 
 // Rubric-based LLM judge.
 //
@@ -82,16 +83,16 @@ function judgeSettings(samples, judgeModel) {
 // transcript auditability, and optionally retained under --keep-transcripts).
 async function gradeOnce({ task, response, rubric, model, timeoutMs, temperature }) {
   const prompt = buildJudgePrompt({ task, response, rubric });
-  const { text, attempts } = await complete({ system: JUDGE_SYSTEM, prompt, model, maxTokens: 400, timeoutMs, temperature });
+  const { text, attempts, usage } = await complete({ system: JUDGE_SYSTEM, prompt, model, maxTokens: 400, timeoutMs, temperature });
   let parsed;
   try {
     parsed = extractJsonObject(text);
   } catch (_e) {
     // Unsalvageable judge output → conservative 0 (a judge that can't be parsed
     // must never silently "pass"), tagged so the caller can see it happened.
-    return { score: 0, reason: 'judge output unparseable', unparsed: true, raw: String(text || ''), attempts: attempts || 1 };
+    return { score: 0, reason: 'judge output unparseable', unparsed: true, raw: String(text || ''), attempts: attempts || 1, usage };
   }
-  return { score: clamp01(parsed.score), reason: String(parsed.reason || '').slice(0, 300), raw: String(text || ''), attempts: attempts || 1 };
+  return { score: clamp01(parsed.score), reason: String(parsed.reason || '').slice(0, 300), raw: String(text || ''), attempts: attempts || 1, usage };
 }
 
 // Grade a response N times and return the sampled distribution:
@@ -103,6 +104,7 @@ async function gradeSamples({ task, response, rubric, model, samples = 5, timeou
   const scores = [];
   const reasons = [];
   const rawTexts = [];
+  const usages = [];
   let attemptsTotal = 0;
   for (let i = 0; i < samples; i++) {
     let r;
@@ -116,6 +118,7 @@ async function gradeSamples({ task, response, rubric, model, samples = 5, timeou
       throw e;
     }
     attemptsTotal += r.attempts || 1;
+    usages.push(r.usage || null);
     scores.push(r.score);
     reasons.push(r.reason);
     rawTexts.push(r.raw || '');
@@ -134,6 +137,11 @@ async function gradeSamples({ task, response, rubric, model, samples = 5, timeou
     model_id: model,
     rubric_hash: rubricHash(rubric),
     attempts: attemptsTotal,
+    // v0.4: the measurement overhead of grading this one case — the SUM over all
+    // N judge calls. Recorded in the receipt as the case's `judge_usage` and
+    // EXCLUDED from every skill-value figure (lib/value.js): it is a cost we
+    // impose to measure, not a cost of running the skill.
+    usage: sumUsage(usages),
   };
 }