driftproof 0.5.0 → 0.7.1

package/README.md

@@ -1,7 +1,7 @@
 <!-- SPDX-License-Identifier: Apache-2.0 -->
 # Driftproof
 
-**Continuous, model-version-bound verification of agent skills.**
+**A dated proof that this skill, this hash, this model, still helps.**
 
 [![driftproof](https://img.shields.io/endpoint?url=https://driftproofhq.com/badges/commit-message-conventions.json)](https://driftproofhq.com)
 &nbsp;— live badge for the bundled `commit-message-conventions` example, generated from its own receipt.
@@ -9,15 +9,17 @@
 Driftproof is an open **receipt spec** plus a **runner** that measures whether an
 agent skill actually helps — by running the skill's eval suite **with** and
 **without** the skill on a named model version, judging each case several times to
-get a confidence band, and emitting a signed, dated **receipt**. Diff two receipts
+get a confidence band, and emitting a **hash-verified**, dated **receipt**. Diff two receipts
 across model releases and you get a **drift report**.
 
 Driftproof **consumes** the [`agentskills.io/evals`](https://agentskills.io) eval
 format; it does not invent its own.
 
-📊 **Four published reports** (each re-derived from committed receipts, nothing
-hand-entered), spanning three report types that share one band-based, floor-gated
-verdict rule and differ only in what moves underneath the skill:
+📊 **Seven published reports** (each re-derived from committed receipts, nothing
+hand-entered), spanning six published report types, the newest being instrument
+re-measurement. All seven share one band-based, floor-gated verdict rule and
+differ in what moves underneath the skill — or, in the value report, in which
+axes are measured; or, in the newest, in the instrument itself:
 
 - **[Report #001](https://driftproofhq.com/reports/001/)** — *release drift*: ten
   public agent skills across a current-vs-previous Sonnet release; 9 of 10 moved
@@ -31,10 +33,41 @@ verdict rule and differ only in what moves underneath the skill:
   `claude-opus-5` (flagship) vs `claude-fable-5` (frontier tier);
   3 durable, 5 tier-dependent, 0 regressions, 2 no effect — encoded expertise
   survives the frontier tier.
+- **[Report #005](https://driftproofhq.com/reports/005/)** — *value*: what a skill
+  *costs* to run, on three axes (accuracy, cost, latency); the same ten suites on
+  three substrates (`claude-sonnet-5`, `claude-fable-5`, `gpt-5.6-sol`) —
+  14 of 30 cells cleared the floor on aggregate: 10 carry a price and 4 report a
+  saving instead, having improved quality while reducing cost.
+  *Three of those cells carry an amendment (v1.1, applied when Report #006
+  published): their lifts rest on single-draw baselines since shown
+  unstable. Cause-agnostic, no corrected figures offered, and the cost-driver and
+  substrate-disagreement findings below are unaffected.*
+- **[Report #006](https://driftproofhq.com/reports/006/)** — *revision drift*: the pinned skill revision against the one
+  upstream ships today, on a held substrate. **The reuse premise was tested and
+  refused: 3 of 3 cells returned no verdict**, each blocked by its own baseline
+  control. A 120-call probe found generation-level sampling noise 3.2× and 7.5×
+  larger than the judge-level noise this instrument actually samples — enough to
+  account for every gap the controls saw without any other cause being
+  established — and the receipt spec gains generation sampling as a result. **No
+  cause is asserted**; the control proves non-reproduction and cannot say why.
+  *The tally a refusal carries: 3 cells, 0 measured, 3 refused.*
+- **[Report #007](https://driftproofhq.com/reports/007/)** — *instrument
+  re-measurement*: the three cells Report #005 published for these skills, run
+  again with the generation sampled adaptively instead of once and with the call
+  timeout the surface policy declares. **No cell separates**: every lift is
+  smaller than its own band, and the 21 cases read 3 improved, 0 regressed,
+  18 no effect, 0 not measured. Five of six comparisons against the archive were
+  **refused** on their baseline control. The instrument defect it reports is its
+  own: a declared 300 s timeout had been shadowed by a `120000` literal since
+  2026-07-27, and the truncated run it caused measured *less* variance than the
+  clean re-run, which is the direction that flatters an instrument. Both runs are
+  published, the broken one as evidence. Amends #005 to v1.2 and #006 to v1.1.
 
 ✍️ The launch essay, **[Three model releases later: what actually happens to agent
-skills](https://driftproofhq.com/writing/three-releases/)**, reads the first three
-reports together.
+skills](https://driftproofhq.com/writing/three-releases/)**, reads all seven reports
+together: what moves underneath a skill, what the skill costs to run, and what a
+corrected instrument did to three published results. Revised 2026-09-01; every
+figure in it is gate-checked against the report page it cites.
 
 ## Why
 
@@ -46,7 +79,7 @@ re-checks.
 
 **Verdicts age because the substrate moves.** Driftproof exists to keep the
 verdict current: cheap, repeatable, hash-stamped measurements bound to a specific
-model version, so "does this skill still help?" has a dated, reproducible answer
+model version, so "does this skill still help?" has a dated, verifiable answer
 instead of a stale one.
 
 The hard part isn't running an eval once — it's making the number **credible
@@ -55,6 +88,15 @@ by more than the drift you're trying to detect. Driftproof's answer is to **samp
 the judge and report confidence bands**, and to **only claim a regression when the
 bands don't overlap**. A tool that cries wolf is worse than no tool.
 
+**A verdict without a price is half an answer.** The same receipts price the
+marginal cost of a skill firing, and Report #005 found the dominant cost driver is
+not the skill's own text but the input it causes the model to pull in: across those
+30 cells the input delta tracks cost at `r = +0.92` while the skill's own length
+tracks it at only `r = +0.33`, and one 738-token skill drew 34× its own size in
+extra input. Identical token deltas also price very differently across substrates —
+the same skill at near-identical deltas costs 3.3× more on `claude-fable-5` than on
+`claude-sonnet-5`, which is exactly their input-rate ratio in the frozen snapshot.
+
 ## Quickstart — receipt for your own skill in ~10 minutes
 
 You need Node ≥ 22 and an `ANTHROPIC_API_KEY`.
@@ -70,8 +112,10 @@ npx driftproof init my-skill
 export CLAUDE_PROVIDER=api
 read -rsp "Anthropic API key: " ANTHROPIC_API_KEY && export ANTHROPIC_API_KEY
 
-# 4. Run the suite with a hard $2 budget (the run is refused up front if it would exceed it)
-npx driftproof run my-skill --models claude-haiku-4-5 --max-usd 2
+# 4. Run the suite. The shipped defaults (DEV_MAX_USD / DEV_MAX_CALLS in config.js)
+#    refuse the run up front if the projection exceeds either; --max-usd and
+#    --max-calls override them.
+npx driftproof run my-skill --models claude-haiku-4-5
 
 # 5. Read the receipt + human summary written to ./receipts/
 cat receipts/*.summary.md
@@ -86,6 +130,15 @@ bands don't overlap. The **effect floor** (0.05, one judge quantization step) is
 minimum real move required before a change counts as more than noise — band
 separation *plus* a floor-sized delta, never either alone.
 
+**What the band does not cover.** A verdict rests on **one generation draw per
+arm**: the band is the spread of the *judge* re-scoring that single response, not
+the spread of the model writing a different one. Report #006 measured the second
+directly and found it larger — draw-to-draw spread up to **sd 0.186** on the 0–1
+scale, against judge-level noise several times smaller. So treat a surprising
+single-run verdict as **provisional and worth re-running** before you act on it.
+Generation sampling lands in the next receipt spec; until it does, this is a
+limit of the instrument, stated rather than implied.
+
 ### Install
 
 ```bash
@@ -148,11 +201,16 @@ The surface and judge settings used are recorded in every receipt.
 
 ### Cost guard
 
-Sampling multiplies calls. Each case costs `2 + 2 × samples` model calls
-(two generations, each judged `samples` times) — 12 calls per case at the default
-`--samples 5`. Driftproof **projects the whole run up front, prints the count, and
-refuses before spending anything** if it would exceed `--max-calls` (default 200).
-The default model list is `haiku` only.
+Sampling multiplies calls on two axes, and the second one is easy to miss. Each
+case costs `SAMPLING.max × (2 + 2 × samples)` model calls: two arms, each **drawn
+up to `SAMPLING.max` times** (generation sampling, receipt spec v0.5), and every
+draw judged `samples` times. Read at a single draw, that formula understates a run
+by the whole draw factor — which is how the shipped caps came to sit a release
+behind the estimator. Driftproof **projects the whole run up front, prints the
+count, and refuses before spending anything** if the projection exceeds the
+per-model call cap (`DEV_MAX_CALLS`) or the dollar budget (`DEV_MAX_USD`) — both
+declared with their derivation in [`config.js`](config.js), both overridable with
+`--max-calls` / `--max-usd`. The default model list is `haiku` only.
 
 ## Receipt anatomy
 
@@ -162,7 +220,7 @@ A receipt is the unit of evidence — one JSON document conforming to
 
 ```jsonc
 {
-  "schema_version": "0.4",
+  "schema_version": "0.5",
   "skill":  { "name": "commit-message-conventions", "version": "0.2.0",
               "content_hash": "…sha256 over SKILL.md + bundled files…" },
   "suite":  { "format": "agentskills.io/evals", "suite_hash": "…", "case_count": 10 },
@@ -171,7 +229,7 @@ A receipt is the unit of evidence — one JSON document conforming to
     "model_release_date": "2025-10-01",
     "provider": "anthropic",
     "surface": "claude-cli",
-    "runner_version": "0.5.0",
+    "runner_version": "0.7.1",
     "date_utc": "2026-07-27T…Z",
     "registry": "registered",
     "transcripts": "hashes-only",
@@ -194,6 +252,12 @@ A receipt is the unit of evidence — one JSON document conforming to
   },
   "comparison": { "with_skill_score": 0.81, "baseline_score": 0.42,
                   "delta": 0.39, "delta_uncertainty": 0.036 },
+  // v0.4 economics, all derived and never composited into one score: "run.pricing_snapshot"
+  // freezes the rates; each case carries "usage" and a separate "judge_usage"; "economics"
+  // holds basis, surface, with_skill/baseline (call_count, mean_input_tokens,
+  // mean_output_tokens, mean_cost_usd_per_call, median_wall_ms + p25/p75/IQR),
+  // skill_incremental_cost_usd_per_call, skill_incremental_cost_usd_per_1k_calls,
+  // output_tokens_delta, median_wall_ms_delta, judge_excluded (const true), judge_overhead.
   "verification_level": "TESTED",
   "receipt_hash": "…sha256 of the canonical receipt with this field removed…"
 }
@@ -209,6 +273,12 @@ Key ideas:
 - **`delta_uncertainty`** is the combined band on the with-skill-vs-baseline lift.
 - **`verification_level`** uses the community lattice: `UNVERIFIED` / `DECLARED` /
   `TESTED` (Driftproof emits `TESTED`). `FORMAL` is reserved.
+- **`economics`** is *derived*, never a second measurement: the token delta is the
+  durable fact, and the dollars are exactly those tokens at the rates frozen into
+  `run.pricing_snapshot`, so a receipt keeps its meaning after a vendor reprices.
+  Judge cost is recorded apart as `judge_usage` and excluded from every skill-value
+  figure (`judge_excluded` is `const true`) — measuring the skill is our cost, not
+  the skill's.
 - **`receipt_hash`** is a self-hash for tamper-evidence (integrity, not yet a key
   signature — see the spec's open questions).
 
@@ -219,7 +289,7 @@ receipts. The rule that keeps it honest: a **regression** (or improvement) is
 claimed **only when the two bands do not overlap**. Overlapping bands are reported
 as **within noise** and never counted as a regression.
 
-## Continuous verification in CI (GitHub Action + badge)
+## Verification in CI (GitHub Action + badge)
 
 Wire drift detection into a repo so the skill is re-checked on every push and when
 the model underneath it changes.
@@ -233,12 +303,13 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
-      - uses: driftproofhq/driftproof@v0.5.0
+      - uses: driftproofhq/driftproof@v0.7.1
         with:
           skill-dir: skills/my-skill
           models: claude-haiku-4-5
-          max-usd: '2'
           api-key: ${{ secrets.ANTHROPIC_API_KEY }}
+          # max-usd:   <n>    # override the dollar budget (default: DEV_MAX_USD in config.js)
+          # max-calls: <n>    # override the per-model call cap (default: DEV_MAX_CALLS)
           # fail-on-regression: 'true'   # (default) fail the job if the skill REGRESSED
 ```
 
@@ -256,7 +327,7 @@ runs).
 JSON object. Commit it somewhere public and point a shields endpoint URL at it:
 
 ```bash
-npx driftproof run skills/my-skill --models claude-haiku-4-5 --max-usd 2 --out receipts
+npx driftproof run skills/my-skill --models claude-haiku-4-5 --out receipts
 npx driftproof badge receipts/*.json --out badges/my-skill.json
 git add badges/my-skill.json && git commit -m "chore: driftproof badge"
 ```
@@ -272,11 +343,14 @@ site, so it reflects a real dated run, not a hand-set color.
 
 ## Reports
 
-Four reports are published, spanning three report types (see the roll at the top
-of this README, and [REPORT-STYLE.md](REPORT-STYLE.md) for the shared chrome
-every report inherits). Each report and every verdict in it are **re-derived from
-the receipts** committed under [`receipts/`](receipts/)
-(`receipts/report-001/` … `receipts/report-004/`) — nothing is hand-entered.
+Seven reports are published, spanning six report types. A report page lives at a
+draft path — `docs/reports/NNN-draft/` — until the publish sequence renames it, and
+`scripts/build-public.sh` excludes every `*-draft/` path from the published tree
+(see the roll at the top of this README, and
+[REPORT-STYLE.md](REPORT-STYLE.md) for the shared chrome every report inherits).
+Each report and every verdict in it are **re-derived from the receipts** committed
+under [`receipts/`](receipts/)
+(`receipts/report-001/` … `receipts/report-006/`) — nothing is hand-entered.
 
 Driftproof does **not** commit third-party skill content. Each `SKILL.md` is
 fetched at run time from a pinned commit and verified by sha256 against
@@ -290,9 +364,10 @@ node scripts/run-report-001.js --concurrency 5   # run both models × with/basel
 node scripts/build-report-001.js                 # re-derive the report from the receipts
 ```
 
-Reports #002–#004 have their own runners
-(`scripts/prepare-report-00N.js`) following the same
-fetch → run → re-derive shape.
+Reports #002–#005 have their own runners
+(`scripts/prepare-report-00N.js` — Report #005's is
+[`scripts/prepare-report-005.js`](scripts/prepare-report-005.js)) following the
+same fetch → run → re-derive shape.
 
 **Model-release triggers are live**: `scripts/release-watch.js` (keyless — it
 reads the public models registry) notices a new model release, re-runs the

package/bin/driftproof

@@ -4,11 +4,12 @@
 
 const fs = require('fs');
 const path = require('path');
-const { PROJECT_NAME, RUNNER_VERSION, DEFAULT_JUDGE_SAMPLES, DEV_MAX_USD } = require('../config');
+const { PROJECT_NAME, RUNNER_VERSION, DEFAULT_JUDGE_SAMPLES, DEV_MAX_USD, DEV_MAX_CALLS } = require('../config');
 const { loadSkill } = require('../lib/skill');
 const { runSkillOnModel, summarizeReceipt, projectCalls } = require('../lib/run');
+const { SAMPLING } = require('../lib/sampling');
 const { validateReceipt, verifyReceiptHash } = require('../lib/receipt');
-const { buildDriftReport } = require('../lib/diff');
+const { buildDriftReport, revisionPairProblem } = require('../lib/diff');
 const { surfaceForModel, isSubscriptionSurface, resolveModel } = require('../lib/provider');
 const { estimateRunCostUSD, BudgetTracker } = require('../lib/cost');
 const { registryStatus } = require('../lib/models');
@@ -76,7 +77,7 @@ USAGE
   ${PROJECT_NAME} run <skill-dir> [--models a,b] [--samples N] [--max-cases N] [--max-calls N]
                         [--judge-model M] [--concurrency N] [--max-usd N]
                         [--keep-transcripts] [--out DIR]
-  ${PROJECT_NAME} diff <receiptA.json> <receiptB.json> [--out FILE]
+  ${PROJECT_NAME} diff <receiptA.json> <receiptB.json> [--out FILE] [--mode release|revision]
   ${PROJECT_NAME} validate <receipt.json>
   ${PROJECT_NAME} badge <receipt.json> [--out FILE] [--github-output]
   ${PROJECT_NAME} import <results.json> --from agent-skills-eval|skillgrade [--out DIR]
@@ -91,8 +92,12 @@ ENV
 NOTES
   - Default models list is 'haiku' only (cheap dev default).
   - Sampled judging: each case does 2 generations + 2×samples judge calls.
-    Default --samples 5 → 12 calls/case; --samples 1 disables bands.
-  - --max-calls is a hard per-run call cap (default 200). The whole run is
+    Default --samples 5 → 12 calls/case; --samples 1 disables bands AND the
+    variance ratio: with one judge sample the per-draw judge spread is zero by
+    construction, so the receipt records variance_ratio: null with
+    variance_ratio_unavailable: "single_judge_sample". Legal, but a run whose
+    ratio you intend to publish needs --samples 2 or more.
+  - --max-calls is a hard per-run call cap (default ${DEV_MAX_CALLS}). The whole run is
     projected up front and REFUSED before any call if it would exceed the cap.
   - --max-usd is a hard dollar budget (default ${DEV_MAX_USD} for dev runs). The projected
     cost is printed up front and the run is REFUSED on BOTH surfaces if it would
@@ -126,7 +131,7 @@ async function cmdRun(positional, flags) {
   const rc = loadRc(skillDir);
   const models = String(flags.models || rc.models || 'haiku').split(',').map((s) => s.trim()).filter(Boolean);
   const maxCases = flags['max-cases'] ? parseInt(flags['max-cases'], 10) : (rc.max_cases != null ? parseInt(rc.max_cases, 10) : null);
-  const maxCalls = flags['max-calls'] ? parseInt(flags['max-calls'], 10) : (rc.max_calls != null ? parseInt(rc.max_calls, 10) : 200);
+  const maxCalls = flags['max-calls'] ? parseInt(flags['max-calls'], 10) : (rc.max_calls != null ? parseInt(rc.max_calls, 10) : DEV_MAX_CALLS);
   const samples = flags.samples ? parseInt(flags.samples, 10) : (rc.samples != null ? parseInt(rc.samples, 10) : DEFAULT_JUDGE_SAMPLES);
   const judgeModel = flags['judge-model'] || rc.judge_model || null;
   const concurrency = flags.concurrency ? parseInt(flags.concurrency, 10) : (rc.concurrency != null ? parseInt(rc.concurrency, 10) : 1);
@@ -137,14 +142,20 @@ async function cmdRun(positional, flags) {
 
   const skill = loadSkill(skillDir);
   const nCases = maxCases ? Math.min(maxCases, skill.suite.caseCount) : skill.suite.caseCount;
-  const perModelCalls = projectCalls(nCases, samples);
+  // v0.5 draws the generation up to SAMPLING.max times per arm, so BOTH the
+  // printed projection and the dollar guard below must be scaled by it. Left at
+  // draws=1 the guard would admit a run costing up to ten times its own
+  // projection — the display would say one number and the runner would refuse
+  // quoting another. Conservative on purpose: refusing a run that would have fit
+  // is recoverable, overspending is not.
+  const perModelCalls = projectCalls(nCases, samples, SAMPLING.max);
   const totalProjected = perModelCalls * models.length;
 
   // Dollar cost guard (Week 3): project the METERED USD cost up front. On the
   // `api` surface this is real spend, so we refuse if it would exceed --max-usd.
   // On `claude-cli` the metered spend is $0 (subscription); the figure is the
   // hypothetical "if run on the metered API" cost — printed, never blocks.
-  const cost = estimateRunCostUSD({ caseCount: nCases, samples, models: models.map((m) => require('../lib/provider').resolveModel(m)), judgeModel: judgeModel || 'haiku' });
+  const cost = estimateRunCostUSD({ caseCount: nCases, draws: SAMPLING.max, samples, models: models.map((m) => require('../lib/provider').resolveModel(m)), judgeModel: judgeModel || 'haiku' });
   // Surface is per-model now (a run may mix an Anthropic and an OpenAI target).
   const surfaces = [...new Set(models.map((m) => surfaceForModel(m)))];
   const surface = surfaces.join(', ');
@@ -250,9 +261,32 @@ function cmdDiff(positional, flags) {
     if (!verifyReceiptHash(r)) console.error(`  ⚠ ${path.basename(p)}: receipt_hash does not verify (tampered or hand-edited)`);
   }
 
-  const labelA = dateStamp(a.run.date_utc);
-  const labelB = dateStamp(b.run.date_utc);
-  const { markdown } = buildDriftReport(a, b, { labelA, labelB });
+  // --mode revision inverts the axis: the skill text is the variable under test
+  // and the substrate is the control. The fields release drift merely warns about
+  // are preconditions here, so a pair that is not a revision pair is REFUSED
+  // (exit 6) rather than rendered with a caveat nobody reads. A differing model
+  // would be release drift wearing a revision label — the one confound this mode
+  // exists to exclude — and an EQUAL content_hash has no revision to measure.
+  const mode = flags.mode || 'release';
+  if (mode !== 'release' && mode !== 'revision') {
+    console.error(`unknown --mode "${mode}" — supported: release, revision`);
+    process.exit(2);
+  }
+  if (mode === 'revision') {
+    const problem = revisionPairProblem(a, b);
+    if (problem) {
+      const why = problem === 'skill.content_hash'
+        ? 'the two receipts carry the SAME skill.content_hash — there is no revision between them to measure'
+        : `${problem} differs between the two receipts — revision drift requires the substrate to be held fixed, and a differing ${problem} would confound the revision with release drift`;
+      console.error(`  ✗ REFUSED (--mode revision): ${why}.`);
+      console.error(`    Compare these two with the default release mode, or supply a pair that differs only in skill.content_hash.`);
+      process.exit(6);
+    }
+  }
+
+  const labelA = mode === 'revision' ? `pinned (${dateStamp(a.run.date_utc)})` : dateStamp(a.run.date_utc);
+  const labelB = mode === 'revision' ? `current (${dateStamp(b.run.date_utc)})` : dateStamp(b.run.date_utc);
+  const { markdown } = buildDriftReport(a, b, { labelA, labelB, mode });
 
   if (flags.out) {
     fs.writeFileSync(path.resolve(flags.out), markdown);
@@ -292,7 +326,7 @@ function cmdInit(positional) {
   console.log(`\nNext:
   1. Edit ${rel(path.join(dir, 'SKILL.md'))} with your skill's instructions.
   2. Edit ${rel(path.join(dir, 'evals', 'evals.json'))} — replace the 3 example cases (rubrics anchored at 0.80).
-  3. driftproof run ${rel(dir)} --models claude-haiku-4-5 --max-usd 2
+  3. driftproof run ${rel(dir)} --models claude-haiku-4-5
 See AUTHORING.md for how to write a fair suite.`);
 }
 

package/config.js

@@ -9,7 +9,7 @@ const PROJECT_NAME = 'driftproof';
 // Bumped whenever the runner's behaviour or receipt-generation semantics change
 // in a way that could affect results. Recorded into every receipt as
 // run.runner_version so a receipt is reproducible against a known engine.
-const RUNNER_VERSION = '0.5.0';
+const RUNNER_VERSION = '0.7.1';
 
 // The eval format we CONSUME (we deliberately do not invent our own).
 const SUITE_FORMAT = 'agentskills.io/evals';
@@ -29,7 +29,7 @@ const SUITE_FORMAT = 'agentskills.io/evals';
 //        at run time so derived dollars stay reproducible), and the derived
 //        `economics` block. v0.3.1 is frozen as receipt.v0.3.1.schema.json;
 //        v0.1/v0.2/v0.3/v0.3.1 receipts all still load.
-const RECEIPT_SCHEMA_VERSION = '0.4';
+const RECEIPT_SCHEMA_VERSION = '0.5';
 
 // Hard USD budget defaults per entry point (Week 4). --max-usd overrides any of
 // these. The projection is refused before any call if it exceeds the cap, on
@@ -37,8 +37,33 @@ const RECEIPT_SCHEMA_VERSION = '0.4';
 //   dev     — an interactive `driftproof run`
 //   report  — a full Report-#001-style suite run (scripts/run-report-001.js)
 //   trigger — a release-trigger-initiated prepare-report run
-const DEV_MAX_USD = 2;
-const REPORT_MAX_USD = 40;
+// Recalibrated 2026-09-01 (spec 018). BOTH literals below were set against a
+// draws=1 projection and were never rescaled when spec 014 (5e08ba2) made the
+// runner draw the generation up to SAMPLING.max times per arm and spec 016 made
+// estimateRunCostUSD require the draw factor. The projection grew 10x; these did
+// not, so `npx driftproof run` refused every suite of two or more cases and the
+// v0.7.0 Action self-test aborted with exit 3. REPORT_MAX_USD was raised 40->300
+// on 2026-08-31 for exactly this reason; that pass missed the dev path, which is
+// the one the Action and every npx user take.
+//
+// DERIVATION — the headroom the pre-sampling defaults carried is preserved, not
+// widened. Bundled 10-case example, 5 judge samples, per-case factor 2+2*5 = 12:
+//   draws=1  ->  120 calls, $0.3525   headroom 200/120 = 1.67x,  $2/$0.3525 = 5.67x
+//   draws=10 -> 1200 calls, $3.5250   1200 * 1.67 = 2000,  $3.5250 * 5.67 = $20.00
+// A 2000-call cap under draws=10 is exactly as tight as 200 was under draws=1.
+// LITERALS ON PURPOSE (spec 018 AC-2): a default derived from the suite in hand
+// can never fire, which retires the guard instead of recalibrating it.
+const DEV_MAX_USD = 20;
+const DEV_MAX_CALLS = 2000;
+// Raised from $40 on 2026-08-31 by the repository owner, in daylight, recorded in
+// DECISIONS. The run did not get more expensive — the projection got honest:
+// spec 016 made estimateRunCostUSD require a draw factor, and v0.5 has drawn the
+// generation up to SAMPLING.max times per arm since spec 014, so the old cap had
+// been passing on a figure up to ten times too small. Set above the ceiling this
+// instrument can currently reach (#007's measured basis at the ceiling is
+// $268.83) rather than just above today's staging figure, because a cap tripped
+// by the next honest run teaches everyone to nudge it.
+const REPORT_MAX_USD = 300;
 const TRIGGER_MAX_USD = 25;
 
 // Default number of judge samples per case in sampled mode. Sampling is what
@@ -56,6 +81,24 @@ const DEFAULT_JUDGE_SAMPLES = 5;
 // spec/RECEIPT.md § "Drift verdict rule" and the report methodology.
 const EFFECT_FLOOR = 0.05;
 
+// ── generation sampling (receipt spec v0.5) ─────────────────────────────────
+//
+// Report #006 measured across-draw spread at sd 0.186 and 0.183 while the
+// instrument sampled only the judge. These are the policy that samples the
+// other axis. Constants, not literals in the runner: lib/sampling.js reads
+// them, so moving one here moves the policy, and the gate asserts that by
+// value rather than by grepping for a name.
+//
+// MIN is 3 because two draws give an sd that is barely a measurement and one
+// gives none at all. MAX is 10: #006's probe used 20 by hand and found the
+// shape at well under half of that, and a per-case ceiling bounds the spend.
+// The SD threshold is the effect floor — a spread wider than the smallest move
+// the verdict rule will call real is exactly when more draws are owed.
+const GENERATION_SAMPLES_MIN = 3;
+const GENERATION_SAMPLES_MAX = 10;
+const GENERATION_SD_THRESHOLD = EFFECT_FLOOR;
+const GENERATION_STABILITY_EPS = 0.01;
+
 // Report #002 is the first CROSS-PROVIDER report: the same suites, the same fixed
 // Haiku judge, run on two substrates — a Claude flagship and a GPT flagship. The
 // GPT flagship is a config constant (not hard-coded across scripts) so a future
@@ -90,7 +133,8 @@ const REPORT_005_JUDGE_MODEL = 'claude-haiku-4-5';
 
 module.exports = {
   PROJECT_NAME, RUNNER_VERSION, SUITE_FORMAT, RECEIPT_SCHEMA_VERSION, DEFAULT_JUDGE_SAMPLES,
-  EFFECT_FLOOR, DEV_MAX_USD, REPORT_MAX_USD, TRIGGER_MAX_USD,
+  EFFECT_FLOOR, DEV_MAX_USD, DEV_MAX_CALLS, REPORT_MAX_USD, TRIGGER_MAX_USD,
+  GENERATION_SAMPLES_MIN, GENERATION_SAMPLES_MAX, GENERATION_SD_THRESHOLD, GENERATION_STABILITY_EPS,
   REPORT_002_CLAUDE_MODEL, REPORT_002_GPT_MODEL, REPORT_002_JUDGE_MODEL,
   REPORT_003_NEW_MODEL, REPORT_003_OLD_MODEL, REPORT_003_JUDGE_MODEL,
   REPORT_004_BASE_MODEL, REPORT_004_FRONTIER_MODEL, REPORT_004_JUDGE_MODEL,

package/lib/canary.js

@@ -0,0 +1,27 @@
+// SPDX-License-Identifier: Apache-2.0
+'use strict';
+
+// Suite canary — receipt spec v0.5.
+//
+// Terminal-Bench 4.0 records a canary string so that a benchmark appearing in
+// training data is detectable. Adopted, translated: our unit of publication is
+// the SUITE, so the canary is per suite, derived from the suite's identity and
+// its case ids rather than randomly assigned, so the same suite yields the same
+// canary on every machine and no registry has to be kept.
+//
+// It is a detection aid, not a control: a canary tells you a suite leaked, it
+// does not stop the leak, and it cannot prove the absence of one.
+
+const crypto = require('crypto');
+
+const NAMESPACE = 'driftproof/suite-canary/v1';
+
+function suiteCanary(suite) {
+  const id = String((suite && (suite.id || suite.name)) || '');
+  const caseIds = ((suite && suite.cases) || []).map((c) => String((c && c.id) || '')).sort();
+  const h = crypto.createHash('sha256').update(`${NAMESPACE}\n${id}\n${caseIds.join('\n')}`).digest('hex');
+  // Formatted as a GUID so it is recognisable as one in a corpus scan.
+  return [h.slice(0, 8), h.slice(8, 12), h.slice(12, 16), h.slice(16, 20), h.slice(20, 32)].join('-');
+}
+
+module.exports = { suiteCanary, NAMESPACE };

package/lib/cost.js

@@ -58,7 +58,25 @@ function perCallCostUSD(modelId, kind) {
 // generated on each target model, judged `samples` times each on `judgeModel`.
 //
 //   returns { totalUSD, perModel: [{ model, usd }], judgeUSD, genUSD, assumptions }
-function estimateRunCostUSD({ caseCount, samples, models, judgeModel }) {
+// THE DRAW FACTOR IS REQUIRED AND HAS NO DEFAULT (spec 016 AC-7).
+//
+// v0.5 draws the generation up to `SAMPLING.max` times per arm, so a run costs
+// its one-draw estimate times the number of draws. `projectCalls` learned this in
+// spec 014 with a `draws` argument DEFAULTING TO 1 "so every existing caller
+// projects exactly what it projected before" — and every existing caller then
+// went on projecting a tenth of the run, silently, for two more loops. Spec 015
+// corrected the CALL projection in three report scripts and left the DOLLAR
+// projection at one draw everywhere, which is the figure a human actually reads
+// before authorising a paid run.
+//
+// A default is what made that invisible, so there is none. Omitting the factor
+// throws, which turns a silent understatement into a loud stop — and every call
+// site has to say what it means, including the ones that legitimately mean 1.
+function estimateRunCostUSD({ caseCount, samples, models, judgeModel, draws }) {
+  if (!Number.isFinite(draws) || draws < 1) {
+    throw new Error('estimateRunCostUSD: `draws` is required and must be >= 1 — pass SAMPLING.max to project a v0.5 run, or 1 to price a single draw deliberately. It is not defaulted, because a default is how the dollar projection stayed at one draw through two loops.');
+  }
+  caseCount = caseCount * draws;
   const judgePrice = priceFor(judgeModel);
   const perModel = [];
   let judgeUSD = 0;
@@ -80,7 +98,7 @@ function estimateRunCostUSD({ caseCount, samples, models, judgeModel }) {
     perModel,
     judgeUSD: round4(judgeUSD),
     genUSD: round4(genUSD),
-    assumptions: { tokens: TOKENS, judgeModel, note: 'rough upper-bound estimate; registry per-MTok pricing; not measured with count_tokens' },
+    assumptions: { tokens: TOKENS, judgeModel, draws, note: 'rough upper-bound estimate; registry per-MTok pricing; not measured with count_tokens' },
   };
 }