driftproof 0.3.0 → 0.5.0

package/spec/RECEIPT.md

@@ -1,5 +1,5 @@
 <!-- SPDX-License-Identifier: Apache-2.0 -->
-# Driftproof receipt — spec v0.3
+# Driftproof receipt — spec v0.4
 
 A **receipt** is a signed, dated record of running one agent skill's eval suite
 **with** and **without** the skill on one model version, with the judge **sampled**
@@ -11,13 +11,117 @@ The machine-readable contract is [`receipt.schema.json`](./receipt.schema.json)
 (JSON Schema, draft 2020-12). This document is the human companion. Where they
 disagree, the schema wins.
 
-**Versioning.** The current schema is v0.3
-([`receipt.schema.json`](./receipt.schema.json)). Prior schemas are kept as
+**Versioning.** The current schema is v0.4
+([`receipt.schema.json`](./receipt.schema.json)). Prior schemas are kept frozen as
+[`receipt.v0.3.1.schema.json`](./receipt.v0.3.1.schema.json),
+[`receipt.v0.3.schema.json`](./receipt.v0.3.schema.json),
 [`receipt.v0.2.schema.json`](./receipt.v0.2.schema.json) and
 [`receipt.v0.1.schema.json`](./receipt.v0.1.schema.json); the validator picks the
-schema by the receipt's own `schema_version`, so v0.1 and v0.2 receipts still load
-and validate. v0.3 is an **additive** bump — every field it adds is required in a
-fresh run, but older receipts are read unchanged against their own schema.
+schema by the receipt's own `schema_version`, so v0.1, v0.2, v0.3 and v0.3.1
+receipts still load and validate — including every receipt behind the four
+published reports, which the gate asserts on each run. v0.4 is an **additive**
+bump: everything it adds is optional, and no earlier receipt is invalidated.
+
+## What changed in v0.4 (economics — what a skill costs to run)
+
+Reports #001–#004 could say whether a skill still helps. They could not say what
+it costs, because the two CLI surfaces we run on were reporting token usage that
+the runner discarded. v0.4 captures it and derives the economics from it.
+
+- **`results.cases[].usage`** — the GENERATION call's usage for that (case, mode)
+  row: `{ input_tokens, output_tokens, cached_tokens, wall_ms }`. `input_tokens`
+  is normalized to the TOTAL input including any cached portion, because the
+  surfaces disagree natively (`claude -p` reports it excluding cache; `codex`
+  includes it). `cached_tokens` is `null` — never `0` — where a surface does not
+  report it. `wall_ms` is measured by the runner around the successful attempt,
+  so it means the same thing on every lane.
+- **`results.cases[].judge_usage`** — the summed usage of the N judge calls that
+  graded that row. This is **measurement overhead we impose, not a cost of
+  running the skill**, and it is excluded from every derived value figure. The
+  schema pins `economics.judge_excluded` to `const: true`, so a receipt cannot
+  claim otherwise.
+- **`run.pricing_snapshot`** — the registry prices **frozen at run time** for the
+  models this run touched. Every derived dollar figure is computed from this
+  snapshot and never from the live registry, so a published receipt does not
+  silently change meaning when a vendor cuts prices later.
+- **`economics`** — the derived block: per-arm mean cost per call, mean input and
+  output tokens, median `wall_ms` with its interquartile range; and the deltas
+  that matter — `skill_incremental_cost_usd_per_call`,
+  `skill_incremental_cost_usd_per_1k_calls`, `output_tokens_delta`,
+  `median_wall_ms_delta`. `basis` states `metered` or `metered-equivalent`
+  (subscription surfaces, where actual metered spend is $0).
+
+**Three axes, never combined.** Accuracy lift, cost, and latency are recorded and
+reported separately. There is deliberately no composite "value score": the axes
+have different units, different error bars, and different owners, so collapsing
+them would manufacture a number no reader could trace back to evidence. The
+presentation rules that follow from this — ratio framings gated on the effect
+floor, latency always carrying its disclosure — are in
+[`REPORT-STYLE.md`](../REPORT-STYLE.md) § "Value-axis presentation rules".
+
+**Read the incremental figures, not the absolute ones.** Both CLI surfaces
+prepend a large fixed harness preamble we do not control (observed ~25k input
+tokens on `claude-cli`, ~11k on `codex`). It inflates absolute per-call cost, but
+it is identical in the with-skill and baseline arms, so it cancels in every Δ.
+
+## Interop-additive revision (Phase 7 — receipts as an open format)
+
+The v0.3.1 schema gained an **additive interop revision** so receipts can be
+**imported** from neighboring eval tools with honest epistemics (see
+[`docs/interop.md`](../docs/interop.md) and the site's `/interop.html`):
+
+- `run.surface` and `run.judge.surface` gain **`"external"`** (the run happened
+  on another tool's harness); `run.transcripts` gains **`"none"`** (nothing
+  retained, not even hashes — only honest on an import).
+- New optional **`run.source`** — provenance of a converted receipt, e.g.
+  `"imported/agent-skills-eval"` or `"imported/skillgrade"`.
+- `skill.content_hash`, `suite.suite_hash`, and per-case `judge.rubric_hash` may
+  be **`null`**, per-case `generation_hash`/`judge_sample_hashes` may be
+  **omitted**, and `comparison.baseline_score`/`delta`/`delta_uncertainty` may be
+  **`null`** (a source tool with no baseline mode) — hashes and baselines are
+  **never fabricated**. `suite.format` may name a non-agentskills format (e.g.
+  `"skillgrade/eval.yaml"`).
+- **The TESTED tightening** (a top-level schema conditional) makes every one of
+  those relaxations available **only below `TESTED`**: a receipt claiming
+  `verification_level: "TESTED"` must still carry the full evidence chain —
+  content/suite hashes, per-case generation + judge-sample hashes, a
+  non-`external` surface, numeric comparison — exactly as before. No previously
+  issued receipt is invalidated, and `TESTED` keeps its meaning.
+- Structural consequence, enforced in code: **drift verdicts require `TESTED`
+  on both sides** — `diff` reports **NOT MEASURED** against a `DECLARED`
+  (imported) receipt, and a `DECLARED` receipt's badge reads *not measured*.
+
+## What changed from v0.3 (multi-provider — spec v0.3.1)
+
+- **`run.provider`** (required) — the two-axis provider the target model ran on:
+  `"anthropic"` or `"openai"` (from the registry `provider`, else inferred from the
+  id). Driftproof is now two-provider: the same suites and the same fixed judge,
+  run on Claude and GPT substrates.
+- **`run.surface`** gains the OpenAI lanes: the enum is now
+  `"api" | "claude-cli" | "openai-api" | "openai-cli"`. `openai-api` = a
+  Chat-Completions-compatible API (base_url configurable); `openai-cli` = the Codex
+  subscription surface (`codex exec`). `run.judge.surface` accepts the same set.
+- **`run.surface_overhead_note`** (optional) — present on the `openai-cli` surface:
+  states the fixed Codex base-instruction preamble (~12–15k input tokens per call)
+  that the harness prepends and does not control. The model id is set by us via
+  `-m` (it is not echoed in the Codex JSONL stream).
+- **Per-case `checks[]`** (optional) — deterministic post-check results, each
+  `{ name, kind, pass }` with `kind ∈ regex | contains | not_contains | min_length`.
+  Structural/regex assertions run on the model output **alongside** the judge and
+  reported as a **separate column** — **supplementary evidence only, never folded
+  into the `outcome`/band verdict.**
+- **`skill.tokens`** (optional) — the estimated token size of the skill's SKILL.md
+  (a coarse `chars/4` proxy, not a model tokenizer), used for the **value-per-token**
+  axis: `delta` per 1k skill tokens. Method documented on the methodology page.
+- **Per-case `case_status`** (optional; default `"ok"`) — `"failed_timeout"` marks a
+  case whose model/judge call persistently timed out after retries. Such a case is
+  **recorded WITHOUT fabricated samples/hashes** and is **excluded from the
+  aggregates** — a band is never invented from a case that did not complete. When
+  any case failed, the run is stamped **`run.status: "incomplete"`** (+
+  `run.failed_case_count`), and a drift/durability report **must exclude an
+  incomplete receipt from verdicts** (listing it honestly as "not measured").
+- These are additive; a v0.3 or earlier receipt reads unchanged against its own
+  frozen schema.
 
 ## Design goals
 
@@ -60,7 +164,7 @@ fresh run, but older receipts are read unchanged against their own schema.
 
 ## Fields
 
-### `schema_version` (string, required) — `"0.3"`.
+### `schema_version` (string, required) — `"0.3.1"`.
 
 ### `skill` (object, required)
 | field | type | notes |
@@ -68,6 +172,7 @@ fresh run, but older receipts are read unchanged against their own schema.
 | `name` | string | From SKILL.md front-matter or its H1. |
 | `version` | string | Skill's declared version. |
 | `content_hash` | sha256 hex | Over `SKILL.md` + every bundled file (the `evals/` dir is **excluded** — hashed separately as `suite_hash`), path + content, path-sorted. |
+| `tokens` | integer | **v0.3.1, optional.** Estimated SKILL.md token size (coarse `chars/4` proxy) for the value-per-token axis. |
 
 ### `suite` (object, required)
 | field | type | notes |
@@ -81,7 +186,9 @@ fresh run, but older receipts are read unchanged against their own schema.
 |---|---|---|
 | `model_id` | string | Canonical model id run on. |
 | `model_release_date` | ISO date or `null` | Release date **if known**, else `null`. |
-| `surface` | `"api"` \| `"claude-cli"` | `api` = Messages API; `claude-cli` = spawned `claude -p` (subscription). |
+| `provider` | `"anthropic"` \| `"openai"` | **v0.3.1.** The two-axis provider (registry `provider`, else inferred from the id). |
+| `surface` | `"api"` \| `"claude-cli"` \| `"openai-api"` \| `"openai-cli"` | `api` = Anthropic Messages API; `claude-cli` = spawned `claude -p` (subscription); `openai-api` = Chat-Completions-compatible API (base_url configurable); `openai-cli` = Codex subscription (`codex exec`). |
+| `surface_overhead_note` | string | **v0.3.1, optional.** On `openai-cli`: the fixed Codex base-instruction preamble (~12–15k input tokens/call) the harness prepends and does not control. |
 | `runner_version` | string | Runner version, for reproducibility. |
 | `date_utc` | ISO 8601 UTC | When the run finished. |
 | `registry` | `"registered"` \| `"unregistered"` | **v0.3.** Whether `model_id` resolved in the model registry (`config/models.json`). |
@@ -111,6 +218,7 @@ receipt says so.
   | `score` | number 0–1 | Alias of `mean` (kept for v0.1 readers). |
   | `threshold` | number or `null` | The case's pass threshold (or `null`). |
   | `reason` | string | One-line judge rationale (optional). |
+  | `checks` | array | **v0.3.1, optional.** Deterministic post-check results, each `{ name, kind, pass }` (`kind ∈ regex/contains/not_contains/min_length`). Run alongside the judge; a **separate column**, **not** folded into `outcome`/the band verdict. |
   | `judge` | object | `{ model_id, rubric_hash }` — who graded and a hash binding the grade to the exact rubric + judge system prompt. |
 - **`aggregates`** — `{ with_skill, baseline }`, each
   `{ case_count, pass_count, borderline_count?, mean_score, stddev }`. Here