@bglocation/tune-context 2.4.0 → 3.0.0

package/.claude-plugin/marketplace.json

@@ -10,7 +10,7 @@
       "name": "tune-context",
       "source": "./",
       "description": "Context-efficiency configurator for Claude Code: doctrine skills (token-efficiency, caveman, cdd, phase-workflow) plus a tune-context configurator that scaffolds a lean CLAUDE.md, subagents, settings and hooks.",
-      "version": "2.4.0",
+      "version": "3.0.0",
       "author": {
         "name": "Szymon Walczak",
         "url": "https://bglocation.dev"

package/.claude-plugin/plugin.json

@@ -2,7 +2,7 @@
   "name": "tune-context",
   "displayName": "tune-context",
   "description": "Context-efficiency configurator for Claude Code: doctrine skills (token-efficiency, caveman, cdd, phase-workflow) plus a tune-context configurator that scaffolds a lean CLAUDE.md, subagents, settings and hooks.",
-  "version": "2.4.0",
+  "version": "3.0.0",
   "author": {
     "name": "Szymon Walczak",
     "url": "https://bglocation.dev"

package/CHANGELOG.md

@@ -5,6 +5,199 @@ All notable changes to `@bglocation/tune-context` are documented here.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [3.0.0] — 2026-10-15
+
+EPIC-010 — the meter of the cost measurement comes from the same installed
+version as the switch, and every artifact names the code that computed it.
+**The major bump is for one deliberately breaking change** — the eval data
+contract (see Changed): rows without a meter stamp and `analyze@1` aggregates
+are refused. Everything else is additive.
+
+### Added
+
+- **`tune-context eval <step>`** (TASK-095) — `prompt-metrics`, `analyze`,
+  `verdict`, `report`, `adoption-report`, named after their modules, no aliases,
+  from any directory. Until now the meter existed only as `node eval/<file>.mjs`,
+  a relative path that resolves in a checkout of this repo and nowhere else, so
+  the switch (from `PATH`) and the meter (from `cwd`) could be two versions.
+  Everything after the step goes verbatim to the step's own parser:
+  `tune-context eval prompt-metrics --help` prints the step's usage. The two
+  doors print the same bytes, held to a baseline written before the refactor.
+- **`tune-context eval verdict`** (TASK-096) — the verdict as JSON on stdout (the
+  object the report embeds) and one line per gate on stderr, ending in the
+  finding. Exit 0 for every verdict, a withheld one included. `verdict` and
+  `report` read their input through one function, so the page cannot accept what
+  the verdict refuses.
+- **The meter stamp** (TASK-097) — rows carry `meter_version`, the aggregate
+  `analyzed_by`, the verdict `judged_by`: the bare release number for an npm
+  install, `<version>+<sha12>` for a checkout, `.dirty` for uncommitted changes in
+  `eval/ cli/ hooks/ bin/ package.json`, `+unknown` where git could not answer.
+  A version number alone would not do: under `npm link` a fix committed before
+  the release carries the number of the bug it fixes. `prompt-metrics` prints
+  `meter <stamp>` as the first line of its summary.
+- **A seventh verdict gate, `meter_mixed`** (frozen by the PO on 2026-10-15) — the
+  whole chain (paired rows, `analyzed_by`, `judged_by`) must carry exactly one
+  stamp that names code; `.dirty`, `+unknown`, a second stamp or a step with none
+  hold the verdict, and the reason names the step that brought the stranger.
+  The six earlier thresholds are untouched.
+- **`tune-context experiment archive --out <dir>`** (TASK-098) — the series'
+  transcripts (every session on an `exp/…` branch) and the ledger, copied out of
+  reach of Claude Code's retention, with `manifest.json` and `SHA256SUMS`
+  (`shasum -a 256 -c`). Append-only: a grown file replaces its copy, any other
+  change refuses the run. A destination inside a git work tree that git does not
+  ignore is refused — a transcript is the whole conversation.
+- **`--ledger <path>`** for `prompt-metrics` and `analyze`, so an archive
+  recounts as `branch+ledger` on any machine.
+
+### Changed
+
+- **The eval data contract — deliberately breaking** (EPIC-010, PO decision 4).
+  Schemas are now `tune-context/analyze@2`, `tune-context/verdict@2` and
+  `tune-context/verdict-thresholds@2`. `analyze` refuses a whole input containing
+  a single row without `meter_version` (exit 1, nothing on stdout, file and line
+  on stderr), and `verdict`/`report` refuse an `analyze@1` aggregate. Scripts
+  that feed rows or aggregates written by an earlier version get exit 1. That is
+  the point: unstamped data cannot say which meter computed it, and the way back
+  to it is recomputing the transcripts — from an `experiment archive`, with the
+  current version or with `npx -p @bglocation/tune-context@<version>`.
+- **The cost evaluation runbook and procedure** run the series from a registry
+  release (`npm link` only for an unreleased fix, from a clean commit), record
+  the meter stamp before the first pair, and archive after every arm.
+- **Skills** — `token-efficiency` and `tune-context` point at
+  `tune-context eval adoption-report` instead of `node eval/…`, which never
+  resolved outside this repo, and `tune-context` names
+  `tune-context eval prompt-metrics` as the source of token cost, where it said
+  only OTEL with a collector had it. Skills are copies: **re-run
+  `tune-context init` after upgrading** to get them.
+- **`adoption-report` points at `tune-context eval prompt-metrics`** (TASK-100) —
+  in `--help` and in the report's closing note on cost, instead of
+  `eval/prompt-metrics.mjs`, a path that resolves only in a checkout of this repo.
+  Likewise `analyze` — its `--help` and the last line of its summary — points at
+  `tune-context eval verdict` instead of a task number that is not in the package.
+
+### Fixed
+
+- **A reader that leaves early is a normal end for every eval step** — `report`
+  writing a long page, or `analyze` an aggregate of many tasks, into `| head`
+  ended in an `EPIPE` stacktrace; only `prompt-metrics` and `verdict` had a
+  guard. One guard now, called by each step.
+- **`report --out` into a directory that does not exist** is one line and exit 1,
+  not a stacktrace.
+
+## [2.5.0] — 2026-09-03
+
+### Fixed
+
+- **`prompt-metrics` charged 45% of all cache-read tokens twice** (found while
+  reviewing TASK-089). A transcript repeats entries verbatim: 11 296 of 36 463
+  local entries (31%) carry a `uuid` already written *earlier in the same file*,
+  6 739 of them assistant turns with a `usage` block, the copies differing only
+  in a cosmetic `slug`. The collector summed both. Local run total drops from
+  $4 566 to $2 323 — an inflation of **1.97×**, several times the effect
+  EPIC-009 is trying to measure. Repeats are now skipped (scoped to one file:
+  zero `uuid`s are shared between files) and counted on stderr, so the drop
+  explains itself rather than looking like a different tool.
+  **Every cost figure this project produced before 2026-08-02 is too high**;
+  `COST_EVALUATION_PROCEDURE.md` §11a carries the correction, including the
+  delegation baseline, which moves from ~0.7 to ~0.3 per session. The frozen
+  §10 thresholds are deliberately left alone — the correction makes the
+  delegation gate stricter relative to the norm, and refitting a threshold to
+  new data is the thing that section forbids.
+
+- **`prompt-metrics` billed one API response as many** (TASK-092, reported by a
+  Claude Code session working in a different project). One response reaches the
+  transcript as *several* entries — one per content block: thinking, text, each
+  `tool_use` — every one with its own `uuid` but carrying an **identical copy of
+  the same `usage` block**. The `uuid` dedup above cannot see this class by
+  construction: those uuids differ by design. Over 19 transcripts, 11 295
+  entries for 5 954 responses. `usage` is now keyed to the RESPONSE
+  (`message.id`, falling back to `requestId`) and the extra entries are counted
+  on stderr. On the same frozen transcripts the run total drops **$1 372.69 →
+  $679.35 (2.02×)**, on top of the 1.97× above.
+  The inflation is **not flat** across token classes (cache-read 1.82×, output
+  2.17×, cache-write 2.39×, input 2.45×) and does **not** cancel in a ratio of
+  two arms: the factor *is* the number of content blocks per turn — how many
+  tools the model reached for, which is the variable the doctrine sets out to
+  move. Per-session spread 1.66×–2.24×.
+  The guard sits **after** the `tool_use` loop, because those blocks are *not*
+  duplicated (6 403 blocks, 6 403 distinct ids): skipping whole entries would
+  have dropped ~2/3 of every tool call and gutted the adoption half of the
+  analysis. All three turn counters are guarded — `api_turns`, `sidechain_turns`
+  and the unattributed bucket — and an entry with neither `message.id` nor
+  `requestId` is **billed** and reported, never assumed unique.
+  **Every cost figure this project produced before 2026-09-03 is too high
+  again**, including the frozen EPIC-009 rows under `docs/` and the TASK-090
+  verdict computed from them. The skew correlates with the arm, so that verdict
+  rests on numbers that have to be recomputed before it means anything.
+
+- **The unattributed bucket dropped tool calls and server-tool cost**
+  (TASK-093). An entry whose `parentUuid` chain does not resolve never reaches
+  the row path, so its `tool_use` blocks reached no counter at all, and its
+  `usage.server_tool_use` was dropped by `addUsage()`, which copies token fields
+  only. The two are different in kind: the tool calls are an **adoption**
+  denominator, not money — a tool call is paid for in tokens, and those were
+  always billed here — while web search is billed **per request** ($10/1000), so
+  counting the entry's tokens does not cover it. Measured: 1 tool call in 66 765
+  entries, and **0** web searches in the whole local corpus, on 30 061 entries
+  that carry a `server_tool_use` block — the cost hole is dormant, not closed.
+  The bucket now carries `tool_calls`, `agent_spawns`, `web_search_requests` and
+  `web_fetch_requests`, printed on stderr only when non-zero, and an
+  unattributed subagent spawn is named as UNCAPTURED cost (class 3) rather than
+  going missing from the one counter that sizes that blind spot. `server_tool_use`
+  is counted once per response (it rides inside `usage`, so it *is* duplicated);
+  the `tool_use` blocks are counted per entry (they are not). Reconciled against
+  an independent counter over three populations: tool calls now agree **to the
+  unit** (6 403 / 99 / 18) without the manual correction the TASK-092 review
+  still needed.
+
+### Added
+
+- **`eval/report.mjs` — the measurement as one self-contained HTML file**
+  (TASK-091, EPIC-009). `node eval/report.mjs --aggregate aggregate.json --out
+  report.html`: data embedded as JSON, charts as inline SVG, **no network and no
+  build step** — it opens from `file://` on a machine with no internet, which is
+  where it will actually be read. The failure it is built against is not a wrong
+  number but a right number about the wrong thing: a chart persuades harder than
+  the caveat under it, so **when a TASK-090 gate holds the arm-comparison chart
+  is not drawn at all** — not greyed out, not captioned. The per-pair costs stay
+  in a table, because those are what a repeat run is planned from. The order
+  effect is printed the same size and in the same place as the arm effect (§11b),
+  a token count never appears without its five classes, the two pools are
+  rendered by two calls that share no accumulator, and the honesty counters —
+  rejected rows with reasons, sessions dropped for switching arm, the delegation
+  cost that is a floor rather than a total — are a section rather than a footer.
+  Visuals follow the `dataviz` skill: palette validated in light and dark,
+  zero-baseline axes, hairline chrome. 22 tests, one group per acceptance
+  criterion, and the load-bearing one asserts the **absence** of a chart.
+
+- **`eval/analyze.mjs` — the aggregate per arm** (TASK-089, EPIC-009). Rows are
+  data; this is the step between them and the sentence "that arm cost less" —
+  and it deliberately stops before the sentence, because the thresholds that
+  would justify it are frozen by the PO in TASK-090. The paired measurement
+  (branch `exp/<task>/<arm>`) and ordinary work (attributed by the TASK-087
+  ledger) come out as two objects with **different shapes** and no total above
+  them, so the two populations cannot be added by accident. Per arm: prompts,
+  sessions, pairs, cost as median **and** spread (a single mean hides the one
+  prompt that read 200 k tokens), `work_s` rather than time-since-submission,
+  tokens always in five classes, tools, delegations, model/effort distribution,
+  and `pass_order` per pair so the order effect can be subtracted from the arm
+  effect (§11b).
+- **Discarding is output, not silence.** Every row lands in an arm or in a named
+  counter, and `rows_in = rows_used + rows_rejected` is asserted by a test. A
+  session whose rows disagree about the arm is dropped **whole** (the arm is a
+  property of the session: `settings.json` and MCP load at startup), a branch
+  that mimics the convention without matching it goes to a counter rather than
+  to the nearest plausible arm, and an unpriced row leaves the cost but stays in
+  the sample.
+- **Subagent cost is now measured from below** (TASK-089 criterion 8). The
+  result of an `Agent` call carries its own `usage` and `resolvedModel`, so each
+  delegation is priced with the existing table and rides on the row as
+  `delegations[]` — deduplicated by `agentId`, because the identical entry is
+  written to several transcript files and without that the figure comes out 2×.
+  It is a **lower bound** and is never added to the parent session's cost; the
+  spawns that left no priced result are counted beside it, since that part is
+  cost we have no figure for at all.
+
 ## [2.4.0] — 2026-07-31
 
 ### Added
@@ -541,6 +734,8 @@ action needed on upgrade.
   (`explore-cheap`, `reviewer`), the PreCompact / SessionStart / PostToolUse hook
   suite, and the adoption report (`eval/adoption-report.mjs`).
 
+[3.0.0]: https://gitlab.com/bglocation/tune-context/-/tags/v3.0.0
+[2.5.0]: https://gitlab.com/bglocation/tune-context/-/tags/v2.5.0
 [2.4.0]: https://gitlab.com/bglocation/tune-context/-/tags/v2.4.0
 [2.3.1]: https://gitlab.com/bglocation/tune-context/-/tags/v2.3.1
 [2.3.0]: https://gitlab.com/bglocation/tune-context/-/tags/v2.3.0

package/README.md

@@ -6,7 +6,8 @@ scaffolds a cost-optimal, high-quality setup (lean CLAUDE.md + skills + subagent
 + settings + hooks + MCP).
 
 **Self-contained** — zero dependency on any host repo. Ships two ways: a Claude
-Code **plugin** and an **npm CLI** (`tune-context init`), both under
+Code **plugin** and an **npm CLI** (`tune-context init`; plus `verify`, and
+`experiment` and `eval` for measuring the doctrine's cost), both under
 `@bglocation`. Incubated inside the [`code-search-mcp`](https://gitlab.com/bglocation/code-search-mcp)
 project, now extracted to its own repository.
 
@@ -16,9 +17,9 @@ tune-context/
 ├── skills/            # token-efficiency, caveman, cdd, phase-workflow, tune-context (+ templates/)
 ├── agents/            # explore-cheap, reviewer (model right-sizing)
 ├── hooks/             # PreCompact state-doc, SessionStart re-inject, adoption log, context reminder + hooks.json
-├── bin/ + cli/        # `tune-context init` — deterministic detect → generate → verify
-├── eval/              # adoption-report.mjs (the rag-usage analog), prompt-metrics.mjs, pricing.mjs
-└── scripts/           # verify-pack, smoke-init, sync-tune-context (dev/maintenance)
+├── bin/ + cli/        # `tune-context init | verify | experiment | eval` — deterministic, no model in the loop
+├── eval/              # adoption-report.mjs (the rag-usage analog) + mcp-probe.mjs, prompt-metrics.mjs, pricing.mjs, arm.mjs, analyze.mjs, verdict.mjs, report.mjs
+└── scripts/           # verify-pack, verify-plugin, smoke-init, sync-tune-context (dev/maintenance)
 ```
 
 ## Quick start
@@ -165,7 +166,7 @@ claude --plugin-dir /path/to/tune-context                       # or a local che
 ```
 
 The plugin manifest (`.claude-plugin/plugin.json`) and hook wiring
-(`hooks/hooks.json`, via `${CLAUDE_PLUGIN_ROOT}`) register the four hook events
+(`hooks/hooks.json`, via `${CLAUDE_PLUGIN_ROOT}`) register the five hook events
 automatically; `bin/tune-context` is added to `PATH` while the plugin is enabled.
 The marketplace catalog is [`.claude-plugin/marketplace.json`](.claude-plugin/marketplace.json).
 
@@ -246,7 +247,7 @@ by `tune-context init`), not here.
 ## Measuring adoption (does the doctrine actually get used?)
 `hooks/adoption-log.mjs` logs one JSONL line per event to
 `~/.claude/tune-context/adoption.jsonl` by default (under `CLAUDE_CONFIG_DIR`, the
-override's `tune-context/adoption.jsonl` — see above), and `eval/adoption-report.mjs`
+override's `tune-context/adoption.jsonl` — see above), and `tune-context eval adoption-report`
 turns that into a report — code search (semantic RAG vs the Grep tool, with
 shell-plumbing greps kept out of the RAG share), doctrine skills, subagents, and
 re-read churn — closing with an *earned* verdict: withheld below a PO-frozen floor of
@@ -321,10 +322,10 @@ instead of printing a number the doubling would have faked.
    `~/.claude/tune-context/adoption.jsonl` by default (created on first write; under
    `CLAUDE_CONFIG_DIR` it's the override's `tune-context/adoption.jsonl` instead).
 
-4. **Read the report** any time, from this checkout (or the installed package):
+4. **Read the report** any time, from any directory:
    ```bash
-   node eval/adoption-report.mjs            # default log location
-   node eval/adoption-report.mjs --log <path>
+   tune-context eval adoption-report            # default log location
+   tune-context eval adoption-report --log <path>
    ```
 
 **Stop / opt out:** delete the `PostToolUse` + `SubagentStart` entries from
@@ -340,7 +341,45 @@ contracts-first edits, delegation — is only partly visible too: no `Skill` cal
 but its trace shows up in the report's edit:read ratio, so the verdict never
 reads zero `Skill` calls as "unused."
 
-### Per-prompt metrics from transcripts (`eval/prompt-metrics.mjs`)
+### The meter, from any directory (`tune-context eval`)
+
+The cost measurement has two halves: the **switch** (`tune-context experiment`,
+below) and the **meter** — `prompt-metrics → analyze → verdict → report`. Both
+are one command, so both come from one installed version:
+
+```bash
+tune-context eval --help                    # the steps
+tune-context eval <step> --help             # that step's flags
+```
+
+The step names are the module names (`eval/<step>.mjs`), one to one, no aliases.
+Everything after the step belongs to the step — `tune-context eval
+prompt-metrics --help` prints the step's usage, not the CLI's.
+
+**Every artifact names the code that computed it.** Rows carry `meter_version`,
+the aggregate `analyzed_by`, the verdict `judged_by` — a *meter stamp*, which
+identifies code rather than a release number:
+
+| stamp | when |
+| --- | --- |
+| `3.0.0` | an npm install — immutable on the registry, rebuilt on demand with `npx -p @bglocation/tune-context@3.0.0` |
+| `3.0.0+0123456789ab` | a checkout of this repo, clean in `eval/ cli/ hooks/ bin/ package.json` |
+| `3.0.0+0123456789ab.dirty` | the same with uncommitted changes there (untracked files included) |
+| `3.0.0+unknown` | a checkout git could not read |
+
+A release number alone is not enough: under `npm link` a fix committed before
+the release carries the same number as the bug it fixes, which is how EPIC-009's
+verdict ended up computed by a meter nobody could name. Rows without a stamp —
+anything from before this contract — are refused, not converted: the way back to
+old data is recomputing the transcripts (see
+[`experiment archive`](#keeping-the-evidence-tune-context-experiment-archive)).
+
+*Developer path, from a checkout of this repo only:* `node eval/<step>.mjs` is
+the same code behind a second door — same parser, same bytes on stdout — and
+stamps `<version>+<sha>` (`.dirty` while you have uncommitted changes, which
+holds every verdict by design). A measurement runs through `tune-context eval`.
+
+### Per-prompt metrics from transcripts (`tune-context eval prompt-metrics`)
 
 The adoption report reads *our* hook log. `prompt-metrics` reads the transcripts
 **Claude Code writes itself** (`<config dir>/projects/<slug>/<session>.jsonl`) and
@@ -348,11 +387,14 @@ emits one JSONL row per prompt — time, tokens split by cache class, model,
 effort, tools, delegations, and cost in dollars (`eval/pricing.mjs`):
 
 ```bash
-node eval/prompt-metrics.mjs                              # every transcript
-node eval/prompt-metrics.mjs --transcript <path.jsonl>    # one session
-node eval/prompt-metrics.mjs --dir <dir> > prompts.jsonl  # JSONL on stdout
+tune-context eval prompt-metrics                              # every transcript
+tune-context eval prompt-metrics --transcript <path.jsonl>    # one session
+tune-context eval prompt-metrics --dir <dir> > prompts.jsonl  # JSONL on stdout
 ```
 
+The first line of the summary on stderr is `meter <stamp>` — the line a
+measurement series copies into its run journal before the first pair.
+
 Rows go to stdout, a run summary to stderr — so a pipe stays clean. Two things
 that source makes possible and a hook cannot: it works for **past** sessions, and
 it works when tune-context is switched **off** (a disabled hook records nothing,
@@ -366,6 +408,13 @@ Honest scope, printed on every run:
   is the right yardstick for comparing two arms, and the wrong number to
   reconcile against a statement. The field is named so the qualifier travels
   with the data.
+- **Repeated entries are skipped, and that changed every figure.** A transcript
+  writes some entries twice — 11 296 of 36 463 local ones (31%) repeat a `uuid`
+  already written *in the same file*, 6 739 of them assistant turns with a
+  `usage` block, the copies identical apart from a cosmetic `slug`. Summing both
+  charged 45% of all cache-read tokens twice: run totals from before 2026-08-02
+  are inflated ~1.97×. The skip is scoped to one file (no `uuid` is shared
+  between files) and the count is printed on every run.
 - **Cache-aware, because a naive token sum is off by ~10×.** Cache reads cost
   0.1× and writes 1.25×/2×; the local run carries 4.18 **billion** cache-read
   tokens against 521 k plain input ones — 99.9% of the figure is cache and
@@ -379,11 +428,16 @@ Honest scope, printed on every run:
   (us +10%) are applied; the claude-sonnet-5 introductory price ends in the
   table on 2026-08-31. Rows are priced at the **prompt's own timestamp**, so
   re-running the report later yields the same figure.
-- **Subagent cost is uncaptured, not zero.** Measured over 29 local transcripts:
-  21 `Agent` calls, **zero** sidechain entries — subagent turns never reach the
-  parent transcript. So `agent_spawns` is always counted, and `sidechain_turns`
-  is counted too: if a future Claude Code starts writing them, the number stops
-  being 0 with no code change.
+- **Subagent cost is measured from below, not captured.** Subagent turns never
+  reach the parent transcript (zero sidechain entries over 34 local ones), but
+  the **result** of an `Agent` call carries its own `usage` and `resolvedModel`
+  — so each delegation is priced with the same table and rides on the row as
+  `delegations[]`, deduplicated by `agentId` because the entry is written twice.
+  That figure is a **lower bound**
+  (`cache_read` grows linearly with the subagent's tool calls, which looks like
+  its last turn rather than a sum) and is **never added** to the row's own cost.
+  `agent_spawns` is still counted next to it: the spawns that left no priced
+  result are cost there is no figure for at all.
 - **Queue wait is not work.** `queued_s` (submission → first assistant entry) is
   reported apart from `work_s`, because a prompt sent while the agent is busy
   waits in a queue and that is the user's time. `max_gap_s` exposes the same
@@ -444,6 +498,145 @@ The measurement protocol that uses this — paired runs on `exp/<task>/<arm>`
 branches, frozen prompt script, verdict gates — is
 [COST_EVALUATION_PROCEDURE.md](COST_EVALUATION_PROCEDURE.md).
 
+### Keeping the evidence (`tune-context experiment archive`)
+
+Claude Code deletes transcripts after `cleanupPeriodDays` (30 by default). A
+meter bug found after that cannot be fixed by recomputing — it voids the series.
+So the series' transcripts are archived **after every arm**, together with the
+ledger that assigns them to arms:
+
+```bash
+tune-context experiment archive --out ~/tune-context-archive/series-1
+```
+
+- **What goes in:** every transcript with at least one entry on an `exp/…`
+  branch — including branches that only mimic the convention, so a recount can
+  reject them by name — and the ledger, in the config dir's own layout, plus
+  `manifest.json` (hash, size, branches, Claude Code versions per file) and
+  `SHA256SUMS`, checkable with `shasum -a 256 -c SHA256SUMS`. Ordinary work is
+  not archived: it is not evidence, and it is your whole history.
+- **Append-only.** An unchanged file is skipped; a file that grew (a resumed
+  session, a longer ledger) replaces its copy and is named; any other change
+  refuses the whole run and leaves the archive as it was. A file Claude Code has
+  since deleted stays — that is what the archive is for. Re-running costs
+  nothing.
+- **A transcript is the whole conversation** — contents of the files read,
+  command output, possibly secrets. A destination inside a git work tree that
+  git does not ignore is refused, with no flag to override it: add it to
+  `.gitignore` or archive outside any repository.
+
+Recomputing from the archive works after the originals are gone. `--ledger`
+points both steps at the archived ledger — without it the rows come out
+`exp_arm_source: branch`, which the paired pool does not accept:
+
+```bash
+tune-context eval prompt-metrics --dir ~/tune-context-archive/series-1/projects \
+  --ledger ~/tune-context-archive/series-1/tune-context/experiment.jsonl > rows.jsonl
+tune-context eval analyze --rows rows.jsonl \
+  --ledger ~/tune-context-archive/series-1/tune-context/experiment.jsonl > aggregate.json
+```
+
+To recompute with the meter that computed the original rows instead of the
+current one, run every step with that version:
+`npx -p @bglocation/tune-context@<version> tune-context eval …`.
+
+### Per-arm aggregate (`tune-context eval analyze`)
+
+Rows are data, not an answer. `analyze` turns a stream of them into one
+aggregate per arm — and stops one step short of ruling which arm won, because
+the thresholds that would do the ruling are frozen by the PO, not by the model:
+
+```bash
+tune-context eval prompt-metrics > rows.jsonl
+tune-context eval analyze --rows rows.jsonl > aggregate.json   # summary on stderr
+tune-context eval prompt-metrics | tune-context eval analyze > aggregate.json   # or through a pipe
+```
+
+- **No stamp, no aggregate.** One row without `meter_version` refuses the
+  **whole** input — exit 1, nothing on stdout, file and line of the first three
+  on stderr. A file mixing old and new rows is a pasted-together input, and
+  skipping the old ones would hand the verdict an aggregate that looks complete.
+  Rows from *different* meters are aggregated (the per-pair costs stay visible
+  for planning a repeat); the verdict holds on them.
+
+- **Two populations, never one number.** The paired measurement (branch
+  `exp/<task>/<arm>`, everything else held constant) and ordinary daily work
+  (attributed by the ledger, nothing held constant) come out as two separate
+  objects with different shapes. There is no total above them and no list of
+  pools to fold over — adding them has to be written out by hand and meant.
+- **Discards are the output too.** Every row lands either in an arm or in a
+  named counter, and `rows_in = rows_used + rows_rejected` is asserted by a
+  test. A session whose rows disagree about the arm is dropped **whole** —
+  `settings.json` and MCP servers load at startup, so a session that switched
+  arms was never in one — and a branch that *mimics* `exp/<task>/<arm>` without
+  matching it (`exp/task-089`, `exp/`) goes to a counter, never to the nearest
+  plausible arm.
+- **Median and spread, not the mean alone.** One 44-hour turn or one prompt that
+  read a 200 k-token file moves a mean of six and leaves the median untouched;
+  the gap between the two is itself the finding, so both are reported with
+  p25/p75, IQR and the extremes.
+- **An unpriced row leaves the cost, not the sample.** It stays in `n_prompts`,
+  sits outside the total, and the pool says `cost_complete: false` with the
+  model that caused it named.
+- **`pass_order` per pair.** The second implementation of a task is cheaper for
+  a reason that has nothing to do with the arm, and it pushes the same way as
+  the effect being looked for. The per-pair, per-arm cost is carried so the two
+  can be separated arithmetically (procedure §11b).
+
+### The verdict as data (`tune-context eval verdict`)
+
+May this run say "one arm is cheaper"? Seven gates answer, each with its own
+frozen threshold, and every gate is evaluated every time — a run that fixes one
+problem and reruns should not pay a run to learn about the second:
+
+```bash
+tune-context eval verdict --aggregate aggregate.json --rejected-runs journal.json > verdict.json
+```
+
+stdout is the verdict as JSON — the same object the report embeds — and stderr
+one line per gate (measured against threshold, and why it holds) ending in the
+finding: `withheld: N gate(s) hold — …`, `no-difference` or `cheaper-arm: <arm>`.
+Exit 0 for every verdict, a withheld one included; exit 1 only for input that
+cannot be read.
+
+`--rejected-runs` carries what the run journal recorded as §7 invalidations
+(`[{ session, task, arm, prompts, reason }]`). **Omitting it is not the same as
+passing `[]`**: the first says nobody asked the journal, and the verdict is held
+for exactly that reason — a rejection share whose denominator is unknown is a
+quietly dropped sample wearing a passing gate.
+
+The seventh gate, `meter_mixed` (frozen by the PO on 2026-10-15), passes only
+when the whole chain — the paired rows, `analyzed_by`, `judged_by` — carries
+exactly one stamp, and one that names code: not `.dirty`, not `+unknown`. A
+series therefore runs from a release, or from a clean commit.
+
+### The report (`tune-context eval report`)
+
+One HTML file — data embedded as JSON, charts as inline SVG, no network and no
+build step, so it opens from `file://` on a machine with nothing installed:
+
+```bash
+tune-context eval report --aggregate aggregate.json --out report.html
+tune-context eval report --aggregate aggregate.json --rejected-runs journal.json --out report.html
+```
+
+It reads its input exactly as `eval verdict` does — one reader, so the page can
+never accept what the verdict refuses — and names the stamp of every step of the
+chain in its header. `--rejected-runs` means the same as above.
+
+- **A withheld verdict is the page, not a footnote.** When a gate holds, the
+  arm-comparison chart is *not drawn at all* — not greyed out, not captioned. Two
+  bars side by side assert "this is a comparison" more loudly than any caption
+  can withdraw it. The per-pair costs stay in a table: those are what a repeat
+  run is planned from.
+- **The order effect is printed the same size as the arm effect.** A page that
+  shows one large and the other in small print has already made the claim §11b
+  refuses.
+- **Tokens never appear without their five classes**, the two pools are never
+  added, and the honesty counters — rejected rows and why, sessions dropped for
+  switching arm mid-flight, the delegation cost that is a floor and not a total —
+  are a section of their own rather than a footer.
+
 ### Getting real cost (OTEL) — outside tune-context
 
 tune-context does not set this up for you (deliberate — see TASK-062: it needs

package/bin/tune-context

@@ -8,4 +8,4 @@
 // process.argv.
 import { main } from './tune-context.mjs';
 
-main();
+await main();