@bglocation/tune-context 2.3.1 → 2.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -10,7 +10,7 @@
10
10
  "name": "tune-context",
11
11
  "source": "./",
12
12
  "description": "Context-efficiency configurator for Claude Code: doctrine skills (token-efficiency, caveman, cdd, phase-workflow) plus a tune-context configurator that scaffolds a lean CLAUDE.md, subagents, settings and hooks.",
13
- "version": "2.3.1",
13
+ "version": "2.5.0",
14
14
  "author": {
15
15
  "name": "Szymon Walczak",
16
16
  "url": "https://bglocation.dev"
@@ -2,7 +2,7 @@
2
2
  "name": "tune-context",
3
3
  "displayName": "tune-context",
4
4
  "description": "Context-efficiency configurator for Claude Code: doctrine skills (token-efficiency, caveman, cdd, phase-workflow) plus a tune-context configurator that scaffolds a lean CLAUDE.md, subagents, settings and hooks.",
5
- "version": "2.3.1",
5
+ "version": "2.5.0",
6
6
  "author": {
7
7
  "name": "Szymon Walczak",
8
8
  "url": "https://bglocation.dev"
package/CHANGELOG.md CHANGED
@@ -5,6 +5,197 @@ All notable changes to `@bglocation/tune-context` are documented here.
5
5
  The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
6
6
  and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
7
7
 
8
+ ## [2.5.0] — 2026-09-03
9
+
10
+ ### Fixed
11
+
12
+ - **`prompt-metrics` charged 45% of all cache-read tokens twice** (found while
13
+ reviewing TASK-089). A transcript repeats entries verbatim: 11 296 of 36 463
14
+ local entries (31%) carry a `uuid` already written *earlier in the same file*,
15
+ 6 739 of them assistant turns with a `usage` block, the copies differing only
16
+ in a cosmetic `slug`. The collector summed both. Local run total drops from
17
+ $4 566 to $2 323 — an inflation of **1.97×**, several times the effect
18
+ EPIC-009 is trying to measure. Repeats are now skipped (scoped to one file:
19
+ zero `uuid`s are shared between files) and counted on stderr, so the drop
20
+ explains itself rather than looking like a different tool.
21
+ **Every cost figure this project produced before 2026-08-02 is too high**;
22
+ `COST_EVALUATION_PROCEDURE.md` §11a carries the correction, including the
23
+ delegation baseline, which moves from ~0.7 to ~0.3 per session. The frozen
24
+ §10 thresholds are deliberately left alone — the correction makes the
25
+ delegation gate stricter relative to the norm, and refitting a threshold to
26
+ new data is the thing that section forbids.
27
+
28
+ - **`prompt-metrics` billed one API response as many** (TASK-092, reported by a
29
+ Claude Code session working in a different project). One response reaches the
30
+ transcript as *several* entries — one per content block: thinking, text, each
31
+ `tool_use` — every one with its own `uuid` but carrying an **identical copy of
32
+ the same `usage` block**. The `uuid` dedup above cannot see this class by
33
+ construction: those uuids differ by design. Over 19 transcripts, 11 295
34
+ entries for 5 954 responses. `usage` is now keyed to the RESPONSE
35
+ (`message.id`, falling back to `requestId`) and the extra entries are counted
36
+ on stderr. On the same frozen transcripts the run total drops **$1 372.69 →
37
+ $679.35 (2.02×)**, on top of the 1.97× above.
38
+ The inflation is **not flat** across token classes (cache-read 1.82×, output
39
+ 2.17×, cache-write 2.39×, input 2.45×) and does **not** cancel in a ratio of
40
+ two arms: the factor *is* the number of content blocks per turn — how many
41
+ tools the model reached for, which is the variable the doctrine sets out to
42
+ move. Per-session spread 1.66×–2.24×.
43
+ The guard sits **after** the `tool_use` loop, because those blocks are *not*
44
+ duplicated (6 403 blocks, 6 403 distinct ids): skipping whole entries would
45
+ have dropped ~2/3 of every tool call and gutted the adoption half of the
46
+ analysis. All three turn counters are guarded — `api_turns`, `sidechain_turns`
47
+ and the unattributed bucket — and an entry with neither `message.id` nor
48
+ `requestId` is **billed** and reported, never assumed unique.
49
+ **Every cost figure this project produced before 2026-09-03 is too high
50
+ again**, including the frozen EPIC-009 rows under `docs/` and the TASK-090
51
+ verdict computed from them. The skew correlates with the arm, so that verdict
52
+ rests on numbers that have to be recomputed before it means anything.
53
+
54
+ - **The unattributed bucket dropped tool calls and server-tool cost**
55
+ (TASK-093). An entry whose `parentUuid` chain does not resolve never reaches
56
+ the row path, so its `tool_use` blocks reached no counter at all, and its
57
+ `usage.server_tool_use` was dropped by `addUsage()`, which copies token fields
58
+ only. The two are different in kind: the tool calls are an **adoption**
59
+ denominator, not money — a tool call is paid for in tokens, and those were
60
+ always billed here — while web search is billed **per request** ($10/1000), so
61
+ counting the entry's tokens does not cover it. Measured: 1 tool call in 66 765
62
+ entries, and **0** web searches in the whole local corpus, on 30 061 entries
63
+ that carry a `server_tool_use` block — the cost hole is dormant, not closed.
64
+ The bucket now carries `tool_calls`, `agent_spawns`, `web_search_requests` and
65
+ `web_fetch_requests`, printed on stderr only when non-zero, and an
66
+ unattributed subagent spawn is named as UNCAPTURED cost (class 3) rather than
67
+ going missing from the one counter that sizes that blind spot. `server_tool_use`
68
+ is counted once per response (it rides inside `usage`, so it *is* duplicated);
69
+ the `tool_use` blocks are counted per entry (they are not). Reconciled against
70
+ an independent counter over three populations: tool calls now agree **to the
71
+ unit** (6 403 / 99 / 18) without the manual correction the TASK-092 review
72
+ still needed.
73
+
74
+ ### Added
75
+
76
+ - **`eval/report.mjs` — the measurement as one self-contained HTML file**
77
+ (TASK-091, EPIC-009). `node eval/report.mjs --aggregate aggregate.json --out
78
+ report.html`: data embedded as JSON, charts as inline SVG, **no network and no
79
+ build step** — it opens from `file://` on a machine with no internet, which is
80
+ where it will actually be read. The failure it is built against is not a wrong
81
+ number but a right number about the wrong thing: a chart persuades harder than
82
+ the caveat under it, so **when a TASK-090 gate holds the arm-comparison chart
83
+ is not drawn at all** — not greyed out, not captioned. The per-pair costs stay
84
+ in a table, because those are what a repeat run is planned from. The order
85
+ effect is printed the same size and in the same place as the arm effect (§11b),
86
+ a token count never appears without its five classes, the two pools are
87
+ rendered by two calls that share no accumulator, and the honesty counters —
88
+ rejected rows with reasons, sessions dropped for switching arm, the delegation
89
+ cost that is a floor rather than a total — are a section rather than a footer.
90
+ Visuals follow the `dataviz` skill: palette validated in light and dark,
91
+ zero-baseline axes, hairline chrome. 22 tests, one group per acceptance
92
+ criterion, and the load-bearing one asserts the **absence** of a chart.
93
+
94
+ - **`eval/analyze.mjs` — the aggregate per arm** (TASK-089, EPIC-009). Rows are
95
+ data; this is the step between them and the sentence "that arm cost less" —
96
+ and it deliberately stops before the sentence, because the thresholds that
97
+ would justify it are frozen by the PO in TASK-090. The paired measurement
98
+ (branch `exp/<task>/<arm>`) and ordinary work (attributed by the TASK-087
99
+ ledger) come out as two objects with **different shapes** and no total above
100
+ them, so the two populations cannot be added by accident. Per arm: prompts,
101
+ sessions, pairs, cost as median **and** spread (a single mean hides the one
102
+ prompt that read 200 k tokens), `work_s` rather than time-since-submission,
103
+ tokens always in five classes, tools, delegations, model/effort distribution,
104
+ and `pass_order` per pair so the order effect can be subtracted from the arm
105
+ effect (§11b).
106
+ - **Discarding is output, not silence.** Every row lands in an arm or in a named
107
+ counter, and `rows_in = rows_used + rows_rejected` is asserted by a test. A
108
+ session whose rows disagree about the arm is dropped **whole** (the arm is a
109
+ property of the session: `settings.json` and MCP load at startup), a branch
110
+ that mimics the convention without matching it goes to a counter rather than
111
+ to the nearest plausible arm, and an unpriced row leaves the cost but stays in
112
+ the sample.
113
+ - **Subagent cost is now measured from below** (TASK-089 criterion 8). The
114
+ result of an `Agent` call carries its own `usage` and `resolvedModel`, so each
115
+ delegation is priced with the existing table and rides on the row as
116
+ `delegations[]` — deduplicated by `agentId`, because the identical entry is
117
+ written to several transcript files and without that the figure comes out 2×.
118
+ It is a **lower bound** and is never added to the parent session's cost; the
119
+ spawns that left no priced result are counted beside it, since that part is
120
+ cost we have no figure for at all.
121
+
122
+ ## [2.4.0] — 2026-07-31
123
+
124
+ ### Added
125
+
126
+ - **`tune-context experiment` — the second arm** (TASK-087, EPIC-009). Measuring
127
+ one arm is not measuring: until now there was no way to run the same work
128
+ *without* the doctrine and without RAG, so every figure the reports produced
129
+ was absolute and had nothing to be compared against. `experiment set --tc off
130
+ --rag off` switches off hook registrations, the managed `CLAUDE.md` block, the
131
+ doctrine skills and the agents (the full arm, PO decision) plus the project
132
+ `.mcp.json`; `restore` puts every byte back. Each `set` appends to an
133
+ append-only ledger (`<config dir>/tune-context/experiment.jsonl`) — the source
134
+ of truth an analyzer uses to assign a session to an arm.
135
+ - **`restore` refuses rather than clobbers.** Every touched path is fingerprinted
136
+ and copied aside *before* it is modified — so a `set` interrupted halfway is
137
+ still reversible — and `restore` compares each path against what `set` left
138
+ there. A file edited during the arm (a permission added to `settings.json`,
139
+ text written below the `CLAUDE.md` marker — the TASK-082 scenario) aborts the
140
+ whole restore and reports **what** diverged, not just which file: the first
141
+ differing line with a preview of both sides for a text file, per-file counts
142
+ and names for a directory, plus the path to the saved copy. Refusal is the
143
+ correct outcome here, not a failure.
144
+ - **`verify` understands the off arm.** With an arm set it reports "off" instead
145
+ of a broken install — and stays RED when the arm did *not* fully land, e.g. a
146
+ leftover hook registration still firing. Reporting a correctly disabled
147
+ doctrine as breakage would be the EPIC-006 false-red regression pointing the
148
+ other way. `init` now refuses to run while an arm is set: it would re-install
149
+ what the arm parked and leave `restore` with fingerprints it never wrote.
150
+
151
+ - **Rows say which arm they belong to** (TASK-088, EPIC-009). `prompt-metrics`
152
+ rows from a branch shaped `exp/<task>/<arm>` now carry `exp_task`, `exp_arm`,
153
+ `exp_arm_source`, `exp_ledger_arm` and `pass_order`. The branch says which arm
154
+ was *intended*, the ledger which was *set* — and when they disagree the tool
155
+ does **not** pick a winner: the arm is left unresolved with the reason on the
156
+ row (`conflict:ledger`, `conflict:no-ledger-entry`, `conflict:branch-changed`),
157
+ because silently assigning a run to an arm is exactly the false saving the
158
+ paired measurement exists to prevent. The branch parser is strict for the same
159
+ reason: `exp/task-089/off-by-one` and `feat/exp/task-089/on` resolve to nothing
160
+ rather than to the nearest matching arm.
161
+ - **`pass_order` on the row.** The second implementation of the same task is
162
+ cheaper for a reason that has nothing to do with the arm — the solution is
163
+ already known — and that effect runs in the same direction as the one being
164
+ measured. Recorded per (task, arm) from the start time, it is what lets the
165
+ order effect be subtracted from the arm effect instead of being counted as one.
166
+
167
+ - **`restore` ends a measurement run even when it moved nothing.** The switch
168
+ only ever turns things *off*, so the `on` arm parks no files — and `restore`
169
+ used to refuse there, ledger entry included. That left the `on` arm's window
170
+ open forever: a follow-up a week later, in the same worktree on the same
171
+ `exp/` branch, still resolved to `branch+ledger` and joined the sample, while
172
+ the identical work after an `off` arm was flagged and excluded. A leak on one
173
+ side only is worse than a leak on both, because the two sides are what the
174
+ epic compares. Now `restore` closes the ledger window whether or not it had
175
+ bytes to put back, and refuses only when there is no open arm to close.
176
+
177
+ ### Fixed
178
+
179
+ - **A cost that is not a number is no longer counted as a cost.** `NaN` is
180
+ neither `null` nor `undefined`, so a row carrying one passed every "is there a
181
+ price here?" check, entered the priced count, and turned the whole run's total
182
+ into `$NaN` — every other row's money gone while the counter still claimed to
183
+ have priced them. The published amount is now refused unless it is finite, and
184
+ the roll-up checks each class, not only the total. The way in is closed too:
185
+ the price, tier and geo tables are looked up by *own* key, so a `service_tier`
186
+ spelled `constructor` reads as unknown instead of handing back a function from
187
+ `Object.prototype`.
188
+
189
+ ### Notes
190
+
191
+ - `rag off` covers the **project** `.mcp.json` only. `~/.claude.json` is rewritten
192
+ by Claude Code while it runs, so the switch never touches the user scope —
193
+ `experiment status` says this explicitly instead of implying RAG is globally
194
+ off.
195
+ - The arm belongs to the **session**: `settings.json` and MCP servers load at
196
+ startup, so a `set` applies from the next Claude Code start. Both `set` and
197
+ `status` say so; nothing here restarts anything for you.
198
+
8
199
  ## [2.3.1] — 2026-07-28
9
200
 
10
201
  ### Fixed
@@ -464,6 +655,8 @@ action needed on upgrade.
464
655
  (`explore-cheap`, `reviewer`), the PreCompact / SessionStart / PostToolUse hook
465
656
  suite, and the adoption report (`eval/adoption-report.mjs`).
466
657
 
658
+ [2.5.0]: https://gitlab.com/bglocation/tune-context/-/tags/v2.5.0
659
+ [2.4.0]: https://gitlab.com/bglocation/tune-context/-/tags/v2.4.0
467
660
  [2.3.1]: https://gitlab.com/bglocation/tune-context/-/tags/v2.3.1
468
661
  [2.3.0]: https://gitlab.com/bglocation/tune-context/-/tags/v2.3.0
469
662
  [2.2.0]: https://gitlab.com/bglocation/tune-context/-/tags/v2.2.0
package/README.md CHANGED
@@ -17,7 +17,7 @@ tune-context/
17
17
  ├── agents/ # explore-cheap, reviewer (model right-sizing)
18
18
  ├── hooks/ # PreCompact state-doc, SessionStart re-inject, adoption log, context reminder + hooks.json
19
19
  ├── bin/ + cli/ # `tune-context init` — deterministic detect → generate → verify
20
- ├── eval/ # adoption-report.mjs (the rag-usage analog)
20
+ ├── eval/ # adoption-report.mjs (the rag-usage analog), prompt-metrics.mjs, pricing.mjs, arm.mjs, analyze.mjs, verdict.mjs, report.mjs
21
21
  └── scripts/ # verify-pack, smoke-init, sync-tune-context (dev/maintenance)
22
22
  ```
23
23
 
@@ -332,19 +332,201 @@ instead of printing a number the doubling would have faked.
332
332
  `~/.claude/tune-context/` (or the `CLAUDE_CONFIG_DIR` override) and records only
333
333
  tool names, file paths, and skill/agent names — never full Bash command strings.
334
334
 
335
- **Honest scope:** this is a *behavioral proxy*, not a bill. Token cost lands only
336
- in OTEL (`CLAUDE_CODE_ENABLE_TELEMETRY=1` → `claude_code.cost.usage`, needs a
337
- collector), never a hook; and caveman is an output style with no tool call, so it
335
+ **Honest scope:** this is a *behavioral proxy*, not a bill. The hook log carries
336
+ no tokens, so dollars come from `prompt-metrics` below (or from OTEL, which is
337
+ Anthropic's own figure); and caveman is an output style with no tool call, so it
338
338
  can't be counted this way. Doctrine applied as *method* — narrow reads,
339
339
  contracts-first edits, delegation — is only partly visible too: no `Skill` call,
340
340
  but its trace shows up in the report's edit:read ratio, so the verdict never
341
341
  reads zero `Skill` calls as "unused."
342
342
 
343
+ ### Per-prompt metrics from transcripts (`eval/prompt-metrics.mjs`)
344
+
345
+ The adoption report reads *our* hook log. `prompt-metrics` reads the transcripts
346
+ **Claude Code writes itself** (`<config dir>/projects/<slug>/<session>.jsonl`) and
347
+ emits one JSONL row per prompt — time, tokens split by cache class, model,
348
+ effort, tools, delegations, and cost in dollars (`eval/pricing.mjs`):
349
+
350
+ ```bash
351
+ node eval/prompt-metrics.mjs # every transcript
352
+ node eval/prompt-metrics.mjs --transcript <path.jsonl> # one session
353
+ node eval/prompt-metrics.mjs --dir <dir> > prompts.jsonl # JSONL on stdout
354
+ ```
355
+
356
+ Rows go to stdout, a run summary to stderr — so a pipe stays clean. Two things
357
+ that source makes possible and a hook cannot: it works for **past** sessions, and
358
+ it works when tune-context is switched **off** (a disabled hook records nothing,
359
+ and that is exactly the arm a comparison needs).
360
+
361
+ Honest scope, printed on every run:
362
+
363
+ - **`list_cost_usd` is API list price, not your bill.** A Pro/Max subscription
364
+ charges a flat fee and consumes plan limits; nothing here is billed per token.
365
+ The figure answers *"what would this work have cost through the API"* — which
366
+ is the right yardstick for comparing two arms, and the wrong number to
367
+ reconcile against a statement. The field is named so the qualifier travels
368
+ with the data.
369
+ - **Repeated entries are skipped, and that changed every figure.** A transcript
370
+ writes some entries twice — 11 296 of 36 463 local ones (31%) repeat a `uuid`
371
+ already written *in the same file*, 6 739 of them assistant turns with a
372
+ `usage` block, the copies identical apart from a cosmetic `slug`. Summing both
373
+ charged 45% of all cache-read tokens twice: run totals from before 2026-08-02
374
+ are inflated ~1.97×. The skip is scoped to one file (no `uuid` is shared
375
+ between files) and the count is printed on every run.
376
+ - **Cache-aware, because a naive token sum is off by ~10×.** Cache reads cost
377
+ 0.1× and writes 1.25×/2×; the local run carries 4.18 **billion** cache-read
378
+ tokens against 521 k plain input ones — 99.9% of the figure is cache and
379
+ output. The total never appears without its per-class split, and a model the
380
+ table doesn't know yields `null` plus an `UNPRICED` count, never a zero.
381
+ - **Prices are data, dated, and expire.** `eval/pricing.mjs` holds one table
382
+ taken from the published pricing page with its retrieval date, storing base
383
+ input/output only and deriving the three cache columns from the documented
384
+ 1.25×/2×/0.1× multipliers — a test asserts the derivation reproduces every
385
+ published row. Service tier (batch −50%), fast mode, and `inference_geo`
386
+ (us +10%) are applied; the claude-sonnet-5 introductory price ends in the
387
+ table on 2026-08-31. Rows are priced at the **prompt's own timestamp**, so
388
+ re-running the report later yields the same figure.
389
+ - **Subagent cost is measured from below, not captured.** Subagent turns never
390
+ reach the parent transcript (zero sidechain entries over 34 local ones), but
391
+ the **result** of an `Agent` call carries its own `usage` and `resolvedModel`
392
+ — so each delegation is priced with the same table and rides on the row as
393
+ `delegations[]`, deduplicated by `agentId` because the entry is written twice.
394
+ That figure is a **lower bound**
395
+ (`cache_read` grows linearly with the subagent's tool calls, which looks like
396
+ its last turn rather than a sum) and is **never added** to the row's own cost.
397
+ `agent_spawns` is still counted next to it: the spawns that left no priced
398
+ result are cost there is no figure for at all.
399
+ - **Queue wait is not work.** `queued_s` (submission → first assistant entry) is
400
+ reported apart from `work_s`, because a prompt sent while the agent is busy
401
+ waits in a queue and that is the user's time. `max_gap_s` exposes the same
402
+ thing *inside* a turn — a tool call blocked on a permission prompt lands in
403
+ `work_s` otherwise.
404
+ - **Model and effort change mid-session** (measured), so both ride on every row,
405
+ with a per-model token split when a single prompt used more than one.
406
+ - Entries that can't be attributed to a prompt are **counted and reported**,
407
+ never silently dropped. Same for the two ways the figure can be too low:
408
+ `usage.iterations` reporting more than the `usage` we price from (3 turns
409
+ locally), and web searches, which are billed per request ($10/1000) rather
410
+ than per token and so sit outside the token total. Web *fetch* is free.
411
+
412
+ ### The other arm (`tune-context experiment`)
413
+
414
+ A cost figure with nothing to compare it against is a number, not a measurement.
415
+ This switch turns the doctrine **off** so the same work can be run twice —
416
+ `tc={on,off}` × `rag={on,off}` — and records which arm was in force when:
417
+
418
+ ```bash
419
+ tune-context experiment status # what is set, and what the config actually shows
420
+ tune-context experiment set --tc off --rag off --note "pair 1"
421
+ tune-context experiment restore # byte for byte, or a refusal
422
+ ```
423
+
424
+ - **`tc off` is the full arm** (PO decision, EPIC-009): hook registrations in
425
+ `settings.json`, the managed block in the project `CLAUDE.md`, the doctrine
426
+ skills, the agents. Four interventions in one variable on purpose — at small
427
+ *n* the biggest effect is the easiest to detect; splitting them is the *next*
428
+ experiment.
429
+ - **`rag off` means this project's `.mcp.json`, and nothing else.** The user
430
+ scope (`~/.claude.json`) is rewritten by Claude Code while it runs, so the
431
+ switch never touches it — `status` says so out loud instead of letting you
432
+ assume RAG is off globally. Check `claude mcp list` before a measured run.
433
+ - **Reversibility comes before the change.** Every touched path is fingerprinted
434
+ and copied aside *before* it is modified, so a `set` that dies halfway is still
435
+ undoable. `restore` compares each path against what `set` left there and
436
+ **refuses** if anything changed in between, naming the file *and the
437
+ difference* (first differing line for a text file, changed/added/removed files
438
+ for a directory, and where the saved copy is) — the working session between the
439
+ two commands is exactly when you might add a permission or write below the
440
+ `CLAUDE.md` marker, and silently erasing that is worse than refusing.
441
+ - **The ledger is the source of truth**
442
+ (`<config dir>/tune-context/experiment.jsonl`, append-only). An analyzer
443
+ assigns a session to an arm by timestamp; detecting the arm from a transcript
444
+ is a cross-check, never the verdict. Every `set` appends — including one that
445
+ changes nothing, because a run whose ledger is silent at its start cannot be
446
+ assigned to an arm at all.
447
+ - **The arm belongs to the session.** `settings.json` and MCP servers are read at
448
+ startup, so it applies from the next Claude Code start; flipping it mid-session
449
+ produces a mixed session, which the analysis throws away.
450
+ - `tune-context verify` reads the arm and reports **"off"** rather than a broken
451
+ install — and stays RED if the arm did not fully land. `init` refuses to run
452
+ while an arm is set: it would re-install what the arm parked and leave
453
+ `restore` with nothing it recognises.
454
+
455
+ The measurement protocol that uses this — paired runs on `exp/<task>/<arm>`
456
+ branches, frozen prompt script, verdict gates — is
457
+ [COST_EVALUATION_PROCEDURE.md](COST_EVALUATION_PROCEDURE.md).
458
+
459
+ ### Per-arm aggregate (`eval/analyze.mjs`)
460
+
461
+ Rows are data, not an answer. `analyze` turns a stream of them into one
462
+ aggregate per arm — and stops one step short of ruling which arm won, because
463
+ the thresholds that would do the ruling are frozen by the PO, not by the model:
464
+
465
+ ```bash
466
+ node eval/prompt-metrics.mjs > rows.jsonl
467
+ node eval/analyze.mjs --rows rows.jsonl > aggregate.json # summary on stderr
468
+ ```
469
+
470
+ - **Two populations, never one number.** The paired measurement (branch
471
+ `exp/<task>/<arm>`, everything else held constant) and ordinary daily work
472
+ (attributed by the ledger, nothing held constant) come out as two separate
473
+ objects with different shapes. There is no total above them and no list of
474
+ pools to fold over — adding them has to be written out by hand and meant.
475
+ - **Discards are the output too.** Every row lands either in an arm or in a
476
+ named counter, and `rows_in = rows_used + rows_rejected` is asserted by a
477
+ test. A session whose rows disagree about the arm is dropped **whole** —
478
+ `settings.json` and MCP servers load at startup, so a session that switched
479
+ arms was never in one — and a branch that *mimics* `exp/<task>/<arm>` without
480
+ matching it (`exp/task-089`, `exp/`) goes to a counter, never to the nearest
481
+ plausible arm.
482
+ - **Median and spread, not the mean alone.** One 44-hour turn or one prompt that
483
+ read a 200 k-token file moves a mean of six and leaves the median untouched;
484
+ the gap between the two is itself the finding, so both are reported with
485
+ p25/p75, IQR and the extremes.
486
+ - **An unpriced row leaves the cost, not the sample.** It stays in `n_prompts`,
487
+ sits outside the total, and the pool says `cost_complete: false` with the
488
+ model that caused it named.
489
+ - **`pass_order` per pair.** The second implementation of a task is cheaper for
490
+ a reason that has nothing to do with the arm, and it pushes the same way as
491
+ the effect being looked for. The per-pair, per-arm cost is carried so the two
492
+ can be separated arithmetically (procedure §11b).
493
+
494
+ ### The report (`eval/report.mjs`)
495
+
496
+ One HTML file — data embedded as JSON, charts as inline SVG, no network and no
497
+ build step, so it opens from `file://` on a machine with nothing installed:
498
+
499
+ ```bash
500
+ node eval/analyze.mjs --rows rows.jsonl > aggregate.json
501
+ node eval/report.mjs --aggregate aggregate.json --out report.html
502
+ node eval/report.mjs --aggregate aggregate.json --rejected-runs journal.json --out report.html
503
+ ```
504
+
505
+ `--rejected-runs` carries what the run journal recorded as §7 invalidations
506
+ (`[{ session, task, arm, prompts, reason }]`). **Omitting it is not the same as
507
+ passing `[]`**: the first says nobody asked the journal, and the verdict is held
508
+ for exactly that reason — a rejection share whose denominator is unknown is a
509
+ quietly dropped sample wearing a passing gate.
510
+
511
+ - **A withheld verdict is the page, not a footnote.** When a gate holds, the
512
+ arm-comparison chart is *not drawn at all* — not greyed out, not captioned. Two
513
+ bars side by side assert "this is a comparison" more loudly than any caption
514
+ can withdraw it. The per-pair costs stay in a table: those are what a repeat
515
+ run is planned from.
516
+ - **The order effect is printed the same size as the arm effect.** A page that
517
+ shows one large and the other in small print has already made the claim §11b
518
+ refuses.
519
+ - **Tokens never appear without their five classes**, the two pools are never
520
+ added, and the honesty counters — rejected rows and why, sessions dropped for
521
+ switching arm mid-flight, the delegation cost that is a floor and not a total —
522
+ are a section of their own rather than a footer.
523
+
343
524
  ### Getting real cost (OTEL) — outside tune-context
344
525
 
345
526
  tune-context does not set this up for you (deliberate — see TASK-062: it needs
346
- infra this tool stays zero-infra without). If you want actual dollars instead of
347
- the behavioral proxy above:
527
+ infra this tool stays zero-infra without). `prompt-metrics` above computes cost
528
+ from a public price list, which is an *independent* calculation, not Anthropic's
529
+ invoice — OTEL is the way to check it against the real one:
348
530
 
349
531
  ```bash
350
532
  export CLAUDE_CODE_ENABLE_TELEMETRY=1 # off by default