@bglocation/tune-context 2.4.0 → 2.5.0

package/.claude-plugin/marketplace.json

@@ -10,7 +10,7 @@
       "name": "tune-context",
       "source": "./",
       "description": "Context-efficiency configurator for Claude Code: doctrine skills (token-efficiency, caveman, cdd, phase-workflow) plus a tune-context configurator that scaffolds a lean CLAUDE.md, subagents, settings and hooks.",
-      "version": "2.4.0",
+      "version": "2.5.0",
       "author": {
         "name": "Szymon Walczak",
         "url": "https://bglocation.dev"

package/.claude-plugin/plugin.json

@@ -2,7 +2,7 @@
   "name": "tune-context",
   "displayName": "tune-context",
   "description": "Context-efficiency configurator for Claude Code: doctrine skills (token-efficiency, caveman, cdd, phase-workflow) plus a tune-context configurator that scaffolds a lean CLAUDE.md, subagents, settings and hooks.",
-  "version": "2.4.0",
+  "version": "2.5.0",
   "author": {
     "name": "Szymon Walczak",
     "url": "https://bglocation.dev"

package/CHANGELOG.md

@@ -5,6 +5,120 @@ All notable changes to `@bglocation/tune-context` are documented here.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [2.5.0] — 2026-09-03
+
+### Fixed
+
+- **`prompt-metrics` charged 45% of all cache-read tokens twice** (found while
+  reviewing TASK-089). A transcript repeats entries verbatim: 11 296 of 36 463
+  local entries (31%) carry a `uuid` already written *earlier in the same file*,
+  6 739 of them assistant turns with a `usage` block, the copies differing only
+  in a cosmetic `slug`. The collector summed both. Local run total drops from
+  $4 566 to $2 323 — an inflation of **1.97×**, several times the effect
+  EPIC-009 is trying to measure. Repeats are now skipped (scoped to one file:
+  zero `uuid`s are shared between files) and counted on stderr, so the drop
+  explains itself rather than looking like a different tool.
+  **Every cost figure this project produced before 2026-08-02 is too high**;
+  `COST_EVALUATION_PROCEDURE.md` §11a carries the correction, including the
+  delegation baseline, which moves from ~0.7 to ~0.3 per session. The frozen
+  §10 thresholds are deliberately left alone — the correction makes the
+  delegation gate stricter relative to the norm, and refitting a threshold to
+  new data is the thing that section forbids.
+
+- **`prompt-metrics` billed one API response as many** (TASK-092, reported by a
+  Claude Code session working in a different project). One response reaches the
+  transcript as *several* entries — one per content block: thinking, text, each
+  `tool_use` — every one with its own `uuid` but carrying an **identical copy of
+  the same `usage` block**. The `uuid` dedup above cannot see this class by
+  construction: those uuids differ by design. Over 19 transcripts, 11 295
+  entries for 5 954 responses. `usage` is now keyed to the RESPONSE
+  (`message.id`, falling back to `requestId`) and the extra entries are counted
+  on stderr. On the same frozen transcripts the run total drops **$1 372.69 →
+  $679.35 (2.02×)**, on top of the 1.97× above.
+  The inflation is **not flat** across token classes (cache-read 1.82×, output
+  2.17×, cache-write 2.39×, input 2.45×) and does **not** cancel in a ratio of
+  two arms: the factor *is* the number of content blocks per turn — how many
+  tools the model reached for, which is the variable the doctrine sets out to
+  move. Per-session spread 1.66×–2.24×.
+  The guard sits **after** the `tool_use` loop, because those blocks are *not*
+  duplicated (6 403 blocks, 6 403 distinct ids): skipping whole entries would
+  have dropped ~2/3 of every tool call and gutted the adoption half of the
+  analysis. All three turn counters are guarded — `api_turns`, `sidechain_turns`
+  and the unattributed bucket — and an entry with neither `message.id` nor
+  `requestId` is **billed** and reported, never assumed unique.
+  **Every cost figure this project produced before 2026-09-03 is too high
+  again**, including the frozen EPIC-009 rows under `docs/` and the TASK-090
+  verdict computed from them. The skew correlates with the arm, so that verdict
+  rests on numbers that have to be recomputed before it means anything.
+
+- **The unattributed bucket dropped tool calls and server-tool cost**
+  (TASK-093). An entry whose `parentUuid` chain does not resolve never reaches
+  the row path, so its `tool_use` blocks reached no counter at all, and its
+  `usage.server_tool_use` was dropped by `addUsage()`, which copies token fields
+  only. The two are different in kind: the tool calls are an **adoption**
+  denominator, not money — a tool call is paid for in tokens, and those were
+  always billed here — while web search is billed **per request** ($10/1000), so
+  counting the entry's tokens does not cover it. Measured: 1 tool call in 66 765
+  entries, and **0** web searches in the whole local corpus, on 30 061 entries
+  that carry a `server_tool_use` block — the cost hole is dormant, not closed.
+  The bucket now carries `tool_calls`, `agent_spawns`, `web_search_requests` and
+  `web_fetch_requests`, printed on stderr only when non-zero, and an
+  unattributed subagent spawn is named as UNCAPTURED cost (class 3) rather than
+  going missing from the one counter that sizes that blind spot. `server_tool_use`
+  is counted once per response (it rides inside `usage`, so it *is* duplicated);
+  the `tool_use` blocks are counted per entry (they are not). Reconciled against
+  an independent counter over three populations: tool calls now agree **to the
+  unit** (6 403 / 99 / 18) without the manual correction the TASK-092 review
+  still needed.
+
+### Added
+
+- **`eval/report.mjs` — the measurement as one self-contained HTML file**
+  (TASK-091, EPIC-009). `node eval/report.mjs --aggregate aggregate.json --out
+  report.html`: data embedded as JSON, charts as inline SVG, **no network and no
+  build step** — it opens from `file://` on a machine with no internet, which is
+  where it will actually be read. The failure it is built against is not a wrong
+  number but a right number about the wrong thing: a chart persuades harder than
+  the caveat under it, so **when a TASK-090 gate holds the arm-comparison chart
+  is not drawn at all** — not greyed out, not captioned. The per-pair costs stay
+  in a table, because those are what a repeat run is planned from. The order
+  effect is printed the same size and in the same place as the arm effect (§11b),
+  a token count never appears without its five classes, the two pools are
+  rendered by two calls that share no accumulator, and the honesty counters —
+  rejected rows with reasons, sessions dropped for switching arm, the delegation
+  cost that is a floor rather than a total — are a section rather than a footer.
+  Visuals follow the `dataviz` skill: palette validated in light and dark,
+  zero-baseline axes, hairline chrome. 22 tests, one group per acceptance
+  criterion, and the load-bearing one asserts the **absence** of a chart.
+
+- **`eval/analyze.mjs` — the aggregate per arm** (TASK-089, EPIC-009). Rows are
+  data; this is the step between them and the sentence "that arm cost less" —
+  and it deliberately stops before the sentence, because the thresholds that
+  would justify it are frozen by the PO in TASK-090. The paired measurement
+  (branch `exp/<task>/<arm>`) and ordinary work (attributed by the TASK-087
+  ledger) come out as two objects with **different shapes** and no total above
+  them, so the two populations cannot be added by accident. Per arm: prompts,
+  sessions, pairs, cost as median **and** spread (a single mean hides the one
+  prompt that read 200 k tokens), `work_s` rather than time-since-submission,
+  tokens always in five classes, tools, delegations, model/effort distribution,
+  and `pass_order` per pair so the order effect can be subtracted from the arm
+  effect (§11b).
+- **Discarding is output, not silence.** Every row lands in an arm or in a named
+  counter, and `rows_in = rows_used + rows_rejected` is asserted by a test. A
+  session whose rows disagree about the arm is dropped **whole** (the arm is a
+  property of the session: `settings.json` and MCP load at startup), a branch
+  that mimics the convention without matching it goes to a counter rather than
+  to the nearest plausible arm, and an unpriced row leaves the cost but stays in
+  the sample.
+- **Subagent cost is now measured from below** (TASK-089 criterion 8). The
+  result of an `Agent` call carries its own `usage` and `resolvedModel`, so each
+  delegation is priced with the existing table and rides on the row as
+  `delegations[]` — deduplicated by `agentId`, because the identical entry is
+  written to several transcript files and without that the figure comes out 2×.
+  It is a **lower bound** and is never added to the parent session's cost; the
+  spawns that left no priced result are counted beside it, since that part is
+  cost we have no figure for at all.
+
 ## [2.4.0] — 2026-07-31
 
 ### Added
@@ -541,6 +655,7 @@ action needed on upgrade.
   (`explore-cheap`, `reviewer`), the PreCompact / SessionStart / PostToolUse hook
   suite, and the adoption report (`eval/adoption-report.mjs`).
 
+[2.5.0]: https://gitlab.com/bglocation/tune-context/-/tags/v2.5.0
 [2.4.0]: https://gitlab.com/bglocation/tune-context/-/tags/v2.4.0
 [2.3.1]: https://gitlab.com/bglocation/tune-context/-/tags/v2.3.1
 [2.3.0]: https://gitlab.com/bglocation/tune-context/-/tags/v2.3.0

package/README.md

@@ -17,7 +17,7 @@ tune-context/
 ├── agents/            # explore-cheap, reviewer (model right-sizing)
 ├── hooks/             # PreCompact state-doc, SessionStart re-inject, adoption log, context reminder + hooks.json
 ├── bin/ + cli/        # `tune-context init` — deterministic detect → generate → verify
-├── eval/              # adoption-report.mjs (the rag-usage analog), prompt-metrics.mjs, pricing.mjs
+├── eval/              # adoption-report.mjs (the rag-usage analog), prompt-metrics.mjs, pricing.mjs, arm.mjs, analyze.mjs, verdict.mjs, report.mjs
 └── scripts/           # verify-pack, smoke-init, sync-tune-context (dev/maintenance)
 ```
 
@@ -366,6 +366,13 @@ Honest scope, printed on every run:
   is the right yardstick for comparing two arms, and the wrong number to
   reconcile against a statement. The field is named so the qualifier travels
   with the data.
+- **Repeated entries are skipped, and that changed every figure.** A transcript
+  writes some entries twice — 11 296 of 36 463 local ones (31%) repeat a `uuid`
+  already written *in the same file*, 6 739 of them assistant turns with a
+  `usage` block, the copies identical apart from a cosmetic `slug`. Summing both
+  charged 45% of all cache-read tokens twice: run totals from before 2026-08-02
+  are inflated ~1.97×. The skip is scoped to one file (no `uuid` is shared
+  between files) and the count is printed on every run.
 - **Cache-aware, because a naive token sum is off by ~10×.** Cache reads cost
   0.1× and writes 1.25×/2×; the local run carries 4.18 **billion** cache-read
   tokens against 521 k plain input ones — 99.9% of the figure is cache and
@@ -379,11 +386,16 @@ Honest scope, printed on every run:
   (us +10%) are applied; the claude-sonnet-5 introductory price ends in the
   table on 2026-08-31. Rows are priced at the **prompt's own timestamp**, so
   re-running the report later yields the same figure.
-- **Subagent cost is uncaptured, not zero.** Measured over 29 local transcripts:
-  21 `Agent` calls, **zero** sidechain entries — subagent turns never reach the
-  parent transcript. So `agent_spawns` is always counted, and `sidechain_turns`
-  is counted too: if a future Claude Code starts writing them, the number stops
-  being 0 with no code change.
+- **Subagent cost is measured from below, not captured.** Subagent turns never
+  reach the parent transcript (zero sidechain entries over 34 local ones), but
+  the **result** of an `Agent` call carries its own `usage` and `resolvedModel`
+  — so each delegation is priced with the same table and rides on the row as
+  `delegations[]`, deduplicated by `agentId` because the entry is written twice.
+  That figure is a **lower bound**
+  (`cache_read` grows linearly with the subagent's tool calls, which looks like
+  its last turn rather than a sum) and is **never added** to the row's own cost.
+  `agent_spawns` is still counted next to it: the spawns that left no priced
+  result are cost there is no figure for at all.
 - **Queue wait is not work.** `queued_s` (submission → first assistant entry) is
   reported apart from `work_s`, because a prompt sent while the agent is busy
   waits in a queue and that is the user's time. `max_gap_s` exposes the same
@@ -444,6 +456,71 @@ The measurement protocol that uses this — paired runs on `exp/<task>/<arm>`
 branches, frozen prompt script, verdict gates — is
 [COST_EVALUATION_PROCEDURE.md](COST_EVALUATION_PROCEDURE.md).
 
+### Per-arm aggregate (`eval/analyze.mjs`)
+
+Rows are data, not an answer. `analyze` turns a stream of them into one
+aggregate per arm — and stops one step short of ruling which arm won, because
+the thresholds that would do the ruling are frozen by the PO, not by the model:
+
+```bash
+node eval/prompt-metrics.mjs > rows.jsonl
+node eval/analyze.mjs --rows rows.jsonl > aggregate.json   # summary on stderr
+```
+
+- **Two populations, never one number.** The paired measurement (branch
+  `exp/<task>/<arm>`, everything else held constant) and ordinary daily work
+  (attributed by the ledger, nothing held constant) come out as two separate
+  objects with different shapes. There is no total above them and no list of
+  pools to fold over — adding them has to be written out by hand and meant.
+- **Discards are the output too.** Every row lands either in an arm or in a
+  named counter, and `rows_in = rows_used + rows_rejected` is asserted by a
+  test. A session whose rows disagree about the arm is dropped **whole** —
+  `settings.json` and MCP servers load at startup, so a session that switched
+  arms was never in one — and a branch that *mimics* `exp/<task>/<arm>` without
+  matching it (`exp/task-089`, `exp/`) goes to a counter, never to the nearest
+  plausible arm.
+- **Median and spread, not the mean alone.** One 44-hour turn or one prompt that
+  read a 200 k-token file moves a mean of six and leaves the median untouched;
+  the gap between the two is itself the finding, so both are reported with
+  p25/p75, IQR and the extremes.
+- **An unpriced row leaves the cost, not the sample.** It stays in `n_prompts`,
+  sits outside the total, and the pool says `cost_complete: false` with the
+  model that caused it named.
+- **`pass_order` per pair.** The second implementation of a task is cheaper for
+  a reason that has nothing to do with the arm, and it pushes the same way as
+  the effect being looked for. The per-pair, per-arm cost is carried so the two
+  can be separated arithmetically (procedure §11b).
+
+### The report (`eval/report.mjs`)
+
+One HTML file — data embedded as JSON, charts as inline SVG, no network and no
+build step, so it opens from `file://` on a machine with nothing installed:
+
+```bash
+node eval/analyze.mjs --rows rows.jsonl > aggregate.json
+node eval/report.mjs --aggregate aggregate.json --out report.html
+node eval/report.mjs --aggregate aggregate.json --rejected-runs journal.json --out report.html
+```
+
+`--rejected-runs` carries what the run journal recorded as §7 invalidations
+(`[{ session, task, arm, prompts, reason }]`). **Omitting it is not the same as
+passing `[]`**: the first says nobody asked the journal, and the verdict is held
+for exactly that reason — a rejection share whose denominator is unknown is a
+quietly dropped sample wearing a passing gate.
+
+- **A withheld verdict is the page, not a footnote.** When a gate holds, the
+  arm-comparison chart is *not drawn at all* — not greyed out, not captioned. Two
+  bars side by side assert "this is a comparison" more loudly than any caption
+  can withdraw it. The per-pair costs stay in a table: those are what a repeat
+  run is planned from.
+- **The order effect is printed the same size as the arm effect.** A page that
+  shows one large and the other in small print has already made the claim §11b
+  refuses.
+- **Tokens never appear without their five classes**, the two pools are never
+  added, and the honesty counters — rejected rows and why, sessions dropped for
+  switching arm mid-flight, the delegation cost that is a floor and not a total —
+  are a section of their own rather than a footer.
+
 ### Getting real cost (OTEL) — outside tune-context
 
 tune-context does not set this up for you (deliberate — see TASK-062: it needs

package/cli/experiment.mjs

@@ -190,7 +190,7 @@ export function observeArm({ claudeDir, cwd, pkgRoot }) {
 
   const claudeMdPath = join(cwd, 'CLAUDE.md');
   const claudeMd = readText(claudeMdPath);
-  const hasBlock = claudeMd !== null && claudeMd.includes(START_MARKER) && claudeMd.includes(END_MARKER);
+  const hasBlock = claudeMd !== null && [START_MARKER, END_MARKER].every((m) => claudeMd.includes(m));
 
   const skills = deployed(claudeDir, pkgRoot, 'skills');
   const agents = deployed(claudeDir, pkgRoot, 'agents');
@@ -276,7 +276,7 @@ function planMutations({ claudeDir, cwd, pkgRoot, tc, rag }) {
 
     const claudeMdPath = join(cwd, 'CLAUDE.md');
     const claudeMd = readText(claudeMdPath);
-    if (claudeMd !== null && claudeMd.includes(START_MARKER) && claudeMd.includes(END_MARKER)) {
+    if (claudeMd !== null && [START_MARKER, END_MARKER].every((m) => claudeMd.includes(m))) {
       mutations.push({
         label: 'managed block removed from the project CLAUDE.md',
         path: claudeMdPath,

package/eval/adoption-report.mjs

@@ -539,6 +539,6 @@ function main() {
   process.stdout.write(`${formatReport(aggregate(records))}\n`);
 }
 
-if (process.argv[1] && process.argv[1].endsWith('adoption-report.mjs')) {
+if (process.argv[1]?.endsWith('adoption-report.mjs')) {
   main();
 }