@bglocation/tune-context 2.3.1 → 2.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +1 -1
- package/.claude-plugin/plugin.json +1 -1
- package/CHANGELOG.md +193 -0
- package/README.md +188 -6
- package/bin/tune-context.mjs +241 -15
- package/cli/experiment.mjs +674 -0
- package/cli/managed-block.mjs +24 -0
- package/cli/verify.mjs +51 -4
- package/eval/adoption-report.mjs +9 -7
- package/eval/analyze.mjs +689 -0
- package/eval/arm.mjs +246 -0
- package/eval/pricing.mjs +461 -0
- package/eval/prompt-metrics.mjs +1046 -0
- package/eval/report.mjs +870 -0
- package/eval/verdict.mjs +639 -0
- package/hooks/adoption-log.mjs +1 -1
- package/hooks/context-reminder.mjs +1 -1
- package/hooks/pre-compact-snapshot.mjs +4 -4
- package/hooks/session-start-reinject.mjs +1 -4
- package/package.json +7 -2
|
@@ -10,7 +10,7 @@
|
|
|
10
10
|
"name": "tune-context",
|
|
11
11
|
"source": "./",
|
|
12
12
|
"description": "Context-efficiency configurator for Claude Code: doctrine skills (token-efficiency, caveman, cdd, phase-workflow) plus a tune-context configurator that scaffolds a lean CLAUDE.md, subagents, settings and hooks.",
|
|
13
|
-
"version": "2.
|
|
13
|
+
"version": "2.5.0",
|
|
14
14
|
"author": {
|
|
15
15
|
"name": "Szymon Walczak",
|
|
16
16
|
"url": "https://bglocation.dev"
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
"name": "tune-context",
|
|
3
3
|
"displayName": "tune-context",
|
|
4
4
|
"description": "Context-efficiency configurator for Claude Code: doctrine skills (token-efficiency, caveman, cdd, phase-workflow) plus a tune-context configurator that scaffolds a lean CLAUDE.md, subagents, settings and hooks.",
|
|
5
|
-
"version": "2.
|
|
5
|
+
"version": "2.5.0",
|
|
6
6
|
"author": {
|
|
7
7
|
"name": "Szymon Walczak",
|
|
8
8
|
"url": "https://bglocation.dev"
|
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,197 @@ All notable changes to `@bglocation/tune-context` are documented here.
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
+
## [2.5.0] — 2026-09-03
|
|
9
|
+
|
|
10
|
+
### Fixed
|
|
11
|
+
|
|
12
|
+
- **`prompt-metrics` charged 45% of all cache-read tokens twice** (found while
|
|
13
|
+
reviewing TASK-089). A transcript repeats entries verbatim: 11 296 of 36 463
|
|
14
|
+
local entries (31%) carry a `uuid` already written *earlier in the same file*,
|
|
15
|
+
6 739 of them assistant turns with a `usage` block, the copies differing only
|
|
16
|
+
in a cosmetic `slug`. The collector summed both. Local run total drops from
|
|
17
|
+
$4 566 to $2 323 — an inflation of **1.97×**, several times the effect
|
|
18
|
+
EPIC-009 is trying to measure. Repeats are now skipped (scoped to one file:
|
|
19
|
+
zero `uuid`s are shared between files) and counted on stderr, so the drop
|
|
20
|
+
explains itself rather than looking like a different tool.
|
|
21
|
+
**Every cost figure this project produced before 2026-08-02 is too high**;
|
|
22
|
+
`COST_EVALUATION_PROCEDURE.md` §11a carries the correction, including the
|
|
23
|
+
delegation baseline, which moves from ~0.7 to ~0.3 per session. The frozen
|
|
24
|
+
§10 thresholds are deliberately left alone — the correction makes the
|
|
25
|
+
delegation gate stricter relative to the norm, and refitting a threshold to
|
|
26
|
+
new data is the thing that section forbids.
|
|
27
|
+
|
|
28
|
+
- **`prompt-metrics` billed one API response as many** (TASK-092, reported by a
|
|
29
|
+
Claude Code session working in a different project). One response reaches the
|
|
30
|
+
transcript as *several* entries — one per content block: thinking, text, each
|
|
31
|
+
`tool_use` — every one with its own `uuid` but carrying an **identical copy of
|
|
32
|
+
the same `usage` block**. The `uuid` dedup above cannot see this class by
|
|
33
|
+
construction: those uuids differ by design. Over 19 transcripts, 11 295
|
|
34
|
+
entries for 5 954 responses. `usage` is now keyed to the RESPONSE
|
|
35
|
+
(`message.id`, falling back to `requestId`) and the extra entries are counted
|
|
36
|
+
on stderr. On the same frozen transcripts the run total drops **$1 372.69 →
|
|
37
|
+
$679.35 (2.02×)**, on top of the 1.97× above.
|
|
38
|
+
The inflation is **not flat** across token classes (cache-read 1.82×, output
|
|
39
|
+
2.17×, cache-write 2.39×, input 2.45×) and does **not** cancel in a ratio of
|
|
40
|
+
two arms: the factor *is* the number of content blocks per turn — how many
|
|
41
|
+
tools the model reached for, which is the variable the doctrine sets out to
|
|
42
|
+
move. Per-session spread 1.66×–2.24×.
|
|
43
|
+
The guard sits **after** the `tool_use` loop, because those blocks are *not*
|
|
44
|
+
duplicated (6 403 blocks, 6 403 distinct ids): skipping whole entries would
|
|
45
|
+
have dropped ~2/3 of every tool call and gutted the adoption half of the
|
|
46
|
+
analysis. All three turn counters are guarded — `api_turns`, `sidechain_turns`
|
|
47
|
+
and the unattributed bucket — and an entry with neither `message.id` nor
|
|
48
|
+
`requestId` is **billed** and reported, never assumed unique.
|
|
49
|
+
**Every cost figure this project produced before 2026-09-03 is too high
|
|
50
|
+
again**, including the frozen EPIC-009 rows under `docs/` and the TASK-090
|
|
51
|
+
verdict computed from them. The skew correlates with the arm, so that verdict
|
|
52
|
+
rests on numbers that have to be recomputed before it means anything.
|
|
53
|
+
|
|
54
|
+
- **The unattributed bucket dropped tool calls and server-tool cost**
|
|
55
|
+
(TASK-093). An entry whose `parentUuid` chain does not resolve never reaches
|
|
56
|
+
the row path, so its `tool_use` blocks reached no counter at all, and its
|
|
57
|
+
`usage.server_tool_use` was dropped by `addUsage()`, which copies token fields
|
|
58
|
+
only. The two are different in kind: the tool calls are an **adoption**
|
|
59
|
+
denominator, not money — a tool call is paid for in tokens, and those were
|
|
60
|
+
always billed here — while web search is billed **per request** ($10/1000), so
|
|
61
|
+
counting the entry's tokens does not cover it. Measured: 1 tool call in 66 765
|
|
62
|
+
entries, and **0** web searches in the whole local corpus, on 30 061 entries
|
|
63
|
+
that carry a `server_tool_use` block — the cost hole is dormant, not closed.
|
|
64
|
+
The bucket now carries `tool_calls`, `agent_spawns`, `web_search_requests` and
|
|
65
|
+
`web_fetch_requests`, printed on stderr only when non-zero, and an
|
|
66
|
+
unattributed subagent spawn is named as UNCAPTURED cost (class 3) rather than
|
|
67
|
+
going missing from the one counter that sizes that blind spot. `server_tool_use`
|
|
68
|
+
is counted once per response (it rides inside `usage`, so it *is* duplicated);
|
|
69
|
+
the `tool_use` blocks are counted per entry (they are not). Reconciled against
|
|
70
|
+
an independent counter over three populations: tool calls now agree **to the
|
|
71
|
+
unit** (6 403 / 99 / 18) without the manual correction the TASK-092 review
|
|
72
|
+
still needed.
|
|
73
|
+
|
|
74
|
+
### Added
|
|
75
|
+
|
|
76
|
+
- **`eval/report.mjs` — the measurement as one self-contained HTML file**
|
|
77
|
+
(TASK-091, EPIC-009). `node eval/report.mjs --aggregate aggregate.json --out
|
|
78
|
+
report.html`: data embedded as JSON, charts as inline SVG, **no network and no
|
|
79
|
+
build step** — it opens from `file://` on a machine with no internet, which is
|
|
80
|
+
where it will actually be read. The failure it is built against is not a wrong
|
|
81
|
+
number but a right number about the wrong thing: a chart persuades harder than
|
|
82
|
+
the caveat under it, so **when a TASK-090 gate holds the arm-comparison chart
|
|
83
|
+
is not drawn at all** — not greyed out, not captioned. The per-pair costs stay
|
|
84
|
+
in a table, because those are what a repeat run is planned from. The order
|
|
85
|
+
effect is printed the same size and in the same place as the arm effect (§11b),
|
|
86
|
+
a token count never appears without its five classes, the two pools are
|
|
87
|
+
rendered by two calls that share no accumulator, and the honesty counters —
|
|
88
|
+
rejected rows with reasons, sessions dropped for switching arm, the delegation
|
|
89
|
+
cost that is a floor rather than a total — are a section rather than a footer.
|
|
90
|
+
Visuals follow the `dataviz` skill: palette validated in light and dark,
|
|
91
|
+
zero-baseline axes, hairline chrome. 22 tests, one group per acceptance
|
|
92
|
+
criterion, and the load-bearing one asserts the **absence** of a chart.
|
|
93
|
+
|
|
94
|
+
- **`eval/analyze.mjs` — the aggregate per arm** (TASK-089, EPIC-009). Rows are
|
|
95
|
+
data; this is the step between them and the sentence "that arm cost less" —
|
|
96
|
+
and it deliberately stops before the sentence, because the thresholds that
|
|
97
|
+
would justify it are frozen by the PO in TASK-090. The paired measurement
|
|
98
|
+
(branch `exp/<task>/<arm>`) and ordinary work (attributed by the TASK-087
|
|
99
|
+
ledger) come out as two objects with **different shapes** and no total above
|
|
100
|
+
them, so the two populations cannot be added by accident. Per arm: prompts,
|
|
101
|
+
sessions, pairs, cost as median **and** spread (a single mean hides the one
|
|
102
|
+
prompt that read 200 k tokens), `work_s` rather than time-since-submission,
|
|
103
|
+
tokens always in five classes, tools, delegations, model/effort distribution,
|
|
104
|
+
and `pass_order` per pair so the order effect can be subtracted from the arm
|
|
105
|
+
effect (§11b).
|
|
106
|
+
- **Discarding is output, not silence.** Every row lands in an arm or in a named
|
|
107
|
+
counter, and `rows_in = rows_used + rows_rejected` is asserted by a test. A
|
|
108
|
+
session whose rows disagree about the arm is dropped **whole** (the arm is a
|
|
109
|
+
property of the session: `settings.json` and MCP load at startup), a branch
|
|
110
|
+
that mimics the convention without matching it goes to a counter rather than
|
|
111
|
+
to the nearest plausible arm, and an unpriced row leaves the cost but stays in
|
|
112
|
+
the sample.
|
|
113
|
+
- **Subagent cost is now measured from below** (TASK-089 criterion 8). The
|
|
114
|
+
result of an `Agent` call carries its own `usage` and `resolvedModel`, so each
|
|
115
|
+
delegation is priced with the existing table and rides on the row as
|
|
116
|
+
`delegations[]` — deduplicated by `agentId`, because the identical entry is
|
|
117
|
+
written to several transcript files and without that the figure comes out 2×.
|
|
118
|
+
It is a **lower bound** and is never added to the parent session's cost; the
|
|
119
|
+
spawns that left no priced result are counted beside it, since that part is
|
|
120
|
+
cost we have no figure for at all.
|
|
121
|
+
|
|
122
|
+
## [2.4.0] — 2026-07-31
|
|
123
|
+
|
|
124
|
+
### Added
|
|
125
|
+
|
|
126
|
+
- **`tune-context experiment` — the second arm** (TASK-087, EPIC-009). Measuring
|
|
127
|
+
one arm is not measuring: until now there was no way to run the same work
|
|
128
|
+
*without* the doctrine and without RAG, so every figure the reports produced
|
|
129
|
+
was absolute and had nothing to be compared against. `experiment set --tc off
|
|
130
|
+
--rag off` switches off hook registrations, the managed `CLAUDE.md` block, the
|
|
131
|
+
doctrine skills and the agents (the full arm, PO decision) plus the project
|
|
132
|
+
`.mcp.json`; `restore` puts every byte back. Each `set` appends to an
|
|
133
|
+
append-only ledger (`<config dir>/tune-context/experiment.jsonl`) — the source
|
|
134
|
+
of truth an analyzer uses to assign a session to an arm.
|
|
135
|
+
- **`restore` refuses rather than clobbers.** Every touched path is fingerprinted
|
|
136
|
+
and copied aside *before* it is modified — so a `set` interrupted halfway is
|
|
137
|
+
still reversible — and `restore` compares each path against what `set` left
|
|
138
|
+
there. A file edited during the arm (a permission added to `settings.json`,
|
|
139
|
+
text written below the `CLAUDE.md` marker — the TASK-082 scenario) aborts the
|
|
140
|
+
whole restore and reports **what** diverged, not just which file: the first
|
|
141
|
+
differing line with a preview of both sides for a text file, per-file counts
|
|
142
|
+
and names for a directory, plus the path to the saved copy. Refusal is the
|
|
143
|
+
correct outcome here, not a failure.
|
|
144
|
+
- **`verify` understands the off arm.** With an arm set it reports "off" instead
|
|
145
|
+
of a broken install — and stays RED when the arm did *not* fully land, e.g. a
|
|
146
|
+
leftover hook registration still firing. Reporting a correctly disabled
|
|
147
|
+
doctrine as breakage would be the EPIC-006 false-red regression pointing the
|
|
148
|
+
other way. `init` now refuses to run while an arm is set: it would re-install
|
|
149
|
+
what the arm parked and leave `restore` with fingerprints it never wrote.
|
|
150
|
+
|
|
151
|
+
- **Rows say which arm they belong to** (TASK-088, EPIC-009). `prompt-metrics`
|
|
152
|
+
rows from a branch shaped `exp/<task>/<arm>` now carry `exp_task`, `exp_arm`,
|
|
153
|
+
`exp_arm_source`, `exp_ledger_arm` and `pass_order`. The branch says which arm
|
|
154
|
+
was *intended*, the ledger which was *set* — and when they disagree the tool
|
|
155
|
+
does **not** pick a winner: the arm is left unresolved with the reason on the
|
|
156
|
+
row (`conflict:ledger`, `conflict:no-ledger-entry`, `conflict:branch-changed`),
|
|
157
|
+
because silently assigning a run to an arm is exactly the false saving the
|
|
158
|
+
paired measurement exists to prevent. The branch parser is strict for the same
|
|
159
|
+
reason: `exp/task-089/off-by-one` and `feat/exp/task-089/on` resolve to nothing
|
|
160
|
+
rather than to the nearest matching arm.
|
|
161
|
+
- **`pass_order` on the row.** The second implementation of the same task is
|
|
162
|
+
cheaper for a reason that has nothing to do with the arm — the solution is
|
|
163
|
+
already known — and that effect runs in the same direction as the one being
|
|
164
|
+
measured. Recorded per (task, arm) from the start time, it is what lets the
|
|
165
|
+
order effect be subtracted from the arm effect instead of being counted as one.
|
|
166
|
+
|
|
167
|
+
- **`restore` ends a measurement run even when it moved nothing.** The switch
|
|
168
|
+
only ever turns things *off*, so the `on` arm parks no files — and `restore`
|
|
169
|
+
used to refuse there, ledger entry included. That left the `on` arm's window
|
|
170
|
+
open forever: a follow-up a week later, in the same worktree on the same
|
|
171
|
+
`exp/` branch, still resolved to `branch+ledger` and joined the sample, while
|
|
172
|
+
the identical work after an `off` arm was flagged and excluded. A leak on one
|
|
173
|
+
side only is worse than a leak on both, because the two sides are what the
|
|
174
|
+
epic compares. Now `restore` closes the ledger window whether or not it had
|
|
175
|
+
bytes to put back, and refuses only when there is no open arm to close.
|
|
176
|
+
|
|
177
|
+
### Fixed
|
|
178
|
+
|
|
179
|
+
- **A cost that is not a number is no longer counted as a cost.** `NaN` is
|
|
180
|
+
neither `null` nor `undefined`, so a row carrying one passed every "is there a
|
|
181
|
+
price here?" check, entered the priced count, and turned the whole run's total
|
|
182
|
+
into `$NaN` — every other row's money gone while the counter still claimed to
|
|
183
|
+
have priced them. The published amount is now refused unless it is finite, and
|
|
184
|
+
the roll-up checks each class, not only the total. The way in is closed too:
|
|
185
|
+
the price, tier and geo tables are looked up by *own* key, so a `service_tier`
|
|
186
|
+
spelled `constructor` reads as unknown instead of handing back a function from
|
|
187
|
+
`Object.prototype`.
|
|
188
|
+
|
|
189
|
+
### Notes
|
|
190
|
+
|
|
191
|
+
- `rag off` covers the **project** `.mcp.json` only. `~/.claude.json` is rewritten
|
|
192
|
+
by Claude Code while it runs, so the switch never touches the user scope —
|
|
193
|
+
`experiment status` says this explicitly instead of implying RAG is globally
|
|
194
|
+
off.
|
|
195
|
+
- The arm belongs to the **session**: `settings.json` and MCP servers load at
|
|
196
|
+
startup, so a `set` applies from the next Claude Code start. Both `set` and
|
|
197
|
+
`status` say so; nothing here restarts anything for you.
|
|
198
|
+
|
|
8
199
|
## [2.3.1] — 2026-07-28
|
|
9
200
|
|
|
10
201
|
### Fixed
|
|
@@ -464,6 +655,8 @@ action needed on upgrade.
|
|
|
464
655
|
(`explore-cheap`, `reviewer`), the PreCompact / SessionStart / PostToolUse hook
|
|
465
656
|
suite, and the adoption report (`eval/adoption-report.mjs`).
|
|
466
657
|
|
|
658
|
+
[2.5.0]: https://gitlab.com/bglocation/tune-context/-/tags/v2.5.0
|
|
659
|
+
[2.4.0]: https://gitlab.com/bglocation/tune-context/-/tags/v2.4.0
|
|
467
660
|
[2.3.1]: https://gitlab.com/bglocation/tune-context/-/tags/v2.3.1
|
|
468
661
|
[2.3.0]: https://gitlab.com/bglocation/tune-context/-/tags/v2.3.0
|
|
469
662
|
[2.2.0]: https://gitlab.com/bglocation/tune-context/-/tags/v2.2.0
|
package/README.md
CHANGED
|
@@ -17,7 +17,7 @@ tune-context/
|
|
|
17
17
|
├── agents/ # explore-cheap, reviewer (model right-sizing)
|
|
18
18
|
├── hooks/ # PreCompact state-doc, SessionStart re-inject, adoption log, context reminder + hooks.json
|
|
19
19
|
├── bin/ + cli/ # `tune-context init` — deterministic detect → generate → verify
|
|
20
|
-
├── eval/ # adoption-report.mjs (the rag-usage analog)
|
|
20
|
+
├── eval/ # adoption-report.mjs (the rag-usage analog), prompt-metrics.mjs, pricing.mjs, arm.mjs, analyze.mjs, verdict.mjs, report.mjs
|
|
21
21
|
└── scripts/ # verify-pack, smoke-init, sync-tune-context (dev/maintenance)
|
|
22
22
|
```
|
|
23
23
|
|
|
@@ -332,19 +332,201 @@ instead of printing a number the doubling would have faked.
|
|
|
332
332
|
`~/.claude/tune-context/` (or the `CLAUDE_CONFIG_DIR` override) and records only
|
|
333
333
|
tool names, file paths, and skill/agent names — never full Bash command strings.
|
|
334
334
|
|
|
335
|
-
**Honest scope:** this is a *behavioral proxy*, not a bill.
|
|
336
|
-
|
|
337
|
-
|
|
335
|
+
**Honest scope:** this is a *behavioral proxy*, not a bill. The hook log carries
|
|
336
|
+
no tokens, so dollars come from `prompt-metrics` below (or from OTEL, which is
|
|
337
|
+
Anthropic's own figure); and caveman is an output style with no tool call, so it
|
|
338
338
|
can't be counted this way. Doctrine applied as *method* — narrow reads,
|
|
339
339
|
contracts-first edits, delegation — is only partly visible too: no `Skill` call,
|
|
340
340
|
but its trace shows up in the report's edit:read ratio, so the verdict never
|
|
341
341
|
reads zero `Skill` calls as "unused."
|
|
342
342
|
|
|
343
|
+
### Per-prompt metrics from transcripts (`eval/prompt-metrics.mjs`)
|
|
344
|
+
|
|
345
|
+
The adoption report reads *our* hook log. `prompt-metrics` reads the transcripts
|
|
346
|
+
**Claude Code writes itself** (`<config dir>/projects/<slug>/<session>.jsonl`) and
|
|
347
|
+
emits one JSONL row per prompt — time, tokens split by cache class, model,
|
|
348
|
+
effort, tools, delegations, and cost in dollars (`eval/pricing.mjs`):
|
|
349
|
+
|
|
350
|
+
```bash
|
|
351
|
+
node eval/prompt-metrics.mjs # every transcript
|
|
352
|
+
node eval/prompt-metrics.mjs --transcript <path.jsonl> # one session
|
|
353
|
+
node eval/prompt-metrics.mjs --dir <dir> > prompts.jsonl # JSONL on stdout
|
|
354
|
+
```
|
|
355
|
+
|
|
356
|
+
Rows go to stdout, a run summary to stderr — so a pipe stays clean. Two things
|
|
357
|
+
that source makes possible and a hook cannot: it works for **past** sessions, and
|
|
358
|
+
it works when tune-context is switched **off** (a disabled hook records nothing,
|
|
359
|
+
and that is exactly the arm a comparison needs).
|
|
360
|
+
|
|
361
|
+
Honest scope, printed on every run:
|
|
362
|
+
|
|
363
|
+
- **`list_cost_usd` is API list price, not your bill.** A Pro/Max subscription
|
|
364
|
+
charges a flat fee and consumes plan limits; nothing here is billed per token.
|
|
365
|
+
The figure answers *"what would this work have cost through the API"* — which
|
|
366
|
+
is the right yardstick for comparing two arms, and the wrong number to
|
|
367
|
+
reconcile against a statement. The field is named so the qualifier travels
|
|
368
|
+
with the data.
|
|
369
|
+
- **Repeated entries are skipped, and that changed every figure.** A transcript
|
|
370
|
+
writes some entries twice — 11 296 of 36 463 local ones (31%) repeat a `uuid`
|
|
371
|
+
already written *in the same file*, 6 739 of them assistant turns with a
|
|
372
|
+
`usage` block, the copies identical apart from a cosmetic `slug`. Summing both
|
|
373
|
+
charged 45% of all cache-read tokens twice: run totals from before 2026-08-02
|
|
374
|
+
are inflated ~1.97×. The skip is scoped to one file (no `uuid` is shared
|
|
375
|
+
between files) and the count is printed on every run.
|
|
376
|
+
- **Cache-aware, because a naive token sum is off by ~10×.** Cache reads cost
|
|
377
|
+
0.1× and writes 1.25×/2×; the local run carries 4.18 **billion** cache-read
|
|
378
|
+
tokens against 521 k plain input ones — 99.9% of the figure is cache and
|
|
379
|
+
output. The total never appears without its per-class split, and a model the
|
|
380
|
+
table doesn't know yields `null` plus an `UNPRICED` count, never a zero.
|
|
381
|
+
- **Prices are data, dated, and expire.** `eval/pricing.mjs` holds one table
|
|
382
|
+
taken from the published pricing page with its retrieval date, storing base
|
|
383
|
+
input/output only and deriving the three cache columns from the documented
|
|
384
|
+
1.25×/2×/0.1× multipliers — a test asserts the derivation reproduces every
|
|
385
|
+
published row. Service tier (batch −50%), fast mode, and `inference_geo`
|
|
386
|
+
(us +10%) are applied; the claude-sonnet-5 introductory price ends in the
|
|
387
|
+
table on 2026-08-31. Rows are priced at the **prompt's own timestamp**, so
|
|
388
|
+
re-running the report later yields the same figure.
|
|
389
|
+
- **Subagent cost is measured from below, not captured.** Subagent turns never
|
|
390
|
+
reach the parent transcript (zero sidechain entries over 34 local ones), but
|
|
391
|
+
the **result** of an `Agent` call carries its own `usage` and `resolvedModel`
|
|
392
|
+
— so each delegation is priced with the same table and rides on the row as
|
|
393
|
+
`delegations[]`, deduplicated by `agentId` because the entry is written twice.
|
|
394
|
+
That figure is a **lower bound**
|
|
395
|
+
(`cache_read` grows linearly with the subagent's tool calls, which looks like
|
|
396
|
+
its last turn rather than a sum) and is **never added** to the row's own cost.
|
|
397
|
+
`agent_spawns` is still counted next to it: the spawns that left no priced
|
|
398
|
+
result are cost there is no figure for at all.
|
|
399
|
+
- **Queue wait is not work.** `queued_s` (submission → first assistant entry) is
|
|
400
|
+
reported apart from `work_s`, because a prompt sent while the agent is busy
|
|
401
|
+
waits in a queue and that is the user's time. `max_gap_s` exposes the same
|
|
402
|
+
thing *inside* a turn — a tool call blocked on a permission prompt lands in
|
|
403
|
+
`work_s` otherwise.
|
|
404
|
+
- **Model and effort change mid-session** (measured), so both ride on every row,
|
|
405
|
+
with a per-model token split when a single prompt used more than one.
|
|
406
|
+
- Entries that can't be attributed to a prompt are **counted and reported**,
|
|
407
|
+
never silently dropped. Same for the two ways the figure can be too low:
|
|
408
|
+
`usage.iterations` reporting more than the `usage` we price from (3 turns
|
|
409
|
+
locally), and web searches, which are billed per request ($10/1000) rather
|
|
410
|
+
than per token and so sit outside the token total. Web *fetch* is free.
|
|
411
|
+
|
|
412
|
+
### The other arm (`tune-context experiment`)
|
|
413
|
+
|
|
414
|
+
A cost figure with nothing to compare it against is a number, not a measurement.
|
|
415
|
+
This switch turns the doctrine **off** so the same work can be run twice —
|
|
416
|
+
`tc={on,off}` × `rag={on,off}` — and records which arm was in force when:
|
|
417
|
+
|
|
418
|
+
```bash
|
|
419
|
+
tune-context experiment status # what is set, and what the config actually shows
|
|
420
|
+
tune-context experiment set --tc off --rag off --note "pair 1"
|
|
421
|
+
tune-context experiment restore # byte for byte, or a refusal
|
|
422
|
+
```
|
|
423
|
+
|
|
424
|
+
- **`tc off` is the full arm** (PO decision, EPIC-009): hook registrations in
|
|
425
|
+
`settings.json`, the managed block in the project `CLAUDE.md`, the doctrine
|
|
426
|
+
skills, the agents. Four interventions in one variable on purpose — at small
|
|
427
|
+
*n* the biggest effect is the easiest to detect; splitting them is the *next*
|
|
428
|
+
experiment.
|
|
429
|
+
- **`rag off` means this project's `.mcp.json`, and nothing else.** The user
|
|
430
|
+
scope (`~/.claude.json`) is rewritten by Claude Code while it runs, so the
|
|
431
|
+
switch never touches it — `status` says so out loud instead of letting you
|
|
432
|
+
assume RAG is off globally. Check `claude mcp list` before a measured run.
|
|
433
|
+
- **Reversibility comes before the change.** Every touched path is fingerprinted
|
|
434
|
+
and copied aside *before* it is modified, so a `set` that dies halfway is still
|
|
435
|
+
undoable. `restore` compares each path against what `set` left there and
|
|
436
|
+
**refuses** if anything changed in between, naming the file *and the
|
|
437
|
+
difference* (first differing line for a text file, changed/added/removed files
|
|
438
|
+
for a directory, and where the saved copy is) — the working session between the
|
|
439
|
+
two commands is exactly when you might add a permission or write below the
|
|
440
|
+
`CLAUDE.md` marker, and silently erasing that is worse than refusing.
|
|
441
|
+
- **The ledger is the source of truth**
|
|
442
|
+
(`<config dir>/tune-context/experiment.jsonl`, append-only). An analyzer
|
|
443
|
+
assigns a session to an arm by timestamp; detecting the arm from a transcript
|
|
444
|
+
is a cross-check, never the verdict. Every `set` appends — including one that
|
|
445
|
+
changes nothing, because a run whose ledger is silent at its start cannot be
|
|
446
|
+
assigned to an arm at all.
|
|
447
|
+
- **The arm belongs to the session.** `settings.json` and MCP servers are read at
|
|
448
|
+
startup, so it applies from the next Claude Code start; flipping it mid-session
|
|
449
|
+
produces a mixed session, which the analysis throws away.
|
|
450
|
+
- `tune-context verify` reads the arm and reports **"off"** rather than a broken
|
|
451
|
+
install — and stays RED if the arm did not fully land. `init` refuses to run
|
|
452
|
+
while an arm is set: it would re-install what the arm parked and leave
|
|
453
|
+
`restore` with nothing it recognises.
|
|
454
|
+
|
|
455
|
+
The measurement protocol that uses this — paired runs on `exp/<task>/<arm>`
|
|
456
|
+
branches, frozen prompt script, verdict gates — is
|
|
457
|
+
[COST_EVALUATION_PROCEDURE.md](COST_EVALUATION_PROCEDURE.md).
|
|
458
|
+
|
|
459
|
+
### Per-arm aggregate (`eval/analyze.mjs`)
|
|
460
|
+
|
|
461
|
+
Rows are data, not an answer. `analyze` turns a stream of them into one
|
|
462
|
+
aggregate per arm — and stops one step short of ruling which arm won, because
|
|
463
|
+
the thresholds that would do the ruling are frozen by the PO, not by the model:
|
|
464
|
+
|
|
465
|
+
```bash
|
|
466
|
+
node eval/prompt-metrics.mjs > rows.jsonl
|
|
467
|
+
node eval/analyze.mjs --rows rows.jsonl > aggregate.json # summary on stderr
|
|
468
|
+
```
|
|
469
|
+
|
|
470
|
+
- **Two populations, never one number.** The paired measurement (branch
|
|
471
|
+
`exp/<task>/<arm>`, everything else held constant) and ordinary daily work
|
|
472
|
+
(attributed by the ledger, nothing held constant) come out as two separate
|
|
473
|
+
objects with different shapes. There is no total above them and no list of
|
|
474
|
+
pools to fold over — adding them has to be written out by hand and meant.
|
|
475
|
+
- **Discards are the output too.** Every row lands either in an arm or in a
|
|
476
|
+
named counter, and `rows_in = rows_used + rows_rejected` is asserted by a
|
|
477
|
+
test. A session whose rows disagree about the arm is dropped **whole** —
|
|
478
|
+
`settings.json` and MCP servers load at startup, so a session that switched
|
|
479
|
+
arms was never in one — and a branch that *mimics* `exp/<task>/<arm>` without
|
|
480
|
+
matching it (`exp/task-089`, `exp/`) goes to a counter, never to the nearest
|
|
481
|
+
plausible arm.
|
|
482
|
+
- **Median and spread, not the mean alone.** One 44-hour turn or one prompt that
|
|
483
|
+
read a 200 k-token file moves a mean of six and leaves the median untouched;
|
|
484
|
+
the gap between the two is itself the finding, so both are reported with
|
|
485
|
+
p25/p75, IQR and the extremes.
|
|
486
|
+
- **An unpriced row leaves the cost, not the sample.** It stays in `n_prompts`,
|
|
487
|
+
sits outside the total, and the pool says `cost_complete: false` with the
|
|
488
|
+
model that caused it named.
|
|
489
|
+
- **`pass_order` per pair.** The second implementation of a task is cheaper for
|
|
490
|
+
a reason that has nothing to do with the arm, and it pushes the same way as
|
|
491
|
+
the effect being looked for. The per-pair, per-arm cost is carried so the two
|
|
492
|
+
can be separated arithmetically (procedure §11b).
|
|
493
|
+
|
|
494
|
+
### The report (`eval/report.mjs`)
|
|
495
|
+
|
|
496
|
+
One HTML file — data embedded as JSON, charts as inline SVG, no network and no
|
|
497
|
+
build step, so it opens from `file://` on a machine with nothing installed:
|
|
498
|
+
|
|
499
|
+
```bash
|
|
500
|
+
node eval/analyze.mjs --rows rows.jsonl > aggregate.json
|
|
501
|
+
node eval/report.mjs --aggregate aggregate.json --out report.html
|
|
502
|
+
node eval/report.mjs --aggregate aggregate.json --rejected-runs journal.json --out report.html
|
|
503
|
+
```
|
|
504
|
+
|
|
505
|
+
`--rejected-runs` carries what the run journal recorded as §7 invalidations
|
|
506
|
+
(`[{ session, task, arm, prompts, reason }]`). **Omitting it is not the same as
|
|
507
|
+
passing `[]`**: the first says nobody asked the journal, and the verdict is held
|
|
508
|
+
for exactly that reason — a rejection share whose denominator is unknown is a
|
|
509
|
+
quietly dropped sample wearing a passing gate.
|
|
510
|
+
|
|
511
|
+
- **A withheld verdict is the page, not a footnote.** When a gate holds, the
|
|
512
|
+
arm-comparison chart is *not drawn at all* — not greyed out, not captioned. Two
|
|
513
|
+
bars side by side assert "this is a comparison" more loudly than any caption
|
|
514
|
+
can withdraw it. The per-pair costs stay in a table: those are what a repeat
|
|
515
|
+
run is planned from.
|
|
516
|
+
- **The order effect is printed the same size as the arm effect.** A page that
|
|
517
|
+
shows one large and the other in small print has already made the claim §11b
|
|
518
|
+
refuses.
|
|
519
|
+
- **Tokens never appear without their five classes**, the two pools are never
|
|
520
|
+
added, and the honesty counters — rejected rows and why, sessions dropped for
|
|
521
|
+
switching arm mid-flight, the delegation cost that is a floor and not a total —
|
|
522
|
+
are a section of their own rather than a footer.
|
|
523
|
+
|
|
343
524
|
### Getting real cost (OTEL) — outside tune-context
|
|
344
525
|
|
|
345
526
|
tune-context does not set this up for you (deliberate — see TASK-062: it needs
|
|
346
|
-
infra this tool stays zero-infra without).
|
|
347
|
-
|
|
527
|
+
infra this tool stays zero-infra without). `prompt-metrics` above computes cost
|
|
528
|
+
from a public price list, which is an *independent* calculation, not Anthropic's
|
|
529
|
+
invoice — OTEL is the way to check it against the real one:
|
|
348
530
|
|
|
349
531
|
```bash
|
|
350
532
|
export CLAUDE_CODE_ENABLE_TELEMETRY=1 # off by default
|