claude-token-saver 3.12.1 → 3.14.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.en.md +19 -1
- package/README.md +29 -1
- package/package.json +1 -1
- package/src/commands/route-scan.js +52 -0
- package/src/cost.js +23 -0
- package/src/formatters/statusline.js +15 -1
- package/src/model-alias.js +12 -3
- package/src/model-rules.js +15 -0
- package/src/route-scan.js +107 -15
- package/src/savings-ledger.js +64 -13
package/README.en.md
CHANGED
|
@@ -199,6 +199,7 @@ claude-token-saver route-scan # scan (24h cache) + tiered can
|
|
|
199
199
|
claude-token-saver harness promote R1 --project # promote candidate R1 to a model-fitting rule
|
|
200
200
|
claude-token-saver route-scan dismiss 1 # not interested — won't resurface
|
|
201
201
|
claude-token-saver route-scan rules # list model-fitting rules (rm <N> to remove)
|
|
202
|
+
claude-token-saver route-scan savings # the savings ledger — which rule moved work off which model, onto which
|
|
202
203
|
```
|
|
203
204
|
|
|
204
205
|
Dig deeper: **tier criteria & research evidence** → [docs/TIER_CRITERIA.md](./docs/TIER_CRITERIA.md) (Korean) · **rule-file mechanics, scan triggers, subagent setup** → [docs/ROUTE_SCAN.md](./docs/ROUTE_SCAN.md) (Korean + English)
|
|
@@ -214,11 +215,15 @@ For environments the learner cannot reach, write the mapping yourself in `<userD
|
|
|
214
215
|
```jsonc
|
|
215
216
|
{
|
|
216
217
|
"modelAliases": {
|
|
217
|
-
"arn:aws:bedrock:*:*:application-inference-profile/<PROFILE_ID>": "claude-opus-5"
|
|
218
|
+
"arn:aws:bedrock:*:*:application-inference-profile/<PROFILE_ID>": "claude-opus-5",
|
|
219
|
+
"prod-large": "claude-opus-5", // house aliases map the same way
|
|
220
|
+
"team-*": "claude-haiku-4-5"
|
|
218
221
|
}
|
|
219
222
|
}
|
|
220
223
|
```
|
|
221
224
|
|
|
225
|
+
**Map house aliases that carry no family name** (`prod-large`, `team-fast`) here too. Shapes that keep the family name are recognized as-is — Bedrock (`anthropic.claude-opus-4-5-v1:0`), Vertex (`claude-opus-4-5@20251101`), and the 1M suffix (`claude-sonnet-4-5[1m]`) — but an alias without one cannot be priced. Rather than report a wrong figure, routing-savings **drops those runs from the aggregate** (both sides of the comparison must be recognizable); one line in the table above brings them back.
|
|
226
|
+
|
|
222
227
|
That file holds internal identifiers in plain text — do not commit it. On a direct-API machine it is never created and behaviour is unchanged.
|
|
223
228
|
|
|
224
229
|
## Spike issue codes
|
|
@@ -301,6 +306,19 @@ Also update `statusLine.command` in `~/.claude/settings.json` to `claude-token-s
|
|
|
301
306
|
|
|
302
307
|
## Release notes
|
|
303
308
|
|
|
309
|
+
### v3.14.0 (2026-08-22)
|
|
310
|
+
- **The statusline headline names the model changes behind the total** — `🔀 Routing saved weekly $1.4 · monthly $2.1 · total $2.1 | fable→sonnet 1× $0.72 · opus→haiku 1× $0.57 …`. Versions are dropped (`claude-opus-4-5-20251101-v1:0` → `opus`) since they bump constantly and add nothing here. **Every** pair is listed rather than a top-N: the amounts sit next to a total, so a truncated list misstates what that total is made of. Families collapse the list on their own, so it stays short without being cut.
|
|
311
|
+
- **New `route-scan savings`** — the evidence behind the headline: a per-model-change rollup (from, to, runs, dollars) and the per-run log (date, amount, model change, the rule that caused it).
|
|
312
|
+
- **Baselines are the model that handled a category most, not the priciest seen** — transcripts routinely carry more than one model (a mid-session switch), and taking the priciest let a single Fable record set the baseline for a category Opus had handled thirty times, inflating every later saving. Ties break toward the pricier model. Baselines written under the old definition are recomputed once.
|
|
313
|
+
- **The ledger is written after the rule refresh** — writing it first made a changed baseline take two scans to settle: the first stored the new baseline but billed against the old one.
|
|
314
|
+
|
|
315
|
+
### v3.13.0 (2026-08-22)
|
|
316
|
+
- **Routing savings are now priced "model before the rule → model that ran it"** — previously the counterfactual was the session's priciest model, which booked a saving even when that model had never handled work of this shape. Each rule now remembers the **model that handled its category before promotion** (its baseline) and prices only against that. The baseline is sticky once set: as a rule takes effect fewer episodes stay on the expensive model, so a recomputed baseline would drift down and shrink the very saving the rule produces.
|
|
317
|
+
- **Delegations no rule covers are excluded** — `Explore`, hand-written agents, and plugin subagents were all being counted, crediting this tool for routing it never did. Ledger events now carry `rule` / `from` / `to`, so every dollar is traceable to the rule and model change behind it.
|
|
318
|
+
- **Ledger schema version 2** — v1 events used the old counterfactual and are discarded rather than migrated; the next `route-scan` refills whatever is genuinely attributable.
|
|
319
|
+
- **House alias model ids no longer price as Sonnet by accident** — when a gateway reports an alias carrying no family name (`prod-large`), the pricing table's default made it Sonnet, fabricating or erasing savings. Both sides of a comparison must now be recognizable ids. Bedrock (`anthropic.claude-opus-4-5-v1:0`), Vertex (`claude-opus-4-5@20251101`), and the `[1m]` suffix are recognized as-is; house aliases come back into the aggregate with one line in `modelAliases` (wildcards supported).
|
|
320
|
+
- `harness check` also reports CLAUDE.md size and whether a `.claudeignore` exists (advisory; the 🅷 score is unaffected).
|
|
321
|
+
|
|
304
322
|
### v3.12.1 (2026-08-22)
|
|
305
323
|
- **Routing-savings headline reads more clearly** — the amounts are now in the savings green (the one unambiguously good number on the line), `wk`/`mo`/`all` are spelled out as `weekly`/`monthly`/`total`, and the period now leads the amount. Three bare amounts in a row read as a single number until the eye found the trailing marker.
|
|
306
324
|
|
package/README.md
CHANGED
|
@@ -179,6 +179,17 @@ claude-token-saver route-scan # 스캔 (24h 캐시) + 티어
|
|
|
179
179
|
claude-token-saver harness promote R1 --project # 후보 R1을 모델 피팅 룰로 등록
|
|
180
180
|
claude-token-saver route-scan dismiss 1 # 관심 없으면 무시 (재스캔에도 안 뜸)
|
|
181
181
|
claude-token-saver route-scan rules # 등록된 모델 피팅 룰 목록 (rm <N>으로 제거)
|
|
182
|
+
claude-token-saver route-scan savings # 절감 원장 — 어느 룰이 어떤 모델에서 어떤 모델로 옮겼는지
|
|
183
|
+
```
|
|
184
|
+
|
|
185
|
+
`route-scan savings`는 statusline의 `🔀 Routing saved` 한 줄 뒤에 있는 근거를 그대로 보여줍니다. 모델 이동별 합계와 실행별 내역이 함께 나오므로, 금액이 어디서 나왔는지 추적할 수 있습니다.
|
|
186
|
+
|
|
187
|
+
```
|
|
188
|
+
🔀 라우팅 절감 — 주간 $1.40 · 월간 $2.09 · 누적 $2.09
|
|
189
|
+
|
|
190
|
+
모델 이동별:
|
|
191
|
+
claude-fable-5 → claude-sonnet-5 — 1회, $0.72
|
|
192
|
+
claude-opus-5 → claude-haiku-4-5 — 1회, $0.57
|
|
182
193
|
```
|
|
183
194
|
|
|
184
195
|
더 알아보기: **티어 기준·리서치 근거** → [docs/TIER_CRITERIA.md](./docs/TIER_CRITERIA.md) · **룰 파일 구조·스캔 트리거·서브에이전트 준비** → [docs/ROUTE_SCAN.md](./docs/ROUTE_SCAN.md)
|
|
@@ -194,11 +205,15 @@ v3.10.0부터는 프로파일 ID를 역할(main·opus·sonnet·haiku)로 되돌
|
|
|
194
205
|
```jsonc
|
|
195
206
|
{
|
|
196
207
|
"modelAliases": {
|
|
197
|
-
"arn:aws:bedrock:*:*:application-inference-profile/<PROFILE_ID>": "claude-opus-5"
|
|
208
|
+
"arn:aws:bedrock:*:*:application-inference-profile/<PROFILE_ID>": "claude-opus-5",
|
|
209
|
+
"prod-large": "claude-opus-5", // 사내 별칭도 같은 방식으로 매핑됩니다
|
|
210
|
+
"team-*": "claude-haiku-4-5"
|
|
198
211
|
}
|
|
199
212
|
}
|
|
200
213
|
```
|
|
201
214
|
|
|
215
|
+
**모델명에 `opus`·`sonnet`·`haiku`·`fable` 이 들어 있지 않은 사내 별칭**(`prod-large`, `team-fast` 등)도 이 표로 매핑하십시오. Bedrock(`anthropic.claude-opus-4-5-v1:0`)·Vertex(`claude-opus-4-5@20251101`)·1M 접미사(`claude-sonnet-4-5[1m]`) 같이 계열명이 남아 있는 형태는 그대로 인식되지만, 계열명이 사라진 별칭은 가격표가 알아볼 수 없습니다. 이 경우 라우팅 절감 계산은 **틀린 금액을 내놓는 대신 그 실행을 집계에서 제외**하며(비교 양쪽 모두 인식 가능한 이름이어야 합니다), 위 표에 한 줄 추가하면 다시 집계에 들어옵니다.
|
|
216
|
+
|
|
202
217
|
이 파일에는 사내 식별자가 평문으로 남으므로 저장소에 커밋하지 마십시오. 게이트웨이를 쓰지 않는 환경에서는 파일이 아예 만들어지지 않고 기존 동작이 그대로 유지됩니다.
|
|
203
218
|
|
|
204
219
|
## 토큰 급증 원인 코드
|
|
@@ -257,6 +272,19 @@ npm uninstall -g claude-cache-monitor && npm i -g claude-token-saver
|
|
|
257
272
|
|
|
258
273
|
## 릴리스 노트
|
|
259
274
|
|
|
275
|
+
### v3.14.0 (2026-08-22)
|
|
276
|
+
- **statusline 헤드라인이 모델 이동을 함께 보여줍니다** — `🔀 Routing saved weekly $1.4 · monthly $2.1 · total $2.1 | fable→sonnet 1× $0.72 · opus→haiku 1× $0.57 …`. 버전 숫자는 계속 올라가고 statusline에서는 잡음이라 계열명만 남깁니다(`claude-opus-4-5-20251101-v1:0` → `opus`). 이동 목록은 **자르지 않고 전부** 표시합니다 — 금액이 `total` 옆에 붙어 있어서 일부만 보이면 합계를 잘못 말하게 됩니다. 계열 단위로 접히면 조합 수가 원래 많지 않아 줄은 짧게 유지됩니다.
|
|
277
|
+
- **`route-scan savings` 추가** — 헤드라인 뒤에 있는 근거를 그대로 조회합니다. 모델 이동별 합계(어느 모델에서 어느 모델로 몇 회, 얼마)와 실행별 내역(날짜·금액·모델 이동·해당 룰)이 함께 나옵니다.
|
|
278
|
+
- **기준 모델을 최고가가 아니라 '가장 많이 처리한 모델'로 정합니다** — 한 트랜스크립트에 모델이 섞이는 일이 흔한데(세션 중 모델 전환 등) 최고가를 고르면 Fable 기록 한 건이 Opus가 서른 번 처리한 카테고리의 기준을 차지해 이후 절감액을 전부 부풀렸습니다. 동수일 때만 비싼 쪽으로 기웁니다. 옛 정의로 굳은 기준은 1회 재계산됩니다.
|
|
279
|
+
- **원장 기록을 룰 갱신 뒤로 옮겼습니다** — 앞서 기록하면 기준이 바뀐 스캔이 새 기준을 저장하면서 청구는 옛 기준으로 해, 합계가 두 번째 스캔에야 맞았습니다.
|
|
280
|
+
|
|
281
|
+
### v3.13.0 (2026-08-22)
|
|
282
|
+
- **라우팅 절감액의 기준을 '승격 전 모델 → 위임 모델' 차액으로 바꿨습니다** — 이전에는 세션의 최상위 모델을 반사실로 잡아, 그 모델이 해당 유형을 실제로 처리한 적이 없어도 차액을 절감으로 기록했습니다. 이제 각 룰이 **승격 전 그 유형을 직접 처리하던 모델**(baseline)을 기억하고, 그 기준 대비로만 계산합니다. baseline은 한 번 정해지면 고정됩니다 — 룰이 효력을 낼수록 직접 처리 사례가 줄어 기준이 흘러내리고, 그러면 룰이 만든 절감이 스스로 작아지기 때문입니다.
|
|
283
|
+
- **룰이 커버하지 않는 위임은 집계에서 뺐습니다** — `Explore`, 직접 만든 에이전트, 플러그인 에이전트처럼 이 도구와 무관하게 돌던 서브에이전트 실행까지 절감으로 잡히고 있었습니다. 도구가 라우팅하지 않은 작업의 절감을 도구 성과로 표시하면 안 됩니다. 원장 이벤트에 `rule`/`from`/`to`를 남겨 어느 룰이 어떤 모델 차이를 만들었는지 추적할 수 있습니다.
|
|
284
|
+
- **원장 스키마 version 2** — 반사실 기준이 달라진 만큼 v1 항목은 마이그레이션 없이 폐기합니다(두 의미를 한 합계에 섞을 수 없습니다). 다음 `route-scan`이 귀속 가능한 절감만 다시 채웁니다.
|
|
285
|
+
- **사내 별칭 모델명이 조용히 Sonnet으로 계산되지 않습니다** — 게이트웨이가 계열명 없는 별칭(`prod-large` 등)을 모델명으로 기록하면 가격표 기본값이 걸려 Sonnet으로 계산됐고, 그 결과 없는 절감이 생기거나 있는 절감이 지워졌습니다. 이제 비교 양쪽 모두 계열명이 남아 있는 id일 때만 집계합니다. Bedrock(`anthropic.claude-opus-4-5-v1:0`)·Vertex(`claude-opus-4-5@20251101`)·1M 접미사(`claude-sonnet-4-5[1m]`)는 그대로 인식되고, 사내 별칭은 `profile-map.json`의 `modelAliases`에 한 줄 추가하면 집계에 복귀합니다(와일드카드 가능).
|
|
286
|
+
- `harness check`가 CLAUDE.md 크기와 `.claudeignore` 유무를 함께 보고합니다(자문 정보, 🅷 점수에는 미반영).
|
|
287
|
+
|
|
260
288
|
### v3.12.1 (2026-08-22)
|
|
261
289
|
- **라우팅 절감 헤드라인의 가독성 정리** — 금액을 절감 녹색으로 칠하고(그 줄에서 유일하게 명확한 호재입니다), `wk`·`mo`·`all` 축약을 `weekly`·`monthly`·`total`로 풀었으며, 금액이 앞서던 순서를 뒤집어 기간이 먼저 오게 했습니다. 금액 셋이 연달아 나오면 뒤따르는 기간 표시를 찾기 전까지 한 덩어리로 읽혔습니다.
|
|
262
290
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "claude-token-saver",
|
|
3
|
-
"version": "3.
|
|
3
|
+
"version": "3.14.0",
|
|
4
4
|
"description": "Route the easy work your expensive Claude model keeps repeating down to haiku/sonnet — post-hoc session analysis, no realtime router, no extra LLM calls.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
@@ -10,6 +10,58 @@ export async function run({ args, hasFlag, numArg }) {
|
|
|
10
10
|
const { userLanguage } = await import('../config.js');
|
|
11
11
|
const lang = userLanguage();
|
|
12
12
|
|
|
13
|
+
// route-scan savings — the delegation ledger behind the statusline's
|
|
14
|
+
// "Routing saved" headline. The headline is one number; this is the
|
|
15
|
+
// evidence for it: which rule fired, which model the work moved off, and
|
|
16
|
+
// which model actually ran it.
|
|
17
|
+
if (args[1] === 'savings') {
|
|
18
|
+
const { loadLedger, delegationSavedTotals } = await import('../savings-ledger.js');
|
|
19
|
+
const events = Object.entries(loadLedger().events)
|
|
20
|
+
.map(([key, e]) => ({ key, ...e }))
|
|
21
|
+
.sort((a, b) => b.ts - a.ts);
|
|
22
|
+
if (events.length === 0) {
|
|
23
|
+
console.log(lang === 'ko'
|
|
24
|
+
? '기록된 라우팅 절감 없음. 승격된 룰이 실제로 위임을 일으킨 뒤 `route-scan --refresh` 를 돌리면 채워집니다.'
|
|
25
|
+
: 'No routing savings recorded yet. Promote a rule, let it delegate, then run `route-scan --refresh`.');
|
|
26
|
+
return;
|
|
27
|
+
}
|
|
28
|
+
const t = delegationSavedTotals();
|
|
29
|
+
const money = (v) => `$${v.toFixed(2)}`;
|
|
30
|
+
console.log(lang === 'ko'
|
|
31
|
+
? `🔀 라우팅 절감 — 주간 ${money(t.week)} · 월간 ${money(t.month)} · 누적 ${money(t.total)}`
|
|
32
|
+
: `🔀 Routing saved — weekly ${money(t.week)} · monthly ${money(t.month)} · total ${money(t.total)}`);
|
|
33
|
+
|
|
34
|
+
// Per model-pair rollup first: the "what moved where" question is what
|
|
35
|
+
// this view exists to answer, and it is easier to read than the log.
|
|
36
|
+
const pairs = new Map();
|
|
37
|
+
for (const e of events) {
|
|
38
|
+
const k = `${e.from || '?'} → ${e.to || '?'}`;
|
|
39
|
+
const p = pairs.get(k) || { runs: 0, usd: 0 };
|
|
40
|
+
p.runs += 1;
|
|
41
|
+
p.usd += Number(e.usd) || 0;
|
|
42
|
+
pairs.set(k, p);
|
|
43
|
+
}
|
|
44
|
+
console.log('');
|
|
45
|
+
console.log(lang === 'ko' ? '모델 이동별:' : 'By model change:');
|
|
46
|
+
for (const [k, p] of [...pairs].sort((a, b) => b[1].usd - a[1].usd)) {
|
|
47
|
+
const runs = lang === 'ko' ? `${p.runs}회` : `${p.runs} run${p.runs === 1 ? '' : 's'}`;
|
|
48
|
+
console.log(` ${k} — ${runs}, ${money(p.usd)}`);
|
|
49
|
+
}
|
|
50
|
+
|
|
51
|
+
console.log('');
|
|
52
|
+
console.log(lang === 'ko' ? '실행별 (최근순):' : 'By run (newest first):');
|
|
53
|
+
for (const e of events) {
|
|
54
|
+
const when = new Date(e.ts).toISOString().slice(0, 10);
|
|
55
|
+
console.log(` ${when} ${money(Number(e.usd) || 0).padStart(7)} ${e.from || '?'} → ${e.to || '?'}`);
|
|
56
|
+
console.log(` ${lang === 'ko' ? '룰' : 'rule'}: ${e.rule || '(unattributed)'}`);
|
|
57
|
+
}
|
|
58
|
+
console.log('');
|
|
59
|
+
console.log(lang === 'ko'
|
|
60
|
+
? '금액은 "룰 승격 전 그 유형을 처리하던 모델"과 실제 실행 모델의 가격 차이입니다 (토큰 수는 고정 가정).'
|
|
61
|
+
: 'Each amount is the price gap between the model that handled this category before the rule and the model that actually ran it (token counts held constant).');
|
|
62
|
+
return;
|
|
63
|
+
}
|
|
64
|
+
|
|
13
65
|
// route-scan rules [rm <N>] — the model-fitting rule registry (rules
|
|
14
66
|
// promoted from candidates; auto-refreshed from logs on every rescan).
|
|
15
67
|
if (args[1] === 'rules') {
|
package/src/cost.js
CHANGED
|
@@ -128,6 +128,29 @@ export function isUnknownModel(model) {
|
|
|
128
128
|
return !model || String(model).toLowerCase() === 'unknown';
|
|
129
129
|
}
|
|
130
130
|
|
|
131
|
+
/**
|
|
132
|
+
* True when the id actually names a Claude family this table can price, as
|
|
133
|
+
* opposed to falling through to the Sonnet default.
|
|
134
|
+
*
|
|
135
|
+
* `detectPricingTier` must keep defaulting — a plain cost estimate is better
|
|
136
|
+
* off guessing Sonnet than refusing to answer. But anything that compares two
|
|
137
|
+
* models must not: behind a company gateway an id can be a house alias
|
|
138
|
+
* (`prod-large`, `team-fast`) carrying no family name, and pricing that as
|
|
139
|
+
* Sonnet silently invents or erases a delegation saving. Callers that need a
|
|
140
|
+
* real comparison gate on this and skip when it is false.
|
|
141
|
+
*
|
|
142
|
+
* Covers the shapes gateways actually emit — Bedrock
|
|
143
|
+
* (`anthropic.claude-opus-4-5-v1:0`, `us.anthropic.…`), Vertex
|
|
144
|
+
* (`claude-opus-4-5@20251101`), and the `[1m]` context suffix — because all of
|
|
145
|
+
* them keep the family name in the string. House aliases that do not are
|
|
146
|
+
* exactly what this returns false for; map those in profile-map.json's
|
|
147
|
+
* `modelAliases`.
|
|
148
|
+
*/
|
|
149
|
+
export function isRecognizedModelId(model) {
|
|
150
|
+
if (isUnknownModel(model)) return false;
|
|
151
|
+
return /fable|mythos|opus|sonnet|haiku/i.test(String(model));
|
|
152
|
+
}
|
|
153
|
+
|
|
131
154
|
export function modelRank(model) {
|
|
132
155
|
// -1 sits below every real tier, so worthDelegating() rejects it and
|
|
133
156
|
// tierForRank() attributes no saving to it: the run leaves the aggregate
|
|
@@ -291,11 +291,25 @@ export function formatReport(data, { color = true, verbose = false, timer = true
|
|
|
291
291
|
const part = (usd, label) =>
|
|
292
292
|
`${c(GRAY)}${label}${c(RESET)} ${c(GREEN)}${formatMoney(Number(usd) || 0)}${c(RESET)}`;
|
|
293
293
|
const head = isIcon ? '🔀 Routing saved' : 'Routing saved';
|
|
294
|
+
// Model changes behind the total, family-level and version-free: `opus →
|
|
295
|
+
// haiku 2× $0.6`. Versions bump constantly and add nothing here — the
|
|
296
|
+
// shape of the trade is the point.
|
|
297
|
+
//
|
|
298
|
+
// Every pair is listed, not a top-N: the amounts are shown next to a
|
|
299
|
+
// total, so a truncated list reads as "this is what the total is made of"
|
|
300
|
+
// and quietly misstates it. Families collapse the list on their own —
|
|
301
|
+
// there are only so many tier-to-tier moves — so it stays short without
|
|
302
|
+
// being cut.
|
|
303
|
+
const pairs = Array.isArray(totals.pairs) ? totals.pairs : [];
|
|
304
|
+
const pairText = pairs
|
|
305
|
+
.map((p) => `${c(GRAY)}${p.from}→${p.to}${c(RESET)} ${c(GRAY)}${p.runs}×${c(RESET)} ${c(GREEN)}${formatMoney(p.usd)}${c(RESET)}`)
|
|
306
|
+
.join(` ${c(GRAY)}·${c(RESET)} `);
|
|
294
307
|
totalsLine =
|
|
295
308
|
`${c(GREEN)}${c(BOLD)}${head}${c(RESET)} ` +
|
|
296
309
|
`${part(totals.week, 'weekly')} ${c(GRAY)}·${c(RESET)} ` +
|
|
297
310
|
`${part(totals.month, 'monthly')} ${c(GRAY)}·${c(RESET)} ` +
|
|
298
|
-
`${part(totals.total, 'total')}
|
|
311
|
+
`${part(totals.total, 'total')}` +
|
|
312
|
+
(pairText ? ` ${c(GRAY)}|${c(RESET)} ${pairText}` : '');
|
|
299
313
|
}
|
|
300
314
|
|
|
301
315
|
// Period label honors hour-precision configs (`mode 6h` → "6h", `mode 1d` → "1d").
|
package/src/model-alias.js
CHANGED
|
@@ -134,8 +134,10 @@ function overrideAlias(model, map) {
|
|
|
134
134
|
for (const [pattern, alias] of Object.entries(map.modelAliases || {})) {
|
|
135
135
|
if (globMatch(pattern, model)) return alias;
|
|
136
136
|
// Overrides are usually written without the LiteLLM `converse/` prefix.
|
|
137
|
-
|
|
138
|
-
|
|
137
|
+
// Only meaningful for ARNs — on a plain id indexOf returns -1 and the
|
|
138
|
+
// slice would hand the matcher a single trailing character.
|
|
139
|
+
const at = model.indexOf('arn:aws:bedrock:');
|
|
140
|
+
if (at > 0 && globMatch(pattern, model.slice(at))) return alias;
|
|
139
141
|
}
|
|
140
142
|
return null;
|
|
141
143
|
}
|
|
@@ -150,13 +152,20 @@ function overrideAlias(model, map) {
|
|
|
150
152
|
export function resolveModelAlias(rawModel, { env = process.env } = {}) {
|
|
151
153
|
if (!rawModel) return UNKNOWN_MODEL;
|
|
152
154
|
const model = String(rawModel);
|
|
153
|
-
if (!isGatewayModelId(model)) return model;
|
|
154
155
|
|
|
155
156
|
const map = loadProfileMap();
|
|
156
157
|
|
|
158
|
+
// Overrides are consulted for EVERY id, not just ARNs. A gateway can be
|
|
159
|
+
// configured to report a house alias (`prod-large`, `team-fast`) that names
|
|
160
|
+
// no Claude family at all; those never reach the ARN branch below, and left
|
|
161
|
+
// alone they price as Sonnet by default. One `modelAliases` entry maps them
|
|
162
|
+
// back. Ids that already name a family are left untouched — an override
|
|
163
|
+
// pattern has to match before anything changes.
|
|
157
164
|
const override = overrideAlias(model, map);
|
|
158
165
|
if (override) return override;
|
|
159
166
|
|
|
167
|
+
if (!isGatewayModelId(model)) return model;
|
|
168
|
+
|
|
160
169
|
const pid = profileIdFrom(model);
|
|
161
170
|
if (!pid) return UNKNOWN_MODEL;
|
|
162
171
|
|
package/src/model-rules.js
CHANGED
|
@@ -340,6 +340,21 @@ export function refreshModelRules(episodeStats, delegatedStats = new Map(), { no
|
|
|
340
340
|
if (s) {
|
|
341
341
|
r.count = s.count;
|
|
342
342
|
r.errRate = s.epCount > 0 ? s.errCount / s.epCount : 0;
|
|
343
|
+
// Baseline = the model that handled this category before the rule moved
|
|
344
|
+
// it. Sticky once set: as a rule takes effect, fewer episodes stay on the
|
|
345
|
+
// expensive model, so a recomputed baseline would drift downward and
|
|
346
|
+
// shrink the very savings the rule is producing.
|
|
347
|
+
//
|
|
348
|
+
// `baselineSource` marks which definition produced it. Baselines written
|
|
349
|
+
// before the definition changed from "priciest model seen" to "model that
|
|
350
|
+
// handled the most episodes" are recomputed once — a single stray record
|
|
351
|
+
// of a pricier model could otherwise hold the baseline above what the
|
|
352
|
+
// category was really running on, inflating every saving priced against it.
|
|
353
|
+
if (s.baselineModel && (!r.baselineModel || r.baselineSource !== 'dominant')) {
|
|
354
|
+
r.baselineModel = s.baselineModel;
|
|
355
|
+
r.baselineSource = 'dominant';
|
|
356
|
+
changed = true;
|
|
357
|
+
}
|
|
343
358
|
}
|
|
344
359
|
// Window snapshot, not a running total: these describe the current scan
|
|
345
360
|
// window so a rule that stopped firing decays to zero instead of coasting
|
package/src/route-scan.js
CHANGED
|
@@ -23,7 +23,7 @@ import { userDataDir } from './paths.js';
|
|
|
23
23
|
import { discoverSessionFiles } from './parser.js';
|
|
24
24
|
import { collectSessionRecords } from './session-records.js';
|
|
25
25
|
import { collectSubagentRuns, indexRuns, runsForEpisode } from './subagent-records.js';
|
|
26
|
-
import { estimateCost, modelRank, TIER_TARGET_RANK, tierForRank } from './cost.js';
|
|
26
|
+
import { estimateCost, modelRank, isRecognizedModelId, TIER_TARGET_RANK, tierForRank } from './cost.js';
|
|
27
27
|
import { learnProfileMapping, resetModelAliasCache } from './model-alias.js';
|
|
28
28
|
import { agentPhrase, agentPhraseEn } from './agents.js';
|
|
29
29
|
|
|
@@ -262,6 +262,24 @@ export function episodeRank(ep) {
|
|
|
262
262
|
* net loss. (Replaces the old boolean "is it haiku?" test, which could not
|
|
263
263
|
* tell a Sonnet session from a Fable one.)
|
|
264
264
|
*/
|
|
265
|
+
/**
|
|
266
|
+
* The model a category was mostly handled by, from a { model → episodes } map.
|
|
267
|
+
* Ties break toward the pricier model: with no majority either way, the more
|
|
268
|
+
* expensive reading of "what this used to cost" is the one worth stating.
|
|
269
|
+
* Returns null for an empty map, which callers read as "no baseline yet".
|
|
270
|
+
*/
|
|
271
|
+
export function dominantModel(counts) {
|
|
272
|
+
let best = null;
|
|
273
|
+
let bestN = 0;
|
|
274
|
+
for (const [model, n] of Object.entries(counts || {})) {
|
|
275
|
+
if (n > bestN || (n === bestN && best && modelRank(model) > modelRank(best))) {
|
|
276
|
+
best = model;
|
|
277
|
+
bestN = n;
|
|
278
|
+
}
|
|
279
|
+
}
|
|
280
|
+
return best;
|
|
281
|
+
}
|
|
282
|
+
|
|
265
283
|
export function worthDelegating(tier, rank) {
|
|
266
284
|
const target = TIER_TARGET_RANK[tier];
|
|
267
285
|
return target !== undefined && rank > target;
|
|
@@ -379,10 +397,29 @@ export async function runRouteScan({ days = 14 } = {}) {
|
|
|
379
397
|
const episodeStats = new Map(); // "category|project" (+ "category|*") → outcome stats
|
|
380
398
|
let tieredEpisodes = 0;
|
|
381
399
|
const bumpStats = (key, ep) => {
|
|
382
|
-
const s = episodeStats.get(key) || { count: 0, errCount: 0, epCount: 0 };
|
|
400
|
+
const s = episodeStats.get(key) || { count: 0, errCount: 0, epCount: 0, baselineModel: null, modelCounts: {} };
|
|
383
401
|
s.count += 1;
|
|
384
402
|
s.epCount += 1;
|
|
385
403
|
if (ep.errors > 0) s.errCount += 1;
|
|
404
|
+
// Baseline model: what actually handled this category BEFORE any rule sent
|
|
405
|
+
// it elsewhere. This is the only honest counterfactual for "routing saved
|
|
406
|
+
// money" — the session's priciest model is not, since it may never have
|
|
407
|
+
// touched work of this shape.
|
|
408
|
+
//
|
|
409
|
+
// Counted, not maxed. A transcript routinely carries more than one model
|
|
410
|
+
// (the user switches mid-session, a compaction pass runs elsewhere), and
|
|
411
|
+
// taking the priciest of them would let a single Fable record set the
|
|
412
|
+
// baseline for a category that Opus handled thirty times — inflating every
|
|
413
|
+
// later saving. The model that handled the most episodes is the one the
|
|
414
|
+
// rule actually replaced; price breaks a tie.
|
|
415
|
+
for (const m of ep.models) {
|
|
416
|
+
// Only ids the pricing table really recognizes may become a baseline —
|
|
417
|
+
// an unresolved gateway id or a house alias would be priced as Sonnet
|
|
418
|
+
// and quietly rewrite every saving computed against it.
|
|
419
|
+
if (!isRecognizedModelId(m)) continue;
|
|
420
|
+
s.modelCounts[m] = (s.modelCounts[m] || 0) + 1;
|
|
421
|
+
}
|
|
422
|
+
s.baselineModel = dominantModel(s.modelCounts);
|
|
386
423
|
episodeStats.set(key, s);
|
|
387
424
|
};
|
|
388
425
|
|
|
@@ -454,7 +491,33 @@ export async function runRouteScan({ days = 14 } = {}) {
|
|
|
454
491
|
// Ledger events feed the statusline's weekly/monthly "Routing saved"
|
|
455
492
|
// totals. Keyed by run transcript path so overlapping re-scans upsert the
|
|
456
493
|
// same event instead of double-counting it.
|
|
494
|
+
//
|
|
495
|
+
// What counts as a routing saving is narrower than what counts for
|
|
496
|
+
// rule-health above. A saving is the price difference a REGISTERED RULE
|
|
497
|
+
// caused: the model that used to handle this category (the rule's baseline,
|
|
498
|
+
// learned from episodes an expensive model handled directly) versus the
|
|
499
|
+
// model the run actually used. Subagent runs that no rule covers —
|
|
500
|
+
// Explore, a hand-written agent, a plugin's own subagent — would have gone
|
|
501
|
+
// to the same cheap model with or without this tool, so attributing their
|
|
502
|
+
// savings here would credit the tool for work it did not route.
|
|
457
503
|
const ledgerEvents = [];
|
|
504
|
+
let ledgerRules = [];
|
|
505
|
+
try {
|
|
506
|
+
const { loadModelRules } = await import('./model-rules.js');
|
|
507
|
+
ledgerRules = loadModelRules().rules.filter((r) => r.status !== 'off');
|
|
508
|
+
} catch { /* no registry → no attributable savings, which is the honest zero */ }
|
|
509
|
+
// A rule's baseline: what it stored at promotion, else what this scan still
|
|
510
|
+
// observes handling the category directly (a rule promoted before baselines
|
|
511
|
+
// existed backfills on the next refresh).
|
|
512
|
+
const baselineFor = (rule, tier, catId, projectDir) => {
|
|
513
|
+
if (rule.baselineModel) return rule.baselineModel;
|
|
514
|
+
const s = episodeStats.get(`${tier}|${catId}|${projectDir}`)
|
|
515
|
+
|| (rule.scope === 'global' ? episodeStats.get(`${tier}|${catId}|*`) : null);
|
|
516
|
+
return s?.baselineModel || null;
|
|
517
|
+
};
|
|
518
|
+
const ruleForRun = (tier, catId, projectDir) => ledgerRules.find((r) =>
|
|
519
|
+
r.tier === tier && r.category === catId &&
|
|
520
|
+
(r.scope === 'global' || r.project === projectDir));
|
|
458
521
|
for (const [sessionPath, index] of runIndexBySession) {
|
|
459
522
|
const used = new Set();
|
|
460
523
|
for (const { ep, projectDir, sessionPath: epSession } of all) {
|
|
@@ -478,22 +541,17 @@ export async function runRouteScan({ days = 14 } = {}) {
|
|
|
478
541
|
const saved = runSaving(run, mainModel);
|
|
479
542
|
bumpDelegated(`${runTier}|${cat.id}|${projectDir}`, run, saved);
|
|
480
543
|
bumpDelegated(`${runTier}|${cat.id}|*`, run, saved);
|
|
481
|
-
|
|
482
|
-
|
|
483
|
-
|
|
484
|
-
|
|
485
|
-
|
|
486
|
-
|
|
487
|
-
|
|
544
|
+
// Routing saving: priced against the rule's baseline (before → after),
|
|
545
|
+
// not against whatever the session's priciest model happened to be.
|
|
546
|
+
const rule = ruleForRun(runTier, cat.id, projectDir);
|
|
547
|
+
if (!rule) continue; // no rule routed this run — not our saving to claim
|
|
548
|
+
// Priced below, after this scan's baselines have been written back to
|
|
549
|
+
// the registry — see the note at the ledger write.
|
|
550
|
+
if (!isRecognizedModelId(run.model)) continue;
|
|
551
|
+
ledgerEvents.push({ run, rule, tier: runTier, catId: cat.id, projectDir });
|
|
488
552
|
}
|
|
489
553
|
}
|
|
490
554
|
}
|
|
491
|
-
try {
|
|
492
|
-
const { recordDelegationEvents } = await import('./savings-ledger.js');
|
|
493
|
-
recordDelegationEvents(ledgerEvents);
|
|
494
|
-
} catch {
|
|
495
|
-
// ledger write failure only delays the statusline totals, never the scan
|
|
496
|
-
}
|
|
497
555
|
|
|
498
556
|
// Keep prior dismissed/promoted signatures across rescans.
|
|
499
557
|
const prev = readRouteScan();
|
|
@@ -600,6 +658,40 @@ export async function runRouteScan({ days = 14 } = {}) {
|
|
|
600
658
|
refreshModelRules(episodeStats, delegatedStats, { now: cache.scannedAt });
|
|
601
659
|
} catch { /* registry unwritable — scan result still valid */ }
|
|
602
660
|
|
|
661
|
+
// Ledger last, so savings are priced against the baselines this scan just
|
|
662
|
+
// wrote. Pricing before the refresh made a changed baseline take two scans
|
|
663
|
+
// to show up: the first wrote the new baseline but billed against the old
|
|
664
|
+
// one, and the totals only settled on the second.
|
|
665
|
+
try {
|
|
666
|
+
const { recordDelegationEvents } = await import('./savings-ledger.js');
|
|
667
|
+
const { loadModelRules } = await import('./model-rules.js');
|
|
668
|
+
const fresh = loadModelRules().rules;
|
|
669
|
+
const priced = [];
|
|
670
|
+
for (const e of ledgerEvents) {
|
|
671
|
+
const rule = fresh.find((r) => r.signature === e.rule.signature) || e.rule;
|
|
672
|
+
const baseline = baselineFor(rule, e.tier, e.catId, e.projectDir);
|
|
673
|
+
// Both sides of the comparison must be ids the pricing table really
|
|
674
|
+
// recognizes. A house alias from a company gateway prices as Sonnet by
|
|
675
|
+
// default, which would fabricate a saving against a cheap baseline or
|
|
676
|
+
// erase a real one — worse than showing nothing. Map such ids in
|
|
677
|
+
// profile-map.json's `modelAliases` to bring these runs back in.
|
|
678
|
+
if (!baseline || !isRecognizedModelId(baseline)) continue;
|
|
679
|
+
const usd = runSaving(e.run, baseline);
|
|
680
|
+
if (usd <= 0) continue;
|
|
681
|
+
priced.push({
|
|
682
|
+
key: e.run.path,
|
|
683
|
+
ts: e.run.endedAt ?? e.run.startedAt ?? Date.now(),
|
|
684
|
+
usd,
|
|
685
|
+
rule: rule.signature,
|
|
686
|
+
from: baseline,
|
|
687
|
+
to: e.run.model,
|
|
688
|
+
});
|
|
689
|
+
}
|
|
690
|
+
recordDelegationEvents(priced);
|
|
691
|
+
} catch {
|
|
692
|
+
// ledger write failure only delays the statusline totals, never the scan
|
|
693
|
+
}
|
|
694
|
+
|
|
603
695
|
return cache;
|
|
604
696
|
}
|
|
605
697
|
|
package/src/savings-ledger.js
CHANGED
|
@@ -8,7 +8,15 @@
|
|
|
8
8
|
* upsert instead of double-counting.
|
|
9
9
|
*
|
|
10
10
|
* File: <userDataDir>/delegation-ledger.json
|
|
11
|
-
* { "
|
|
11
|
+
* { "version": 2,
|
|
12
|
+
* "events": { "<run path>": { "ts", "usd", "rule", "from", "to" } } }
|
|
13
|
+
*
|
|
14
|
+
* `from`/`to` are the models the routing decision moved between, and `usd` is
|
|
15
|
+
* the price difference between them for this run's tokens. Version 1 priced
|
|
16
|
+
* every downgraded subagent run against the session's priciest model, which
|
|
17
|
+
* credited the tool for runs no rule of its own had routed; those events are
|
|
18
|
+
* discarded rather than migrated, since the number they carry cannot be
|
|
19
|
+
* recomputed without a rescan (which route-scan does anyway).
|
|
12
20
|
*
|
|
13
21
|
* Best-effort like every other state file here: an unreadable ledger reads as
|
|
14
22
|
* empty, and the statusline renders totals of 0 as "no chip".
|
|
@@ -25,32 +33,48 @@ export function ledgerPath() {
|
|
|
25
33
|
return join(userDataDir(), 'delegation-ledger.json');
|
|
26
34
|
}
|
|
27
35
|
|
|
36
|
+
export const LEDGER_VERSION = 2;
|
|
37
|
+
|
|
28
38
|
export function loadLedger() {
|
|
29
39
|
try {
|
|
30
40
|
const data = JSON.parse(readFileSync(ledgerPath(), 'utf8'));
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
41
|
+
if (!data || typeof data.events !== 'object' || data.events === null) {
|
|
42
|
+
return { version: LEDGER_VERSION, events: {} };
|
|
43
|
+
}
|
|
44
|
+
// Pre-v2 events were priced against a different counterfactual — drop them
|
|
45
|
+
// instead of mixing two meanings into one total. The next scan rebuilds
|
|
46
|
+
// whatever is still attributable.
|
|
47
|
+
if (data.version !== LEDGER_VERSION) return { version: LEDGER_VERSION, events: {} };
|
|
48
|
+
return data;
|
|
34
49
|
} catch {
|
|
35
|
-
return { events: {} };
|
|
50
|
+
return { version: LEDGER_VERSION, events: {} };
|
|
36
51
|
}
|
|
37
52
|
}
|
|
38
53
|
|
|
39
54
|
/**
|
|
40
|
-
* Upsert saving events. `events` is an array of
|
|
41
|
-
* is the run's transcript path
|
|
42
|
-
*
|
|
55
|
+
* Upsert saving events. `events` is an array of
|
|
56
|
+
* { key, ts, usd, rule, from, to } where `key` is the run's transcript path
|
|
57
|
+
* (unique per subagent run) and `from`/`to` name the models the routing
|
|
58
|
+
* decision moved between. Zero-saving runs are skipped — they carry no
|
|
59
|
+
* information the totals care about.
|
|
43
60
|
*/
|
|
44
61
|
export function recordDelegationEvents(events) {
|
|
45
62
|
if (!Array.isArray(events) || events.length === 0) return;
|
|
46
63
|
const data = loadLedger();
|
|
64
|
+
data.version = LEDGER_VERSION;
|
|
47
65
|
let changed = false;
|
|
48
66
|
for (const e of events) {
|
|
49
67
|
if (!e || !e.key || !(Number(e.usd) > 0) || !Number.isFinite(e.ts)) continue;
|
|
50
68
|
const prev = data.events[e.key];
|
|
51
69
|
const usd = Math.round(Number(e.usd) * 10000) / 10000;
|
|
52
70
|
if (prev && prev.ts === e.ts && prev.usd === usd) continue;
|
|
53
|
-
data.events[e.key] = {
|
|
71
|
+
data.events[e.key] = {
|
|
72
|
+
ts: e.ts,
|
|
73
|
+
usd,
|
|
74
|
+
...(e.rule ? { rule: e.rule } : {}),
|
|
75
|
+
...(e.from ? { from: e.from } : {}),
|
|
76
|
+
...(e.to ? { to: e.to } : {}),
|
|
77
|
+
};
|
|
54
78
|
changed = true;
|
|
55
79
|
}
|
|
56
80
|
if (!changed) return;
|
|
@@ -64,11 +88,31 @@ export function recordDelegationEvents(events) {
|
|
|
64
88
|
}
|
|
65
89
|
|
|
66
90
|
/**
|
|
67
|
-
*
|
|
68
|
-
*
|
|
91
|
+
* Family name of a model id, with the version dropped: `claude-opus-5` and
|
|
92
|
+
* `claude-opus-4-5-20251101-v1:0` both read as `opus`.
|
|
93
|
+
*
|
|
94
|
+
* Versions move constantly, and on a statusline the digits are noise — what
|
|
95
|
+
* the reader wants is the shape of the trade ("opus work now runs on haiku").
|
|
96
|
+
* Falls back to the id itself so an unmapped name is visible rather than
|
|
97
|
+
* silently folded into another family.
|
|
98
|
+
*/
|
|
99
|
+
export function modelFamily(model) {
|
|
100
|
+
const m = String(model || '').toLowerCase();
|
|
101
|
+
for (const f of ['fable', 'mythos', 'opus', 'sonnet', 'haiku']) {
|
|
102
|
+
if (m.includes(f)) return f;
|
|
103
|
+
}
|
|
104
|
+
return String(model || '?');
|
|
105
|
+
}
|
|
106
|
+
|
|
107
|
+
/**
|
|
108
|
+
* Rolling totals: last 7 days, last 30 days, and lifetime, plus `pairs` — the
|
|
109
|
+
* lifetime rollup by family-level model change, priciest first. `now` is
|
|
110
|
+
* injectable for tests. Never throws — an unreadable ledger yields zeros.
|
|
69
111
|
*/
|
|
70
112
|
export function delegationSavedTotals(now = Date.now()) {
|
|
71
|
-
const
|
|
113
|
+
const empty = () => ({ week: 0, month: 0, total: 0, pairs: [] });
|
|
114
|
+
const totals = empty();
|
|
115
|
+
const byPair = new Map();
|
|
72
116
|
try {
|
|
73
117
|
for (const e of Object.values(loadLedger().events)) {
|
|
74
118
|
const usd = Number(e.usd) || 0;
|
|
@@ -78,9 +122,16 @@ export function delegationSavedTotals(now = Date.now()) {
|
|
|
78
122
|
if (now - e.ts <= WEEK_MS) totals.week += usd;
|
|
79
123
|
if (now - e.ts <= MONTH_MS) totals.month += usd;
|
|
80
124
|
}
|
|
125
|
+
if (!e.from || !e.to) continue;
|
|
126
|
+
const key = `${modelFamily(e.from)}→${modelFamily(e.to)}`;
|
|
127
|
+
const p = byPair.get(key) || { from: modelFamily(e.from), to: modelFamily(e.to), runs: 0, usd: 0 };
|
|
128
|
+
p.runs += 1;
|
|
129
|
+
p.usd += usd;
|
|
130
|
+
byPair.set(key, p);
|
|
81
131
|
}
|
|
82
132
|
} catch {
|
|
83
|
-
return
|
|
133
|
+
return empty();
|
|
84
134
|
}
|
|
135
|
+
totals.pairs = [...byPair.values()].sort((a, b) => b.usd - a.usd);
|
|
85
136
|
return totals;
|
|
86
137
|
}
|