claude-token-saver 3.8.1 → 3.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.en.md +14 -0
- package/README.md +14 -0
- package/package.json +1 -1
- package/src/brief.js +27 -11
- package/src/commands/harness.js +3 -0
- package/src/commands/route-scan.js +32 -6
- package/src/cost.js +39 -0
- package/src/model-rules.js +115 -15
- package/src/route-scan.js +136 -8
- package/src/session-records.js +17 -1
- package/src/subagent-records.js +185 -0
package/README.en.md
CHANGED
|
@@ -263,6 +263,20 @@ Also update `statusLine.command` in `~/.claude/settings.json` to `claude-token-s
|
|
|
263
263
|
|
|
264
264
|
## Release notes
|
|
265
265
|
|
|
266
|
+
### v3.9.0 (2026-08-01)
|
|
267
|
+
|
|
268
|
+
Compared this tool's design against manifest.build's "Everyone is building LLM routers, we deprecated ours" (a retrospective on 4 months across 7,000 users) and closed the four of their failure modes that were not already ruled out here. Full comparison in [TIER_CRITERIA.md §3.9](./docs/TIER_CRITERIA.md).
|
|
269
|
+
|
|
270
|
+
- **rule-health now watches real delegations** — the old error rate's denominator was "episodes the expensive model handled directly that merely *looked* delegable by shape". It never measured whether a rule succeeded **when it actually fired**, so a rule could fail every single time without moving the signal. Claude Code writes each subagent run to `<session>/subagents/`, and its meta `toolUseId` joins exactly onto the parent's Task call — so the outcome of a real delegation is now counted directly. Once 5 measured runs accumulate, they replace the shape-based proxy, and the warning says which evidence it is using. (Author's 14-day log: 4 of 106 sessions, 18 runs, 100% join rate.)
|
|
271
|
+
- **Per-rule savings** — the tokens a delegated run spent are re-priced at the session model's rate, and the difference is reported over the 14-day window as `~$` in `route-scan rules` and `ratchet-model.md`. You can only prune rules that aren't worth their keep if you can see which ones those are. A rule with no measured delegations shows `—`, not `$0` — the two mean opposite things.
|
|
272
|
+
- **Tiers are now relative to the session model** — the only gate used to be "is it haiku?", so a Sonnet session produced T1 rules telling it to delegate to Sonnet: a subagent rebuilding context for zero price difference, which is a net loss. Candidates are now only created when the target tier is genuinely cheaper (haiku 0 · sonnet 1 · opus 2 · fable 3).
|
|
273
|
+
- **Probe-then-commit budgets** — a rule is derived from statistics but **fires on the request text alone**, and most of what makes a task hard only shows up after the first tool call. That gap can't be closed, so the cost of getting it wrong is capped instead: every rule now carries a calibrated limit (T2: 8 tool calls / p25 output, T1: p75 output), and a run likely to exceed it — or hitting an error — stops there, reports partial progress, and hands back to the main model. The clause is composed at render time rather than baked into the stored rule, so **rules promoted before this release get it too**, and the promote preview can never drift from what lands in the file.
|
|
274
|
+
|
|
275
|
+
### v3.8.2 (2026-08-01)
|
|
276
|
+
- **Fixed: the context tier never came back down after compaction** — the tier was kept as a high-water mark, so a session that compacted at 80% stayed at tier 1 even though its context had just been emptied, and it refilled to the cap with no signal at all. The tier now follows the measurement down and warns again on the next climb.
|
|
277
|
+
- **Both windows are named** — with `autoCompactWindow` at 400k the briefing measured against 400k (80%) while Claude Code's own display measured against 1M (33%); two irreconcilable numbers for one session. The text now reads `자동 압축 창(400k)의 80% (… 화면의 1M 창 기준으로는 33%)`.
|
|
278
|
+
- **No "start a new session" advice when a compact window is set** — that threshold is exactly where compaction runs on its own, so the briefing suggests writing decisions and next steps to a file instead.
|
|
279
|
+
|
|
266
280
|
### v3.8.1 (2026-07-31)
|
|
267
281
|
- **Fixed: 1M sessions were judged against a 200k window** — the briefing inferred the window from the largest request seen so far, so a 1M session counted as 200k until it had already grown past 250k. At 160k of input it announced "past 80% of the 200k window" — really 16%. The window now comes from the configured model id, and when `autoCompactWindow` is set that is where the session actually turns over, so the percentage is measured against it (the text says `(autoCompactWindow 기준)`). The observed-size heuristic remains only as the fallback for an unreadable model id.
|
|
268
282
|
|
package/README.md
CHANGED
|
@@ -220,6 +220,20 @@ npm uninstall -g claude-cache-monitor && npm i -g claude-token-saver
|
|
|
220
220
|
|
|
221
221
|
## 릴리스 노트
|
|
222
222
|
|
|
223
|
+
### v3.9.0 (2026-08-01)
|
|
224
|
+
|
|
225
|
+
manifest.build의 "다들 LLM 라우터 만드는데 우리는 폐기했다"(7천 사용자·4개월 실사용 회고)와 이 도구의 설계를 대조해, 그쪽 실패 요인 중 아직 안 막혀 있던 것 4개를 메웠습니다. 자세한 대조는 [TIER_CRITERIA.md §3.9](./docs/TIER_CRITERIA.md).
|
|
226
|
+
|
|
227
|
+
- **rule-health가 이제 실제 위임 결과를 봅니다** — 기존 에러율의 분모는 "비싼 모델이 직접 처리했는데 형태상 위임 가능해 보이던 에피소드"였습니다. 즉 룰이 **실제로 발동했을 때 잘 됐는지는 한 번도 재지 않았고**, 매번 실패하는 룰이 있어도 신호가 안 움직였습니다. Claude Code가 서브에이전트 실행을 `<세션>/subagents/`에 따로 남기고 그 메타의 `toolUseId`가 부모의 Task 호출과 정확히 맞물리므로, 이제 실제 위임의 성패를 직접 셉니다. 실측 5건 이상 쌓인 룰은 추정 대신 실측으로 판정하고, 경고에도 어느 쪽 근거인지 표시합니다. (저자 로그 14일: 106세션 중 4세션·18런, 조인 성공률 100%)
|
|
228
|
+
- **룰별 절감액 표시** — 위임 실행의 토큰을 세션 모델 단가로 되돌린 차액을 14일 창으로 계산해 `route-scan rules`와 `ratchet-model.md`에 `~$` 표기로 붙입니다. 값어치 없는 룰이 눈에 보여야 정리할 수 있습니다. 위임 기록이 없는 룰은 `$0`이 아니라 `—` — 둘은 정반대를 뜻합니다.
|
|
229
|
+
- **세션 모델 기준 상대 티어** — 기존 게이트가 "haiku인가?" 하나뿐이라, Sonnet 세션에서도 "sonnet한테 위임하라"는 T1 룰이 만들어졌습니다. 컨텍스트만 새로 쌓고 단가 차이는 0인 순손해입니다. 이제 목표 티어가 실제로 더 싼 경우에만 후보를 만듭니다 (haiku 0 · sonnet 1 · opus 2 · fable 3).
|
|
230
|
+
- **위임 예산 문구(probe-then-commit)** — 룰은 통계로 만들어지지만 **발동은 요청 텍스트만 보고** 일어나고, 난이도는 대개 첫 도구 호출 뒤에야 드러납니다. 이 불일치는 못 없애니 오판 비용에 상한을 겁니다: 각 룰에 캘리브레이션된 상한(T2는 호출 8회·출력 p25, T1은 출력 p75)이 붙고, 넘길 것 같거나 에러가 나면 서브에이전트가 멈춰 진행분만 보고하고 메인 모델이 이어받습니다. 문구는 저장된 룰에 굽지 않고 렌더 시점에 조립하므로 **예전에 등록한 룰도 자동으로 적용**받고, promote 프리뷰와 실제 파일이 어긋날 수 없습니다.
|
|
231
|
+
|
|
232
|
+
### v3.8.2 (2026-08-01)
|
|
233
|
+
- **압축 뒤에도 경고 티어가 안 내려가던 문제 수정** — 컨텍스트 티어를 최고치로만 기억해서, 80%에서 자동 압축이 돌아 컨텍스트가 다시 비어도 티어가 1로 남았습니다. 그 세션은 창을 다시 꽉 채워도 아무 신호를 못 받았습니다. 이제 측정치가 내려가면 티어도 같이 내려가고, 다시 차오르면 정상적으로 경고합니다.
|
|
234
|
+
- **두 가지 창 표기 혼선 정리** — `autoCompactWindow`를 40만으로 잡으면 브리핑은 40만 기준(80%)인데 Claude Code 화면은 1M 기준(33%)이라, 같은 세션의 두 숫자가 서로 안 맞아 보였습니다. 이제 `자동 압축 창(400k)의 80%(… 화면의 1M 창 기준으로는 33%)`처럼 둘 다 적습니다.
|
|
235
|
+
- **압축 창이 설정돼 있으면 "새 세션 시작" 권고를 하지 않습니다** — 그 지점은 압축이 자동으로 처리하는 지점이라, 대신 결정·다음 할 일을 파일에 남기라고 안내합니다.
|
|
236
|
+
|
|
223
237
|
### v3.8.1 (2026-07-31)
|
|
224
238
|
- **1M 세션을 200k 창으로 오판하던 브리핑 버그 수정** — 세션 창을 "지금까지 본 가장 큰 요청"으로 추정해서, 1M 세션이라도 25만 토큰을 넘기 전까지는 200k로 취급했습니다. 그래서 입력 160k에서 "200k 창의 80%를 넘었습니다" 경고가 떴습니다(실제로는 16%). 이제 설정된 모델 ID로 창을 판정하고, `autoCompactWindow`가 잡혀 있으면 그 값이 실제로 세션이 넘어가는 지점이므로 그쪽을 기준으로 %를 계산합니다(문구에도 `(autoCompactWindow 기준)` 표기). 모델 ID를 못 읽는 경우에만 기존 관측치 추정으로 되돌아갑니다.
|
|
225
239
|
|
package/package.json
CHANGED
package/src/brief.js
CHANGED
|
@@ -85,18 +85,19 @@ function saveState(state, now) {
|
|
|
85
85
|
* is unreadable (env override, settings we do not resolve).
|
|
86
86
|
*/
|
|
87
87
|
export function ctxWindowFor(observedMax = 0, root = process.cwd()) {
|
|
88
|
-
let
|
|
88
|
+
let modelWindow = observedMax > WINDOW_1M_MIN_INPUT ? 1_000_000 : 200_000;
|
|
89
|
+
let window = modelWindow;
|
|
89
90
|
let compactCapped = false;
|
|
90
91
|
try {
|
|
91
92
|
const { model } = resolveModelId(root);
|
|
92
|
-
if (isOneMillionModel(model)) window = 1_000_000;
|
|
93
|
+
if (isOneMillionModel(model)) modelWindow = window = 1_000_000;
|
|
93
94
|
const cap = effectiveWindow(root).value;
|
|
94
95
|
if (cap !== null && cap < window) {
|
|
95
96
|
window = cap;
|
|
96
97
|
compactCapped = true;
|
|
97
98
|
}
|
|
98
99
|
} catch { /* settings unreadable — the observed-size fallback still holds */ }
|
|
99
|
-
return { window, compactCapped };
|
|
100
|
+
return { window, modelWindow, compactCapped };
|
|
100
101
|
}
|
|
101
102
|
|
|
102
103
|
/**
|
|
@@ -129,8 +130,8 @@ export function sessionCtx(transcriptPath, { root = process.cwd() } = {}) {
|
|
|
129
130
|
if (total > 0) { input = total; maxInput = Math.max(maxInput, total); }
|
|
130
131
|
}
|
|
131
132
|
if (input == null) return null;
|
|
132
|
-
const { window, compactCapped } = ctxWindowFor(maxInput, root);
|
|
133
|
-
return { input, window, compactCapped, pct: input / window };
|
|
133
|
+
const { window, modelWindow, compactCapped } = ctxWindowFor(maxInput, root);
|
|
134
|
+
return { input, window, modelWindow, compactCapped, pct: input / window };
|
|
134
135
|
}
|
|
135
136
|
|
|
136
137
|
function ctxTierOf(pct) {
|
|
@@ -175,15 +176,30 @@ export async function runBrief({ sessionId, transcriptPath, now = Date.now() })
|
|
|
175
176
|
const ctx = transcriptPath ? sessionCtx(transcriptPath) : null;
|
|
176
177
|
if (ctx) {
|
|
177
178
|
const tier = ctxTierOf(ctx.pct);
|
|
179
|
+
// The tier is not monotonic: auto-compaction drops the live context back to
|
|
180
|
+
// a fraction of the window, which starts a new fill cycle. Holding the old
|
|
181
|
+
// high-water tier meant a session that compacted at 80% was never warned
|
|
182
|
+
// again — it silently refilled to the cap with no signal at all.
|
|
183
|
+
if (tier < (s.ctxTier || 0)) s.ctxTier = tier;
|
|
178
184
|
if (tier > (s.ctxTier || 0)) {
|
|
179
|
-
// Name
|
|
180
|
-
//
|
|
181
|
-
//
|
|
185
|
+
// Name both denominators when they differ. Claude Code's own UI counts
|
|
186
|
+
// against the model window, so a bare "400k 창의 80%" reads as wrong to
|
|
187
|
+
// anyone looking at a statusline that says 32% of 1M — same session,
|
|
188
|
+
// two different windows, no way to reconcile them from the text alone.
|
|
182
189
|
const winLabel = ctx.window >= 1_000_000 ? '1M' : fmtK(ctx.window);
|
|
183
|
-
const
|
|
190
|
+
const modelPct = Math.round((ctx.input / ctx.modelWindow) * 100);
|
|
191
|
+
const modelLabel = ctx.modelWindow >= 1_000_000 ? '1M' : fmtK(ctx.modelWindow);
|
|
192
|
+
const capNote = ctx.compactCapped ? `, 화면의 ${modelLabel} 창 기준으로는 ${modelPct}%` : '';
|
|
193
|
+
// With autoCompactWindow set, crossing the threshold means compaction is
|
|
194
|
+
// about to run on its own. Telling the user to start a new session there
|
|
195
|
+
// would be advice for a problem the setting already handles.
|
|
184
196
|
items.push(tier === 2
|
|
185
|
-
?
|
|
186
|
-
|
|
197
|
+
? (ctx.compactCapped
|
|
198
|
+
? `이 세션의 컨텍스트가 자동 압축 창(${winLabel})의 95%를 넘었습니다(직전 요청 입력 ${fmtK(ctx.input)}${capNote}). 곧 자동 압축이 돌아 이전 대화가 요약으로 바뀝니다 — 지금 단계를 마무리하고 이어서 할 일은 파일에 적어두면 압축 뒤에도 안전합니다.`
|
|
199
|
+
: `이 세션의 컨텍스트가 ${winLabel} 창의 95%를 넘었습니다(직전 요청 입력 ${fmtK(ctx.input)}). 곧 자동 압축으로 맥락 손실이 생길 수 있으니, 진행 중인 작업을 일단락하고 새 세션을 시작하는 편이 좋습니다.`)
|
|
200
|
+
: (ctx.compactCapped
|
|
201
|
+
? `이 세션의 컨텍스트가 자동 압축 창(${winLabel})의 80%를 넘었습니다(직전 요청 입력 ${fmtK(ctx.input)}${capNote}). 설정해 둔 압축 지점이 가까워졌습니다 — 압축은 알아서 돌아가니 새 세션을 서두를 필요는 없고, 여기까지의 결정과 다음 할 일만 파일에 남겨두면 됩니다.`
|
|
202
|
+
: `이 세션의 컨텍스트가 ${winLabel} 창의 80%를 넘었습니다(직전 요청 입력 ${fmtK(ctx.input)}). 이후 요청은 비용이 커지는 구간입니다 — 작업이 일단락되면 새 세션 시작을 권합니다.`));
|
|
187
203
|
s.ctxTier = tier;
|
|
188
204
|
}
|
|
189
205
|
}
|
package/src/commands/harness.js
CHANGED
|
@@ -228,6 +228,9 @@ export async function run({ args, hasFlag }) {
|
|
|
228
228
|
rule: lang === 'ko' ? routeCandidate.rule : (routeCandidate.ruleEn || routeCandidate.rule),
|
|
229
229
|
example: routeCandidate.example,
|
|
230
230
|
count: routeCandidate.count,
|
|
231
|
+
// Calibrated budget snapshot — ratchet-model.md restates it when it
|
|
232
|
+
// merges a category's T2 and T1 rules into one conditional rule.
|
|
233
|
+
budget: routeCandidate.budget || null,
|
|
231
234
|
promotedAt: new Date().toISOString().slice(0, 10),
|
|
232
235
|
lastSeen: new Date().toISOString().slice(0, 10),
|
|
233
236
|
});
|
|
@@ -40,7 +40,18 @@ export async function run({ args, hasFlag, numArg }) {
|
|
|
40
40
|
? `${r.tier} (${rs.tierLabel(r.tier)}) · ${rs.scopeLabel(r.scope)} · 반복 ${r.count || 0}회 · 에러율 ${Math.round((r.errRate || 0) * 100)}%`
|
|
41
41
|
: `${r.tier} (${rs.tierLabel(r.tier, 'en')}) · ${rs.scopeLabel(r.scope, 'en')} · seen ×${r.count || 0} · err ${Math.round((r.errRate || 0) * 100)}%`;
|
|
42
42
|
console.log(` #${i + 1} ${stat}${health}`);
|
|
43
|
-
|
|
43
|
+
// Measured outcome of the rule actually firing, plus what it saved.
|
|
44
|
+
// A rule with no measured delegations shows "—", never "$0.00": the
|
|
45
|
+
// two mean opposite things (no data vs. data showing no value).
|
|
46
|
+
const measured = r.delegatedRuns
|
|
47
|
+
? (lang === 'ko'
|
|
48
|
+
? `실제 위임 ${r.delegatedRuns}건 · 에러율 ${Math.round((r.delegatedErrRate || 0) * 100)}% · 절감 ~$${(r.savedUsd || 0).toFixed(2)}`
|
|
49
|
+
: `measured ×${r.delegatedRuns} · err ${Math.round((r.delegatedErrRate || 0) * 100)}% · saved ~$${(r.savedUsd || 0).toFixed(2)}`)
|
|
50
|
+
: (lang === 'ko' ? '실제 위임 기록 — (아직 없음)' : 'measured delegations — (none yet)');
|
|
51
|
+
console.log(` ${measured}`);
|
|
52
|
+
// Same composer the md file uses, so what is listed here is exactly
|
|
53
|
+
// what the model reads.
|
|
54
|
+
console.log(` ${mr.composeRuleText(r.rule, r, lang)}`);
|
|
44
55
|
});
|
|
45
56
|
console.log(lang === 'ko'
|
|
46
57
|
? '\n제거: claude-token-saver route-scan rules rm <N>'
|
|
@@ -79,12 +90,17 @@ export async function run({ args, hasFlag, numArg }) {
|
|
|
79
90
|
} catch (e) { debug('route-scan:spawn-refresh', e); /* stale cache is still usable below */ }
|
|
80
91
|
}
|
|
81
92
|
const open = rs.openCandidates(cache);
|
|
93
|
+
// Rule text shown to the model must be composed the same way the md file
|
|
94
|
+
// composes it, budget clause included — otherwise the briefing promises
|
|
95
|
+
// one rule and the file carries another.
|
|
96
|
+
const mrHook = await import('../model-rules.js');
|
|
97
|
+
const composed = (base, c) => mrHook.composeRuleText(base, c, lang);
|
|
82
98
|
// Registered rules whose delegated-category error rate crossed the
|
|
83
99
|
// health threshold since promotion — the user approved these, so a
|
|
84
100
|
// status change must be briefed, not just written into the md file.
|
|
85
101
|
let reviewRules = [];
|
|
86
102
|
try {
|
|
87
|
-
const mr =
|
|
103
|
+
const mr = mrHook;
|
|
88
104
|
reviewRules = mr.loadModelRules().rules
|
|
89
105
|
.map((r, i) => ({ ...r, n: i + 1 }))
|
|
90
106
|
.filter((r) => r.status === 'review');
|
|
@@ -105,7 +121,7 @@ export async function run({ args, hasFlag, numArg }) {
|
|
|
105
121
|
for (const c of open) {
|
|
106
122
|
const tier = c.tier || 'T2';
|
|
107
123
|
const label = lang === 'ko' ? c.label : (c.labelEn || c.label);
|
|
108
|
-
const rule = lang === 'ko' ? c.rule : (c.ruleEn || c.rule);
|
|
124
|
+
const rule = composed(lang === 'ko' ? c.rule : (c.ruleEn || c.rule), c);
|
|
109
125
|
if (lang === 'ko') {
|
|
110
126
|
lines.push(` 후보 R${c.id} — "${label}" 유형, ${c.count}회 반복 (프로젝트: ${c.project})`);
|
|
111
127
|
lines.push(` 판정: ${tier} (${rs.tierLabel(tier)}) → ${c.agent} 서브에이전트 위임 권장 · 적용 범위 제안: ${rs.scopeLabel(c.suggestedScope)}`);
|
|
@@ -133,9 +149,17 @@ export async function run({ args, hasFlag, numArg }) {
|
|
|
133
149
|
? '[claude-token-saver rule-health] 사용자가 승인한 위임 룰 중, 위임 대상 유형의 최근 에러율이 기준(20%)을 넘어 재검토가 필요한 룰이 있습니다 — 사용자에게 브리핑하고 조건 좁히기/제거를 상의하세요:'
|
|
134
150
|
: '[claude-token-saver rule-health] Some user-approved delegation rules now exceed the 20% error-rate threshold for their delegated category — brief the user and discuss narrowing or removing them:');
|
|
135
151
|
for (const r of reviewRules) {
|
|
152
|
+
// Say WHICH signal tripped: a measured delegation failure rate is a
|
|
153
|
+
// much stronger claim than the shape-based proxy, and the user's
|
|
154
|
+
// decision (narrow vs. remove) depends on knowing which it is.
|
|
155
|
+
const measured = r.healthSource === 'delegated';
|
|
156
|
+
const rate = Math.round(((measured ? r.delegatedErrRate : r.errRate) || 0) * 100);
|
|
157
|
+
const evidence = lang === 'ko'
|
|
158
|
+
? (measured ? `실제 위임 ${r.delegatedRuns}건 실측 에러율 ${rate}%` : `유형 에러율 ${rate}% (형태 기반 추정)`)
|
|
159
|
+
: (measured ? `${rate}% measured across ${r.delegatedRuns} real delegations` : `${rate}% for the category (shape-based proxy)`);
|
|
136
160
|
lines.push(lang === 'ko'
|
|
137
|
-
? ` 룰 #${r.n} (${r.tier} ${rs.tierLabel(r.tier)} · ${rs.scopeLabel(r.scope)}) —
|
|
138
|
-
: ` rule #${r.n} (${r.tier} ${rs.tierLabel(r.tier, 'en')} · ${rs.scopeLabel(r.scope, 'en')}) —
|
|
161
|
+
? ` 룰 #${r.n} (${r.tier} ${rs.tierLabel(r.tier)} · ${rs.scopeLabel(r.scope)}) — ${evidence}`
|
|
162
|
+
: ` rule #${r.n} (${r.tier} ${rs.tierLabel(r.tier, 'en')} · ${rs.scopeLabel(r.scope, 'en')}) — ${evidence}`);
|
|
139
163
|
lines.push(` "${r.rule}"`);
|
|
140
164
|
}
|
|
141
165
|
lines.push(lang === 'ko'
|
|
@@ -191,7 +215,9 @@ export async function run({ args, hasFlag, numArg }) {
|
|
|
191
215
|
console.log(` verdict: ${tier} (${rs.tierLabel(tier, 'en')}) → delegate to ${c.agent} · suggested scope: ${rs.scopeLabel(c.suggestedScope, 'en')}`);
|
|
192
216
|
}
|
|
193
217
|
console.log(` ${lang === 'ko' ? '예시' : 'example'}: "${c.example}"`);
|
|
194
|
-
|
|
218
|
+
const mrList = await import('../model-rules.js');
|
|
219
|
+
const base = lang === 'ko' ? c.rule : (c.ruleEn || c.rule);
|
|
220
|
+
console.log(` ${lang === 'ko' ? '룰' : 'rule'}: ${mrList.composeRuleText(base, c, lang)}`);
|
|
195
221
|
}
|
|
196
222
|
console.log('');
|
|
197
223
|
console.log(lang === 'ko' ? '등록 / 무시:' : 'Promote / dismiss:');
|
package/src/cost.js
CHANGED
|
@@ -100,6 +100,45 @@ function detectPricingTier(model) {
|
|
|
100
100
|
return 'claude-sonnet';
|
|
101
101
|
}
|
|
102
102
|
|
|
103
|
+
/**
|
|
104
|
+
* Relative price rank of a model's pricing tier. Delegation only pays off
|
|
105
|
+
* when the target tier is genuinely cheaper than the model that did the work,
|
|
106
|
+
* so route-scan needs an ORDER, not just "is it haiku?" — a Sonnet session
|
|
107
|
+
* must not produce "delegate to Sonnet" rules (the subagent would rebuild
|
|
108
|
+
* context for zero price difference).
|
|
109
|
+
*/
|
|
110
|
+
const TIER_RANK = {
|
|
111
|
+
'claude-fable-5': 3,
|
|
112
|
+
'claude-opus-legacy': 2,
|
|
113
|
+
'claude-opus-new': 2,
|
|
114
|
+
'claude-sonnet': 1,
|
|
115
|
+
'claude-haiku-4-5': 0,
|
|
116
|
+
'claude-haiku-3-5': 0,
|
|
117
|
+
'claude-haiku-3': 0,
|
|
118
|
+
};
|
|
119
|
+
|
|
120
|
+
export function modelRank(model) {
|
|
121
|
+
const rank = TIER_RANK[detectPricingTier(model)];
|
|
122
|
+
// Unknown ids fall through detectPricingTier to the Sonnet tier; ranking
|
|
123
|
+
// them 1 keeps the conservative reading (cheap enough that a Sonnet-target
|
|
124
|
+
// rule is not worth it, expensive enough that a haiku one still is).
|
|
125
|
+
return rank ?? 1;
|
|
126
|
+
}
|
|
127
|
+
|
|
128
|
+
/** Price rank each delegation tier targets: T2 → haiku, T1 → sonnet. */
|
|
129
|
+
export const TIER_TARGET_RANK = { T2: 0, T1: 1 };
|
|
130
|
+
|
|
131
|
+
/**
|
|
132
|
+
* Which delegation tier a subagent run at this price rank represents.
|
|
133
|
+
* null = the run was NOT a downgrade (same tier or higher), so it carries no
|
|
134
|
+
* delegation saving to attribute to a rule.
|
|
135
|
+
*/
|
|
136
|
+
export function tierForRank(rank) {
|
|
137
|
+
if (rank === 0) return 'T2';
|
|
138
|
+
if (rank === 1) return 'T1';
|
|
139
|
+
return null;
|
|
140
|
+
}
|
|
141
|
+
|
|
103
142
|
function tokensToMillions(n) {
|
|
104
143
|
return n / 1_000_000;
|
|
105
144
|
}
|
package/src/model-rules.js
CHANGED
|
@@ -42,6 +42,46 @@ export const HEALTH_ERR_RATE = 0.2;
|
|
|
42
42
|
// noise (1 error in 4 episodes = 25% — instant flag), so the review flag is
|
|
43
43
|
// withheld until the sample is large enough to mean something.
|
|
44
44
|
export const HEALTH_MIN_SAMPLE = 10;
|
|
45
|
+
// Measured delegations needed before the real outcome replaces the proxy.
|
|
46
|
+
// Lower than HEALTH_MIN_SAMPLE on purpose: a delegated run is DIRECT evidence
|
|
47
|
+
// of the rule firing, so far fewer are needed to mean something than episodes
|
|
48
|
+
// that merely resemble the rule's shape.
|
|
49
|
+
export const HEALTH_MIN_SAMPLE_DELEGATED = 5;
|
|
50
|
+
|
|
51
|
+
// Fallback budget for rules promoted before budgets were recorded — matches
|
|
52
|
+
// route-scan's pre-calibration defaults. Kept as local constants rather than
|
|
53
|
+
// imported: route-scan imports this module, and a static back-import would
|
|
54
|
+
// close the cycle.
|
|
55
|
+
const DEFAULT_BUDGET = { T2: { calls: 8, out: 1500 }, T1: { calls: null, out: 8000 } };
|
|
56
|
+
|
|
57
|
+
/** The cap half of a budget clause, e.g. "도구 호출 8회·출력 1500 토큰". */
|
|
58
|
+
export function budgetCapPhrase(rule, lang = userLanguage()) {
|
|
59
|
+
const b = rule.budget || DEFAULT_BUDGET[rule.tier] || DEFAULT_BUDGET.T2;
|
|
60
|
+
if (lang === 'ko') {
|
|
61
|
+
return b.calls ? `도구 호출 ${b.calls}회·출력 ${b.out} 토큰` : `출력 ${b.out} 토큰`;
|
|
62
|
+
}
|
|
63
|
+
return b.calls ? `${b.calls} tool calls / ${b.out} output tokens` : `${b.out} output tokens`;
|
|
64
|
+
}
|
|
65
|
+
|
|
66
|
+
/**
|
|
67
|
+
* Append the probe-then-commit budget to a rule's text.
|
|
68
|
+
*
|
|
69
|
+
* Kept here — not baked into the stored rule string — so it is composed in ONE
|
|
70
|
+
* place: rules promoted before budgets existed gain the clause too, and the
|
|
71
|
+
* promote preview and the written file can never drift apart (both call this).
|
|
72
|
+
*
|
|
73
|
+
* Why the clause matters: a rule is chosen from past statistics but FIRES on
|
|
74
|
+
* the request text alone, and difficulty mostly surfaces after the first tool
|
|
75
|
+
* call. Naming a cap turns an unavoidable mis-fire into a bounded probe
|
|
76
|
+
* instead of a cheap model grinding at work it cannot finish.
|
|
77
|
+
*/
|
|
78
|
+
export function composeRuleText(baseText, rule, lang = userLanguage()) {
|
|
79
|
+
const cap = budgetCapPhrase(rule, lang);
|
|
80
|
+
const clause = lang === 'ko'
|
|
81
|
+
? `위임 상한은 ${cap}이며, 넘길 것 같거나 에러가 나면 거기서 멈춰 진행분만 보고하고 메인 모델이 이어받는다. 세션 모델이 이미 위임 목표와 같은 급 이하면 위임하지 않는다`
|
|
82
|
+
: `Cap the run at ${cap}; if it looks likely to exceed that or hits an error, it stops there and reports partial progress while the main model takes over. Skip delegation entirely when the session model is already at or below the target tier`;
|
|
83
|
+
return `${baseText.replace(/\s*$/, '')}. ${clause}`;
|
|
84
|
+
}
|
|
45
85
|
|
|
46
86
|
// See the note in route-scan.js — paths.js is the only place that resolves
|
|
47
87
|
// this, so an XDG_CONFIG_HOME override moves every state file together.
|
|
@@ -124,10 +164,34 @@ export function renderModelRatchet(rules, lang = userLanguage()) {
|
|
|
124
164
|
'## Rules',
|
|
125
165
|
'',
|
|
126
166
|
];
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
const
|
|
167
|
+
// Which number drove the review flag matters to the reader: a measured
|
|
168
|
+
// delegated error rate is evidence the rule itself is failing, while the
|
|
169
|
+
// proxy only says work of that shape tends to go wrong.
|
|
170
|
+
const pct = (v) => Math.round((v || 0) * 100);
|
|
171
|
+
const healthOf = (r) => {
|
|
172
|
+
if (r.status !== 'review') return '';
|
|
173
|
+
const measured = r.healthSource === 'delegated';
|
|
174
|
+
const rate = pct(measured ? r.delegatedErrRate : r.errRate);
|
|
175
|
+
if (ko) {
|
|
176
|
+
return measured
|
|
177
|
+
? ` ⚠ rule-health: 실제 위임 ${r.delegatedRuns}건 중 에러율 ${rate}% — 조건을 좁히거나 제거 검토`
|
|
178
|
+
: ` ⚠ rule-health: 최근 위임 대상 에러율 ${rate}% — 조건을 좁히거나 제거 검토`;
|
|
179
|
+
}
|
|
180
|
+
return measured
|
|
181
|
+
? ` ⚠ rule-health: ${rate}% error rate across ${r.delegatedRuns} measured delegations — narrow the condition or remove`
|
|
182
|
+
: ` ⚠ rule-health: recent error rate ${rate}% for the delegated category — narrow the condition or remove`;
|
|
183
|
+
};
|
|
184
|
+
const statsOf = (r) => {
|
|
185
|
+
const base = `×${r.count || 0}, err ${pct(r.errRate)}%, seen ${r.lastSeen || r.promotedAt}`;
|
|
186
|
+
if (!r.delegatedRuns) return base;
|
|
187
|
+
const saved = r.savedUsd ? `, saved ~$${r.savedUsd.toFixed(2)}` : '';
|
|
188
|
+
return `${base}, delegated ×${r.delegatedRuns} err ${pct(r.delegatedErrRate)}%${saved}`;
|
|
189
|
+
};
|
|
190
|
+
// Merged T2+T1 rules carry two caps, so they state both once rather than
|
|
191
|
+
// repeating the whole stop-condition per tier.
|
|
192
|
+
const mergedBudget = (t2, t1) => ko
|
|
193
|
+
? `위임 상한은 haiku ${budgetCapPhrase(t2, 'ko')}, sonnet ${budgetCapPhrase(t1, 'ko')}이며, 넘길 것 같거나 에러가 나면 거기서 멈춰 진행분만 보고하고 메인 모델이 이어받는다. 세션 모델이 이미 위임 목표와 같은 급 이하면 위임하지 않는다`
|
|
194
|
+
: `Cap haiku runs at ${budgetCapPhrase(t2, 'en')} and sonnet runs at ${budgetCapPhrase(t1, 'en')}; a run likely to exceed its cap, or hitting an error, stops there and reports partial progress while the main model takes over. Skip delegation entirely when the session model is already at or below the target tier`;
|
|
131
195
|
|
|
132
196
|
// A category can carry both a T2 (haiku) and a T1 (sonnet) rule. Tier is
|
|
133
197
|
// only known after an episode finishes, so two separate bullets give the
|
|
@@ -147,16 +211,18 @@ export function renderModelRatchet(rules, lang = userLanguage()) {
|
|
|
147
211
|
const rule = ko
|
|
148
212
|
? `"${t2.label}" 유형 요청은 기본적으로 ${agentPhrase(t2.agent)} 서브에이전트로 위임한다(예: "${t2.example}"). ` +
|
|
149
213
|
`여러 단계·여러 파일 수정이 얽힌 중간 난도 요청(예: "${t1.example}")은 model: sonnet 서브에이전트로 위임한다. ` +
|
|
150
|
-
`설계 판단·배포·스토어 제출 같은 비가역 작업이 섞이거나 위임 중 에러가 반복되면 위임하지 말고 메인 모델이 직접
|
|
214
|
+
`설계 판단·배포·스토어 제출 같은 비가역 작업이 섞이거나 위임 중 에러가 반복되면 위임하지 말고 메인 모델이 직접 처리한다. ` +
|
|
215
|
+
mergedBudget(t2, t1)
|
|
151
216
|
: `Delegate "${t2.labelEn || t2.label}" requests to ${agentPhraseEn(t2.agent)} by default (e.g. "${t2.example}"). ` +
|
|
152
217
|
`Escalate moderate ones that span multiple steps or file edits (e.g. "${t1.example}") to a model: sonnet subagent. ` +
|
|
153
|
-
`Do not delegate at all — handle it on the main model — when the request mixes in design judgement or irreversible work (deploy, release, store submission), or when errors repeat during delegation
|
|
218
|
+
`Do not delegate at all — handle it on the main model — when the request mixes in design judgement or irreversible work (deploy, release, store submission), or when errors repeat during delegation. ` +
|
|
219
|
+
mergedBudget(t2, t1);
|
|
154
220
|
lines.push(`- ${rule}${healthOf(t2)}${healthOf(t1)} <!-- T2 ${statsOf(t2)} / T1 ${statsOf(t1)} -->`);
|
|
155
221
|
for (const r of group) {
|
|
156
|
-
if (r !== t2 && r !== t1) lines.push(`- ${r.rule}${healthOf(r)} <!-- ${statsOf(r)} -->`);
|
|
222
|
+
if (r !== t2 && r !== t1) lines.push(`- ${composeRuleText(r.rule, r, lang)}${healthOf(r)} <!-- ${statsOf(r)} -->`);
|
|
157
223
|
}
|
|
158
224
|
} else {
|
|
159
|
-
for (const r of group) lines.push(`- ${r.rule}${healthOf(r)} <!-- ${statsOf(r)} -->`);
|
|
225
|
+
for (const r of group) lines.push(`- ${composeRuleText(r.rule, r, lang)}${healthOf(r)} <!-- ${statsOf(r)} -->`);
|
|
160
226
|
}
|
|
161
227
|
}
|
|
162
228
|
return lines.join('\n') + '\n';
|
|
@@ -229,17 +295,51 @@ export function syncAllFiles({ previousPaths = [] } = {}) {
|
|
|
229
295
|
* an expensive model handled directly that still look T1/T2 by shape (tier
|
|
230
296
|
* judged with the error signal zeroed; see route-scan's rule-health pass).
|
|
231
297
|
*/
|
|
232
|
-
export function refreshModelRules(episodeStats, { now } = {}) {
|
|
298
|
+
export function refreshModelRules(episodeStats, delegatedStats = new Map(), { now } = {}) {
|
|
233
299
|
const data = loadModelRules();
|
|
234
300
|
let changed = false;
|
|
301
|
+
const pick = (stats, r) => stats.get(`${r.tier}|${r.category}|${r.project}`)
|
|
302
|
+
|| (r.scope === 'global' ? stats.get(`${r.tier}|${r.category}|*`) : null);
|
|
303
|
+
|
|
235
304
|
for (const r of data.rules) {
|
|
236
|
-
const s = episodeStats
|
|
237
|
-
|
|
238
|
-
if (!s)
|
|
239
|
-
|
|
240
|
-
|
|
305
|
+
const s = pick(episodeStats, r);
|
|
306
|
+
const d = pick(delegatedStats, r);
|
|
307
|
+
if (!s && !d) {
|
|
308
|
+
// A rule whose category didn't appear at all this window keeps its last
|
|
309
|
+
// known recurrence, but its measured-delegation fields must still read
|
|
310
|
+
// as "nothing measured" rather than stay undefined — the CLI and the
|
|
311
|
+
// rendered md both branch on them.
|
|
312
|
+
r.delegatedRuns = r.delegatedRuns ?? 0;
|
|
313
|
+
r.delegatedErrRate = r.delegatedErrRate ?? 0;
|
|
314
|
+
r.savedUsd = r.savedUsd ?? 0;
|
|
315
|
+
r.healthSource = r.healthSource ?? 'proxy';
|
|
316
|
+
continue;
|
|
317
|
+
}
|
|
318
|
+
|
|
319
|
+
if (s) {
|
|
320
|
+
r.count = s.count;
|
|
321
|
+
r.errRate = s.epCount > 0 ? s.errCount / s.epCount : 0;
|
|
322
|
+
}
|
|
323
|
+
// Window snapshot, not a running total: these describe the current scan
|
|
324
|
+
// window so a rule that stopped firing decays to zero instead of coasting
|
|
325
|
+
// on old credit.
|
|
326
|
+
r.delegatedRuns = d ? d.runs : 0;
|
|
327
|
+
r.delegatedErrRate = d && d.runs > 0 ? d.errRuns / d.runs : 0;
|
|
328
|
+
r.savedUsd = d ? Math.round(d.savedUsd * 100) / 100 : 0;
|
|
241
329
|
r.lastSeen = now || r.lastSeen;
|
|
242
|
-
|
|
330
|
+
|
|
331
|
+
// Measured outcome beats the proxy once there is enough of it. The proxy
|
|
332
|
+
// asks "does work SHAPED like this tend to fail?"; the measurement asks
|
|
333
|
+
// "does this rule fail when it actually fires?" — only the second can
|
|
334
|
+
// catch a rule that is mis-firing on requests it should never have taken.
|
|
335
|
+
if (r.delegatedRuns >= HEALTH_MIN_SAMPLE_DELEGATED) {
|
|
336
|
+
r.healthSource = 'delegated';
|
|
337
|
+
r.status = r.delegatedErrRate > HEALTH_ERR_RATE ? 'review' : 'active';
|
|
338
|
+
} else {
|
|
339
|
+
r.healthSource = 'proxy';
|
|
340
|
+
r.status = s && r.errRate > HEALTH_ERR_RATE && s.epCount >= HEALTH_MIN_SAMPLE
|
|
341
|
+
? 'review' : 'active';
|
|
342
|
+
}
|
|
243
343
|
changed = true;
|
|
244
344
|
}
|
|
245
345
|
if (changed) {
|
package/src/route-scan.js
CHANGED
|
@@ -22,6 +22,8 @@ import { join } from 'node:path';
|
|
|
22
22
|
import { userDataDir } from './paths.js';
|
|
23
23
|
import { discoverSessionFiles } from './parser.js';
|
|
24
24
|
import { collectSessionRecords } from './session-records.js';
|
|
25
|
+
import { collectSubagentRuns, indexRuns, runsForEpisode } from './subagent-records.js';
|
|
26
|
+
import { estimateCost, modelRank, TIER_TARGET_RANK, tierForRank } from './cost.js';
|
|
25
27
|
import { agentPhrase, agentPhraseEn } from './agents.js';
|
|
26
28
|
|
|
27
29
|
// ── Tier bands (docs/TIER_CRITERIA.md §3) ────────────────────────────────
|
|
@@ -211,7 +213,13 @@ function toEpisodes(records) {
|
|
|
211
213
|
for (const r of records) {
|
|
212
214
|
const text = (r.userText || '').trim();
|
|
213
215
|
if (!cur || cur.text !== text) {
|
|
214
|
-
cur = {
|
|
216
|
+
cur = {
|
|
217
|
+
text, calls: 0, out: 0, mutating: 0, errors: 0, delegated: 0,
|
|
218
|
+
models: new Set(), cwd: '', tools: {},
|
|
219
|
+
// Delegation attribution (subagent-records): exact join key, plus the
|
|
220
|
+
// episode's time span for the fallback when a run has no meta file.
|
|
221
|
+
delegationToolUseIds: [], startedAt: null, endedAt: null,
|
|
222
|
+
};
|
|
215
223
|
episodes.push(cur);
|
|
216
224
|
}
|
|
217
225
|
cur.calls += 1;
|
|
@@ -222,14 +230,61 @@ function toEpisodes(records) {
|
|
|
222
230
|
for (const [name, n] of Object.entries(r.toolCounts || {})) {
|
|
223
231
|
cur.tools[name] = (cur.tools[name] || 0) + n;
|
|
224
232
|
}
|
|
233
|
+
for (const id of r.delegationToolUseIds || []) cur.delegationToolUseIds.push(id);
|
|
234
|
+
if (r.timestamp) {
|
|
235
|
+
const t = Date.parse(r.timestamp);
|
|
236
|
+
if (Number.isFinite(t)) {
|
|
237
|
+
if (cur.startedAt === null || t < cur.startedAt) cur.startedAt = t;
|
|
238
|
+
if (cur.endedAt === null || t > cur.endedAt) cur.endedAt = t;
|
|
239
|
+
}
|
|
240
|
+
}
|
|
225
241
|
cur.models.add(r.model);
|
|
226
242
|
if (!cur.cwd && r.cwd) cur.cwd = r.cwd;
|
|
227
243
|
}
|
|
228
244
|
return episodes;
|
|
229
245
|
}
|
|
230
246
|
|
|
231
|
-
|
|
232
|
-
|
|
247
|
+
/**
|
|
248
|
+
* Price rank of the model that actually handled the episode (the most
|
|
249
|
+
* expensive one, when a session switched models mid-episode).
|
|
250
|
+
*/
|
|
251
|
+
export function episodeRank(ep) {
|
|
252
|
+
let rank = -1;
|
|
253
|
+
for (const m of ep.models) rank = Math.max(rank, modelRank(m));
|
|
254
|
+
return rank;
|
|
255
|
+
}
|
|
256
|
+
|
|
257
|
+
/**
|
|
258
|
+
* Delegation only pays when the target tier is strictly cheaper than what ran
|
|
259
|
+
* the work. Without this a Sonnet session produced "delegate to sonnet" T1
|
|
260
|
+
* rules — a subagent rebuilding context for zero price difference, which is a
|
|
261
|
+
* net loss. (Replaces the old boolean "is it haiku?" test, which could not
|
|
262
|
+
* tell a Sonnet session from a Fable one.)
|
|
263
|
+
*/
|
|
264
|
+
export function worthDelegating(tier, rank) {
|
|
265
|
+
const target = TIER_TARGET_RANK[tier];
|
|
266
|
+
return target !== undefined && rank > target;
|
|
267
|
+
}
|
|
268
|
+
|
|
269
|
+
/**
|
|
270
|
+
* USD a delegated run saved versus the session model doing the same work.
|
|
271
|
+
* Approximation, deliberately stated as one: it holds token counts constant,
|
|
272
|
+
* which a cheaper model would not reproduce exactly. Directionally right and
|
|
273
|
+
* enough to rank rules by value, so it is rendered as "~$X".
|
|
274
|
+
*/
|
|
275
|
+
export function runSaving(run, mainModel) {
|
|
276
|
+
if (!run.model || !mainModel) return 0;
|
|
277
|
+
const totals = {
|
|
278
|
+
input: run.input,
|
|
279
|
+
cacheCreation: run.cacheCreation,
|
|
280
|
+
cacheRead: run.cacheRead,
|
|
281
|
+
ephemeral5m: run.ephemeral5m,
|
|
282
|
+
ephemeral1h: run.ephemeral1h,
|
|
283
|
+
output: run.out,
|
|
284
|
+
};
|
|
285
|
+
const actual = estimateCost(totals, run.model).actual;
|
|
286
|
+
const counterfactual = estimateCost(totals, mainModel).actual;
|
|
287
|
+
return Math.max(0, counterfactual - actual);
|
|
233
288
|
}
|
|
234
289
|
|
|
235
290
|
const clamp = (v, [lo, hi]) => Math.min(hi, Math.max(lo, v));
|
|
@@ -278,8 +333,9 @@ export async function runRouteScan({ days = 14 } = {}) {
|
|
|
278
333
|
|
|
279
334
|
// Pass 1 — collect episodes (needed up front: thresholds are calibrated
|
|
280
335
|
// from the full window's output distribution before any tiering).
|
|
281
|
-
const all = []; // { ep, projectDir }
|
|
336
|
+
const all = []; // { ep, projectDir, sessionPath }
|
|
282
337
|
let dataBytes = 0; // window size snapshot — the rescan gate diffs against it
|
|
338
|
+
const runIndexBySession = new Map(); // sessionPath → indexRuns() result
|
|
283
339
|
for (const f of files) {
|
|
284
340
|
let records;
|
|
285
341
|
try {
|
|
@@ -290,7 +346,17 @@ export async function runRouteScan({ days = 14 } = {}) {
|
|
|
290
346
|
}
|
|
291
347
|
for (const ep of toEpisodes(records)) {
|
|
292
348
|
if (!ep.text) continue;
|
|
293
|
-
all.push({ ep, projectDir: f.projectDir });
|
|
349
|
+
all.push({ ep, projectDir: f.projectDir, sessionPath: f.path });
|
|
350
|
+
}
|
|
351
|
+
// Subagent transcripts of this session — the real outcome of every
|
|
352
|
+
// delegation it made. Best-effort: sessions that never delegated have no
|
|
353
|
+
// directory and cost one failed readdir.
|
|
354
|
+
const runs = await collectSubagentRuns(f.path);
|
|
355
|
+
if (runs.length > 0) {
|
|
356
|
+
runIndexBySession.set(f.path, indexRuns(runs));
|
|
357
|
+
// Subagent bytes count toward the window size so the rescan gate stays
|
|
358
|
+
// accurate for delegation-heavy workloads.
|
|
359
|
+
for (const r of runs) dataBytes += r.bytes || 0;
|
|
294
360
|
}
|
|
295
361
|
}
|
|
296
362
|
const totalEpisodes = all.length;
|
|
@@ -311,7 +377,8 @@ export async function runRouteScan({ days = 14 } = {}) {
|
|
|
311
377
|
|
|
312
378
|
for (const { ep, projectDir } of all) {
|
|
313
379
|
if (isSkippable(ep.text)) continue;
|
|
314
|
-
|
|
380
|
+
const epRank = episodeRank(ep);
|
|
381
|
+
if (!worthDelegating('T2', epRank)) continue; // already at the cheapest tier
|
|
315
382
|
const cat = categorize(ep.text, ep.tools);
|
|
316
383
|
if (cat) {
|
|
317
384
|
// rule-health denominator: episodes that LOOK delegable by shape
|
|
@@ -323,13 +390,16 @@ export async function runRouteScan({ days = 14 } = {}) {
|
|
|
323
390
|
// and sharing one category-wide stat would double-count every episode
|
|
324
391
|
// into both rules (identical ×N / err% on unrelated tiers).
|
|
325
392
|
const shapeTier = tierOf({ ...ep, errors: 0 }, cat, thresholds);
|
|
326
|
-
|
|
393
|
+
// Same rank gate as the candidate path below — a denominator counting
|
|
394
|
+
// episodes that can't produce a rule would skew that rule's error rate.
|
|
395
|
+
if ((shapeTier === 'T1' || shapeTier === 'T2') && worthDelegating(shapeTier, epRank)) {
|
|
327
396
|
bumpStats(`${shapeTier}|${cat.id}|${projectDir}`, ep);
|
|
328
397
|
bumpStats(`${shapeTier}|${cat.id}|*`, ep);
|
|
329
398
|
}
|
|
330
399
|
}
|
|
331
400
|
const tier = tierOf(ep, cat, thresholds);
|
|
332
401
|
if (tier !== 'T1' && tier !== 'T2') continue;
|
|
402
|
+
if (!worthDelegating(tier, epRank)) continue;
|
|
333
403
|
tieredEpisodes += 1;
|
|
334
404
|
const key = `${tier}|${cat.id}|${projectDir}`;
|
|
335
405
|
const g = groups.get(key) || {
|
|
@@ -353,6 +423,50 @@ export async function runRouteScan({ days = 14 } = {}) {
|
|
|
353
423
|
groups.set(key, g);
|
|
354
424
|
}
|
|
355
425
|
|
|
426
|
+
// Pass 3 — measured delegation outcomes (rule-health v2). Episodes that
|
|
427
|
+
// DID delegate are excluded from tiering by design (tierOf returns T0 when
|
|
428
|
+
// ep.delegated > 0: there is nothing left to route). But they are exactly
|
|
429
|
+
// where a promoted rule's real success rate lives, so they get their own
|
|
430
|
+
// pass: join each episode to the subagent transcripts it spawned, and file
|
|
431
|
+
// the outcome under the tier that run's model represents — a haiku run is a
|
|
432
|
+
// T2 rule firing, a sonnet run a T1 one. Runs that were not a downgrade
|
|
433
|
+
// (same tier or higher) carry no delegation saving and are skipped.
|
|
434
|
+
const delegatedStats = new Map(); // "tier|category|project" → outcome aggregate
|
|
435
|
+
const bumpDelegated = (key, run, saved) => {
|
|
436
|
+
const d = delegatedStats.get(key) || { runs: 0, errRuns: 0, outTokens: 0, savedUsd: 0 };
|
|
437
|
+
d.runs += 1;
|
|
438
|
+
if (run.toolErrors > 0) d.errRuns += 1;
|
|
439
|
+
d.outTokens += run.out || 0;
|
|
440
|
+
d.savedUsd += saved;
|
|
441
|
+
delegatedStats.set(key, d);
|
|
442
|
+
};
|
|
443
|
+
for (const [sessionPath, index] of runIndexBySession) {
|
|
444
|
+
const used = new Set();
|
|
445
|
+
for (const { ep, projectDir, sessionPath: epSession } of all) {
|
|
446
|
+
if (epSession !== sessionPath) continue;
|
|
447
|
+
if (!ep.delegationToolUseIds.length && index.unjoined.length === 0) continue;
|
|
448
|
+
const runs = runsForEpisode(index, ep, used);
|
|
449
|
+
if (runs.length === 0) continue;
|
|
450
|
+
const cat = categorize(ep.text, ep.tools);
|
|
451
|
+
if (!cat) continue;
|
|
452
|
+
// Counterfactual = the priciest model on the episode, i.e. what would
|
|
453
|
+
// have done the work had it not been handed off.
|
|
454
|
+
let mainModel = null;
|
|
455
|
+
let mainRank = -1;
|
|
456
|
+
for (const m of ep.models) {
|
|
457
|
+
const r = modelRank(m);
|
|
458
|
+
if (r > mainRank) { mainRank = r; mainModel = m; }
|
|
459
|
+
}
|
|
460
|
+
for (const run of runs) {
|
|
461
|
+
const runTier = tierForRank(modelRank(run.model));
|
|
462
|
+
if (!runTier || !worthDelegating(runTier, mainRank)) continue;
|
|
463
|
+
const saved = runSaving(run, mainModel);
|
|
464
|
+
bumpDelegated(`${runTier}|${cat.id}|${projectDir}`, run, saved);
|
|
465
|
+
bumpDelegated(`${runTier}|${cat.id}|*`, run, saved);
|
|
466
|
+
}
|
|
467
|
+
}
|
|
468
|
+
}
|
|
469
|
+
|
|
356
470
|
// Keep prior dismissed/promoted signatures across rescans.
|
|
357
471
|
const prev = readRouteScan();
|
|
358
472
|
const resolved = new Set(prev?.resolved || []);
|
|
@@ -373,6 +487,16 @@ export async function runRouteScan({ days = 14 } = {}) {
|
|
|
373
487
|
// Both languages are computed at scan time and stored on the candidate, so
|
|
374
488
|
// switching `language` later re-renders (and promotes) correctly without
|
|
375
489
|
// waiting for a rescan.
|
|
490
|
+
// The probe-then-commit budget is deliberately NOT baked into this text:
|
|
491
|
+
// model-rules composes it on (composeRuleText), so rules promoted before
|
|
492
|
+
// budgets existed gain the clause too, and the promote preview can never
|
|
493
|
+
// drift from what lands in the file. What the candidate carries is the
|
|
494
|
+
// calibrated budget itself — the user's own thresholds, snapshotted at scan
|
|
495
|
+
// time rather than hardcoded downstream.
|
|
496
|
+
const budgetOf = (g) => ({
|
|
497
|
+
calls: g.tier === 'T2' ? T2_MAX_CALLS + 2 : null,
|
|
498
|
+
out: g.tier === 'T2' ? thresholds.t2Out : thresholds.t1Out,
|
|
499
|
+
});
|
|
376
500
|
const ruleText = (g) => g.tier === 'T2'
|
|
377
501
|
? `"${g.label}" 유형의 단순 요청(예: "${g.example}")은 ${agentPhrase(g.agent)} 서브에이전트로 위임한다 (설계 판단·배포·스토어 제출 같은 비가역 작업이 섞이면 위임하지 않음)`
|
|
378
502
|
: `"${g.label}" 유형의 중간 난도 요청(예: "${g.example}")은 model: sonnet 서브에이전트로 위임한다 (설계 판단·비가역 작업·반복 에러 발생 시 메인 모델이 이어받음)`;
|
|
@@ -400,6 +524,10 @@ export async function runRouteScan({ days = 14 } = {}) {
|
|
|
400
524
|
count: g.count,
|
|
401
525
|
models: [...g.models],
|
|
402
526
|
example: g.example,
|
|
527
|
+
// Snapshot of the calibrated budget this rule was written against, so
|
|
528
|
+
// the merged T2+T1 rendering in ratchet-model.md can restate it without
|
|
529
|
+
// re-running a scan.
|
|
530
|
+
budget: budgetOf(g),
|
|
403
531
|
// Concentrated in one project dir → project rule; the scan groups by
|
|
404
532
|
// project already, so scope suggestion is per-candidate 'project' unless
|
|
405
533
|
// the same category recurs across 2+ projects (then 'global').
|
|
@@ -441,7 +569,7 @@ export async function runRouteScan({ days = 14 } = {}) {
|
|
|
441
569
|
// rates, and rule-health flags — and rewrites their managed blocks.
|
|
442
570
|
try {
|
|
443
571
|
const { refreshModelRules } = await import('./model-rules.js');
|
|
444
|
-
refreshModelRules(episodeStats, { now: cache.scannedAt });
|
|
572
|
+
refreshModelRules(episodeStats, delegatedStats, { now: cache.scannedAt });
|
|
445
573
|
} catch { /* registry unwritable — scan result still valid */ }
|
|
446
574
|
|
|
447
575
|
return cache;
|
package/src/session-records.js
CHANGED
|
@@ -112,11 +112,18 @@ export async function collectSessionRecords(filePath, { includeContent = true }
|
|
|
112
112
|
// Per-tool call histogram — what the call actually DID. route-scan's
|
|
113
113
|
// categorizer trusts this over the prompt's wording (behavior-first).
|
|
114
114
|
const toolCounts = { ...(prev?.toolCounts || {}) };
|
|
115
|
+
// tool_use ids of Task/Agent calls: the join key to the subagent
|
|
116
|
+
// transcripts under <session>/subagents/*.meta.json, which is how the
|
|
117
|
+
// outcome of a delegation is measured (rule-health v2).
|
|
118
|
+
const delegationToolUseIds = [...(prev?.delegationToolUseIds || [])];
|
|
115
119
|
if (Array.isArray(msg.content)) {
|
|
116
120
|
for (const b of msg.content) {
|
|
117
121
|
if (!b || b.type !== 'tool_use') continue;
|
|
118
122
|
if (MUTATING_TOOLS.has(b.name)) mutatingToolCalls += 1;
|
|
119
|
-
if (DELEGATION_TOOLS.has(b.name))
|
|
123
|
+
if (DELEGATION_TOOLS.has(b.name)) {
|
|
124
|
+
delegationCalls += 1;
|
|
125
|
+
if (typeof b.id === 'string') delegationToolUseIds.push(b.id);
|
|
126
|
+
}
|
|
120
127
|
if (typeof b.name === 'string') toolCounts[b.name] = (toolCounts[b.name] || 0) + 1;
|
|
121
128
|
}
|
|
122
129
|
}
|
|
@@ -129,6 +136,14 @@ export async function collectSessionRecords(filePath, { includeContent = true }
|
|
|
129
136
|
(usage.cache_creation_input_tokens || 0) +
|
|
130
137
|
(usage.cache_read_input_tokens || 0),
|
|
131
138
|
completion_tokens: usage.output_tokens || 0,
|
|
139
|
+
// Per-bucket split, kept alongside the collapsed prompt_tokens: pricing
|
|
140
|
+
// differs per bucket, so costing a delegated run against what the
|
|
141
|
+
// session model would have charged needs them separated.
|
|
142
|
+
input_tokens: usage.input_tokens || 0,
|
|
143
|
+
cache_creation_tokens: usage.cache_creation_input_tokens || 0,
|
|
144
|
+
cache_read_tokens: usage.cache_read_input_tokens || 0,
|
|
145
|
+
ephemeral5m: usage.cache_creation?.ephemeral_5m_input_tokens || 0,
|
|
146
|
+
ephemeral1h: usage.cache_creation?.ephemeral_1h_input_tokens || 0,
|
|
132
147
|
depth,
|
|
133
148
|
userText: includeContent ? lastUserText : '',
|
|
134
149
|
assistantText: includeContent ? contentText(msg.content) : '',
|
|
@@ -136,6 +151,7 @@ export async function collectSessionRecords(filePath, { includeContent = true }
|
|
|
136
151
|
// Entries of the same request accumulate tool blocks and errors.
|
|
137
152
|
mutatingToolCalls: (prev?.mutatingToolCalls || 0) + mutatingToolCalls,
|
|
138
153
|
delegationCalls: (prev?.delegationCalls || 0) + delegationCalls,
|
|
154
|
+
delegationToolUseIds,
|
|
139
155
|
toolErrors: prev?.toolErrors || 0,
|
|
140
156
|
toolCounts,
|
|
141
157
|
};
|
|
@@ -0,0 +1,185 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* subagent-records — read the transcripts of subagent runs a session spawned.
|
|
3
|
+
*
|
|
4
|
+
* Why this exists: rule-health used to be a PROXY. It measured the error rate
|
|
5
|
+
* of episodes the expensive model handled DIRECTLY that merely *looked*
|
|
6
|
+
* delegable by shape — never the outcome of an actual delegation. So a rule
|
|
7
|
+
* could be quietly failing every time it fired and the signal would not move.
|
|
8
|
+
* Claude Code writes each subagent run to its own transcript, which makes the
|
|
9
|
+
* real outcome measurable:
|
|
10
|
+
*
|
|
11
|
+
* ~/.claude/projects/<munged>/<sessionId>/subagents/agent-<id>.jsonl
|
|
12
|
+
* ~/.claude/projects/<munged>/<sessionId>/subagents/agent-<id>.meta.json
|
|
13
|
+
*
|
|
14
|
+
* The .jsonl is byte-identical in shape to a main transcript (`isSidechain:
|
|
15
|
+
* true`, assistant entries carrying `message.model` + `message.usage`,
|
|
16
|
+
* tool_result blocks carrying `is_error`), so collectSessionRecords parses it
|
|
17
|
+
* unchanged — including the rejection / self-corrected error filters, which
|
|
18
|
+
* must apply here for the same reason they apply to main sessions.
|
|
19
|
+
*
|
|
20
|
+
* The .meta.json carries `{ agentType, description, toolUseId, spawnDepth }`.
|
|
21
|
+
* `toolUseId` is the join key back to the Task/Agent tool_use block in the
|
|
22
|
+
* parent transcript, which is how a run is attributed to the episode (and
|
|
23
|
+
* therefore the category) that caused it.
|
|
24
|
+
*
|
|
25
|
+
* Everything is best-effort: this layout is a Claude Code internal, so a
|
|
26
|
+
* missing directory, an absent meta file, or an unparseable line degrades to
|
|
27
|
+
* less data, never to a throw. route-scan keeps its proxy signal as fallback.
|
|
28
|
+
*/
|
|
29
|
+
|
|
30
|
+
import { readdir, readFile, stat } from 'node:fs/promises';
|
|
31
|
+
import { join, dirname, basename } from 'node:path';
|
|
32
|
+
import { collectSessionRecords, normalizeModelId } from './session-records.js';
|
|
33
|
+
|
|
34
|
+
/** Directory holding a session's subagent transcripts (may not exist). */
|
|
35
|
+
export function subagentDirFor(sessionPath) {
|
|
36
|
+
return join(dirname(sessionPath), basename(sessionPath, '.jsonl'), 'subagents');
|
|
37
|
+
}
|
|
38
|
+
|
|
39
|
+
async function readMeta(metaPath) {
|
|
40
|
+
try {
|
|
41
|
+
return JSON.parse(await readFile(metaPath, 'utf8'));
|
|
42
|
+
} catch {
|
|
43
|
+
return null;
|
|
44
|
+
}
|
|
45
|
+
}
|
|
46
|
+
|
|
47
|
+
/**
|
|
48
|
+
* A run's model when several appear in one transcript (a harness retry can
|
|
49
|
+
* switch tiers mid-run): the one that produced the most output, since that is
|
|
50
|
+
* what dominates both the bill and the counterfactual.
|
|
51
|
+
*/
|
|
52
|
+
function primaryModel(byModelOut) {
|
|
53
|
+
let best = null;
|
|
54
|
+
let bestOut = -1;
|
|
55
|
+
for (const [model, out] of byModelOut) {
|
|
56
|
+
if (out > bestOut) { best = model; bestOut = out; }
|
|
57
|
+
}
|
|
58
|
+
return best;
|
|
59
|
+
}
|
|
60
|
+
|
|
61
|
+
/**
|
|
62
|
+
* Aggregate one subagent transcript into a single run record.
|
|
63
|
+
* Returns null when the file carries no billable API call.
|
|
64
|
+
*/
|
|
65
|
+
export async function collectSubagentRun(jsonlPath) {
|
|
66
|
+
let records;
|
|
67
|
+
try {
|
|
68
|
+
records = await collectSessionRecords(jsonlPath, { includeContent: false });
|
|
69
|
+
} catch {
|
|
70
|
+
return null;
|
|
71
|
+
}
|
|
72
|
+
if (records.length === 0) return null;
|
|
73
|
+
|
|
74
|
+
const run = {
|
|
75
|
+
path: jsonlPath,
|
|
76
|
+
agentId: basename(jsonlPath, '.jsonl').replace(/^agent-/, ''),
|
|
77
|
+
agentType: null,
|
|
78
|
+
toolUseId: null,
|
|
79
|
+
spawnDepth: null,
|
|
80
|
+
model: null,
|
|
81
|
+
calls: records.length,
|
|
82
|
+
out: 0,
|
|
83
|
+
input: 0,
|
|
84
|
+
cacheCreation: 0,
|
|
85
|
+
cacheRead: 0,
|
|
86
|
+
ephemeral5m: 0,
|
|
87
|
+
ephemeral1h: 0,
|
|
88
|
+
toolErrors: 0,
|
|
89
|
+
startedAt: null,
|
|
90
|
+
endedAt: null,
|
|
91
|
+
bytes: 0,
|
|
92
|
+
};
|
|
93
|
+
const byModelOut = new Map();
|
|
94
|
+
for (const r of records) {
|
|
95
|
+
run.out += r.completion_tokens || 0;
|
|
96
|
+
run.input += r.input_tokens || 0;
|
|
97
|
+
run.cacheCreation += r.cache_creation_tokens || 0;
|
|
98
|
+
run.cacheRead += r.cache_read_tokens || 0;
|
|
99
|
+
run.ephemeral5m += r.ephemeral5m || 0;
|
|
100
|
+
run.ephemeral1h += r.ephemeral1h || 0;
|
|
101
|
+
run.toolErrors += r.toolErrors || 0;
|
|
102
|
+
const model = normalizeModelId(r.model);
|
|
103
|
+
byModelOut.set(model, (byModelOut.get(model) || 0) + (r.completion_tokens || 0));
|
|
104
|
+
if (r.timestamp) {
|
|
105
|
+
const t = Date.parse(r.timestamp);
|
|
106
|
+
if (Number.isFinite(t)) {
|
|
107
|
+
if (run.startedAt === null || t < run.startedAt) run.startedAt = t;
|
|
108
|
+
if (run.endedAt === null || t > run.endedAt) run.endedAt = t;
|
|
109
|
+
}
|
|
110
|
+
}
|
|
111
|
+
}
|
|
112
|
+
run.model = primaryModel(byModelOut);
|
|
113
|
+
|
|
114
|
+
const meta = await readMeta(jsonlPath.replace(/\.jsonl$/, '.meta.json'));
|
|
115
|
+
if (meta) {
|
|
116
|
+
run.agentType = meta.agentType ?? null;
|
|
117
|
+
run.toolUseId = meta.toolUseId ?? null;
|
|
118
|
+
run.spawnDepth = meta.spawnDepth ?? null;
|
|
119
|
+
}
|
|
120
|
+
try {
|
|
121
|
+
run.bytes = (await stat(jsonlPath)).size;
|
|
122
|
+
} catch { /* size only feeds the rescan gate — 0 is a safe under-estimate */ }
|
|
123
|
+
|
|
124
|
+
return run;
|
|
125
|
+
}
|
|
126
|
+
|
|
127
|
+
/**
|
|
128
|
+
* All subagent runs a session spawned. Empty array when the session never
|
|
129
|
+
* delegated (the common case) or the directory is unreadable.
|
|
130
|
+
*/
|
|
131
|
+
export async function collectSubagentRuns(sessionPath) {
|
|
132
|
+
const dir = subagentDirFor(sessionPath);
|
|
133
|
+
let entries;
|
|
134
|
+
try {
|
|
135
|
+
entries = await readdir(dir);
|
|
136
|
+
} catch {
|
|
137
|
+
return [];
|
|
138
|
+
}
|
|
139
|
+
const runs = [];
|
|
140
|
+
for (const e of entries) {
|
|
141
|
+
if (!e.endsWith('.jsonl')) continue;
|
|
142
|
+
const run = await collectSubagentRun(join(dir, e));
|
|
143
|
+
if (run) runs.push(run);
|
|
144
|
+
}
|
|
145
|
+
return runs;
|
|
146
|
+
}
|
|
147
|
+
|
|
148
|
+
/**
|
|
149
|
+
* Index a session's runs for attribution: exact join by tool_use id first,
|
|
150
|
+
* with the un-joinable ones kept aside for the timestamp fallback (a run
|
|
151
|
+
* whose .meta.json is missing or predates toolUseId still happened, and
|
|
152
|
+
* dropping it would silently under-count a rule's real error rate).
|
|
153
|
+
*/
|
|
154
|
+
export function indexRuns(runs) {
|
|
155
|
+
const byToolUse = new Map();
|
|
156
|
+
const unjoined = [];
|
|
157
|
+
for (const r of runs) {
|
|
158
|
+
if (r.toolUseId) byToolUse.set(r.toolUseId, r);
|
|
159
|
+
else unjoined.push(r);
|
|
160
|
+
}
|
|
161
|
+
return { byToolUse, unjoined, all: runs };
|
|
162
|
+
}
|
|
163
|
+
|
|
164
|
+
/**
|
|
165
|
+
* Runs attributable to one episode. Exact tool_use ids win; anything left
|
|
166
|
+
* unjoined is matched by start time falling inside the episode's span. `used`
|
|
167
|
+
* is a shared Set of run paths across the session so no run is counted twice
|
|
168
|
+
* (an episode's span can overlap a neighbouring episode's runs).
|
|
169
|
+
*/
|
|
170
|
+
export function runsForEpisode(index, ep, used) {
|
|
171
|
+
const out = [];
|
|
172
|
+
for (const id of ep.delegationToolUseIds || []) {
|
|
173
|
+
const r = index.byToolUse.get(id);
|
|
174
|
+
if (r && !used.has(r.path)) { used.add(r.path); out.push(r); }
|
|
175
|
+
}
|
|
176
|
+
if (ep.startedAt === null || ep.endedAt === null) return out;
|
|
177
|
+
for (const r of index.unjoined) {
|
|
178
|
+
if (used.has(r.path) || r.startedAt === null) continue;
|
|
179
|
+
if (r.startedAt >= ep.startedAt && r.startedAt <= ep.endedAt) {
|
|
180
|
+
used.add(r.path);
|
|
181
|
+
out.push(r);
|
|
182
|
+
}
|
|
183
|
+
}
|
|
184
|
+
return out;
|
|
185
|
+
}
|