claude-token-saver 3.8.2 โ 3.9.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.en.md +18 -4
- package/README.md +17 -3
- package/package.json +1 -1
- package/src/brief.js +1 -1
- package/src/commands/compact-window.js +12 -9
- package/src/commands/harness.js +3 -0
- package/src/commands/route-scan.js +32 -6
- package/src/compact-window.js +19 -6
- package/src/cost.js +39 -0
- package/src/model-rules.js +115 -15
- package/src/route-scan.js +136 -8
- package/src/session-records.js +17 -1
- package/src/subagent-records.js +185 -0
package/README.en.md
CHANGED
|
@@ -144,19 +144,21 @@ An auto `.bak` is kept, but **the session context that earned the rule its place
|
|
|
144
144
|
|
|
145
145
|
## ๐ฆ compact-window โ pin where a 1M session compacts
|
|
146
146
|
|
|
147
|
-
Claude Code compacts when usage approaches `min(autoCompactWindow, model max context)`. On a 1M window, with that value unset, compaction only fires near 800k โ and until then every request re-bills the whole context.
|
|
147
|
+
Claude Code compacts when usage approaches `min(autoCompactWindow, model max context)`. On a 1M window, with that value unset, compaction only fires near 800k โ and until then every request re-bills the whole context. **1M is too large; the recommendation is a 400kโ700k band** โ 2โ3.5x a 200k session's headroom for the genuinely large pastes, with the runaway tail cut off.
|
|
148
|
+
|
|
149
|
+
**Anything inside the band is left alone.** 400k is the floor where the saving beats the extra compactions, and long sessions often want more room than that. Only an unset window, or one above 700k, is warned about (a smaller one is a deliberate, more aggressive choice).
|
|
148
150
|
|
|
149
151
|
**200k sessions are never warned** โ their window is already at or below 200k, so the setting cannot change anything.
|
|
150
152
|
|
|
151
153
|
```bash
|
|
152
154
|
claude-token-saver compact-window # status (model, window, value, source)
|
|
153
|
-
claude-token-saver compact-window set --global # pin
|
|
155
|
+
claude-token-saver compact-window set --global # pin 500k (mid-band) in ~/.claude/settings.json
|
|
154
156
|
claude-token-saver compact-window set --project # pin it in <root>/.claude/settings.json
|
|
155
|
-
claude-token-saver compact-window set --global --value
|
|
157
|
+
claude-token-saver compact-window set --global --value 600k # explicit value (100kโ1M)
|
|
156
158
|
claude-token-saver compact-window off | on # toggle the warning
|
|
157
159
|
```
|
|
158
160
|
|
|
159
|
-
- On a 1M model with the value unset or above
|
|
161
|
+
- On a 1M model with the value unset or above 700k, the statusline shows `๐
ทโ compact-window?` and the session briefing hands the model the exact registration command.
|
|
160
162
|
- Scope (`--global`/`--project`) is **required** for `set` โ a global settings file is never edited on a guess.
|
|
161
163
|
- Every other key in `settings.json` is preserved and a `.bak` is written first. Malformed JSON aborts the write untouched.
|
|
162
164
|
- An exported `CLAUDE_CODE_AUTO_COMPACT_WINDOW` beats settings.json; `set` detects that and says so.
|
|
@@ -263,6 +265,18 @@ Also update `statusLine.command` in `~/.claude/settings.json` to `claude-token-s
|
|
|
263
265
|
|
|
264
266
|
## Release notes
|
|
265
267
|
|
|
268
|
+
### v3.9.1 (2026-08-01)
|
|
269
|
+
- **compact-window now recommends a 400kโ700k band instead of a single 400k** โ 400k proved too tight in practice and compacted too often. The advice is a range now, and **a window inside it (or below it) is never warned about**; only an unset value or one above 700k raises `๐
ทโ compact-window?` and the briefing. `set` defaults to 500k (mid-band); pick your own with `--value 600k`.
|
|
270
|
+
|
|
271
|
+
### v3.9.0 (2026-08-01)
|
|
272
|
+
|
|
273
|
+
Compared this tool's design against manifest.build's "Everyone is building LLM routers, we deprecated ours" (a retrospective on 4 months across 7,000 users) and closed the four of their failure modes that were not already ruled out here. Full comparison in [TIER_CRITERIA.md ยง3.9](./docs/TIER_CRITERIA.md).
|
|
274
|
+
|
|
275
|
+
- **rule-health now watches real delegations** โ the old error rate's denominator was "episodes the expensive model handled directly that merely *looked* delegable by shape". It never measured whether a rule succeeded **when it actually fired**, so a rule could fail every single time without moving the signal. Claude Code writes each subagent run to `<session>/subagents/`, and its meta `toolUseId` joins exactly onto the parent's Task call โ so the outcome of a real delegation is now counted directly. Once 5 measured runs accumulate, they replace the shape-based proxy, and the warning says which evidence it is using. (Author's 14-day log: 4 of 106 sessions, 18 runs, 100% join rate.)
|
|
276
|
+
- **Per-rule savings** โ the tokens a delegated run spent are re-priced at the session model's rate, and the difference is reported over the 14-day window as `~$` in `route-scan rules` and `ratchet-model.md`. You can only prune rules that aren't worth their keep if you can see which ones those are. A rule with no measured delegations shows `โ`, not `$0` โ the two mean opposite things.
|
|
277
|
+
- **Tiers are now relative to the session model** โ the only gate used to be "is it haiku?", so a Sonnet session produced T1 rules telling it to delegate to Sonnet: a subagent rebuilding context for zero price difference, which is a net loss. Candidates are now only created when the target tier is genuinely cheaper (haiku 0 ยท sonnet 1 ยท opus 2 ยท fable 3).
|
|
278
|
+
- **Probe-then-commit budgets** โ a rule is derived from statistics but **fires on the request text alone**, and most of what makes a task hard only shows up after the first tool call. That gap can't be closed, so the cost of getting it wrong is capped instead: every rule now carries a calibrated limit (T2: 8 tool calls / p25 output, T1: p75 output), and a run likely to exceed it โ or hitting an error โ stops there, reports partial progress, and hands back to the main model. The clause is composed at render time rather than baked into the stored rule, so **rules promoted before this release get it too**, and the promote preview can never drift from what lands in the file.
|
|
279
|
+
|
|
266
280
|
### v3.8.2 (2026-08-01)
|
|
267
281
|
- **Fixed: the context tier never came back down after compaction** โ the tier was kept as a high-water mark, so a session that compacted at 80% stayed at tier 1 even though its context had just been emptied, and it refilled to the cap with no signal at all. The tier now follows the measurement down and warns again on the next climb.
|
|
268
282
|
- **Both windows are named** โ with `autoCompactWindow` at 400k the briefing measured against 400k (80%) while Claude Code's own display measured against 1M (33%); two irreconcilable numbers for one session. The text now reads `์๋ ์์ถ ์ฐฝ(400k)์ 80% (โฆ ํ๋ฉด์ 1M ์ฐฝ ๊ธฐ์ค์ผ๋ก๋ 33%)`.
|
package/README.md
CHANGED
|
@@ -125,15 +125,17 @@ ratchet์ ๊ฐ์น๋ **ํ ๋ฐฉํฅ ๋์ **์ ์์ต๋๋ค. ๋ฃฐ์ ๊ฐ๋ณ๊ฒ ์ง
|
|
|
125
125
|
|
|
126
126
|
## ๐ฆ compact-window โ 1M ์ปจํ
์คํธ์ ์๋ ์์ถ ์ง์ ๊ณ ์
|
|
127
127
|
|
|
128
|
-
Claude Code๋ `min(autoCompactWindow, ๋ชจ๋ธ ์ต๋ ์ฐฝ)`์ ๊ฐ๊น์์ง๋ฉด ๋ํ๋ฅผ ์๋ ์์ถํฉ๋๋ค. 1M ์ฐฝ์ ์ฐ๋ฉด ์ด ๊ฐ์ด ์กํ ์์ง ์์ ํ 80๋ง ํ ํฐ ๊ทผ์ฒ๊น์ง ๊ฐ์์ผ ์์ถ์ด ๊ฑธ๋ฆฌ๊ณ , ๊ทธ์ ๊น์ง ๋ชจ๋ ์์ฒญ์ด ์ ์ฒด ์ปจํ
์คํธ๋ฅผ ํต์งธ๋ก ์ฌ๊ณผ๊ธํฉ๋๋ค. 40
|
|
128
|
+
Claude Code๋ `min(autoCompactWindow, ๋ชจ๋ธ ์ต๋ ์ฐฝ)`์ ๊ฐ๊น์์ง๋ฉด ๋ํ๋ฅผ ์๋ ์์ถํฉ๋๋ค. 1M ์ฐฝ์ ์ฐ๋ฉด ์ด ๊ฐ์ด ์กํ ์์ง ์์ ํ 80๋ง ํ ํฐ ๊ทผ์ฒ๊น์ง ๊ฐ์์ผ ์์ถ์ด ๊ฑธ๋ฆฌ๊ณ , ๊ทธ์ ๊น์ง ๋ชจ๋ ์์ฒญ์ด ์ ์ฒด ์ปจํ
์คํธ๋ฅผ ํต์งธ๋ก ์ฌ๊ณผ๊ธํฉ๋๋ค. **1M์ ๋๋ฌด ํฌ๋ 40๋ง~70๋ง ๋ฒ์๋ฅผ ๊ถ์ฅํฉ๋๋ค** โ ํฐ ๋ถ์ฌ๋ฃ๊ธฐ์ฉ ์ฌ์ ๋ 200k ์ธ์
์ 2~3.5๋ฐฐ๋ก ๋จ๊ธฐ๋ฉด์ ๊ผฌ๋ฆฌ๋ง ์๋ผ๋
๋๋ค.
|
|
129
|
+
|
|
130
|
+
**๊ถ์ฅ ๋ฒ์ ์์ด๋ฉด ๊ฒฝ๊ณ ํ์ง ์์ต๋๋ค.** 40๋ง์ ์ ๊ฐ์ด ์์ถ ํ์๋ฅผ ์ด๊ธฐ๋ ํํ์ด๊ณ , ๊ธด ์ธ์
์ ๊ทธ๋ณด๋ค ์ฌ์ ๊ฐ ๋ ํ์ํ ๊ฒฝ์ฐ๊ฐ ๋ง์ต๋๋ค. ๋ฏธ์ค์ ์ด๊ฑฐ๋ 70๋ง์ ๋์ ๋๋ง ์๋ฆฝ๋๋ค(๊ทธ๋ณด๋ค ๋ฎ๊ฒ ์ก์ ๊ฑด ๋ ๊ณต๊ฒฉ์ ์ผ๋ก ์๋ผ๊ฒ ๋ค๋ ์ ํ์ด๋ผ ๊ทธ๋ฅ ๋ก๋๋ค).
|
|
129
131
|
|
|
130
132
|
**200k ์ปจํ
์คํธ๋ ๊ฒฝ๊ณ ๋์์ด ์๋๋๋ค** โ ์ฐฝ์ด ์ด๋ฏธ 200k ์ดํ๋ผ ์ด ์ค์ ์ด ๋ฐ๊ฟ ๊ฒ ์์ต๋๋ค.
|
|
131
133
|
|
|
132
134
|
```bash
|
|
133
135
|
claude-token-saver compact-window # ํ์ฌ ์ํ (๋ชจ๋ธยท์ฐฝยท์ค์ ๊ฐยท์ถ์ฒ)
|
|
134
|
-
claude-token-saver compact-window set --global # ~/.claude/settings.json ์
|
|
136
|
+
claude-token-saver compact-window set --global # ~/.claude/settings.json ์ 50๋ง ๊ณ ์ (๋ฒ์ ์ค๊ฐ)
|
|
135
137
|
claude-token-saver compact-window set --project # <root>/.claude/settings.json ์ ๊ณ ์
|
|
136
|
-
claude-token-saver compact-window set --global --value
|
|
138
|
+
claude-token-saver compact-window set --global --value 600k # ๊ฐ ์ง์ ์ง์ (10๋ง~1M)
|
|
137
139
|
claude-token-saver compact-window off | on # ๊ฒฝ๊ณ ํ์ ํ ๊ธ
|
|
138
140
|
```
|
|
139
141
|
|
|
@@ -220,6 +222,18 @@ npm uninstall -g claude-cache-monitor && npm i -g claude-token-saver
|
|
|
220
222
|
|
|
221
223
|
## ๋ฆด๋ฆฌ์ค ๋
ธํธ
|
|
222
224
|
|
|
225
|
+
### v3.9.1 (2026-08-01)
|
|
226
|
+
- **compact-window ๊ถ์ฅ๊ฐ์ด ๋จ์ผ 40๋ง์์ 40๋ง~70๋ง ๋ฒ์๋ก** โ 40๋ง์ ์ค์ฌ์ฉ์์ ๋๋ฌด ๋นก๋นกํด ์์ถ์ด ์ฆ์์ต๋๋ค. ์ด์ ๋ฒ์๋ฅผ ์ ์ํ๊ณ , **๊ทธ ์(๋๋ ๊ทธ๋ณด๋ค ๋ฎ๊ฒ) ์ก์๋ ์ธ์
์ ๊ฒฝ๊ณ ํ์ง ์์ต๋๋ค.** ๋ฏธ์ค์ ์ด๊ฑฐ๋ 70๋ง ์ด๊ณผ์ผ ๋๋ง `๐
ทโ compact-window?`์ ๋ธ๋ฆฌํ์ด ๋น๋๋ค. `set`์ ๊ธฐ๋ณธ๊ฐ๋ ๋ฒ์ ์ค๊ฐ์ธ 50๋ง์ผ๋ก ์ฌ๋ ธ๊ณ , ์ํ๋ ๊ฐ์ `--value 600k`๋ก ์ง์ ํฉ๋๋ค.
|
|
227
|
+
|
|
228
|
+
### v3.9.0 (2026-08-01)
|
|
229
|
+
|
|
230
|
+
manifest.build์ "๋ค๋ค LLM ๋ผ์ฐํฐ ๋ง๋๋๋ฐ ์ฐ๋ฆฌ๋ ํ๊ธฐํ๋ค"(7์ฒ ์ฌ์ฉ์ยท4๊ฐ์ ์ค์ฌ์ฉ ํ๊ณ )์ ์ด ๋๊ตฌ์ ์ค๊ณ๋ฅผ ๋์กฐํด, ๊ทธ์ชฝ ์คํจ ์์ธ ์ค ์์ง ์ ๋งํ ์๋ ๊ฒ 4๊ฐ๋ฅผ ๋ฉ์ ์ต๋๋ค. ์์ธํ ๋์กฐ๋ [TIER_CRITERIA.md ยง3.9](./docs/TIER_CRITERIA.md).
|
|
231
|
+
|
|
232
|
+
- **rule-health๊ฐ ์ด์ ์ค์ ์์ ๊ฒฐ๊ณผ๋ฅผ ๋ด
๋๋ค** โ ๊ธฐ์กด ์๋ฌ์จ์ ๋ถ๋ชจ๋ "๋น์ผ ๋ชจ๋ธ์ด ์ง์ ์ฒ๋ฆฌํ๋๋ฐ ํํ์ ์์ ๊ฐ๋ฅํด ๋ณด์ด๋ ์ํผ์๋"์์ต๋๋ค. ์ฆ ๋ฃฐ์ด **์ค์ ๋ก ๋ฐ๋ํ์ ๋ ์ ๋๋์ง๋ ํ ๋ฒ๋ ์ฌ์ง ์์๊ณ **, ๋งค๋ฒ ์คํจํ๋ ๋ฃฐ์ด ์์ด๋ ์ ํธ๊ฐ ์ ์์ง์์ต๋๋ค. Claude Code๊ฐ ์๋ธ์์ด์ ํธ ์คํ์ `<์ธ์
>/subagents/`์ ๋ฐ๋ก ๋จ๊ธฐ๊ณ ๊ทธ ๋ฉํ์ `toolUseId`๊ฐ ๋ถ๋ชจ์ Task ํธ์ถ๊ณผ ์ ํํ ๋ง๋ฌผ๋ฆฌ๋ฏ๋ก, ์ด์ ์ค์ ์์์ ์ฑํจ๋ฅผ ์ง์ ์
๋๋ค. ์ค์ธก 5๊ฑด ์ด์ ์์ธ ๋ฃฐ์ ์ถ์ ๋์ ์ค์ธก์ผ๋ก ํ์ ํ๊ณ , ๊ฒฝ๊ณ ์๋ ์ด๋ ์ชฝ ๊ทผ๊ฑฐ์ธ์ง ํ์ํฉ๋๋ค. (์ ์ ๋ก๊ทธ 14์ผ: 106์ธ์
์ค 4์ธ์
ยท18๋ฐ, ์กฐ์ธ ์ฑ๊ณต๋ฅ 100%)
|
|
233
|
+
- **๋ฃฐ๋ณ ์ ๊ฐ์ก ํ์** โ ์์ ์คํ์ ํ ํฐ์ ์ธ์
๋ชจ๋ธ ๋จ๊ฐ๋ก ๋๋๋ฆฐ ์ฐจ์ก์ 14์ผ ์ฐฝ์ผ๋ก ๊ณ์ฐํด `route-scan rules`์ `ratchet-model.md`์ `~$` ํ๊ธฐ๋ก ๋ถ์
๋๋ค. ๊ฐ์ด์น ์๋ ๋ฃฐ์ด ๋์ ๋ณด์ฌ์ผ ์ ๋ฆฌํ ์ ์์ต๋๋ค. ์์ ๊ธฐ๋ก์ด ์๋ ๋ฃฐ์ `$0`์ด ์๋๋ผ `โ` โ ๋์ ์ ๋ฐ๋๋ฅผ ๋ปํฉ๋๋ค.
|
|
234
|
+
- **์ธ์
๋ชจ๋ธ ๊ธฐ์ค ์๋ ํฐ์ด** โ ๊ธฐ์กด ๊ฒ์ดํธ๊ฐ "haiku์ธ๊ฐ?" ํ๋๋ฟ์ด๋ผ, Sonnet ์ธ์
์์๋ "sonnetํํ
์์ํ๋ผ"๋ T1 ๋ฃฐ์ด ๋ง๋ค์ด์ก์ต๋๋ค. ์ปจํ
์คํธ๋ง ์๋ก ์๊ณ ๋จ๊ฐ ์ฐจ์ด๋ 0์ธ ์์ํด์
๋๋ค. ์ด์ ๋ชฉํ ํฐ์ด๊ฐ ์ค์ ๋ก ๋ ์ผ ๊ฒฝ์ฐ์๋ง ํ๋ณด๋ฅผ ๋ง๋ญ๋๋ค (haiku 0 ยท sonnet 1 ยท opus 2 ยท fable 3).
|
|
235
|
+
- **์์ ์์ฐ ๋ฌธ๊ตฌ(probe-then-commit)** โ ๋ฃฐ์ ํต๊ณ๋ก ๋ง๋ค์ด์ง์ง๋ง **๋ฐ๋์ ์์ฒญ ํ
์คํธ๋ง ๋ณด๊ณ ** ์ผ์ด๋๊ณ , ๋์ด๋๋ ๋๊ฐ ์ฒซ ๋๊ตฌ ํธ์ถ ๋ค์์ผ ๋๋ฌ๋ฉ๋๋ค. ์ด ๋ถ์ผ์น๋ ๋ชป ์์ ๋ ์คํ ๋น์ฉ์ ์ํ์ ๊ฒ๋๋ค: ๊ฐ ๋ฃฐ์ ์บ๋ฆฌ๋ธ๋ ์ด์
๋ ์ํ(T2๋ ํธ์ถ 8ํยท์ถ๋ ฅ p25, T1์ ์ถ๋ ฅ p75)์ด ๋ถ๊ณ , ๋๊ธธ ๊ฒ ๊ฐ๊ฑฐ๋ ์๋ฌ๊ฐ ๋๋ฉด ์๋ธ์์ด์ ํธ๊ฐ ๋ฉ์ถฐ ์งํ๋ถ๋ง ๋ณด๊ณ ํ๊ณ ๋ฉ์ธ ๋ชจ๋ธ์ด ์ด์ด๋ฐ์ต๋๋ค. ๋ฌธ๊ตฌ๋ ์ ์ฅ๋ ๋ฃฐ์ ๊ตฝ์ง ์๊ณ ๋ ๋ ์์ ์ ์กฐ๋ฆฝํ๋ฏ๋ก **์์ ์ ๋ฑ๋กํ ๋ฃฐ๋ ์๋์ผ๋ก ์ ์ฉ**๋ฐ๊ณ , promote ํ๋ฆฌ๋ทฐ์ ์ค์ ํ์ผ์ด ์ด๊ธ๋ ์ ์์ต๋๋ค.
|
|
236
|
+
|
|
223
237
|
### v3.8.2 (2026-08-01)
|
|
224
238
|
- **์์ถ ๋ค์๋ ๊ฒฝ๊ณ ํฐ์ด๊ฐ ์ ๋ด๋ ค๊ฐ๋ ๋ฌธ์ ์์ ** โ ์ปจํ
์คํธ ํฐ์ด๋ฅผ ์ต๊ณ ์น๋ก๋ง ๊ธฐ์ตํด์, 80%์์ ์๋ ์์ถ์ด ๋์ ์ปจํ
์คํธ๊ฐ ๋ค์ ๋น์ด๋ ํฐ์ด๊ฐ 1๋ก ๋จ์์ต๋๋ค. ๊ทธ ์ธ์
์ ์ฐฝ์ ๋ค์ ๊ฝ ์ฑ์๋ ์๋ฌด ์ ํธ๋ฅผ ๋ชป ๋ฐ์์ต๋๋ค. ์ด์ ์ธก์ ์น๊ฐ ๋ด๋ ค๊ฐ๋ฉด ํฐ์ด๋ ๊ฐ์ด ๋ด๋ ค๊ฐ๊ณ , ๋ค์ ์ฐจ์ค๋ฅด๋ฉด ์ ์์ ์ผ๋ก ๊ฒฝ๊ณ ํฉ๋๋ค.
|
|
225
239
|
- **๋ ๊ฐ์ง ์ฐฝ ํ๊ธฐ ํผ์ ์ ๋ฆฌ** โ `autoCompactWindow`๋ฅผ 40๋ง์ผ๋ก ์ก์ผ๋ฉด ๋ธ๋ฆฌํ์ 40๋ง ๊ธฐ์ค(80%)์ธ๋ฐ Claude Code ํ๋ฉด์ 1M ๊ธฐ์ค(33%)์ด๋ผ, ๊ฐ์ ์ธ์
์ ๋ ์ซ์๊ฐ ์๋ก ์ ๋ง์ ๋ณด์์ต๋๋ค. ์ด์ `์๋ ์์ถ ์ฐฝ(400k)์ 80%(โฆ ํ๋ฉด์ 1M ์ฐฝ ๊ธฐ์ค์ผ๋ก๋ 33%)`์ฒ๋ผ ๋ ๋ค ์ ์ต๋๋ค.
|
package/package.json
CHANGED
package/src/brief.js
CHANGED
|
@@ -246,7 +246,7 @@ export async function runBrief({ sessionId, transcriptPath, now = Date.now() })
|
|
|
246
246
|
briefed.add(sig);
|
|
247
247
|
s.briefed = [...briefed];
|
|
248
248
|
const now = st.window ? `ํ์ฌ ${fmtK(st.window)}` : 'ํ์ฌ ๋ฏธ์ค์ ';
|
|
249
|
-
items.push(`1M ์ปจํ
์คํธ ๋ชจ๋ธ(${st.model})์ธ๋ฐ autoCompactWindow๊ฐ ${now}์
๋๋ค โ ์๋ ์์ถ์ด 80๋ง ํ ํฐ ๊ทผ์ฒ์์์ผ ๊ฑธ๋ ค ๊ทธ์ ๊น์ง ๋ชจ๋ ์์ฒญ์ด ์ ์ฒด ์ปจํ
์คํธ๋ฅผ ์ฌ๊ณผ๊ธํฉ๋๋ค.
|
|
249
|
+
items.push(`1M ์ปจํ
์คํธ ๋ชจ๋ธ(${st.model})์ธ๋ฐ autoCompactWindow๊ฐ ${now}์
๋๋ค โ ์๋ ์์ถ์ด 80๋ง ํ ํฐ ๊ทผ์ฒ์์์ผ ๊ฑธ๋ ค ๊ทธ์ ๊น์ง ๋ชจ๋ ์์ฒญ์ด ์ ์ฒด ์ปจํ
์คํธ๋ฅผ ์ฌ๊ณผ๊ธํฉ๋๋ค. 1M์ ๋๋ฌด ํฌ๋ ${fmtK(st.recommendedMin)}~${fmtK(st.recommendedMax)} ๋ฒ์๋ฅผ ๊ถ์ฅํฉ๋๋ค(๊ทธ ๋ฒ์ ์์ด๋ฉด ๊ฒฝ๊ณ ํ์ง ์์ต๋๋ค). 1M ์ฐฝ ์์ฒด๋ ๊ทธ๋๋ก ๋๊ณ ์์ถ ์์ ๋ง ์๋น๊น๋๋ค. ๋ฑ๋ก: claude-token-saver compact-window set --global|--project [--value ${fmtK(st.recommendedMax)}] (๊ธฐ๋ณธ ${fmtK(st.recommended)}, ์ ์ฉ ๋ฒ์๋ ์ฌ์ฉ์์๊ฒ ํ์ธ) / ๋๊ธฐ: compact-window off`);
|
|
250
250
|
}
|
|
251
251
|
}
|
|
252
252
|
} catch { /* settings unreadable โ other briefings above still apply */ }
|
|
@@ -1,9 +1,9 @@
|
|
|
1
1
|
/**
|
|
2
2
|
* Subcommand: compact-window โ audit / pin Claude Code's `autoCompactWindow`.
|
|
3
3
|
* claude-token-saver compact-window # status
|
|
4
|
-
* claude-token-saver compact-window set --global # pin
|
|
5
|
-
* claude-token-saver compact-window set --project # pin
|
|
6
|
-
* claude-token-saver compact-window set --global --value
|
|
4
|
+
* claude-token-saver compact-window set --global # pin 500k in ~/.claude/settings.json
|
|
5
|
+
* claude-token-saver compact-window set --project # pin 500k in <root>/.claude/settings.json
|
|
6
|
+
* claude-token-saver compact-window set --global --value 600k
|
|
7
7
|
* claude-token-saver compact-window off | on # toggle the statusline warning
|
|
8
8
|
*
|
|
9
9
|
* Scope is deliberately explicit for `set`: writing a global settings.json is
|
|
@@ -69,22 +69,25 @@ export async function run({ args, hasFlag }) {
|
|
|
69
69
|
return;
|
|
70
70
|
}
|
|
71
71
|
if (s.ok) {
|
|
72
|
-
console.log(ko
|
|
72
|
+
console.log(ko
|
|
73
|
+
? `\nโ
์์ถ ์ฐฝ์ด ๊ถ์ฅ ๋ฒ์(${fmt(s.recommendedMin)}~${fmt(s.recommendedMax)}) ์ํ ์ดํ์
๋๋ค.`
|
|
74
|
+
: `\nโ
Compaction window is at or below the top of the recommended ${fmt(s.recommendedMin)}โ${fmt(s.recommendedMax)} band.`);
|
|
73
75
|
return;
|
|
74
76
|
}
|
|
75
77
|
console.log(ko
|
|
76
78
|
? `\nโ 1M ์ปจํ
์คํธ์ธ๋ฐ autoCompactWindow๊ฐ ${s.reason === 'unset' ? '์ค์ ๋์ง ์์์ต๋๋ค' : `${fmt(s.window)}๋ก ๋๋ฌด ํฝ๋๋ค`} โ ์๋ ์์ถ์ด 80๋ง ํ ํฐ ๊ทผ์ฒ์์์ผ ๊ฑธ๋ฆฝ๋๋ค.`
|
|
77
79
|
: `\nโ 1M context with autoCompactWindow ${s.reason === 'unset' ? 'unset' : `at ${fmt(s.window)}`} โ compaction only fires near 800k.`);
|
|
78
80
|
console.log(ko
|
|
79
|
-
?
|
|
80
|
-
:
|
|
81
|
-
console.log(
|
|
82
|
-
console.log(
|
|
81
|
+
? ` ๊ทธ ์ ๊น์ง ๋ชจ๋ ์์ฒญ์ด ์ ์ฒด ์ปจํ
์คํธ๋ฅผ ์ฌ๊ณผ๊ธํฉ๋๋ค. 1M์ ๋๋ฌด ํฌ๋ ${fmt(s.recommendedMin)}~${fmt(s.recommendedMax)} ๋ฒ์๋ฅผ ๊ถ์ฅํฉ๋๋ค (๊ธฐ๋ณธ๊ฐ ${fmt(s.recommended)}, --value๋ก ์กฐ์ ). 1M ์ฐฝ ์์ฒด๋ ๊ทธ๋๋ก ๋๊ณ ์์ถ ์์ ๋ง ์๋น๊น๋๋ค.`
|
|
82
|
+
: ` Until then every request re-bills the whole context. 1M is too large โ pick something in ${fmt(s.recommendedMin)}โ${fmt(s.recommendedMax)} (default ${fmt(s.recommended)}, override with --value). The 1M window itself stays.`);
|
|
83
|
+
console.log(`\n claude-token-saver compact-window set --global (~/.claude/settings.json, ${fmt(s.recommended)})`);
|
|
84
|
+
console.log(` claude-token-saver compact-window set --project (<root>/.claude/settings.json, ${fmt(s.recommended)})`);
|
|
85
|
+
console.log(` claude-token-saver compact-window set --global --value ${fmt(s.recommendedMax)}`);
|
|
83
86
|
console.log(ko ? ' (์ ์ฉ ๋ฒ์๋ ์ฌ์ฉ์์๊ฒ ๋จผ์ ํ์ธํ ๊ฒ)' : ' (confirm the scope with the user first)');
|
|
84
87
|
return;
|
|
85
88
|
}
|
|
86
89
|
|
|
87
90
|
console.error(`Unknown compact-window subcommand: ${sub}`);
|
|
88
|
-
console.error('Usage: claude-token-saver compact-window [status|set --global|--project [--value
|
|
91
|
+
console.error('Usage: claude-token-saver compact-window [status|set --global|--project [--value 500k]|off|on]');
|
|
89
92
|
process.exit(1);
|
|
90
93
|
}
|
package/src/commands/harness.js
CHANGED
|
@@ -228,6 +228,9 @@ export async function run({ args, hasFlag }) {
|
|
|
228
228
|
rule: lang === 'ko' ? routeCandidate.rule : (routeCandidate.ruleEn || routeCandidate.rule),
|
|
229
229
|
example: routeCandidate.example,
|
|
230
230
|
count: routeCandidate.count,
|
|
231
|
+
// Calibrated budget snapshot โ ratchet-model.md restates it when it
|
|
232
|
+
// merges a category's T2 and T1 rules into one conditional rule.
|
|
233
|
+
budget: routeCandidate.budget || null,
|
|
231
234
|
promotedAt: new Date().toISOString().slice(0, 10),
|
|
232
235
|
lastSeen: new Date().toISOString().slice(0, 10),
|
|
233
236
|
});
|
|
@@ -40,7 +40,18 @@ export async function run({ args, hasFlag, numArg }) {
|
|
|
40
40
|
? `${r.tier} (${rs.tierLabel(r.tier)}) ยท ${rs.scopeLabel(r.scope)} ยท ๋ฐ๋ณต ${r.count || 0}ํ ยท ์๋ฌ์จ ${Math.round((r.errRate || 0) * 100)}%`
|
|
41
41
|
: `${r.tier} (${rs.tierLabel(r.tier, 'en')}) ยท ${rs.scopeLabel(r.scope, 'en')} ยท seen ร${r.count || 0} ยท err ${Math.round((r.errRate || 0) * 100)}%`;
|
|
42
42
|
console.log(` #${i + 1} ${stat}${health}`);
|
|
43
|
-
|
|
43
|
+
// Measured outcome of the rule actually firing, plus what it saved.
|
|
44
|
+
// A rule with no measured delegations shows "โ", never "$0.00": the
|
|
45
|
+
// two mean opposite things (no data vs. data showing no value).
|
|
46
|
+
const measured = r.delegatedRuns
|
|
47
|
+
? (lang === 'ko'
|
|
48
|
+
? `์ค์ ์์ ${r.delegatedRuns}๊ฑด ยท ์๋ฌ์จ ${Math.round((r.delegatedErrRate || 0) * 100)}% ยท ์ ๊ฐ ~$${(r.savedUsd || 0).toFixed(2)}`
|
|
49
|
+
: `measured ร${r.delegatedRuns} ยท err ${Math.round((r.delegatedErrRate || 0) * 100)}% ยท saved ~$${(r.savedUsd || 0).toFixed(2)}`)
|
|
50
|
+
: (lang === 'ko' ? '์ค์ ์์ ๊ธฐ๋ก โ (์์ง ์์)' : 'measured delegations โ (none yet)');
|
|
51
|
+
console.log(` ${measured}`);
|
|
52
|
+
// Same composer the md file uses, so what is listed here is exactly
|
|
53
|
+
// what the model reads.
|
|
54
|
+
console.log(` ${mr.composeRuleText(r.rule, r, lang)}`);
|
|
44
55
|
});
|
|
45
56
|
console.log(lang === 'ko'
|
|
46
57
|
? '\n์ ๊ฑฐ: claude-token-saver route-scan rules rm <N>'
|
|
@@ -79,12 +90,17 @@ export async function run({ args, hasFlag, numArg }) {
|
|
|
79
90
|
} catch (e) { debug('route-scan:spawn-refresh', e); /* stale cache is still usable below */ }
|
|
80
91
|
}
|
|
81
92
|
const open = rs.openCandidates(cache);
|
|
93
|
+
// Rule text shown to the model must be composed the same way the md file
|
|
94
|
+
// composes it, budget clause included โ otherwise the briefing promises
|
|
95
|
+
// one rule and the file carries another.
|
|
96
|
+
const mrHook = await import('../model-rules.js');
|
|
97
|
+
const composed = (base, c) => mrHook.composeRuleText(base, c, lang);
|
|
82
98
|
// Registered rules whose delegated-category error rate crossed the
|
|
83
99
|
// health threshold since promotion โ the user approved these, so a
|
|
84
100
|
// status change must be briefed, not just written into the md file.
|
|
85
101
|
let reviewRules = [];
|
|
86
102
|
try {
|
|
87
|
-
const mr =
|
|
103
|
+
const mr = mrHook;
|
|
88
104
|
reviewRules = mr.loadModelRules().rules
|
|
89
105
|
.map((r, i) => ({ ...r, n: i + 1 }))
|
|
90
106
|
.filter((r) => r.status === 'review');
|
|
@@ -105,7 +121,7 @@ export async function run({ args, hasFlag, numArg }) {
|
|
|
105
121
|
for (const c of open) {
|
|
106
122
|
const tier = c.tier || 'T2';
|
|
107
123
|
const label = lang === 'ko' ? c.label : (c.labelEn || c.label);
|
|
108
|
-
const rule = lang === 'ko' ? c.rule : (c.ruleEn || c.rule);
|
|
124
|
+
const rule = composed(lang === 'ko' ? c.rule : (c.ruleEn || c.rule), c);
|
|
109
125
|
if (lang === 'ko') {
|
|
110
126
|
lines.push(` ํ๋ณด R${c.id} โ "${label}" ์ ํ, ${c.count}ํ ๋ฐ๋ณต (ํ๋ก์ ํธ: ${c.project})`);
|
|
111
127
|
lines.push(` ํ์ : ${tier} (${rs.tierLabel(tier)}) โ ${c.agent} ์๋ธ์์ด์ ํธ ์์ ๊ถ์ฅ ยท ์ ์ฉ ๋ฒ์ ์ ์: ${rs.scopeLabel(c.suggestedScope)}`);
|
|
@@ -133,9 +149,17 @@ export async function run({ args, hasFlag, numArg }) {
|
|
|
133
149
|
? '[claude-token-saver rule-health] ์ฌ์ฉ์๊ฐ ์น์ธํ ์์ ๋ฃฐ ์ค, ์์ ๋์ ์ ํ์ ์ต๊ทผ ์๋ฌ์จ์ด ๊ธฐ์ค(20%)์ ๋์ด ์ฌ๊ฒํ ๊ฐ ํ์ํ ๋ฃฐ์ด ์์ต๋๋ค โ ์ฌ์ฉ์์๊ฒ ๋ธ๋ฆฌํํ๊ณ ์กฐ๊ฑด ์ขํ๊ธฐ/์ ๊ฑฐ๋ฅผ ์์ํ์ธ์:'
|
|
134
150
|
: '[claude-token-saver rule-health] Some user-approved delegation rules now exceed the 20% error-rate threshold for their delegated category โ brief the user and discuss narrowing or removing them:');
|
|
135
151
|
for (const r of reviewRules) {
|
|
152
|
+
// Say WHICH signal tripped: a measured delegation failure rate is a
|
|
153
|
+
// much stronger claim than the shape-based proxy, and the user's
|
|
154
|
+
// decision (narrow vs. remove) depends on knowing which it is.
|
|
155
|
+
const measured = r.healthSource === 'delegated';
|
|
156
|
+
const rate = Math.round(((measured ? r.delegatedErrRate : r.errRate) || 0) * 100);
|
|
157
|
+
const evidence = lang === 'ko'
|
|
158
|
+
? (measured ? `์ค์ ์์ ${r.delegatedRuns}๊ฑด ์ค์ธก ์๋ฌ์จ ${rate}%` : `์ ํ ์๋ฌ์จ ${rate}% (ํํ ๊ธฐ๋ฐ ์ถ์ )`)
|
|
159
|
+
: (measured ? `${rate}% measured across ${r.delegatedRuns} real delegations` : `${rate}% for the category (shape-based proxy)`);
|
|
136
160
|
lines.push(lang === 'ko'
|
|
137
|
-
? ` ๋ฃฐ #${r.n} (${r.tier} ${rs.tierLabel(r.tier)} ยท ${rs.scopeLabel(r.scope)}) โ
|
|
138
|
-
: ` rule #${r.n} (${r.tier} ${rs.tierLabel(r.tier, 'en')} ยท ${rs.scopeLabel(r.scope, 'en')}) โ
|
|
161
|
+
? ` ๋ฃฐ #${r.n} (${r.tier} ${rs.tierLabel(r.tier)} ยท ${rs.scopeLabel(r.scope)}) โ ${evidence}`
|
|
162
|
+
: ` rule #${r.n} (${r.tier} ${rs.tierLabel(r.tier, 'en')} ยท ${rs.scopeLabel(r.scope, 'en')}) โ ${evidence}`);
|
|
139
163
|
lines.push(` "${r.rule}"`);
|
|
140
164
|
}
|
|
141
165
|
lines.push(lang === 'ko'
|
|
@@ -191,7 +215,9 @@ export async function run({ args, hasFlag, numArg }) {
|
|
|
191
215
|
console.log(` verdict: ${tier} (${rs.tierLabel(tier, 'en')}) โ delegate to ${c.agent} ยท suggested scope: ${rs.scopeLabel(c.suggestedScope, 'en')}`);
|
|
192
216
|
}
|
|
193
217
|
console.log(` ${lang === 'ko' ? '์์' : 'example'}: "${c.example}"`);
|
|
194
|
-
|
|
218
|
+
const mrList = await import('../model-rules.js');
|
|
219
|
+
const base = lang === 'ko' ? c.rule : (c.ruleEn || c.rule);
|
|
220
|
+
console.log(` ${lang === 'ko' ? '๋ฃฐ' : 'rule'}: ${mrList.composeRuleText(base, c, lang)}`);
|
|
195
221
|
}
|
|
196
222
|
console.log('');
|
|
197
223
|
console.log(lang === 'ko' ? '๋ฑ๋ก / ๋ฌด์:' : 'Promote / dismiss:');
|
package/src/compact-window.js
CHANGED
|
@@ -5,8 +5,13 @@
|
|
|
5
5
|
* `min(settings.autoCompactWindow, model max context)`. On a 1M window the
|
|
6
6
|
* default lets a session grow past 800k before compaction ever fires, so every
|
|
7
7
|
* later request re-bills a context most sessions never needed. Pinning
|
|
8
|
-
* `autoCompactWindow
|
|
9
|
-
* the genuinely large ones while capping the runaway tail.
|
|
8
|
+
* `autoCompactWindow` somewhere in 400kโ700k keeps 2โ3.5x the headroom of a
|
|
9
|
+
* 200k session for the genuinely large ones while capping the runaway tail.
|
|
10
|
+
*
|
|
11
|
+
* The recommendation is a range, not a number: 400k is the floor where the
|
|
12
|
+
* saving is worth the extra compactions, and in practice long sessions often
|
|
13
|
+
* want more room than that. Anything at or below 700k is left alone โ only an
|
|
14
|
+
* unset window, or one above 700k, is a real config defect.
|
|
10
15
|
*
|
|
11
16
|
* 200k sessions are exempt by design: their window is already <= 200k, so the
|
|
12
17
|
* setting changes nothing and a warning would be pure noise.
|
|
@@ -28,7 +33,13 @@ import { debug } from './debug.js';
|
|
|
28
33
|
// the app would honor.
|
|
29
34
|
export const MIN_WINDOW = 100_000;
|
|
30
35
|
export const MAX_WINDOW = 1_000_000;
|
|
31
|
-
|
|
36
|
+
// Recommended band. Warn only outside it on the high side (or when unset) โ
|
|
37
|
+
// a window below RECOMMENDED_MIN is a deliberate, more aggressive choice and
|
|
38
|
+
// costs nothing, so it stays silent.
|
|
39
|
+
export const RECOMMENDED_MIN = 400_000;
|
|
40
|
+
export const RECOMMENDED_MAX = 700_000;
|
|
41
|
+
// What `compact-window set` writes when no --value is given: middle of the band.
|
|
42
|
+
export const RECOMMENDED_WINDOW = 500_000;
|
|
32
43
|
|
|
33
44
|
// A model id whose context is the 1M variant. Claude Code spells it as a
|
|
34
45
|
// suffix on the model id (`claude-opus-5[1m]`); the beta header form
|
|
@@ -116,10 +127,10 @@ export function effectiveWindow(root) {
|
|
|
116
127
|
/**
|
|
117
128
|
* Full audit result. `ok` is true when there is nothing to warn about โ either
|
|
118
129
|
* the session is not on a 1M model (exempt), or the window is already pinned at
|
|
119
|
-
* or below the recommended
|
|
130
|
+
* or below the top of the recommended band (700k).
|
|
120
131
|
*
|
|
121
132
|
* `reason` names why it is not ok: 'unset' (no autoCompactWindow anywhere) or
|
|
122
|
-
* 'too-large' (set, but above
|
|
133
|
+
* 'too-large' (set, but above 700k โ still lets context run away).
|
|
123
134
|
*/
|
|
124
135
|
export function compactWindowStatus({ root = process.cwd() } = {}) {
|
|
125
136
|
const { model, source: modelSource } = resolveModelId(root);
|
|
@@ -133,10 +144,12 @@ export function compactWindowStatus({ root = process.cwd() } = {}) {
|
|
|
133
144
|
windowSource: win.source,
|
|
134
145
|
windowPath: win.path || null,
|
|
135
146
|
recommended: RECOMMENDED_WINDOW,
|
|
147
|
+
recommendedMin: RECOMMENDED_MIN,
|
|
148
|
+
recommendedMax: RECOMMENDED_MAX,
|
|
136
149
|
};
|
|
137
150
|
if (!is1m) return { ...base, ok: true, reason: 'not-1m' };
|
|
138
151
|
if (win.value === null) return { ...base, ok: false, reason: 'unset' };
|
|
139
|
-
if (win.value >
|
|
152
|
+
if (win.value > RECOMMENDED_MAX) return { ...base, ok: false, reason: 'too-large' };
|
|
140
153
|
return { ...base, ok: true, reason: 'configured' };
|
|
141
154
|
}
|
|
142
155
|
|
package/src/cost.js
CHANGED
|
@@ -100,6 +100,45 @@ function detectPricingTier(model) {
|
|
|
100
100
|
return 'claude-sonnet';
|
|
101
101
|
}
|
|
102
102
|
|
|
103
|
+
/**
|
|
104
|
+
* Relative price rank of a model's pricing tier. Delegation only pays off
|
|
105
|
+
* when the target tier is genuinely cheaper than the model that did the work,
|
|
106
|
+
* so route-scan needs an ORDER, not just "is it haiku?" โ a Sonnet session
|
|
107
|
+
* must not produce "delegate to Sonnet" rules (the subagent would rebuild
|
|
108
|
+
* context for zero price difference).
|
|
109
|
+
*/
|
|
110
|
+
const TIER_RANK = {
|
|
111
|
+
'claude-fable-5': 3,
|
|
112
|
+
'claude-opus-legacy': 2,
|
|
113
|
+
'claude-opus-new': 2,
|
|
114
|
+
'claude-sonnet': 1,
|
|
115
|
+
'claude-haiku-4-5': 0,
|
|
116
|
+
'claude-haiku-3-5': 0,
|
|
117
|
+
'claude-haiku-3': 0,
|
|
118
|
+
};
|
|
119
|
+
|
|
120
|
+
export function modelRank(model) {
|
|
121
|
+
const rank = TIER_RANK[detectPricingTier(model)];
|
|
122
|
+
// Unknown ids fall through detectPricingTier to the Sonnet tier; ranking
|
|
123
|
+
// them 1 keeps the conservative reading (cheap enough that a Sonnet-target
|
|
124
|
+
// rule is not worth it, expensive enough that a haiku one still is).
|
|
125
|
+
return rank ?? 1;
|
|
126
|
+
}
|
|
127
|
+
|
|
128
|
+
/** Price rank each delegation tier targets: T2 โ haiku, T1 โ sonnet. */
|
|
129
|
+
export const TIER_TARGET_RANK = { T2: 0, T1: 1 };
|
|
130
|
+
|
|
131
|
+
/**
|
|
132
|
+
* Which delegation tier a subagent run at this price rank represents.
|
|
133
|
+
* null = the run was NOT a downgrade (same tier or higher), so it carries no
|
|
134
|
+
* delegation saving to attribute to a rule.
|
|
135
|
+
*/
|
|
136
|
+
export function tierForRank(rank) {
|
|
137
|
+
if (rank === 0) return 'T2';
|
|
138
|
+
if (rank === 1) return 'T1';
|
|
139
|
+
return null;
|
|
140
|
+
}
|
|
141
|
+
|
|
103
142
|
function tokensToMillions(n) {
|
|
104
143
|
return n / 1_000_000;
|
|
105
144
|
}
|
package/src/model-rules.js
CHANGED
|
@@ -42,6 +42,46 @@ export const HEALTH_ERR_RATE = 0.2;
|
|
|
42
42
|
// noise (1 error in 4 episodes = 25% โ instant flag), so the review flag is
|
|
43
43
|
// withheld until the sample is large enough to mean something.
|
|
44
44
|
export const HEALTH_MIN_SAMPLE = 10;
|
|
45
|
+
// Measured delegations needed before the real outcome replaces the proxy.
|
|
46
|
+
// Lower than HEALTH_MIN_SAMPLE on purpose: a delegated run is DIRECT evidence
|
|
47
|
+
// of the rule firing, so far fewer are needed to mean something than episodes
|
|
48
|
+
// that merely resemble the rule's shape.
|
|
49
|
+
export const HEALTH_MIN_SAMPLE_DELEGATED = 5;
|
|
50
|
+
|
|
51
|
+
// Fallback budget for rules promoted before budgets were recorded โ matches
|
|
52
|
+
// route-scan's pre-calibration defaults. Kept as local constants rather than
|
|
53
|
+
// imported: route-scan imports this module, and a static back-import would
|
|
54
|
+
// close the cycle.
|
|
55
|
+
const DEFAULT_BUDGET = { T2: { calls: 8, out: 1500 }, T1: { calls: null, out: 8000 } };
|
|
56
|
+
|
|
57
|
+
/** The cap half of a budget clause, e.g. "๋๊ตฌ ํธ์ถ 8ํยท์ถ๋ ฅ 1500 ํ ํฐ". */
|
|
58
|
+
export function budgetCapPhrase(rule, lang = userLanguage()) {
|
|
59
|
+
const b = rule.budget || DEFAULT_BUDGET[rule.tier] || DEFAULT_BUDGET.T2;
|
|
60
|
+
if (lang === 'ko') {
|
|
61
|
+
return b.calls ? `๋๊ตฌ ํธ์ถ ${b.calls}ํยท์ถ๋ ฅ ${b.out} ํ ํฐ` : `์ถ๋ ฅ ${b.out} ํ ํฐ`;
|
|
62
|
+
}
|
|
63
|
+
return b.calls ? `${b.calls} tool calls / ${b.out} output tokens` : `${b.out} output tokens`;
|
|
64
|
+
}
|
|
65
|
+
|
|
66
|
+
/**
|
|
67
|
+
* Append the probe-then-commit budget to a rule's text.
|
|
68
|
+
*
|
|
69
|
+
* Kept here โ not baked into the stored rule string โ so it is composed in ONE
|
|
70
|
+
* place: rules promoted before budgets existed gain the clause too, and the
|
|
71
|
+
* promote preview and the written file can never drift apart (both call this).
|
|
72
|
+
*
|
|
73
|
+
* Why the clause matters: a rule is chosen from past statistics but FIRES on
|
|
74
|
+
* the request text alone, and difficulty mostly surfaces after the first tool
|
|
75
|
+
* call. Naming a cap turns an unavoidable mis-fire into a bounded probe
|
|
76
|
+
* instead of a cheap model grinding at work it cannot finish.
|
|
77
|
+
*/
|
|
78
|
+
export function composeRuleText(baseText, rule, lang = userLanguage()) {
|
|
79
|
+
const cap = budgetCapPhrase(rule, lang);
|
|
80
|
+
const clause = lang === 'ko'
|
|
81
|
+
? `์์ ์ํ์ ${cap}์ด๋ฉฐ, ๋๊ธธ ๊ฒ ๊ฐ๊ฑฐ๋ ์๋ฌ๊ฐ ๋๋ฉด ๊ฑฐ๊ธฐ์ ๋ฉ์ถฐ ์งํ๋ถ๋ง ๋ณด๊ณ ํ๊ณ ๋ฉ์ธ ๋ชจ๋ธ์ด ์ด์ด๋ฐ๋๋ค. ์ธ์
๋ชจ๋ธ์ด ์ด๋ฏธ ์์ ๋ชฉํ์ ๊ฐ์ ๊ธ ์ดํ๋ฉด ์์ํ์ง ์๋๋ค`
|
|
82
|
+
: `Cap the run at ${cap}; if it looks likely to exceed that or hits an error, it stops there and reports partial progress while the main model takes over. Skip delegation entirely when the session model is already at or below the target tier`;
|
|
83
|
+
return `${baseText.replace(/\s*$/, '')}. ${clause}`;
|
|
84
|
+
}
|
|
45
85
|
|
|
46
86
|
// See the note in route-scan.js โ paths.js is the only place that resolves
|
|
47
87
|
// this, so an XDG_CONFIG_HOME override moves every state file together.
|
|
@@ -124,10 +164,34 @@ export function renderModelRatchet(rules, lang = userLanguage()) {
|
|
|
124
164
|
'## Rules',
|
|
125
165
|
'',
|
|
126
166
|
];
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
const
|
|
167
|
+
// Which number drove the review flag matters to the reader: a measured
|
|
168
|
+
// delegated error rate is evidence the rule itself is failing, while the
|
|
169
|
+
// proxy only says work of that shape tends to go wrong.
|
|
170
|
+
const pct = (v) => Math.round((v || 0) * 100);
|
|
171
|
+
const healthOf = (r) => {
|
|
172
|
+
if (r.status !== 'review') return '';
|
|
173
|
+
const measured = r.healthSource === 'delegated';
|
|
174
|
+
const rate = pct(measured ? r.delegatedErrRate : r.errRate);
|
|
175
|
+
if (ko) {
|
|
176
|
+
return measured
|
|
177
|
+
? ` โ rule-health: ์ค์ ์์ ${r.delegatedRuns}๊ฑด ์ค ์๋ฌ์จ ${rate}% โ ์กฐ๊ฑด์ ์ขํ๊ฑฐ๋ ์ ๊ฑฐ ๊ฒํ `
|
|
178
|
+
: ` โ rule-health: ์ต๊ทผ ์์ ๋์ ์๋ฌ์จ ${rate}% โ ์กฐ๊ฑด์ ์ขํ๊ฑฐ๋ ์ ๊ฑฐ ๊ฒํ `;
|
|
179
|
+
}
|
|
180
|
+
return measured
|
|
181
|
+
? ` โ rule-health: ${rate}% error rate across ${r.delegatedRuns} measured delegations โ narrow the condition or remove`
|
|
182
|
+
: ` โ rule-health: recent error rate ${rate}% for the delegated category โ narrow the condition or remove`;
|
|
183
|
+
};
|
|
184
|
+
const statsOf = (r) => {
|
|
185
|
+
const base = `ร${r.count || 0}, err ${pct(r.errRate)}%, seen ${r.lastSeen || r.promotedAt}`;
|
|
186
|
+
if (!r.delegatedRuns) return base;
|
|
187
|
+
const saved = r.savedUsd ? `, saved ~$${r.savedUsd.toFixed(2)}` : '';
|
|
188
|
+
return `${base}, delegated ร${r.delegatedRuns} err ${pct(r.delegatedErrRate)}%${saved}`;
|
|
189
|
+
};
|
|
190
|
+
// Merged T2+T1 rules carry two caps, so they state both once rather than
|
|
191
|
+
// repeating the whole stop-condition per tier.
|
|
192
|
+
const mergedBudget = (t2, t1) => ko
|
|
193
|
+
? `์์ ์ํ์ haiku ${budgetCapPhrase(t2, 'ko')}, sonnet ${budgetCapPhrase(t1, 'ko')}์ด๋ฉฐ, ๋๊ธธ ๊ฒ ๊ฐ๊ฑฐ๋ ์๋ฌ๊ฐ ๋๋ฉด ๊ฑฐ๊ธฐ์ ๋ฉ์ถฐ ์งํ๋ถ๋ง ๋ณด๊ณ ํ๊ณ ๋ฉ์ธ ๋ชจ๋ธ์ด ์ด์ด๋ฐ๋๋ค. ์ธ์
๋ชจ๋ธ์ด ์ด๋ฏธ ์์ ๋ชฉํ์ ๊ฐ์ ๊ธ ์ดํ๋ฉด ์์ํ์ง ์๋๋ค`
|
|
194
|
+
: `Cap haiku runs at ${budgetCapPhrase(t2, 'en')} and sonnet runs at ${budgetCapPhrase(t1, 'en')}; a run likely to exceed its cap, or hitting an error, stops there and reports partial progress while the main model takes over. Skip delegation entirely when the session model is already at or below the target tier`;
|
|
131
195
|
|
|
132
196
|
// A category can carry both a T2 (haiku) and a T1 (sonnet) rule. Tier is
|
|
133
197
|
// only known after an episode finishes, so two separate bullets give the
|
|
@@ -147,16 +211,18 @@ export function renderModelRatchet(rules, lang = userLanguage()) {
|
|
|
147
211
|
const rule = ko
|
|
148
212
|
? `"${t2.label}" ์ ํ ์์ฒญ์ ๊ธฐ๋ณธ์ ์ผ๋ก ${agentPhrase(t2.agent)} ์๋ธ์์ด์ ํธ๋ก ์์ํ๋ค(์: "${t2.example}"). ` +
|
|
149
213
|
`์ฌ๋ฌ ๋จ๊ณยท์ฌ๋ฌ ํ์ผ ์์ ์ด ์ฝํ ์ค๊ฐ ๋๋ ์์ฒญ(์: "${t1.example}")์ model: sonnet ์๋ธ์์ด์ ํธ๋ก ์์ํ๋ค. ` +
|
|
150
|
-
`์ค๊ณ ํ๋จยท๋ฐฐํฌยท์คํ ์ด ์ ์ถ ๊ฐ์ ๋น๊ฐ์ญ ์์
์ด ์์ด๊ฑฐ๋ ์์ ์ค ์๋ฌ๊ฐ ๋ฐ๋ณต๋๋ฉด ์์ํ์ง ๋ง๊ณ ๋ฉ์ธ ๋ชจ๋ธ์ด ์ง์
|
|
214
|
+
`์ค๊ณ ํ๋จยท๋ฐฐํฌยท์คํ ์ด ์ ์ถ ๊ฐ์ ๋น๊ฐ์ญ ์์
์ด ์์ด๊ฑฐ๋ ์์ ์ค ์๋ฌ๊ฐ ๋ฐ๋ณต๋๋ฉด ์์ํ์ง ๋ง๊ณ ๋ฉ์ธ ๋ชจ๋ธ์ด ์ง์ ์ฒ๋ฆฌํ๋ค. ` +
|
|
215
|
+
mergedBudget(t2, t1)
|
|
151
216
|
: `Delegate "${t2.labelEn || t2.label}" requests to ${agentPhraseEn(t2.agent)} by default (e.g. "${t2.example}"). ` +
|
|
152
217
|
`Escalate moderate ones that span multiple steps or file edits (e.g. "${t1.example}") to a model: sonnet subagent. ` +
|
|
153
|
-
`Do not delegate at all โ handle it on the main model โ when the request mixes in design judgement or irreversible work (deploy, release, store submission), or when errors repeat during delegation
|
|
218
|
+
`Do not delegate at all โ handle it on the main model โ when the request mixes in design judgement or irreversible work (deploy, release, store submission), or when errors repeat during delegation. ` +
|
|
219
|
+
mergedBudget(t2, t1);
|
|
154
220
|
lines.push(`- ${rule}${healthOf(t2)}${healthOf(t1)} <!-- T2 ${statsOf(t2)} / T1 ${statsOf(t1)} -->`);
|
|
155
221
|
for (const r of group) {
|
|
156
|
-
if (r !== t2 && r !== t1) lines.push(`- ${r.rule}${healthOf(r)} <!-- ${statsOf(r)} -->`);
|
|
222
|
+
if (r !== t2 && r !== t1) lines.push(`- ${composeRuleText(r.rule, r, lang)}${healthOf(r)} <!-- ${statsOf(r)} -->`);
|
|
157
223
|
}
|
|
158
224
|
} else {
|
|
159
|
-
for (const r of group) lines.push(`- ${r.rule}${healthOf(r)} <!-- ${statsOf(r)} -->`);
|
|
225
|
+
for (const r of group) lines.push(`- ${composeRuleText(r.rule, r, lang)}${healthOf(r)} <!-- ${statsOf(r)} -->`);
|
|
160
226
|
}
|
|
161
227
|
}
|
|
162
228
|
return lines.join('\n') + '\n';
|
|
@@ -229,17 +295,51 @@ export function syncAllFiles({ previousPaths = [] } = {}) {
|
|
|
229
295
|
* an expensive model handled directly that still look T1/T2 by shape (tier
|
|
230
296
|
* judged with the error signal zeroed; see route-scan's rule-health pass).
|
|
231
297
|
*/
|
|
232
|
-
export function refreshModelRules(episodeStats, { now } = {}) {
|
|
298
|
+
export function refreshModelRules(episodeStats, delegatedStats = new Map(), { now } = {}) {
|
|
233
299
|
const data = loadModelRules();
|
|
234
300
|
let changed = false;
|
|
301
|
+
const pick = (stats, r) => stats.get(`${r.tier}|${r.category}|${r.project}`)
|
|
302
|
+
|| (r.scope === 'global' ? stats.get(`${r.tier}|${r.category}|*`) : null);
|
|
303
|
+
|
|
235
304
|
for (const r of data.rules) {
|
|
236
|
-
const s = episodeStats
|
|
237
|
-
|
|
238
|
-
if (!s)
|
|
239
|
-
|
|
240
|
-
|
|
305
|
+
const s = pick(episodeStats, r);
|
|
306
|
+
const d = pick(delegatedStats, r);
|
|
307
|
+
if (!s && !d) {
|
|
308
|
+
// A rule whose category didn't appear at all this window keeps its last
|
|
309
|
+
// known recurrence, but its measured-delegation fields must still read
|
|
310
|
+
// as "nothing measured" rather than stay undefined โ the CLI and the
|
|
311
|
+
// rendered md both branch on them.
|
|
312
|
+
r.delegatedRuns = r.delegatedRuns ?? 0;
|
|
313
|
+
r.delegatedErrRate = r.delegatedErrRate ?? 0;
|
|
314
|
+
r.savedUsd = r.savedUsd ?? 0;
|
|
315
|
+
r.healthSource = r.healthSource ?? 'proxy';
|
|
316
|
+
continue;
|
|
317
|
+
}
|
|
318
|
+
|
|
319
|
+
if (s) {
|
|
320
|
+
r.count = s.count;
|
|
321
|
+
r.errRate = s.epCount > 0 ? s.errCount / s.epCount : 0;
|
|
322
|
+
}
|
|
323
|
+
// Window snapshot, not a running total: these describe the current scan
|
|
324
|
+
// window so a rule that stopped firing decays to zero instead of coasting
|
|
325
|
+
// on old credit.
|
|
326
|
+
r.delegatedRuns = d ? d.runs : 0;
|
|
327
|
+
r.delegatedErrRate = d && d.runs > 0 ? d.errRuns / d.runs : 0;
|
|
328
|
+
r.savedUsd = d ? Math.round(d.savedUsd * 100) / 100 : 0;
|
|
241
329
|
r.lastSeen = now || r.lastSeen;
|
|
242
|
-
|
|
330
|
+
|
|
331
|
+
// Measured outcome beats the proxy once there is enough of it. The proxy
|
|
332
|
+
// asks "does work SHAPED like this tend to fail?"; the measurement asks
|
|
333
|
+
// "does this rule fail when it actually fires?" โ only the second can
|
|
334
|
+
// catch a rule that is mis-firing on requests it should never have taken.
|
|
335
|
+
if (r.delegatedRuns >= HEALTH_MIN_SAMPLE_DELEGATED) {
|
|
336
|
+
r.healthSource = 'delegated';
|
|
337
|
+
r.status = r.delegatedErrRate > HEALTH_ERR_RATE ? 'review' : 'active';
|
|
338
|
+
} else {
|
|
339
|
+
r.healthSource = 'proxy';
|
|
340
|
+
r.status = s && r.errRate > HEALTH_ERR_RATE && s.epCount >= HEALTH_MIN_SAMPLE
|
|
341
|
+
? 'review' : 'active';
|
|
342
|
+
}
|
|
243
343
|
changed = true;
|
|
244
344
|
}
|
|
245
345
|
if (changed) {
|
package/src/route-scan.js
CHANGED
|
@@ -22,6 +22,8 @@ import { join } from 'node:path';
|
|
|
22
22
|
import { userDataDir } from './paths.js';
|
|
23
23
|
import { discoverSessionFiles } from './parser.js';
|
|
24
24
|
import { collectSessionRecords } from './session-records.js';
|
|
25
|
+
import { collectSubagentRuns, indexRuns, runsForEpisode } from './subagent-records.js';
|
|
26
|
+
import { estimateCost, modelRank, TIER_TARGET_RANK, tierForRank } from './cost.js';
|
|
25
27
|
import { agentPhrase, agentPhraseEn } from './agents.js';
|
|
26
28
|
|
|
27
29
|
// โโ Tier bands (docs/TIER_CRITERIA.md ยง3) โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
|
@@ -211,7 +213,13 @@ function toEpisodes(records) {
|
|
|
211
213
|
for (const r of records) {
|
|
212
214
|
const text = (r.userText || '').trim();
|
|
213
215
|
if (!cur || cur.text !== text) {
|
|
214
|
-
cur = {
|
|
216
|
+
cur = {
|
|
217
|
+
text, calls: 0, out: 0, mutating: 0, errors: 0, delegated: 0,
|
|
218
|
+
models: new Set(), cwd: '', tools: {},
|
|
219
|
+
// Delegation attribution (subagent-records): exact join key, plus the
|
|
220
|
+
// episode's time span for the fallback when a run has no meta file.
|
|
221
|
+
delegationToolUseIds: [], startedAt: null, endedAt: null,
|
|
222
|
+
};
|
|
215
223
|
episodes.push(cur);
|
|
216
224
|
}
|
|
217
225
|
cur.calls += 1;
|
|
@@ -222,14 +230,61 @@ function toEpisodes(records) {
|
|
|
222
230
|
for (const [name, n] of Object.entries(r.toolCounts || {})) {
|
|
223
231
|
cur.tools[name] = (cur.tools[name] || 0) + n;
|
|
224
232
|
}
|
|
233
|
+
for (const id of r.delegationToolUseIds || []) cur.delegationToolUseIds.push(id);
|
|
234
|
+
if (r.timestamp) {
|
|
235
|
+
const t = Date.parse(r.timestamp);
|
|
236
|
+
if (Number.isFinite(t)) {
|
|
237
|
+
if (cur.startedAt === null || t < cur.startedAt) cur.startedAt = t;
|
|
238
|
+
if (cur.endedAt === null || t > cur.endedAt) cur.endedAt = t;
|
|
239
|
+
}
|
|
240
|
+
}
|
|
225
241
|
cur.models.add(r.model);
|
|
226
242
|
if (!cur.cwd && r.cwd) cur.cwd = r.cwd;
|
|
227
243
|
}
|
|
228
244
|
return episodes;
|
|
229
245
|
}
|
|
230
246
|
|
|
231
|
-
|
|
232
|
-
|
|
247
|
+
/**
|
|
248
|
+
* Price rank of the model that actually handled the episode (the most
|
|
249
|
+
* expensive one, when a session switched models mid-episode).
|
|
250
|
+
*/
|
|
251
|
+
export function episodeRank(ep) {
|
|
252
|
+
let rank = -1;
|
|
253
|
+
for (const m of ep.models) rank = Math.max(rank, modelRank(m));
|
|
254
|
+
return rank;
|
|
255
|
+
}
|
|
256
|
+
|
|
257
|
+
/**
|
|
258
|
+
* Delegation only pays when the target tier is strictly cheaper than what ran
|
|
259
|
+
* the work. Without this a Sonnet session produced "delegate to sonnet" T1
|
|
260
|
+
* rules โ a subagent rebuilding context for zero price difference, which is a
|
|
261
|
+
* net loss. (Replaces the old boolean "is it haiku?" test, which could not
|
|
262
|
+
* tell a Sonnet session from a Fable one.)
|
|
263
|
+
*/
|
|
264
|
+
export function worthDelegating(tier, rank) {
|
|
265
|
+
const target = TIER_TARGET_RANK[tier];
|
|
266
|
+
return target !== undefined && rank > target;
|
|
267
|
+
}
|
|
268
|
+
|
|
269
|
+
/**
|
|
270
|
+
* USD a delegated run saved versus the session model doing the same work.
|
|
271
|
+
* Approximation, deliberately stated as one: it holds token counts constant,
|
|
272
|
+
* which a cheaper model would not reproduce exactly. Directionally right and
|
|
273
|
+
* enough to rank rules by value, so it is rendered as "~$X".
|
|
274
|
+
*/
|
|
275
|
+
export function runSaving(run, mainModel) {
|
|
276
|
+
if (!run.model || !mainModel) return 0;
|
|
277
|
+
const totals = {
|
|
278
|
+
input: run.input,
|
|
279
|
+
cacheCreation: run.cacheCreation,
|
|
280
|
+
cacheRead: run.cacheRead,
|
|
281
|
+
ephemeral5m: run.ephemeral5m,
|
|
282
|
+
ephemeral1h: run.ephemeral1h,
|
|
283
|
+
output: run.out,
|
|
284
|
+
};
|
|
285
|
+
const actual = estimateCost(totals, run.model).actual;
|
|
286
|
+
const counterfactual = estimateCost(totals, mainModel).actual;
|
|
287
|
+
return Math.max(0, counterfactual - actual);
|
|
233
288
|
}
|
|
234
289
|
|
|
235
290
|
const clamp = (v, [lo, hi]) => Math.min(hi, Math.max(lo, v));
|
|
@@ -278,8 +333,9 @@ export async function runRouteScan({ days = 14 } = {}) {
|
|
|
278
333
|
|
|
279
334
|
// Pass 1 โ collect episodes (needed up front: thresholds are calibrated
|
|
280
335
|
// from the full window's output distribution before any tiering).
|
|
281
|
-
const all = []; // { ep, projectDir }
|
|
336
|
+
const all = []; // { ep, projectDir, sessionPath }
|
|
282
337
|
let dataBytes = 0; // window size snapshot โ the rescan gate diffs against it
|
|
338
|
+
const runIndexBySession = new Map(); // sessionPath โ indexRuns() result
|
|
283
339
|
for (const f of files) {
|
|
284
340
|
let records;
|
|
285
341
|
try {
|
|
@@ -290,7 +346,17 @@ export async function runRouteScan({ days = 14 } = {}) {
|
|
|
290
346
|
}
|
|
291
347
|
for (const ep of toEpisodes(records)) {
|
|
292
348
|
if (!ep.text) continue;
|
|
293
|
-
all.push({ ep, projectDir: f.projectDir });
|
|
349
|
+
all.push({ ep, projectDir: f.projectDir, sessionPath: f.path });
|
|
350
|
+
}
|
|
351
|
+
// Subagent transcripts of this session โ the real outcome of every
|
|
352
|
+
// delegation it made. Best-effort: sessions that never delegated have no
|
|
353
|
+
// directory and cost one failed readdir.
|
|
354
|
+
const runs = await collectSubagentRuns(f.path);
|
|
355
|
+
if (runs.length > 0) {
|
|
356
|
+
runIndexBySession.set(f.path, indexRuns(runs));
|
|
357
|
+
// Subagent bytes count toward the window size so the rescan gate stays
|
|
358
|
+
// accurate for delegation-heavy workloads.
|
|
359
|
+
for (const r of runs) dataBytes += r.bytes || 0;
|
|
294
360
|
}
|
|
295
361
|
}
|
|
296
362
|
const totalEpisodes = all.length;
|
|
@@ -311,7 +377,8 @@ export async function runRouteScan({ days = 14 } = {}) {
|
|
|
311
377
|
|
|
312
378
|
for (const { ep, projectDir } of all) {
|
|
313
379
|
if (isSkippable(ep.text)) continue;
|
|
314
|
-
|
|
380
|
+
const epRank = episodeRank(ep);
|
|
381
|
+
if (!worthDelegating('T2', epRank)) continue; // already at the cheapest tier
|
|
315
382
|
const cat = categorize(ep.text, ep.tools);
|
|
316
383
|
if (cat) {
|
|
317
384
|
// rule-health denominator: episodes that LOOK delegable by shape
|
|
@@ -323,13 +390,16 @@ export async function runRouteScan({ days = 14 } = {}) {
|
|
|
323
390
|
// and sharing one category-wide stat would double-count every episode
|
|
324
391
|
// into both rules (identical รN / err% on unrelated tiers).
|
|
325
392
|
const shapeTier = tierOf({ ...ep, errors: 0 }, cat, thresholds);
|
|
326
|
-
|
|
393
|
+
// Same rank gate as the candidate path below โ a denominator counting
|
|
394
|
+
// episodes that can't produce a rule would skew that rule's error rate.
|
|
395
|
+
if ((shapeTier === 'T1' || shapeTier === 'T2') && worthDelegating(shapeTier, epRank)) {
|
|
327
396
|
bumpStats(`${shapeTier}|${cat.id}|${projectDir}`, ep);
|
|
328
397
|
bumpStats(`${shapeTier}|${cat.id}|*`, ep);
|
|
329
398
|
}
|
|
330
399
|
}
|
|
331
400
|
const tier = tierOf(ep, cat, thresholds);
|
|
332
401
|
if (tier !== 'T1' && tier !== 'T2') continue;
|
|
402
|
+
if (!worthDelegating(tier, epRank)) continue;
|
|
333
403
|
tieredEpisodes += 1;
|
|
334
404
|
const key = `${tier}|${cat.id}|${projectDir}`;
|
|
335
405
|
const g = groups.get(key) || {
|
|
@@ -353,6 +423,50 @@ export async function runRouteScan({ days = 14 } = {}) {
|
|
|
353
423
|
groups.set(key, g);
|
|
354
424
|
}
|
|
355
425
|
|
|
426
|
+
// Pass 3 โ measured delegation outcomes (rule-health v2). Episodes that
|
|
427
|
+
// DID delegate are excluded from tiering by design (tierOf returns T0 when
|
|
428
|
+
// ep.delegated > 0: there is nothing left to route). But they are exactly
|
|
429
|
+
// where a promoted rule's real success rate lives, so they get their own
|
|
430
|
+
// pass: join each episode to the subagent transcripts it spawned, and file
|
|
431
|
+
// the outcome under the tier that run's model represents โ a haiku run is a
|
|
432
|
+
// T2 rule firing, a sonnet run a T1 one. Runs that were not a downgrade
|
|
433
|
+
// (same tier or higher) carry no delegation saving and are skipped.
|
|
434
|
+
const delegatedStats = new Map(); // "tier|category|project" โ outcome aggregate
|
|
435
|
+
const bumpDelegated = (key, run, saved) => {
|
|
436
|
+
const d = delegatedStats.get(key) || { runs: 0, errRuns: 0, outTokens: 0, savedUsd: 0 };
|
|
437
|
+
d.runs += 1;
|
|
438
|
+
if (run.toolErrors > 0) d.errRuns += 1;
|
|
439
|
+
d.outTokens += run.out || 0;
|
|
440
|
+
d.savedUsd += saved;
|
|
441
|
+
delegatedStats.set(key, d);
|
|
442
|
+
};
|
|
443
|
+
for (const [sessionPath, index] of runIndexBySession) {
|
|
444
|
+
const used = new Set();
|
|
445
|
+
for (const { ep, projectDir, sessionPath: epSession } of all) {
|
|
446
|
+
if (epSession !== sessionPath) continue;
|
|
447
|
+
if (!ep.delegationToolUseIds.length && index.unjoined.length === 0) continue;
|
|
448
|
+
const runs = runsForEpisode(index, ep, used);
|
|
449
|
+
if (runs.length === 0) continue;
|
|
450
|
+
const cat = categorize(ep.text, ep.tools);
|
|
451
|
+
if (!cat) continue;
|
|
452
|
+
// Counterfactual = the priciest model on the episode, i.e. what would
|
|
453
|
+
// have done the work had it not been handed off.
|
|
454
|
+
let mainModel = null;
|
|
455
|
+
let mainRank = -1;
|
|
456
|
+
for (const m of ep.models) {
|
|
457
|
+
const r = modelRank(m);
|
|
458
|
+
if (r > mainRank) { mainRank = r; mainModel = m; }
|
|
459
|
+
}
|
|
460
|
+
for (const run of runs) {
|
|
461
|
+
const runTier = tierForRank(modelRank(run.model));
|
|
462
|
+
if (!runTier || !worthDelegating(runTier, mainRank)) continue;
|
|
463
|
+
const saved = runSaving(run, mainModel);
|
|
464
|
+
bumpDelegated(`${runTier}|${cat.id}|${projectDir}`, run, saved);
|
|
465
|
+
bumpDelegated(`${runTier}|${cat.id}|*`, run, saved);
|
|
466
|
+
}
|
|
467
|
+
}
|
|
468
|
+
}
|
|
469
|
+
|
|
356
470
|
// Keep prior dismissed/promoted signatures across rescans.
|
|
357
471
|
const prev = readRouteScan();
|
|
358
472
|
const resolved = new Set(prev?.resolved || []);
|
|
@@ -373,6 +487,16 @@ export async function runRouteScan({ days = 14 } = {}) {
|
|
|
373
487
|
// Both languages are computed at scan time and stored on the candidate, so
|
|
374
488
|
// switching `language` later re-renders (and promotes) correctly without
|
|
375
489
|
// waiting for a rescan.
|
|
490
|
+
// The probe-then-commit budget is deliberately NOT baked into this text:
|
|
491
|
+
// model-rules composes it on (composeRuleText), so rules promoted before
|
|
492
|
+
// budgets existed gain the clause too, and the promote preview can never
|
|
493
|
+
// drift from what lands in the file. What the candidate carries is the
|
|
494
|
+
// calibrated budget itself โ the user's own thresholds, snapshotted at scan
|
|
495
|
+
// time rather than hardcoded downstream.
|
|
496
|
+
const budgetOf = (g) => ({
|
|
497
|
+
calls: g.tier === 'T2' ? T2_MAX_CALLS + 2 : null,
|
|
498
|
+
out: g.tier === 'T2' ? thresholds.t2Out : thresholds.t1Out,
|
|
499
|
+
});
|
|
376
500
|
const ruleText = (g) => g.tier === 'T2'
|
|
377
501
|
? `"${g.label}" ์ ํ์ ๋จ์ ์์ฒญ(์: "${g.example}")์ ${agentPhrase(g.agent)} ์๋ธ์์ด์ ํธ๋ก ์์ํ๋ค (์ค๊ณ ํ๋จยท๋ฐฐํฌยท์คํ ์ด ์ ์ถ ๊ฐ์ ๋น๊ฐ์ญ ์์
์ด ์์ด๋ฉด ์์ํ์ง ์์)`
|
|
378
502
|
: `"${g.label}" ์ ํ์ ์ค๊ฐ ๋๋ ์์ฒญ(์: "${g.example}")์ model: sonnet ์๋ธ์์ด์ ํธ๋ก ์์ํ๋ค (์ค๊ณ ํ๋จยท๋น๊ฐ์ญ ์์
ยท๋ฐ๋ณต ์๋ฌ ๋ฐ์ ์ ๋ฉ์ธ ๋ชจ๋ธ์ด ์ด์ด๋ฐ์)`;
|
|
@@ -400,6 +524,10 @@ export async function runRouteScan({ days = 14 } = {}) {
|
|
|
400
524
|
count: g.count,
|
|
401
525
|
models: [...g.models],
|
|
402
526
|
example: g.example,
|
|
527
|
+
// Snapshot of the calibrated budget this rule was written against, so
|
|
528
|
+
// the merged T2+T1 rendering in ratchet-model.md can restate it without
|
|
529
|
+
// re-running a scan.
|
|
530
|
+
budget: budgetOf(g),
|
|
403
531
|
// Concentrated in one project dir โ project rule; the scan groups by
|
|
404
532
|
// project already, so scope suggestion is per-candidate 'project' unless
|
|
405
533
|
// the same category recurs across 2+ projects (then 'global').
|
|
@@ -441,7 +569,7 @@ export async function runRouteScan({ days = 14 } = {}) {
|
|
|
441
569
|
// rates, and rule-health flags โ and rewrites their managed blocks.
|
|
442
570
|
try {
|
|
443
571
|
const { refreshModelRules } = await import('./model-rules.js');
|
|
444
|
-
refreshModelRules(episodeStats, { now: cache.scannedAt });
|
|
572
|
+
refreshModelRules(episodeStats, delegatedStats, { now: cache.scannedAt });
|
|
445
573
|
} catch { /* registry unwritable โ scan result still valid */ }
|
|
446
574
|
|
|
447
575
|
return cache;
|
package/src/session-records.js
CHANGED
|
@@ -112,11 +112,18 @@ export async function collectSessionRecords(filePath, { includeContent = true }
|
|
|
112
112
|
// Per-tool call histogram โ what the call actually DID. route-scan's
|
|
113
113
|
// categorizer trusts this over the prompt's wording (behavior-first).
|
|
114
114
|
const toolCounts = { ...(prev?.toolCounts || {}) };
|
|
115
|
+
// tool_use ids of Task/Agent calls: the join key to the subagent
|
|
116
|
+
// transcripts under <session>/subagents/*.meta.json, which is how the
|
|
117
|
+
// outcome of a delegation is measured (rule-health v2).
|
|
118
|
+
const delegationToolUseIds = [...(prev?.delegationToolUseIds || [])];
|
|
115
119
|
if (Array.isArray(msg.content)) {
|
|
116
120
|
for (const b of msg.content) {
|
|
117
121
|
if (!b || b.type !== 'tool_use') continue;
|
|
118
122
|
if (MUTATING_TOOLS.has(b.name)) mutatingToolCalls += 1;
|
|
119
|
-
if (DELEGATION_TOOLS.has(b.name))
|
|
123
|
+
if (DELEGATION_TOOLS.has(b.name)) {
|
|
124
|
+
delegationCalls += 1;
|
|
125
|
+
if (typeof b.id === 'string') delegationToolUseIds.push(b.id);
|
|
126
|
+
}
|
|
120
127
|
if (typeof b.name === 'string') toolCounts[b.name] = (toolCounts[b.name] || 0) + 1;
|
|
121
128
|
}
|
|
122
129
|
}
|
|
@@ -129,6 +136,14 @@ export async function collectSessionRecords(filePath, { includeContent = true }
|
|
|
129
136
|
(usage.cache_creation_input_tokens || 0) +
|
|
130
137
|
(usage.cache_read_input_tokens || 0),
|
|
131
138
|
completion_tokens: usage.output_tokens || 0,
|
|
139
|
+
// Per-bucket split, kept alongside the collapsed prompt_tokens: pricing
|
|
140
|
+
// differs per bucket, so costing a delegated run against what the
|
|
141
|
+
// session model would have charged needs them separated.
|
|
142
|
+
input_tokens: usage.input_tokens || 0,
|
|
143
|
+
cache_creation_tokens: usage.cache_creation_input_tokens || 0,
|
|
144
|
+
cache_read_tokens: usage.cache_read_input_tokens || 0,
|
|
145
|
+
ephemeral5m: usage.cache_creation?.ephemeral_5m_input_tokens || 0,
|
|
146
|
+
ephemeral1h: usage.cache_creation?.ephemeral_1h_input_tokens || 0,
|
|
132
147
|
depth,
|
|
133
148
|
userText: includeContent ? lastUserText : '',
|
|
134
149
|
assistantText: includeContent ? contentText(msg.content) : '',
|
|
@@ -136,6 +151,7 @@ export async function collectSessionRecords(filePath, { includeContent = true }
|
|
|
136
151
|
// Entries of the same request accumulate tool blocks and errors.
|
|
137
152
|
mutatingToolCalls: (prev?.mutatingToolCalls || 0) + mutatingToolCalls,
|
|
138
153
|
delegationCalls: (prev?.delegationCalls || 0) + delegationCalls,
|
|
154
|
+
delegationToolUseIds,
|
|
139
155
|
toolErrors: prev?.toolErrors || 0,
|
|
140
156
|
toolCounts,
|
|
141
157
|
};
|
|
@@ -0,0 +1,185 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* subagent-records โ read the transcripts of subagent runs a session spawned.
|
|
3
|
+
*
|
|
4
|
+
* Why this exists: rule-health used to be a PROXY. It measured the error rate
|
|
5
|
+
* of episodes the expensive model handled DIRECTLY that merely *looked*
|
|
6
|
+
* delegable by shape โ never the outcome of an actual delegation. So a rule
|
|
7
|
+
* could be quietly failing every time it fired and the signal would not move.
|
|
8
|
+
* Claude Code writes each subagent run to its own transcript, which makes the
|
|
9
|
+
* real outcome measurable:
|
|
10
|
+
*
|
|
11
|
+
* ~/.claude/projects/<munged>/<sessionId>/subagents/agent-<id>.jsonl
|
|
12
|
+
* ~/.claude/projects/<munged>/<sessionId>/subagents/agent-<id>.meta.json
|
|
13
|
+
*
|
|
14
|
+
* The .jsonl is byte-identical in shape to a main transcript (`isSidechain:
|
|
15
|
+
* true`, assistant entries carrying `message.model` + `message.usage`,
|
|
16
|
+
* tool_result blocks carrying `is_error`), so collectSessionRecords parses it
|
|
17
|
+
* unchanged โ including the rejection / self-corrected error filters, which
|
|
18
|
+
* must apply here for the same reason they apply to main sessions.
|
|
19
|
+
*
|
|
20
|
+
* The .meta.json carries `{ agentType, description, toolUseId, spawnDepth }`.
|
|
21
|
+
* `toolUseId` is the join key back to the Task/Agent tool_use block in the
|
|
22
|
+
* parent transcript, which is how a run is attributed to the episode (and
|
|
23
|
+
* therefore the category) that caused it.
|
|
24
|
+
*
|
|
25
|
+
* Everything is best-effort: this layout is a Claude Code internal, so a
|
|
26
|
+
* missing directory, an absent meta file, or an unparseable line degrades to
|
|
27
|
+
* less data, never to a throw. route-scan keeps its proxy signal as fallback.
|
|
28
|
+
*/
|
|
29
|
+
|
|
30
|
+
import { readdir, readFile, stat } from 'node:fs/promises';
|
|
31
|
+
import { join, dirname, basename } from 'node:path';
|
|
32
|
+
import { collectSessionRecords, normalizeModelId } from './session-records.js';
|
|
33
|
+
|
|
34
|
+
/** Directory holding a session's subagent transcripts (may not exist). */
|
|
35
|
+
export function subagentDirFor(sessionPath) {
|
|
36
|
+
return join(dirname(sessionPath), basename(sessionPath, '.jsonl'), 'subagents');
|
|
37
|
+
}
|
|
38
|
+
|
|
39
|
+
async function readMeta(metaPath) {
|
|
40
|
+
try {
|
|
41
|
+
return JSON.parse(await readFile(metaPath, 'utf8'));
|
|
42
|
+
} catch {
|
|
43
|
+
return null;
|
|
44
|
+
}
|
|
45
|
+
}
|
|
46
|
+
|
|
47
|
+
/**
|
|
48
|
+
* A run's model when several appear in one transcript (a harness retry can
|
|
49
|
+
* switch tiers mid-run): the one that produced the most output, since that is
|
|
50
|
+
* what dominates both the bill and the counterfactual.
|
|
51
|
+
*/
|
|
52
|
+
function primaryModel(byModelOut) {
|
|
53
|
+
let best = null;
|
|
54
|
+
let bestOut = -1;
|
|
55
|
+
for (const [model, out] of byModelOut) {
|
|
56
|
+
if (out > bestOut) { best = model; bestOut = out; }
|
|
57
|
+
}
|
|
58
|
+
return best;
|
|
59
|
+
}
|
|
60
|
+
|
|
61
|
+
/**
|
|
62
|
+
* Aggregate one subagent transcript into a single run record.
|
|
63
|
+
* Returns null when the file carries no billable API call.
|
|
64
|
+
*/
|
|
65
|
+
export async function collectSubagentRun(jsonlPath) {
|
|
66
|
+
let records;
|
|
67
|
+
try {
|
|
68
|
+
records = await collectSessionRecords(jsonlPath, { includeContent: false });
|
|
69
|
+
} catch {
|
|
70
|
+
return null;
|
|
71
|
+
}
|
|
72
|
+
if (records.length === 0) return null;
|
|
73
|
+
|
|
74
|
+
const run = {
|
|
75
|
+
path: jsonlPath,
|
|
76
|
+
agentId: basename(jsonlPath, '.jsonl').replace(/^agent-/, ''),
|
|
77
|
+
agentType: null,
|
|
78
|
+
toolUseId: null,
|
|
79
|
+
spawnDepth: null,
|
|
80
|
+
model: null,
|
|
81
|
+
calls: records.length,
|
|
82
|
+
out: 0,
|
|
83
|
+
input: 0,
|
|
84
|
+
cacheCreation: 0,
|
|
85
|
+
cacheRead: 0,
|
|
86
|
+
ephemeral5m: 0,
|
|
87
|
+
ephemeral1h: 0,
|
|
88
|
+
toolErrors: 0,
|
|
89
|
+
startedAt: null,
|
|
90
|
+
endedAt: null,
|
|
91
|
+
bytes: 0,
|
|
92
|
+
};
|
|
93
|
+
const byModelOut = new Map();
|
|
94
|
+
for (const r of records) {
|
|
95
|
+
run.out += r.completion_tokens || 0;
|
|
96
|
+
run.input += r.input_tokens || 0;
|
|
97
|
+
run.cacheCreation += r.cache_creation_tokens || 0;
|
|
98
|
+
run.cacheRead += r.cache_read_tokens || 0;
|
|
99
|
+
run.ephemeral5m += r.ephemeral5m || 0;
|
|
100
|
+
run.ephemeral1h += r.ephemeral1h || 0;
|
|
101
|
+
run.toolErrors += r.toolErrors || 0;
|
|
102
|
+
const model = normalizeModelId(r.model);
|
|
103
|
+
byModelOut.set(model, (byModelOut.get(model) || 0) + (r.completion_tokens || 0));
|
|
104
|
+
if (r.timestamp) {
|
|
105
|
+
const t = Date.parse(r.timestamp);
|
|
106
|
+
if (Number.isFinite(t)) {
|
|
107
|
+
if (run.startedAt === null || t < run.startedAt) run.startedAt = t;
|
|
108
|
+
if (run.endedAt === null || t > run.endedAt) run.endedAt = t;
|
|
109
|
+
}
|
|
110
|
+
}
|
|
111
|
+
}
|
|
112
|
+
run.model = primaryModel(byModelOut);
|
|
113
|
+
|
|
114
|
+
const meta = await readMeta(jsonlPath.replace(/\.jsonl$/, '.meta.json'));
|
|
115
|
+
if (meta) {
|
|
116
|
+
run.agentType = meta.agentType ?? null;
|
|
117
|
+
run.toolUseId = meta.toolUseId ?? null;
|
|
118
|
+
run.spawnDepth = meta.spawnDepth ?? null;
|
|
119
|
+
}
|
|
120
|
+
try {
|
|
121
|
+
run.bytes = (await stat(jsonlPath)).size;
|
|
122
|
+
} catch { /* size only feeds the rescan gate โ 0 is a safe under-estimate */ }
|
|
123
|
+
|
|
124
|
+
return run;
|
|
125
|
+
}
|
|
126
|
+
|
|
127
|
+
/**
|
|
128
|
+
* All subagent runs a session spawned. Empty array when the session never
|
|
129
|
+
* delegated (the common case) or the directory is unreadable.
|
|
130
|
+
*/
|
|
131
|
+
export async function collectSubagentRuns(sessionPath) {
|
|
132
|
+
const dir = subagentDirFor(sessionPath);
|
|
133
|
+
let entries;
|
|
134
|
+
try {
|
|
135
|
+
entries = await readdir(dir);
|
|
136
|
+
} catch {
|
|
137
|
+
return [];
|
|
138
|
+
}
|
|
139
|
+
const runs = [];
|
|
140
|
+
for (const e of entries) {
|
|
141
|
+
if (!e.endsWith('.jsonl')) continue;
|
|
142
|
+
const run = await collectSubagentRun(join(dir, e));
|
|
143
|
+
if (run) runs.push(run);
|
|
144
|
+
}
|
|
145
|
+
return runs;
|
|
146
|
+
}
|
|
147
|
+
|
|
148
|
+
/**
|
|
149
|
+
* Index a session's runs for attribution: exact join by tool_use id first,
|
|
150
|
+
* with the un-joinable ones kept aside for the timestamp fallback (a run
|
|
151
|
+
* whose .meta.json is missing or predates toolUseId still happened, and
|
|
152
|
+
* dropping it would silently under-count a rule's real error rate).
|
|
153
|
+
*/
|
|
154
|
+
export function indexRuns(runs) {
|
|
155
|
+
const byToolUse = new Map();
|
|
156
|
+
const unjoined = [];
|
|
157
|
+
for (const r of runs) {
|
|
158
|
+
if (r.toolUseId) byToolUse.set(r.toolUseId, r);
|
|
159
|
+
else unjoined.push(r);
|
|
160
|
+
}
|
|
161
|
+
return { byToolUse, unjoined, all: runs };
|
|
162
|
+
}
|
|
163
|
+
|
|
164
|
+
/**
|
|
165
|
+
* Runs attributable to one episode. Exact tool_use ids win; anything left
|
|
166
|
+
* unjoined is matched by start time falling inside the episode's span. `used`
|
|
167
|
+
* is a shared Set of run paths across the session so no run is counted twice
|
|
168
|
+
* (an episode's span can overlap a neighbouring episode's runs).
|
|
169
|
+
*/
|
|
170
|
+
export function runsForEpisode(index, ep, used) {
|
|
171
|
+
const out = [];
|
|
172
|
+
for (const id of ep.delegationToolUseIds || []) {
|
|
173
|
+
const r = index.byToolUse.get(id);
|
|
174
|
+
if (r && !used.has(r.path)) { used.add(r.path); out.push(r); }
|
|
175
|
+
}
|
|
176
|
+
if (ep.startedAt === null || ep.endedAt === null) return out;
|
|
177
|
+
for (const r of index.unjoined) {
|
|
178
|
+
if (used.has(r.path) || r.startedAt === null) continue;
|
|
179
|
+
if (r.startedAt >= ep.startedAt && r.startedAt <= ep.endedAt) {
|
|
180
|
+
used.add(r.path);
|
|
181
|
+
out.push(r);
|
|
182
|
+
}
|
|
183
|
+
}
|
|
184
|
+
return out;
|
|
185
|
+
}
|