@alexeiled/claude-router 1.14.0 → 1.16.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/plugin.json +1 -1
- package/CHANGELOG.md +61 -0
- package/README.md +4 -1
- package/docs/architecture.md +17 -3
- package/docs/configuration.md +6 -2
- package/docs/evaluation.md +15 -11
- package/docs/native-router.md +2 -0
- package/docs/router-band.svg +30 -8
- package/docs/router-pane-now.svg +133 -131
- package/docs/router-pane-policy.svg +47 -43
- package/docs/routing-modes.md +143 -0
- package/docs/user-guide.md +17 -2
- package/hooks/native-router.mjs +428 -36
- package/lib/activity-stats.mjs +33 -11
- package/lib/activity.mjs +12 -6
- package/lib/band.mjs +8 -1
- package/lib/config.mjs +72 -17
- package/lib/cost.mjs +3 -0
- package/lib/decisions.mjs +252 -0
- package/lib/display.mjs +46 -5
- package/lib/facts.mjs +125 -4
- package/lib/modes.mjs +98 -0
- package/lib/panel.mjs +41 -3
- package/lib/policy.mjs +7 -4
- package/lib/route.mjs +84 -26
- package/lib/savings.mjs +17 -0
- package/lib/usage.mjs +31 -19
- package/package.json +2 -1
- package/types/index.d.ts +15 -2
|
@@ -0,0 +1,143 @@
|
|
|
1
|
+
# Routing modes
|
|
2
|
+
|
|
3
|
+
One setting trades cost against answer quality: `cost`, `balanced` (the default) or `quality`. The classifier and
|
|
4
|
+
your routes stay as they are. The mode moves the route each tier runs by one fixed step.
|
|
5
|
+
|
|
6
|
+
| Mode | What changes |
|
|
7
|
+
| --- | --- |
|
|
8
|
+
| `cost` | A cheaper route where the turn allows it: the tier below when that tier runs a model with a lower input price, else one effort level less on the same model. Never below `low`. |
|
|
9
|
+
| `balanced` | Nothing: every tier runs its route as configured. |
|
|
10
|
+
| `quality` | One effort level more on the same model, else the route of the tier above. |
|
|
11
|
+
|
|
12
|
+
In every mode:
|
|
13
|
+
|
|
14
|
+
- `micro` turns keep the `micro` route. Only the classifier makes a turn `micro`; `cost` never does.
|
|
15
|
+
- A pin (`/router pin <tier>`) runs that tier's configured route. With routing off (`/router off`, `/model`),
|
|
16
|
+
Claude's model runs, as before.
|
|
17
|
+
- A route never loses its effort: an effort level moves only within what the model lists in `models.<alias>.efforts`,
|
|
18
|
+
and a route with a `null` effort (the session's) or a model without efforts has no effort step.
|
|
19
|
+
|
|
20
|
+
## Choose a mode
|
|
21
|
+
|
|
22
|
+
The mode comes from the first of these that sets one:
|
|
23
|
+
|
|
24
|
+
1. **This session:** `/router mode cost`, `/router mode balanced` or `/router mode quality`. `/router mode default`
|
|
25
|
+
clears it. A resume of the session keeps it; a new session does not.
|
|
26
|
+
2. **The project:** `.claude/router.json` in the directory Claude Code runs in. It may hold only the mode, for
|
|
27
|
+
example `{ "mode": "cost" }`. A file with any other key, a mode that is not one of the three, or JSON that does not
|
|
28
|
+
parse is ignored, and the pane and `/router` say why. Add it to `.gitignore` to keep the choice to yourself.
|
|
29
|
+
3. **The profile:** `"mode"` in `router.json`, or **Mode** on the pane's Policy tab, which saves at once with Undo.
|
|
30
|
+
4. `balanced`.
|
|
31
|
+
|
|
32
|
+
The band, the Now tab and `/router` show the mode and its source, for example `cost (session)`. The prompt footer
|
|
33
|
+
names a mode other than balanced.
|
|
34
|
+
|
|
35
|
+
## When a change applies
|
|
36
|
+
|
|
37
|
+
A change applies from the next turn. It changes the route the router prefers; the route in use moves through the
|
|
38
|
+
same checks as any other switch. On the same task a cheaper route waits until `minDwellRequests` requests have run on
|
|
39
|
+
the route in use and a checkpoint since: a passed check (tests, build, lint), or the success the route's activity can
|
|
40
|
+
show ([Changes during a task](user-guide.md#changes-during-a-task)). A move must also pay back its cache write. Until
|
|
41
|
+
then the band says why the route stays. Pin a tier to run its configured route now.
|
|
42
|
+
|
|
43
|
+
## Policy per mode
|
|
44
|
+
|
|
45
|
+
Each mode except `balanced` replaces some `policy` fields while it is active. The built-in values started as
|
|
46
|
+
codex-router's and are **not tuned** on outcomes. Since 1.16, `cost` upgrades on balanced's bar (2 votes from 0.75):
|
|
47
|
+
with 3 votes from 0.85, the cache write of a cold stronger model raised the bar to about 0.92 from 60k tokens and 0.97
|
|
48
|
+
at 250k, so a cost turn on Haiku kept a task the classifier scored hard at 90%. `cost` still moves down sooner than
|
|
49
|
+
`balanced` (one vote, 80% support, a 4-request dwell):
|
|
50
|
+
|
|
51
|
+
```json
|
|
52
|
+
{
|
|
53
|
+
"modes": {
|
|
54
|
+
"cost": { "policy": { "upgradeVotes": 2, "upgradeBase": 0.75, "jumpConfidence": 0.98, "downgradeVotes": 1,
|
|
55
|
+
"downgradeMass": 0.8, "minDwellRequests": 4, "escalationHoldTurns": 1 } },
|
|
56
|
+
"quality": { "policy": { "upgradeVotes": 1, "upgradeBase": 0.6, "jumpConfidence": 0.9, "downgradeVotes": 3,
|
|
57
|
+
"downgradeMass": 0.95, "minDwellRequests": 16, "escalationHoldTurns": 4 } }
|
|
58
|
+
}
|
|
59
|
+
}
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
Your `modes` entries merge field by field over these. A mode's field beats the same field in `policy`. Only `policy`
|
|
63
|
+
keys are allowed, and the result must pass the same checks as `policy`.
|
|
64
|
+
|
|
65
|
+
## Every decision
|
|
66
|
+
|
|
67
|
+
The route each mode gives a session's first turn, for each classifier answer, through the router's real decision
|
|
68
|
+
path. Generated from the test data by `npm run docs:modes`; `test/mode-matrix.test.mjs` fails when it is stale.
|
|
69
|
+
|
|
70
|
+
<!-- mode-matrix:start -->
|
|
71
|
+
**Built-in routes**
|
|
72
|
+
|
|
73
|
+
| Classifier | Activities | cost | balanced | quality |
|
|
74
|
+
| --- | --- | --- | --- | --- |
|
|
75
|
+
| micro | code, debug, explore, plan, review, ops, docs, uncertain | Haiku · medium | Haiku · medium | Haiku · medium |
|
|
76
|
+
| low | code, debug, plan, review | Sonnet · medium | Sonnet · high | Sonnet · xhigh |
|
|
77
|
+
| low | explore, ops, docs, uncertain | Haiku · medium | Haiku · high | Haiku · xhigh |
|
|
78
|
+
| medium | code, debug, plan, review | Sonnet · high | Opus · medium | Opus · high |
|
|
79
|
+
| medium | explore, docs, uncertain | Haiku · high | Opus · medium | Opus · high |
|
|
80
|
+
| medium | ops | Haiku · medium | Haiku · high | Haiku · xhigh |
|
|
81
|
+
| high | code, debug, explore, plan, review, docs, uncertain | Opus · high | Opus · xhigh | Opus · max |
|
|
82
|
+
| high | ops | Haiku · high | Opus · xhigh | Opus · max |
|
|
83
|
+
| uncertain | code, debug, plan, review | Sonnet · medium | Sonnet · high | Sonnet · xhigh |
|
|
84
|
+
| uncertain | explore, ops, docs, uncertain | Haiku · medium | Haiku · high | Haiku · xhigh |
|
|
85
|
+
|
|
86
|
+
**`routes.high` = Opus · high**
|
|
87
|
+
|
|
88
|
+
| Classifier | Activities | cost | balanced | quality |
|
|
89
|
+
| --- | --- | --- | --- | --- |
|
|
90
|
+
| micro | code, debug, explore, plan, review, ops, docs, uncertain | Haiku · medium | Haiku · medium | Haiku · medium |
|
|
91
|
+
| low | code, debug, plan, review | Sonnet · medium | Sonnet · high | Sonnet · xhigh |
|
|
92
|
+
| low | explore, ops, docs, uncertain | Haiku · medium | Haiku · high | Haiku · xhigh |
|
|
93
|
+
| medium | code, debug, plan, review | Sonnet · high | Opus · medium | Opus · high |
|
|
94
|
+
| medium | explore, docs, uncertain | Haiku · high | Opus · medium | Opus · high |
|
|
95
|
+
| medium | ops | Haiku · medium | Haiku · high | Haiku · xhigh |
|
|
96
|
+
| high | code, debug, explore, plan, review, docs, uncertain | Opus · medium | Opus · high | Opus · xhigh |
|
|
97
|
+
| high | ops | Haiku · high | Opus · high | Opus · xhigh |
|
|
98
|
+
| uncertain | code, debug, plan, review | Sonnet · medium | Sonnet · high | Sonnet · xhigh |
|
|
99
|
+
| uncertain | explore, ops, docs, uncertain | Haiku · medium | Haiku · high | Haiku · xhigh |
|
|
100
|
+
<!-- mode-matrix:end -->
|
|
101
|
+
|
|
102
|
+
## Known trade-offs
|
|
103
|
+
|
|
104
|
+
The steps are fixed rules, with no per-activity exceptions. Two cells go against this router's own model notes
|
|
105
|
+
(`lib/config.mjs`), and replay data should decide them later:
|
|
106
|
+
|
|
107
|
+
- `cost` at `high` for **ops** runs Haiku 5.5 at high (the `medium` ops route), although Haiku trails Opus on
|
|
108
|
+
multi-step tool work. Repeated tool errors cannot escalate it: no tier is above `high`. For such a task, pin `high`
|
|
109
|
+
or switch to `balanced`.
|
|
110
|
+
- `cost` at `low` for **code, debug, plan and review** runs Sonnet 5.5 at medium, which scored below Haiku 5.5 at high
|
|
111
|
+
on FrontierCode and costs more per token.
|
|
112
|
+
|
|
113
|
+
With the built-in routes, `cost` at `low` runs Haiku 5.5 at medium, the same route as `micro`.
|
|
114
|
+
|
|
115
|
+
## Measured impact
|
|
116
|
+
|
|
117
|
+
One 5-turn coding task (read two files, add a function with tests, run the tests, review, add edge-case tests), the
|
|
118
|
+
same prompts in a new session per mode, Jev classifier, built-in routes. Cost is a list-price estimate of the measured
|
|
119
|
+
tokens ([raw results](../experiments/mod-router/results/mode-live-acceptance.json)).
|
|
120
|
+
|
|
121
|
+
| Mode | Replies by route | Switches | Output tokens | Estimated cost |
|
|
122
|
+
| -------- | ----------------------------------- | -------- | ------------- | -------------- |
|
|
123
|
+
| cost | Haiku · medium 2, Sonnet · medium 6 | 1 | 2,860 | $0.166 |
|
|
124
|
+
| balanced | Opus · medium 10 | 0 | 3,986 | $0.323 |
|
|
125
|
+
| quality | Opus · high 8 | 0 | 5,332 | $0.358 |
|
|
126
|
+
|
|
127
|
+
**These are single runs.** The classifier's answers vary from run to run: its confidence that the first turn was
|
|
128
|
+
`micro` was 92%, 74% and 79% in the three runs, so part of the first-turn difference is classifier noise, not the mode.
|
|
129
|
+
Read the numbers as the direction and rough size of the effect, not as a benchmark.
|
|
130
|
+
|
|
131
|
+
- `quality` ran the same model as `balanced`, at the same price per token. All of its extra cost is output: 34% more
|
|
132
|
+
tokens at the higher effort. A list-price estimate can't show this: priced as one nominal turn each, a fixed set of 20
|
|
133
|
+
prompts comes to $2.916 in both modes ([probe](../experiments/mod-router/results/mode-probe-jev.json)).
|
|
134
|
+
- `cost` spent $0.069 of its $0.166 on one cache write, for the move to Sonnet. A longer session spreads that write
|
|
135
|
+
over more turns.
|
|
136
|
+
- Every task ended with the tests passing; this run does not compare the quality of the answers.
|
|
137
|
+
|
|
138
|
+
## Records
|
|
139
|
+
|
|
140
|
+
The decision at the start of each turn writes one line to Claude Code's debug log (`claude --debug`): `router
|
|
141
|
+
decision` with the mode, its source, the classifier's tier, the tier and reason the router chose, and the route. A move
|
|
142
|
+
later in the turn (a review, the turn's tool calls, a context fit) writes none; the usage cells show it. Usage records keep each reply
|
|
143
|
+
under its mode (`mode|activity|tier|model@effort`).
|
package/docs/user-guide.md
CHANGED
|
@@ -194,12 +194,18 @@ The classifier labels a turn from its prompt, and the work can change after that
|
|
|
194
194
|
|
|
195
195
|
- **On every later step of a turn** it reads the turn's tool calls. A turn labelled `ops` or `explore` that edits a code file moves at once to the tier's `code` route (`docs` when it edits only Markdown or `docs/` files). The tools are the evidence, so this needs no classifier support, and the cache cost does not hold it back; only the cash cap on a credits-billed model can. This move only goes up. The toast and the band show it as `↗ coding`.
|
|
196
196
|
- **Every 4 requests inside a turn** (`policy.reviewEveryRequests`) it asks the classifier again, in the background, with the turn's prompt and the recent dialogue. A step never waits for the answer: it applies at the first step after it arrives. Each review sends the classifier the same kind of text a new turn does. `0` turns reviews off.
|
|
197
|
-
- **A new prompt that continues the task
|
|
197
|
+
- **A new prompt that continues the task** is decided the same way as a review. Claude Code stamps each prompt with where it came from. A task notification, an observer's report or a programmatic follow-up is always progress on the running task: the classifier is asked about your last request, with the notification as the newest dialogue line, so it judges the work still to do rather than the notification's wording. A message from you or another agent, or "go on", counts as the running task when the classifier scores it as a continuation. After a resume the router does not know your last request yet, so a notification is judged as it reads.
|
|
198
198
|
|
|
199
199
|
On the running task:
|
|
200
200
|
|
|
201
201
|
- A stronger tier takes the usual votes: two agreeing answers, or one for a clear two-tier jump. A stronger activity takes one answer whose probability clears the cache check. Repeated tool errors still escalate at once.
|
|
202
|
-
- A cheaper tier or activity also waits for a checkpoint: at least 8 requests since the last model change (`policy.minDwellRequests`), and the
|
|
202
|
+
- A cheaper tier or activity also waits for a checkpoint: at least 8 requests since the last model change (`policy.minDwellRequests`), and a success since then that fits the work the route runs for. A passed test, build, lint or type-check always counts. A check is a test runner, type checker or linter (`pytest`, `tsc`, `eslint`), or a launcher running one (`npm test`, `go test`, `make lint`), alone or after `&&`. A check piped into another command (`npm test | tail`), followed by `;` or `|| true`, or sent to the background does not count, because its exit status can hide a failure or is not known yet. Without a passed check, the route's activity decides what counts:
|
|
203
|
+
- `ops`: the last command that ran something succeeded, with no file edit or delegated agent since.
|
|
204
|
+
- `docs`: the last edit succeeded and touched only Markdown, text or `docs/` files, with no command run beside it.
|
|
205
|
+
- `explore`, `plan`, `review`: a turn finished, nothing was edited, written, run or delegated, and the last tool result was not an error.
|
|
206
|
+
- `code`, `debug` and a route with no activity: only a passed check counts.
|
|
207
|
+
|
|
208
|
+
A failed check blocks every one of these, and so does a check whose result is not the check's own (chained, redirected, piped, after `;` or `||`, or in the background). While no checkpoint is met, the route stays: for a tier the band shows `… down after a checkpoint`; a held activity keeps the running one and shows `= fits`. A cheaper activity also needs 90% probability (`policy.downgradeMass`).
|
|
203
209
|
- A cheaper tier lands where the answer as a whole points: when the classifier says `micro` 63% and `low` 30%, the turn moves to `low`, which 93% of the answer supports, instead of staying until `micro` alone reaches 90%.
|
|
204
210
|
- A pinned turn does not move. A new task, a prompt the classifier does not score as a continuation, is routed as a fresh turn. Ollama does not answer the continuation question, so every prompt is a new task to it; reviews inside a turn still apply.
|
|
205
211
|
|
|
@@ -225,11 +231,20 @@ On the **Classifier** tab, **ACROSS SESSIONS · since the last reset** shows the
|
|
|
225
231
|
| `/router off` | Turn routing off and keep Claude's selected model. |
|
|
226
232
|
| `/router pin <micro\|low\|medium\|high>` | Pin the next turn and its tool continuations. Routing on must already be enabled. |
|
|
227
233
|
| `/router activities <off\|on>` | Set the [activity routing](#route-by-activity) mode. Saved to `router.json` at once; **Undo** reverts it. |
|
|
234
|
+
| `/router mode <cost\|balanced\|quality>` | Set this session's [routing mode](routing-modes.md) from the next turn. A resume keeps it. |
|
|
235
|
+
| `/router mode default` | Clear the session's mode: the project's or the profile's applies again. |
|
|
236
|
+
| `/router decisions [n]` | Print this session's last `n` routing decisions, 20 by default. See [Routing decisions](#routing-decisions). |
|
|
228
237
|
| `/model <name>` | Select a model and turn routing off. Use `/router auto` to turn it back on. |
|
|
229
238
|
| **Turn on** / **Turn off** on the band | The same as `/router auto` and `/router off`. |
|
|
230
239
|
| **on** / **off** on the pane (`o` / `f`) | The same as `/router auto` and `/router off`. |
|
|
231
240
|
| **pin** on the Now tab | Pin that tier for the next turn. Routing on must already be enabled. |
|
|
232
241
|
|
|
242
|
+
A routing mode moves each tier's route one step toward cost or quality; pins and `micro` turns are not moved. The band and the Now tab show the mode and where it comes from, such as `cost (session)`. When the mode kept the cell's model and changed only its effort, as `cost` does for Opus at `high`, the band adds `effort only`: the cost estimates price the same tokens on both routes, so that change shows no difference there, though real output length can differ. A change applies from the next turn, and the route in use moves only through the usual checks, so on a running task a cheaper route can wait a few requests and for a checkpoint. **Mode** on the Policy tab sets the profile's mode. See [Routing modes](routing-modes.md).
|
|
243
|
+
|
|
244
|
+
### Routing decisions
|
|
245
|
+
|
|
246
|
+
Each routed turn adds rows to the session's decision record in `$.store` (`decisions:v1:<session id>`): one when the turn starts, one per classifier review or tool-based move inside it, and one when it ends. A row holds the prompt's origin (`composer`, `task-notification`, …, or `unknown`), the classifier's tier and activity with their shares, whether it counted as the running task, the tier, activity, route and reason chosen, and at the end the turn's tool bucket and how many replies were priced. It holds no prompt, tool or file text. `/router decisions [n]` prints them. Each session keeps its newest 60 rows; records of the 20 most recent other sessions stay, for at most 30 days. **Delete history** deletes them all.
|
|
247
|
+
|
|
233
248
|
A pin does not change the next turn after the pinned turn finishes. A fresh session on a model that a tier routes to, or with activity routing `on` a model an activity override uses (Sonnet 5.5 on the defaults), starts with routing on. A fresh session on any other model starts with routing off. `/clear` starts the new session with routing on. Resuming a saved session restores its saved routing on or off.
|
|
234
249
|
|
|
235
250
|
## Edit routes and policy
|