@alexeiled/claude-router 0.6.0 → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/plugin.json +1 -1
- package/CHANGELOG.md +90 -0
- package/README.md +97 -75
- package/docs/architecture.md +435 -0
- package/docs/configuration.md +151 -153
- package/docs/evaluation.md +83 -0
- package/docs/tier-share.svg +73 -0
- package/docs/user-guide.md +177 -92
- package/lib/config.mjs +105 -14
- package/lib/cost.mjs +53 -8
- package/lib/facts.mjs +37 -6
- package/lib/gateway.mjs +33 -30
- package/lib/jev.mjs +7 -4
- package/lib/policy.mjs +16 -12
- package/lib/rewrite.mjs +93 -3
- package/lib/router.mjs +92 -29
- package/lib/runtime.mjs +72 -7
- package/lib/status.mjs +45 -3
- package/lib/store.mjs +2 -1
- package/package.json +3 -2
- package/scripts/ensure-gateway.mjs +34 -8
- package/scripts/gateway.mjs +12 -2
- package/scripts/status.mjs +16 -3
- package/scripts/statusline.mjs +6 -2
- package/docs/design.md +0 -254
package/docs/user-guide.md
CHANGED
|
@@ -1,117 +1,202 @@
|
|
|
1
1
|
# User guide
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
After the [install](../README.md#install), work in Claude Code as usual. The
|
|
4
|
+
router selects the model and the effort for each turn. This guide shows how
|
|
5
|
+
to read what it does, and how to control it.
|
|
6
|
+
|
|
7
|
+
- [Check the setup](#check-the-setup)
|
|
8
|
+
- [A session, step by step](#a-session-step-by-step)
|
|
9
|
+
- [Read the status line](#read-the-status-line)
|
|
10
|
+
- [Read `/router:status`](#read-routerstatus)
|
|
11
|
+
- [Pin a tier](#pin-a-tier)
|
|
12
|
+
- [Read the decision log](#read-the-decision-log)
|
|
13
|
+
- [Update](#update)
|
|
14
|
+
- [Stop using the router](#stop-using-the-router)
|
|
15
|
+
- [Troubleshooting](#troubleshooting)
|
|
16
|
+
|
|
17
|
+
## Check the setup
|
|
18
|
+
|
|
19
|
+
1. Start Claude Code. `/model` shows `Jev Router (auto)` as the model.
|
|
20
|
+
2. Send a prompt. If setup added the status line, it shows the route, for
|
|
21
|
+
example `jev-router ▸ sonnet-5 · high · same-tier`.
|
|
22
|
+
3. Run `/router:status`. It shows `Jev routing: active.`
|
|
23
|
+
|
|
24
|
+
If a step fails, read [Troubleshooting](#troubleshooting).
|
|
25
|
+
|
|
26
|
+
## A session, step by step
|
|
27
|
+
|
|
28
|
+
An example of four turns. Each box is one prompt with its route and reason.
|
|
29
|
+
|
|
30
|
+
```mermaid
|
|
31
|
+
flowchart LR
|
|
32
|
+
T1["<b>1</b> · design a cache layer<br/>Opus · xhigh<br/><i>jump</i>"]
|
|
33
|
+
T2["<b>2</b> · yes, go on<br/>Opus · xhigh<br/><i>continuation</i>"]
|
|
34
|
+
T3["<b>3</b> · rename foo to bar<br/>Opus · xhigh<br/><i>downgrade-pending</i>"]
|
|
35
|
+
T4["<b>4</b> · fix the typo<br/>Sonnet<br/><i>downgrade</i>"]
|
|
36
|
+
T1 --> T2 --> T3 --> T4
|
|
37
|
+
|
|
38
|
+
classDef opus fill:#0d366b,stroke:#0d366b,color:#ffffff
|
|
39
|
+
classDef sonnet fill:#cde2fb,stroke:#2a78d6,color:#0b0b0b
|
|
40
|
+
class T1,T2,T3 opus
|
|
41
|
+
class T4 sonnet
|
|
42
|
+
```
|
|
4
43
|
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
The
|
|
9
|
-
requests it exits, unless a turn waits for a tool result, for example a
|
|
10
|
-
permission prompt that you have not answered. The next prompt starts it again
|
|
11
|
-
(`gateway.idleShutdownMs` in [configuration](configuration.md)).
|
|
44
|
+
1. A confident vote for hard work goes up two tiers at once.
|
|
45
|
+
2. A prompt that continues the task keeps the route.
|
|
46
|
+
3. One vote for a lower tier is not enough. The route stays.
|
|
47
|
+
4. The second vote in a row moves the route down.
|
|
12
48
|
|
|
13
|
-
|
|
49
|
+
The tool calls in a turn keep the route of the turn, so the cache stays warm.
|
|
50
|
+
The [README](../README.md#how-jev-selects-a-tier) gives all the rules.
|
|
14
51
|
|
|
15
|
-
|
|
16
|
-
TYPESAFE_API_KEY=… node scripts/gateway.mjs &
|
|
17
|
-
ANTHROPIC_BASE_URL=http://127.0.0.1:43170 claude --plugin-dir . --model jev-router
|
|
18
|
-
```
|
|
52
|
+
## Read the status line
|
|
19
53
|
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
- A prompt that continues the task keeps the route. Examples: "continue",
|
|
27
|
-
"yes", "now fix the tests".
|
|
28
|
-
- After two failed repairs of the same error, the route goes up one tier. The
|
|
29
|
-
route stays there for two turns. A repair is an edit between the two errors.
|
|
30
|
-
- An upgrade needs two consecutive votes for a higher tier. A jump of two
|
|
31
|
-
tiers with high confidence happens at once. The required confidence goes up
|
|
32
|
-
with the cost to read the context again on the new model.
|
|
33
|
-
- A downgrade needs two consecutive confident votes.
|
|
34
|
-
- A cold switch to a model that bills usage credits is refused when the cache
|
|
35
|
-
write costs more than `policy.cashCapUsd`. Then the strongest plan tier
|
|
36
|
-
serves the turn. Behind the gateway, Claude Code does not show its consent
|
|
37
|
-
prompt for these credits, so the cap is the only guard. No default model
|
|
38
|
-
bills credits; this applies once you add one in `router.json`.
|
|
39
|
-
- When Jev fails or times out, or when there is no key, the baseline tier
|
|
40
|
-
serves the turn. After three failures in a row, new turns skip Jev for one
|
|
41
|
-
minute and then try it once, so an outage costs one slow turn a minute, not
|
|
42
|
-
a second on every turn. `/router:status` shows the pause.
|
|
43
|
-
- When Claude Code sends the same request again (after a 429, a 529 or a
|
|
44
|
-
dropped connection), the gateway keeps the route of that turn. A retry is
|
|
45
|
-
not a new vote and makes no Jev call.
|
|
46
|
-
- Jev receives the prompt and the last six turns of text. Tool results are not
|
|
47
|
-
sent. No other data leaves the machine, except the usual Anthropic request.
|
|
48
|
-
|
|
49
|
-
A Claude Code turn starts at about 100k tokens of system prompt and tool
|
|
50
|
-
definitions, so the first switch to a model is the expensive one.
|
|
51
|
-
|
|
52
|
-
## Pin a tier by hand
|
|
53
|
-
|
|
54
|
-
Type the tier skill as a command. The turn runs on the model of that skill.
|
|
55
|
-
The gateway sends the real model id unchanged.
|
|
54
|
+
```text
|
|
55
|
+
jev-router ▸ opus-5-5 · xhigh · upgrade
|
|
56
|
+
│ │ └─ reason for the route
|
|
57
|
+
│ └─ effort
|
|
58
|
+
└─ model of the last turn
|
|
59
|
+
```
|
|
56
60
|
|
|
61
|
+
| Line | Meaning |
|
|
62
|
+
| ------------------------------------------------ | --------------------------------------------------- |
|
|
63
|
+
| `jev-router ▸ …` | The route of the last turn in this session. |
|
|
64
|
+
| `jev-router: no turn yet` | This session has no routed turn. |
|
|
65
|
+
| `router: gateway off, the next prompt starts it` | The gateway stopped after idle time, or it crashed. |
|
|
66
|
+
|
|
67
|
+
The line changes after each message.
|
|
68
|
+
|
|
69
|
+
**The route changed:**
|
|
70
|
+
|
|
71
|
+
| Reason | Meaning |
|
|
72
|
+
| ------------- | ---------------------------------------------------------------------------- |
|
|
73
|
+
| `upgrade` | Two votes in a row for a higher tier, with enough confidence. |
|
|
74
|
+
| `jump` | One vote confident enough to go up two tiers. |
|
|
75
|
+
| `downgrade` | Two votes in a row for a lower tier, with enough confidence. |
|
|
76
|
+
| `escalation` | The same error came back after an edit. The route went up one tier. |
|
|
77
|
+
| `context-fit` | The selected model cannot hold the context. A larger window serves the turn. |
|
|
78
|
+
| `cash-gate` | The cold-write guard stopped a switch to a `credits` model. |
|
|
79
|
+
|
|
80
|
+
**The route stayed:**
|
|
81
|
+
|
|
82
|
+
| Reason | Meaning |
|
|
83
|
+
| ------------------- | ------------------------------------------------------------------- |
|
|
84
|
+
| `same-tier` | Jev agreed with the current tier. |
|
|
85
|
+
| `continuation` | The prompt continues the task. |
|
|
86
|
+
| `upgrade-pending` | One vote for a higher tier. The next vote decides. |
|
|
87
|
+
| `downgrade-pending` | One vote for a lower tier. The next vote decides. |
|
|
88
|
+
| `hold` | The route stays up for two turns after an escalation. |
|
|
89
|
+
| `uncertain` | Jev did not select a tier. |
|
|
90
|
+
| `no-advice` | No Jev answer: no key, an error, a pause, or a prompt without text. |
|
|
91
|
+
|
|
92
|
+
`forced` means that the gateway was started with `--force-tier`.
|
|
93
|
+
|
|
94
|
+
## Read `/router:status`
|
|
95
|
+
|
|
96
|
+
`/router:status` shows the gateway, the routes and the last turn. For a
|
|
97
|
+
vote, it also shows the numbers:
|
|
98
|
+
|
|
99
|
+
```text
|
|
100
|
+
Router v0.7.0, gateway: http://127.0.0.1:43170, alias `jev-router`.
|
|
101
|
+
Jev routing: active.
|
|
102
|
+
Default tier: low.
|
|
103
|
+
…
|
|
104
|
+
Last turn: low → claude-sonnet-5 at high, reason upgrade-pending, context 84210 tokens, cache reads 83904.
|
|
105
|
+
Why: Jev asked for a higher tier; the route stays until the votes and the mass are enough.
|
|
106
|
+
Estimate: upgrade mass 0.78 against a bar of 0.84, switching tax $0.66 at list prices, 1 vote(s) in a row, cache: candidate unknown, current warm.
|
|
57
107
|
```
|
|
108
|
+
|
|
109
|
+
- **Upgrade mass** is the Jev confidence in a higher tier. It must reach the
|
|
110
|
+
bar.
|
|
111
|
+
- **The bar** grows with the switching tax: the cost to write the context
|
|
112
|
+
into a cold cache. Here Opus has no warm cache.
|
|
113
|
+
- **Cache** is `warm`, `expired` or `unknown`. `unknown` means no response
|
|
114
|
+
for that cache since the session started or the conversation got shorter.
|
|
115
|
+
- **Dollars** are list prices. On a subscription plan, they show the relative
|
|
116
|
+
cost only.
|
|
117
|
+
|
|
118
|
+
## Pin a tier
|
|
119
|
+
|
|
120
|
+
To run one turn on a fixed tier, type the tier skill before the prompt:
|
|
121
|
+
|
|
122
|
+
```text
|
|
58
123
|
/router:high redesign the auth flow
|
|
59
124
|
/router:micro rename foo to bar in this file
|
|
60
125
|
```
|
|
61
126
|
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
## See the current route
|
|
127
|
+
The status line shows the reason `pinned`. Tool calls in that turn stay on the
|
|
128
|
+
tier; the next prompt goes back to Jev.
|
|
65
129
|
|
|
66
|
-
`/
|
|
67
|
-
|
|
130
|
+
`/model <name>` also works, but it stops the routing for the rest of the
|
|
131
|
+
session.
|
|
68
132
|
|
|
69
|
-
|
|
70
|
-
while the session uses the router:
|
|
133
|
+
## Read the decision log
|
|
71
134
|
|
|
72
|
-
|
|
73
|
-
-
|
|
74
|
-
|
|
75
|
-
idle time, or it crashed. Either way the next prompt starts it.
|
|
135
|
+
The gateway writes one JSON line for each routed request to
|
|
136
|
+
`~/.claude/plugins/data/router-alexei-led-claude-router/decisions.jsonl`. The
|
|
137
|
+
log has no prompt text.
|
|
76
138
|
|
|
77
|
-
|
|
139
|
+
| Field | Content |
|
|
140
|
+
| -------------- | ---------------------------------------------------------------------------------- |
|
|
141
|
+
| `tier` | The tier of the request. |
|
|
142
|
+
| `model`, `effort` | The model id and the effort that the gateway sent. `effort` is `null` for a model without effort. |
|
|
143
|
+
| `reason` | The rule that decided. Tool calls show `tool-continuation`. `error`: routing failed and the default tier served. |
|
|
144
|
+
| `advice` | The Jev probabilities for each tier, and for "continues the task". |
|
|
145
|
+
| `estimate` | The numbers of a vote: confidence, bar, switching tax, cache state. |
|
|
146
|
+
| `shadow` | The cost of the Jev choice against the current route. The policy does not use it. |
|
|
147
|
+
| `observed` | The usage of the response: model, effort, context tokens, cache reads, output. |
|
|
148
|
+
| `failed` | Anthropic answered a routed turn with an error: status, tier, model, effort. |
|
|
149
|
+
| `historyBreak` | A compaction or a rewind. The votes and the cache estimates reset. |
|
|
150
|
+
| `cacheReset` | The context shrank by more than 20%. The cache estimates reset. |
|
|
78
151
|
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
The gateway writes one line for each routed request to `decisions.jsonl` in
|
|
82
|
-
the plugin data directory. The directory is
|
|
83
|
-
`~/.claude/plugins/data/router-<marketplace>/`. For a gateway that you started
|
|
84
|
-
by hand, the directory is `$TMPDIR/router/`.
|
|
152
|
+
Two useful queries:
|
|
85
153
|
|
|
86
154
|
```sh
|
|
87
|
-
|
|
155
|
+
LOG=~/.claude/plugins/data/router-alexei-led-claude-router/decisions.jsonl
|
|
156
|
+
jq -r 'select(.observed) | .observed.model' $LOG | sort | uniq -c # requests by model
|
|
157
|
+
jq -r 'select(.reason) | .reason' $LOG | sort | uniq -c | sort -rn # decisions by reason
|
|
158
|
+
jq -c 'select(.failed) | .failed' $LOG # failed turns
|
|
88
159
|
```
|
|
89
160
|
|
|
90
|
-
`
|
|
91
|
-
lines carry the model that answered, the context tokens and the cache reads
|
|
92
|
-
from the response. Compare the estimated and the observed cache reads to tune
|
|
93
|
-
the thresholds.
|
|
94
|
-
|
|
95
|
-
`scripts/transcript-models.sh <transcript.jsonl>` shows the model for each
|
|
161
|
+
`scripts/transcript-models.sh <transcript.jsonl>` shows the model of each
|
|
96
162
|
assistant message in a Claude Code transcript.
|
|
97
163
|
|
|
164
|
+
## Update
|
|
165
|
+
|
|
166
|
+
1. Update the marketplace and the plugin:
|
|
167
|
+
|
|
168
|
+
```sh
|
|
169
|
+
claude plugin marketplace update alexei-led-claude-router
|
|
170
|
+
claude plugin update router@alexei-led-claude-router
|
|
171
|
+
```
|
|
172
|
+
|
|
173
|
+
2. Restart Claude Code. The next prompt replaces the running gateway.
|
|
174
|
+
3. Run `/router:setup` once. The status line command contains the plugin
|
|
175
|
+
version.
|
|
176
|
+
|
|
177
|
+
## Stop using the router
|
|
178
|
+
|
|
179
|
+
1. In `~/.claude/settings.json`, remove `model`, `env.ANTHROPIC_BASE_URL`,
|
|
180
|
+
`env.ENABLE_TOOL_SEARCH`, `env.CLAUDE_CODE_GATEWAY_HINT_HEADERS` and the
|
|
181
|
+
`jev-router[1m]` row of `modelPicker.options`.
|
|
182
|
+
2. If setup changed your status line, restore the old command.
|
|
183
|
+
3. Restart Claude Code.
|
|
184
|
+
4. To remove the plugin, run
|
|
185
|
+
`claude plugin uninstall router@alexei-led-claude-router`.
|
|
186
|
+
|
|
187
|
+
The gateway stops by itself after two hours without requests.
|
|
188
|
+
|
|
98
189
|
## Troubleshooting
|
|
99
190
|
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
- To stop the gateway, run `pkill -f scripts/gateway.mjs`. It releases the port
|
|
113
|
-
at once and finishes the responses it is streaming. The next prompt starts it
|
|
114
|
-
again.
|
|
115
|
-
- Rate limits and overload errors from Anthropic (429, 529) reach Claude Code
|
|
116
|
-
unchanged. Claude Code waits and retries; the gateway does not add a second
|
|
117
|
-
layer of retries.
|
|
191
|
+
| Symptom | Cause | Fix |
|
|
192
|
+
| ----------------------------------------------------------------- | --------------------------------------------- | ------------------------------------------------------------------------------------ |
|
|
193
|
+
| "There's an issue with the selected model (jev-router[1m])" | The session started before the restart. | Restart Claude Code. |
|
|
194
|
+
| Claude Code rejects `jev-router` | The session does not use the gateway. | Run `/router:setup`, then restart Claude Code. |
|
|
195
|
+
| The status line shows `gateway off` after the next prompt | The gateway cannot start, for example because `router.json` has an unknown key. | Run `/router:status` to see the error. Read `gateway.log` next to `decisions.jsonl`. |
|
|
196
|
+
| Every turn has the reason `no-advice` | No Jev key, or Jev fails. | Run `/router:status`. Read the `router:` lines in `gateway.log`. |
|
|
197
|
+
| The effort is not the effort that you set | The model does not accept that effort. | The gateway uses the nearest lower level. Haiku has no effort and no thinking. |
|
|
198
|
+
| 429 or 529 errors | Anthropic rate limits or overload. | Claude Code waits and tries again. The gateway sends these errors through unchanged. |
|
|
199
|
+
| A `router.json` change has no effect | The gateway reads the configuration at start. | Run `pkill -f scripts/gateway.mjs`. The next prompt starts a new gateway. |
|
|
200
|
+
|
|
201
|
+
To see if the gateway runs, open `http://127.0.0.1:43170/v1/models`. The list
|
|
202
|
+
shows the alias.
|
package/lib/config.mjs
CHANGED
|
@@ -2,6 +2,9 @@
|
|
|
2
2
|
|
|
3
3
|
export const TIERS = ['micro', 'low', 'medium', 'high'];
|
|
4
4
|
export const EFFORTS = ['low', 'medium', 'high', 'xhigh', 'max'];
|
|
5
|
+
// Request features that Claude Code uses with its own models and that only some models accept (a beta each):
|
|
6
|
+
// `system` messages after the prompt, per-turn control, and tool additions or removals in the history.
|
|
7
|
+
export const FEATURES = ['mid-conversation-system', 'per-turn-control', 'mid-conversation-tool-changes'];
|
|
5
8
|
|
|
6
9
|
// The alias before 0.4.2. Settings written by an older /router:setup still send it; drop it once they are rare.
|
|
7
10
|
export const LEGACY_ALIAS = 'router';
|
|
@@ -20,34 +23,45 @@ export const DEFAULTS = {
|
|
|
20
23
|
micro: { model: 'haiku' },
|
|
21
24
|
},
|
|
22
25
|
// `id` is sent upstream verbatim. List prices in USD per million tokens; `cacheRead` is absolute,
|
|
23
|
-
// not a multiplier (Opus 5.5 reads at 0.05x input, the rest at the standard 0.1x
|
|
24
|
-
//
|
|
26
|
+
// not a multiplier (Opus 5.5 reads at 0.05x input, the rest at the standard 0.1x). A price change must also change
|
|
27
|
+
// test/fixtures/list-prices.json, with its source and date. `output` feeds only the shadow estimate in the log.
|
|
28
|
+
// `efforts` lists what the model accepts; an empty list means no effort field and no adaptive thinking. A change
|
|
29
|
+
// must also change test/fixtures/effort-support.json, from a new probe against the real API.
|
|
30
|
+
// `features` lists the FEATURES the model accepts; the gateway removes the others from the request. A model
|
|
31
|
+
// without the field gets none: a removed feature is lost, a feature the model rejects fails the turn. A change
|
|
32
|
+
// must also change test/fixtures/model-features.json, from a new probe.
|
|
25
33
|
models: {
|
|
26
34
|
opus: {
|
|
27
35
|
id: 'claude-opus-5-5',
|
|
28
36
|
input: 4,
|
|
37
|
+
output: 20,
|
|
29
38
|
cacheRead: 0.2,
|
|
30
39
|
contextWindow: 1_000_000,
|
|
31
40
|
billing: 'plan',
|
|
32
41
|
efforts: EFFORTS,
|
|
42
|
+
features: FEATURES,
|
|
33
43
|
},
|
|
34
44
|
sonnet: {
|
|
35
45
|
id: 'claude-sonnet-5',
|
|
36
46
|
input: 2,
|
|
47
|
+
output: 10,
|
|
37
48
|
cacheRead: 0.2,
|
|
38
49
|
contextWindow: 1_000_000,
|
|
39
50
|
billing: 'plan',
|
|
40
51
|
efforts: EFFORTS,
|
|
52
|
+
features: ['mid-conversation-system'],
|
|
41
53
|
},
|
|
42
54
|
// `maxOutput` caps `max_tokens`: Claude Code asks for 128K, Haiku 4.5 answers more than 64K with 400.
|
|
43
55
|
haiku: {
|
|
44
56
|
id: 'claude-haiku-4-5',
|
|
45
57
|
input: 1,
|
|
58
|
+
output: 5,
|
|
46
59
|
cacheRead: 0.1,
|
|
47
60
|
contextWindow: 200_000,
|
|
48
61
|
maxOutput: 64_000,
|
|
49
62
|
billing: 'plan',
|
|
50
63
|
efforts: [],
|
|
64
|
+
features: [],
|
|
51
65
|
},
|
|
52
66
|
},
|
|
53
67
|
cache: {
|
|
@@ -76,13 +90,31 @@ export const DEFAULTS = {
|
|
|
76
90
|
|
|
77
91
|
const API_KEY_VAR = 'TYPESAFE_API_KEY';
|
|
78
92
|
|
|
79
|
-
|
|
93
|
+
// Keys a route or a model entry may carry. The other closed sections take their key sets from DEFAULTS.
|
|
94
|
+
const ROUTE_KEYS = ['model', 'effort'];
|
|
95
|
+
const MODEL_KEYS = [
|
|
96
|
+
'id',
|
|
97
|
+
'input',
|
|
98
|
+
'output',
|
|
99
|
+
'cacheRead',
|
|
100
|
+
'contextWindow',
|
|
101
|
+
'maxOutput',
|
|
102
|
+
'billing',
|
|
103
|
+
'efforts',
|
|
104
|
+
'features',
|
|
105
|
+
];
|
|
106
|
+
// A JSON file can hold these as own keys; merged into a plain object they reach the prototype chain.
|
|
107
|
+
const FORBIDDEN_KEYS = new Set(['__proto__', 'constructor', 'prototype']);
|
|
108
|
+
|
|
109
|
+
// `forcedTier` skips Jev and the policy. It comes from the daemon's --force-tier flag, never from env: a project's
|
|
110
|
+
// settings can set a hook's env, and a forced tier changes spend.
|
|
111
|
+
export function loadConfig({ env = {}, userFile = null, forcedTier = null } = {}) {
|
|
112
|
+
if (userFile !== null) checkShape(userFile);
|
|
80
113
|
const config = merge(DEFAULTS, userFile ?? {});
|
|
81
114
|
validate(config);
|
|
82
115
|
const apiKey = env[API_KEY_VAR]?.trim() || null;
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
throw new Error(`ROUTER_FORCE_TIER must be one of ${TIERS.join(', ')}`);
|
|
116
|
+
if (forcedTier !== null && !TIERS.includes(forcedTier))
|
|
117
|
+
throw new Error(`the forced tier must be one of ${TIERS.join(', ')}`);
|
|
86
118
|
return { ...config, apiKey, forcedTier };
|
|
87
119
|
}
|
|
88
120
|
|
|
@@ -95,27 +127,65 @@ function merge(base, override) {
|
|
|
95
127
|
const out = { ...base };
|
|
96
128
|
for (const [key, value] of Object.entries(override)) {
|
|
97
129
|
out[key] =
|
|
98
|
-
key
|
|
130
|
+
Object.hasOwn(base, key) && typeof value === 'object' && value !== null && !Array.isArray(value)
|
|
99
131
|
? merge(base[key], value)
|
|
100
132
|
: value;
|
|
101
133
|
}
|
|
102
134
|
return out;
|
|
103
135
|
}
|
|
104
136
|
|
|
137
|
+
// The raw file, before the merge: only known keys, and every section an object. Messages name the path, never the
|
|
138
|
+
// value. An open map (model aliases, cache TTL labels) takes any key but the forbidden ones.
|
|
139
|
+
function checkShape(file) {
|
|
140
|
+
checkObject(file, 'router.json', Object.keys(DEFAULTS));
|
|
141
|
+
const { gateway, routes, models, cache, policy, jev, context } = file;
|
|
142
|
+
if (gateway !== undefined) checkObject(gateway, 'gateway', Object.keys(DEFAULTS.gateway));
|
|
143
|
+
if (routes !== undefined) {
|
|
144
|
+
checkObject(routes, 'routes', TIERS);
|
|
145
|
+
for (const [tier, route] of Object.entries(routes)) checkObject(route, `routes.${tier}`, ROUTE_KEYS);
|
|
146
|
+
}
|
|
147
|
+
if (models !== undefined) {
|
|
148
|
+
checkObject(models, 'models');
|
|
149
|
+
for (const [alias, model] of Object.entries(models)) checkObject(model, `models.${alias}`, MODEL_KEYS);
|
|
150
|
+
}
|
|
151
|
+
if (cache !== undefined) {
|
|
152
|
+
checkObject(cache, 'cache', Object.keys(DEFAULTS.cache));
|
|
153
|
+
for (const map of ['writeMultiplier', 'ttlMs'])
|
|
154
|
+
if (cache[map] !== undefined) checkObject(cache[map], `cache.${map}`);
|
|
155
|
+
}
|
|
156
|
+
if (policy !== undefined) checkObject(policy, 'policy', Object.keys(DEFAULTS.policy));
|
|
157
|
+
if (jev !== undefined) checkObject(jev, 'jev', Object.keys(DEFAULTS.jev));
|
|
158
|
+
if (context !== undefined) checkObject(context, 'context', Object.keys(DEFAULTS.context));
|
|
159
|
+
}
|
|
160
|
+
|
|
161
|
+
// `allowed` null: an open map.
|
|
162
|
+
function checkObject(value, path, allowed = null) {
|
|
163
|
+
if (typeof value !== 'object' || value === null || Array.isArray(value)) throw new Error(`${path} must be an object`);
|
|
164
|
+
for (const key of Object.keys(value)) {
|
|
165
|
+
const at = path === 'router.json' ? key : `${path}.${key}`;
|
|
166
|
+
if (FORBIDDEN_KEYS.has(key)) throw new Error(`${at} is not allowed`);
|
|
167
|
+
if (allowed && !allowed.includes(key)) throw new Error(`${at} is not a known key`);
|
|
168
|
+
}
|
|
169
|
+
}
|
|
170
|
+
|
|
171
|
+
const isNumber = (value, min, { above = false, integer = false } = {}) =>
|
|
172
|
+
Number.isFinite(value) && (above ? value > min : value >= min) && (!integer || Number.isInteger(value));
|
|
173
|
+
|
|
105
174
|
function validate(config) {
|
|
106
175
|
for (const tier of TIERS) {
|
|
107
176
|
const route = config.routes[tier];
|
|
108
177
|
if (!route || typeof route.model !== 'string') throw new Error(`routes.${tier}.model is required`);
|
|
109
|
-
|
|
110
|
-
if (!model) throw new Error(`routes.${tier}.model "${route.model}" is not in models`);
|
|
178
|
+
if (!Object.hasOwn(config.models, route.model)) throw new Error(`routes.${tier}.model is not in models`);
|
|
111
179
|
if (route.effort !== undefined && !EFFORTS.includes(route.effort))
|
|
112
|
-
throw new Error(`routes.${tier}.effort
|
|
180
|
+
throw new Error(`routes.${tier}.effort must be one of ${EFFORTS.join(', ')}`);
|
|
113
181
|
}
|
|
114
182
|
for (const [alias, model] of Object.entries(config.models)) {
|
|
115
183
|
for (const field of ['input', 'cacheRead', 'contextWindow']) {
|
|
116
184
|
if (!Number.isFinite(model[field]) || model[field] < 0)
|
|
117
185
|
throw new Error(`models.${alias}.${field} must be a non-negative number`);
|
|
118
186
|
}
|
|
187
|
+
if (model.output !== undefined && !(Number.isFinite(model.output) && model.output >= 0))
|
|
188
|
+
throw new Error(`models.${alias}.output must be a non-negative number`);
|
|
119
189
|
if (typeof model.id !== 'string' || !model.id) throw new Error(`models.${alias}.id is required`);
|
|
120
190
|
if (model.maxOutput !== undefined && !(Number.isInteger(model.maxOutput) && model.maxOutput > 0))
|
|
121
191
|
throw new Error(`models.${alias}.maxOutput must be a positive integer`);
|
|
@@ -123,6 +193,11 @@ function validate(config) {
|
|
|
123
193
|
throw new Error(`models.${alias}.billing must be plan or credits`);
|
|
124
194
|
if (!Array.isArray(model.efforts) || model.efforts.some((e) => !EFFORTS.includes(e)))
|
|
125
195
|
throw new Error(`models.${alias}.efforts must list valid effort levels`);
|
|
196
|
+
if (
|
|
197
|
+
model.features !== undefined &&
|
|
198
|
+
!(Array.isArray(model.features) && model.features.every((f) => FEATURES.includes(f)))
|
|
199
|
+
)
|
|
200
|
+
throw new Error(`models.${alias}.features must list known features (${FEATURES.join(', ')})`);
|
|
126
201
|
}
|
|
127
202
|
for (const key of ['baselineTier', 'auxiliaryTier']) {
|
|
128
203
|
if (!TIERS.includes(config.gateway[key])) throw new Error(`gateway.${key} must be one of ${TIERS.join(', ')}`);
|
|
@@ -134,11 +209,27 @@ function validate(config) {
|
|
|
134
209
|
throw new Error('gateway.idleShutdownMs must be a non-negative number (0 keeps the gateway running)');
|
|
135
210
|
const p = config.policy;
|
|
136
211
|
for (const field of ['upgradeBase', 'jumpConfidence', 'downgradeMass', 'continuationMass']) {
|
|
137
|
-
if (!(p[field]
|
|
212
|
+
if (!(isNumber(p[field], 0) && p[field] <= 1)) throw new Error(`policy.${field} must be between 0 and 1`);
|
|
138
213
|
}
|
|
139
214
|
for (const field of ['upgradeVotes', 'downgradeVotes', 'escalationHoldTurns']) {
|
|
140
|
-
if (!
|
|
215
|
+
if (!isNumber(p[field], 1, { integer: true })) throw new Error(`policy.${field} must be a positive integer`);
|
|
216
|
+
}
|
|
217
|
+
if (!isNumber(p.upgradeSlope, 0)) throw new Error('policy.upgradeSlope must be a non-negative number');
|
|
218
|
+
// The pivot divides: tax / (tax + pivot). Zero makes a free switch NaN and silently stops upgrades.
|
|
219
|
+
if (!isNumber(p.upgradePivotUsd, 0, { above: true })) throw new Error('policy.upgradePivotUsd must be positive');
|
|
220
|
+
if (!isNumber(p.cashCapUsd, 0)) throw new Error('policy.cashCapUsd must be a non-negative number');
|
|
221
|
+
const { cache, jev, context } = config;
|
|
222
|
+
for (const map of ['writeMultiplier', 'ttlMs']) {
|
|
223
|
+
for (const [ttl, value] of Object.entries(cache[map]))
|
|
224
|
+
if (!isNumber(value, 0, { above: true })) throw new Error(`cache.${map}.${ttl} must be positive`);
|
|
225
|
+
}
|
|
226
|
+
if (!isNumber(cache.warmMarginMs, 0)) throw new Error('cache.warmMarginMs must be a non-negative number');
|
|
227
|
+
if (!isNumber(jev.timeoutMs, 0, { above: true })) throw new Error('jev.timeoutMs must be positive');
|
|
228
|
+
for (const field of ['endpoint', 'model']) {
|
|
229
|
+
if (typeof jev[field] !== 'string' || !jev[field]) throw new Error(`jev.${field} is required`);
|
|
230
|
+
}
|
|
231
|
+
for (const field of ['recentTurns', 'maxTextChars']) {
|
|
232
|
+
if (!isNumber(context[field], 1, { integer: true })) throw new Error(`context.${field} must be a positive integer`);
|
|
141
233
|
}
|
|
142
|
-
if (
|
|
143
|
-
if (!(config.jev.timeoutMs > 0)) throw new Error('jev.timeoutMs must be positive');
|
|
234
|
+
if (typeof config.log !== 'boolean') throw new Error('log must be true or false');
|
|
144
235
|
}
|
package/lib/cost.mjs
CHANGED
|
@@ -1,22 +1,46 @@
|
|
|
1
|
-
// Cache-aware
|
|
1
|
+
// Cache-aware cost of the next request. Pure arithmetic over configured prices and the gateway's memory.
|
|
2
|
+
import { clampEffort } from './rewrite.mjs';
|
|
3
|
+
|
|
4
|
+
// The prompt cache belongs to a model, and its messages part to the effort too: a top-level effort change rewrites
|
|
5
|
+
// the messages cache. Opus at `high` and Opus at `xhigh` are two caches, not one.
|
|
6
|
+
export function cacheKey(modelId, effort) {
|
|
7
|
+
return effort ? `${modelId}@${effort}` : modelId;
|
|
8
|
+
}
|
|
9
|
+
|
|
10
|
+
// The cache a request routed to `tier` uses. `sentEffort` is what Claude Code sent, for a route that keeps it.
|
|
11
|
+
export function routeCacheKey(config, tier, sentEffort) {
|
|
12
|
+
const model = modelOf(config, tier);
|
|
13
|
+
return cacheKey(model.id, clampEffort(config.routes[tier].effort ?? sentEffort, model.efforts));
|
|
14
|
+
}
|
|
2
15
|
|
|
3
16
|
export function isWarm(modelState, now, cache) {
|
|
4
17
|
if (!modelState) return false;
|
|
5
18
|
return now < modelState.lastAt + cache.ttlMs[modelState.ttl] - cache.warmMarginMs;
|
|
6
19
|
}
|
|
7
20
|
|
|
21
|
+
// For the logs: `unknown` means no response for this cache since the session started or its history broke. The cost
|
|
22
|
+
// arithmetic treats `unknown` and `expired` alike, as a full write.
|
|
23
|
+
export function cacheState(modelState, now, cache) {
|
|
24
|
+
if (!modelState) return 'unknown';
|
|
25
|
+
return isWarm(modelState, now, cache) ? 'warm' : 'expired';
|
|
26
|
+
}
|
|
27
|
+
|
|
28
|
+
function modelOf(config, tier) {
|
|
29
|
+
return config.models[config.routes[tier].model];
|
|
30
|
+
}
|
|
31
|
+
|
|
8
32
|
function ttlFor(config, alias, facts) {
|
|
9
33
|
if (config.models[alias].billing === 'credits') return '5m';
|
|
10
34
|
return facts.lastRequest?.ttl ?? '5m';
|
|
11
35
|
}
|
|
12
36
|
|
|
13
|
-
// USD for the input side of one request
|
|
14
|
-
export function inputCostUsd(config,
|
|
37
|
+
// USD for the input side of one request routed to `tier` with `tokens` of context.
|
|
38
|
+
export function inputCostUsd(config, tier, tokens, facts, now) {
|
|
39
|
+
const alias = config.routes[tier].model;
|
|
15
40
|
const model = config.models[alias];
|
|
16
|
-
const state = facts.models[
|
|
41
|
+
const state = facts.models[routeCacheKey(config, tier, facts.effort)];
|
|
17
42
|
const reusable = isWarm(state, now, config.cache) ? Math.min(state.prefixTokens, tokens) : 0;
|
|
18
|
-
const
|
|
19
|
-
const write = model.input * config.cache.writeMultiplier[ttl];
|
|
43
|
+
const write = model.input * config.cache.writeMultiplier[ttlFor(config, alias, facts)];
|
|
20
44
|
return (model.cacheRead * reusable + write * (tokens - reusable)) / 1e6;
|
|
21
45
|
}
|
|
22
46
|
|
|
@@ -31,9 +55,30 @@ export function nextContextTokens(facts) {
|
|
|
31
55
|
return last ? last.tokens + last.outputTokens : 0;
|
|
32
56
|
}
|
|
33
57
|
|
|
34
|
-
export function switchingTaxUsd(config,
|
|
58
|
+
export function switchingTaxUsd(config, candidateTier, incumbentTier, facts, now) {
|
|
35
59
|
const tokens = nextContextTokens(facts);
|
|
36
60
|
return (
|
|
37
|
-
inputCostUsd(config,
|
|
61
|
+
inputCostUsd(config, candidateTier, tokens, facts, now) - inputCostUsd(config, incumbentTier, tokens, facts, now)
|
|
38
62
|
);
|
|
39
63
|
}
|
|
64
|
+
|
|
65
|
+
// Shadow estimate for the decision log; the policy does not read it. At list prices, so for `plan` models it is a
|
|
66
|
+
// list-price equivalent, not a charge. Output uses the last observed output size for both routes, although effort
|
|
67
|
+
// changes how much a model writes.
|
|
68
|
+
// - nextTurnUsd: candidate minus incumbent for the next request, input at the current cache state plus output.
|
|
69
|
+
// - laterTurnUsd: the same difference for each later turn, once both caches are warm.
|
|
70
|
+
// - paybackTurns: 0 when the switch is cheaper at once, n when later turns repay it after n turns, null when never.
|
|
71
|
+
export function shadowEconomics(config, candidateTier, incumbentTier, facts, now) {
|
|
72
|
+
const candidate = modelOf(config, candidateTier);
|
|
73
|
+
const incumbent = modelOf(config, incumbentTier);
|
|
74
|
+
if (candidate.output === undefined || incumbent.output === undefined) return null;
|
|
75
|
+
const tokens = nextContextTokens(facts);
|
|
76
|
+
const outputTokens = facts.lastRequest?.outputTokens ?? 0;
|
|
77
|
+
const outputUsd = ((candidate.output - incumbent.output) * outputTokens) / 1e6;
|
|
78
|
+
const nextTurnUsd = switchingTaxUsd(config, candidateTier, incumbentTier, facts, now) + outputUsd;
|
|
79
|
+
const laterTurnUsd = ((candidate.cacheRead - incumbent.cacheRead) * tokens) / 1e6 + outputUsd;
|
|
80
|
+
let paybackTurns = null;
|
|
81
|
+
if (nextTurnUsd <= 0 && laterTurnUsd <= 0) paybackTurns = 0;
|
|
82
|
+
else if (nextTurnUsd > 0 && laterTurnUsd < 0) paybackTurns = Math.ceil(nextTurnUsd / -laterTurnUsd);
|
|
83
|
+
return { nextTurnUsd, laterTurnUsd, paybackTurns, outputTokens };
|
|
84
|
+
}
|