@alexeiled/claude-router 0.6.0 → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/plugin.json +1 -1
- package/CHANGELOG.md +90 -0
- package/README.md +97 -75
- package/docs/architecture.md +435 -0
- package/docs/configuration.md +151 -153
- package/docs/evaluation.md +83 -0
- package/docs/tier-share.svg +73 -0
- package/docs/user-guide.md +177 -92
- package/lib/config.mjs +105 -14
- package/lib/cost.mjs +53 -8
- package/lib/facts.mjs +37 -6
- package/lib/gateway.mjs +33 -30
- package/lib/jev.mjs +7 -4
- package/lib/policy.mjs +16 -12
- package/lib/rewrite.mjs +93 -3
- package/lib/router.mjs +92 -29
- package/lib/runtime.mjs +72 -7
- package/lib/status.mjs +45 -3
- package/lib/store.mjs +2 -1
- package/package.json +3 -2
- package/scripts/ensure-gateway.mjs +34 -8
- package/scripts/gateway.mjs +12 -2
- package/scripts/status.mjs +16 -3
- package/scripts/statusline.mjs +6 -2
- package/docs/design.md +0 -254
package/docs/configuration.md
CHANGED
|
@@ -1,22 +1,31 @@
|
|
|
1
1
|
# Configuration
|
|
2
2
|
|
|
3
|
-
## Where
|
|
4
|
-
|
|
5
|
-
|
|
|
6
|
-
|
|
|
7
|
-
|
|
|
8
|
-
|
|
|
9
|
-
|
|
|
10
|
-
|
|
|
11
|
-
|
|
12
|
-
The
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
3
|
+
## Where the settings are
|
|
4
|
+
|
|
5
|
+
| Setting | Location | Written by |
|
|
6
|
+
| ---------------------- | --------------------------------------------------------------- | --------------------------------------- |
|
|
7
|
+
| Jev API key | macOS Keychain, plugin option `typesafe_api_key` | Claude Code, when you enable the plugin |
|
|
8
|
+
| Claude Code settings | `~/.claude/settings.json` | `/router:setup` |
|
|
9
|
+
| Router configuration | `~/.claude/router.json` | You |
|
|
10
|
+
| Data and logs | `~/.claude/plugins/data/router-alexei-led-claude-router/` | The gateway |
|
|
11
|
+
|
|
12
|
+
- The hooks give the key to the gateway as `TYPESAFE_API_KEY`. The key is
|
|
13
|
+
never in a file.
|
|
14
|
+
- The gateway that the hooks start reads `~/.claude/router.json`, or the file
|
|
15
|
+
that `ROUTER_CONFIG` names when it is under `~/.claude/`. A project can set
|
|
16
|
+
environment variables for the plugin hooks, so the hooks take your home
|
|
17
|
+
directory from your OS user account, not from `$HOME`, and ignore
|
|
18
|
+
`ROUTER_CONFIG` for a path outside `~/.claude/`. So a cloned repository
|
|
19
|
+
cannot choose the router configuration (and with it the port, the routes or
|
|
20
|
+
the Jev endpoint) through `router.json`, `ROUTER_CONFIG` or `$HOME`. The
|
|
21
|
+
gateway still inherits the other environment variables of the session that
|
|
22
|
+
started it.
|
|
23
|
+
- The gateway reads `router.json` at start. After a change, run
|
|
24
|
+
`pkill -f scripts/gateway.mjs`. The next prompt starts a new gateway.
|
|
16
25
|
|
|
17
26
|
## Claude Code settings
|
|
18
27
|
|
|
19
|
-
`/router:setup` writes
|
|
28
|
+
`/router:setup` writes these keys to `~/.claude/settings.json`:
|
|
20
29
|
|
|
21
30
|
```json
|
|
22
31
|
{
|
|
@@ -39,164 +48,153 @@ starts it again.
|
|
|
39
48
|
}
|
|
40
49
|
```
|
|
41
50
|
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
catalog. Setup also removes the `ANTHROPIC_CUSTOM_MODEL_OPTION*` keys that
|
|
51
|
-
older versions wrote. Before 0.4.2 the alias was `router`; the gateway still
|
|
52
|
-
routes it, and setup replaces that row.
|
|
53
|
-
|
|
54
|
-
Setup writes the file as its last step. Restart Claude Code after it. Until
|
|
55
|
-
the restart, the session sends `jev-router` to Anthropic and shows "There's an
|
|
56
|
-
issue with the selected model (jev-router[1m])".
|
|
57
|
-
|
|
58
|
-
Claude Code does not know the model `jev-router`, so it assumes a 200K window.
|
|
59
|
-
The `[1m]` suffix declares 1M, the largest `contextWindow` of the default
|
|
60
|
-
routes; Claude Code strips it before the request. `CLAUDE_CODE_MAX_CONTEXT_TOKENS`
|
|
61
|
-
does not work here: Claude Code ignores it for this model. With the suffix,
|
|
62
|
-
Claude Code sends the 1M context beta header on every request; the gateway
|
|
63
|
-
drops it for a model with a smaller window (Haiku answers it with 400). The gateway sends a turn
|
|
64
|
-
only to a model whose window holds the context with room to spare (80%). A
|
|
65
|
-
large session skips Haiku (200K) and goes to Sonnet or Opus. The
|
|
66
|
-
`/router:status` reason for such a turn is `context-fit`.
|
|
67
|
-
|
|
68
|
-
If you agree, it also wraps the status line command:
|
|
51
|
+
| Key | Why |
|
|
52
|
+
| -------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
|
|
53
|
+
| `model` | The `[1m]` suffix tells Claude Code that the window is 1M tokens. Claude Code removes the suffix before the request. |
|
|
54
|
+
| `env.ANTHROPIC_BASE_URL` | Sends the traffic of Claude Code to the gateway. |
|
|
55
|
+
| `env.ENABLE_TOOL_SEARCH` | Keeps MCP tool schemas deferred. Without it, each request carries all schemas: about 50K tokens with the claude.ai connectors. |
|
|
56
|
+
| `env.CLAUDE_CODE_GATEWAY_HINT_HEADERS` | Claude Code sends the [class of each request](architecture.md#request-classes). |
|
|
57
|
+
| `modelPicker` row | Adds `Jev Router (auto)` to `/model`. `behavesAs` names a known model. Without it, Claude Code rejects the alias. |
|
|
58
|
+
| `statusLine` (optional) | Runs your status line command, then adds the route of the last turn. |
|
|
69
59
|
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
"type": "command",
|
|
74
|
-
"command": "node <plugin root>/scripts/statusline.mjs claude-powerline"
|
|
75
|
-
}
|
|
76
|
-
}
|
|
77
|
-
```
|
|
60
|
+
Setup writes the file once, as its last step. Restart Claude Code after it.
|
|
61
|
+
Until the restart, the session shows "There's an issue with the selected
|
|
62
|
+
model (jev-router[1m])".
|
|
78
63
|
|
|
79
|
-
|
|
80
|
-
session, for example `jev-router ▸ opus-5-5 · xhigh`. Without a command
|
|
81
|
-
after it, it prints only that line. The path contains the plugin version, so
|
|
82
|
-
run `/router:setup` again after a plugin update.
|
|
64
|
+
## Router configuration
|
|
83
65
|
|
|
84
|
-
|
|
85
|
-
|
|
66
|
+
Each key in `~/.claude/router.json` is optional. A key replaces the default
|
|
67
|
+
at the same path, and objects merge.
|
|
86
68
|
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
69
|
+
The file is strict. Text that is not valid JSON, an unknown key (a typo such
|
|
70
|
+
as `routs`), a value of the wrong type, or a number out of range stops a new
|
|
71
|
+
gateway from starting. The error names the file and the field, never the
|
|
72
|
+
value. A gateway that already runs keeps serving. When no gateway runs, every
|
|
73
|
+
request fails to connect until you fix the file. The SessionStart hook prints
|
|
74
|
+
the error; the prompt hook is quiet. `/router:status` also shows it: it reads
|
|
75
|
+
the file and starts nothing.
|
|
94
76
|
|
|
95
|
-
|
|
96
|
-
a compaction by a context that shrank by more than 20%.
|
|
97
|
-
|
|
98
|
-
A subagent with `model: inherit` sends `x-claude-code-agent-id` even without
|
|
99
|
-
the hint headers. Each subagent keeps its own routing memory, so its turns do
|
|
100
|
-
not change the route of the main conversation.
|
|
101
|
-
|
|
102
|
-
## Configuration file
|
|
103
|
-
|
|
104
|
-
Each key is optional. A key in the file replaces the default with the same
|
|
105
|
-
path. Nested objects merge.
|
|
77
|
+
For example, to run `low` on Sonnet at `high` effort and give Jev more time:
|
|
106
78
|
|
|
107
79
|
```json
|
|
108
80
|
{
|
|
109
|
-
"
|
|
110
|
-
|
|
111
|
-
"alias": "jev-router",
|
|
112
|
-
"baselineTier": "low",
|
|
113
|
-
"auxiliaryTier": "low",
|
|
114
|
-
"idleShutdownMs": 7200000
|
|
115
|
-
},
|
|
116
|
-
"routes": {
|
|
117
|
-
"high": { "model": "opus", "effort": "xhigh" },
|
|
118
|
-
"medium": { "model": "opus", "effort": "high" },
|
|
119
|
-
"low": { "model": "sonnet" },
|
|
120
|
-
"micro": { "model": "haiku" }
|
|
121
|
-
},
|
|
122
|
-
"models": {
|
|
123
|
-
"sonnet": {
|
|
124
|
-
"id": "claude-sonnet-5",
|
|
125
|
-
"input": 2,
|
|
126
|
-
"cacheRead": 0.2,
|
|
127
|
-
"contextWindow": 1000000,
|
|
128
|
-
"billing": "plan",
|
|
129
|
-
"efforts": ["low", "medium", "high", "xhigh", "max"]
|
|
130
|
-
}
|
|
131
|
-
},
|
|
132
|
-
"policy": {
|
|
133
|
-
"upgradeVotes": 2,
|
|
134
|
-
"upgradeBase": 0.75,
|
|
135
|
-
"upgradeSlope": 0.15,
|
|
136
|
-
"upgradePivotUsd": 0.5,
|
|
137
|
-
"jumpConfidence": 0.95,
|
|
138
|
-
"downgradeVotes": 2,
|
|
139
|
-
"downgradeMass": 0.9,
|
|
140
|
-
"continuationMass": 0.7,
|
|
141
|
-
"escalationHoldTurns": 2,
|
|
142
|
-
"cashCapUsd": 2
|
|
143
|
-
},
|
|
144
|
-
"jev": { "model": "jev-1.13.0", "timeoutMs": 1500 },
|
|
145
|
-
"context": { "recentTurns": 6, "maxTextChars": 1200 },
|
|
146
|
-
"log": true
|
|
81
|
+
"routes": { "low": { "model": "sonnet", "effort": "high" } },
|
|
82
|
+
"jev": { "timeoutMs": 2500 }
|
|
147
83
|
}
|
|
148
84
|
```
|
|
149
85
|
|
|
86
|
+
### gateway
|
|
87
|
+
|
|
88
|
+
| Key | Default | Meaning |
|
|
89
|
+
| ---------------- | ------------ | -------------------------------------------------------------------------------- |
|
|
90
|
+
| `port` | `43170` | The loopback port of the gateway. |
|
|
91
|
+
| `alias` | `jev-router` | The model name that Claude Code sends. |
|
|
92
|
+
| `baselineTier` | `low` | The tier when nothing else decides: a new session, or no Jev answer. |
|
|
93
|
+
| `auxiliaryTier` | `low` | The tier for side requests, for example session titles. |
|
|
94
|
+
| `idleShutdownMs` | `7200000` | The gateway exits after this time without requests. `0` keeps it running. |
|
|
95
|
+
|
|
150
96
|
### routes
|
|
151
97
|
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
98
|
+
| Tier | Default |
|
|
99
|
+
| -------- | ------------------------------------ |
|
|
100
|
+
| `high` | `{ "model": "opus", "effort": "xhigh" }` |
|
|
101
|
+
| `medium` | `{ "model": "opus", "effort": "high" }` |
|
|
102
|
+
| `low` | `{ "model": "sonnet" }` |
|
|
103
|
+
| `micro` | `{ "model": "haiku" }` |
|
|
104
|
+
|
|
105
|
+
`model` is a key of `models`. `effort` is `low`, `medium`, `high`, `xhigh` or
|
|
106
|
+
`max`. Without `effort`, the gateway keeps the effort that Claude Code sent.
|
|
107
|
+
If you change a route, also change the frontmatter of `skills/<tier>/SKILL.md`.
|
|
108
|
+
A test makes sure that they agree.
|
|
156
109
|
|
|
157
110
|
### models
|
|
158
111
|
|
|
159
|
-
| Alias |
|
|
160
|
-
| -------- | ------------------ |
|
|
161
|
-
| `opus` | `claude-opus-5-5` |
|
|
162
|
-
| `sonnet` | `claude-sonnet-5` |
|
|
163
|
-
| `haiku` | `claude-haiku-4-5` |
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
`
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
`
|
|
172
|
-
levels that the model accepts. An empty list
|
|
173
|
-
|
|
112
|
+
| Alias | `id` | `input` | `output` | `cacheRead` | `contextWindow` | `maxOutput` | `billing` | `efforts` | `features` |
|
|
113
|
+
| -------- | ------------------ | ------- | -------- | ----------- | --------------- | ----------- | --------- | --------- | ------------------------- |
|
|
114
|
+
| `opus` | `claude-opus-5-5` | 4 | 20 | 0.2 | 1,000,000 | — | `plan` | all five | all three |
|
|
115
|
+
| `sonnet` | `claude-sonnet-5` | 2 | 10 | 0.2 | 1,000,000 | — | `plan` | all five | `mid-conversation-system` |
|
|
116
|
+
| `haiku` | `claude-haiku-4-5` | 1 | 5 | 0.1 | 200,000 | 64,000 | `plan` | none | none |
|
|
117
|
+
|
|
118
|
+
| Field | Meaning |
|
|
119
|
+
| --------------- | ---------------------------------------------------------------------------------------------------- |
|
|
120
|
+
| `id` | The model id that the gateway sends to Anthropic. |
|
|
121
|
+
| `input`, `output`, `cacheRead` | List prices in USD per million tokens. `output` is optional. Only the `shadow` estimate uses it. |
|
|
122
|
+
| `contextWindow` | The window in tokens. A turn goes only to a model that holds the context at 80% fill. |
|
|
123
|
+
| `maxOutput` | Optional limit for `max_tokens`. Haiku 4.5 rejects more than 64K. |
|
|
124
|
+
| `billing` | `plan` for the subscription limits. `credits` for models that bill usage credits. |
|
|
125
|
+
| `efforts` | The effort levels that the model accepts. An empty list removes effort and thinking. |
|
|
126
|
+
| `features` | The request features of Claude Code that the model accepts: `mid-conversation-system`, `per-turn-control`, `mid-conversation-tool-changes`. The gateway removes the others ([architecture](architecture.md#system-context)). Without the field: none. |
|
|
127
|
+
|
|
128
|
+
The default prices match `test/fixtures/list-prices.json`, which names its
|
|
129
|
+
source and date. The default `efforts` match `test/fixtures/effort-support.json`,
|
|
130
|
+
from a probe against the real API. A test fails when they differ.
|
|
174
131
|
|
|
175
132
|
### policy
|
|
176
133
|
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
|
180
|
-
|
|
|
181
|
-
| `
|
|
182
|
-
| `
|
|
183
|
-
| `
|
|
184
|
-
| `
|
|
185
|
-
| `
|
|
186
|
-
| `
|
|
187
|
-
| `
|
|
188
|
-
| `
|
|
134
|
+
The [switching policy](architecture.md#switching-policy) uses these values.
|
|
135
|
+
|
|
136
|
+
| Key | Default | Meaning |
|
|
137
|
+
| --------------------- | ------- | ------------------------------------------------------------------------------------- |
|
|
138
|
+
| `upgradeVotes` | `2` | Consecutive votes above the current tier before an upgrade. |
|
|
139
|
+
| `upgradeBase` | `0.75` | The upgrade bar when a switch costs nothing. |
|
|
140
|
+
| `upgradeSlope` | `0.15` | How much a switching cost can raise the bar: `base + slope × tax / (tax + pivot)`. |
|
|
141
|
+
| `upgradePivotUsd` | `0.5` | The switching cost that adds half of the slope. |
|
|
142
|
+
| `jumpConfidence` | `0.95` | The confidence for a jump of two tiers without the vote delay. |
|
|
143
|
+
| `downgradeVotes` | `2` | Consecutive votes for a lower tier before a downgrade. |
|
|
144
|
+
| `downgradeMass` | `0.9` | The confidence for a downgrade. |
|
|
145
|
+
| `continuationMass` | `0.7` | The Jev probability for "continues the task" that keeps the route. |
|
|
146
|
+
| `escalationHoldTurns` | `2` | Turns that the route stays up after an escalation. |
|
|
147
|
+
| `cashCapUsd` | `2` | The cold-write guard: the largest first cache write for a switch to a `credits` model. |
|
|
148
|
+
|
|
149
|
+
### cache, jev, context, log
|
|
150
|
+
|
|
151
|
+
| Key | Default | Meaning |
|
|
152
|
+
| -------------------------- | -------------------------------------- | -------------------------------------------------------- |
|
|
153
|
+
| `cache.writeMultiplier` | `{ "5m": 1.25, "1h": 2 }` | The price of a cache write, as a multiple of `input`. |
|
|
154
|
+
| `cache.ttlMs` | `{ "5m": 300000, "1h": 3600000 }` | The lifetime of each cache TTL. |
|
|
155
|
+
| `cache.warmMarginMs` | `30000` | A cache counts as cold this long before it expires. |
|
|
156
|
+
| `jev.endpoint` | `https://api.typesafe.ai/v1/systemone` | The Jev API. |
|
|
157
|
+
| `jev.model` | `jev-1.13.0` | The Jev model. |
|
|
158
|
+
| `jev.timeoutMs` | `1500` | The total time for one Jev answer, one retry included. |
|
|
159
|
+
| `context.recentTurns` | `6` | The number of recent turns that Jev receives. |
|
|
160
|
+
| `context.maxTextChars` | `1200` | The characters of the prompt and of each turn that Jev receives. A longer text keeps its start and end. |
|
|
161
|
+
| `log` | `true` | Write `decisions.jsonl`. |
|
|
189
162
|
|
|
190
163
|
## Environment variables
|
|
191
164
|
|
|
192
|
-
| Variable | Effect
|
|
193
|
-
| -------------------- |
|
|
194
|
-
| `TYPESAFE_API_KEY` | The Jev key.
|
|
195
|
-
| `ROUTER_CONFIG` |
|
|
196
|
-
| `
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
165
|
+
| Variable | Effect |
|
|
166
|
+
| -------------------- | ----------------------------------------------------------------------------------------------- |
|
|
167
|
+
| `TYPESAFE_API_KEY` | The Jev key. The hooks set it from the plugin option. |
|
|
168
|
+
| `ROUTER_CONFIG` | Another path for the router configuration, under `~/.claude/` only. Another path is ignored, with a warning. |
|
|
169
|
+
| `CLAUDE_PLUGIN_DATA` | The data directory. Claude Code sets it for the plugin hooks. |
|
|
170
|
+
|
|
171
|
+
## Command-line flags
|
|
172
|
+
|
|
173
|
+
A project cannot change these: they are part of the command, not of the
|
|
174
|
+
environment.
|
|
175
|
+
|
|
176
|
+
| Flag | Script | Effect |
|
|
177
|
+
| --------------------- | ---------------------------------------- | ----------------------------------------------------------------------- |
|
|
178
|
+
| `--config <path>` | `gateway.mjs`, `ensure-gateway.mjs`, `status.mjs` | The router configuration, at any path. |
|
|
179
|
+
| `--force-tier <tier>` | `gateway.mjs` | `micro`, `low`, `medium` or `high`. Skips Jev and the policy. For tests. |
|
|
180
|
+
|
|
181
|
+
The flags are for a gateway that you start by hand. The hooks and the status
|
|
182
|
+
line do not pass them. To use another file with the hooks, keep it under
|
|
183
|
+
`~/.claude/` and set `ROUTER_CONFIG`.
|
|
184
|
+
|
|
185
|
+
To force a tier, stop the gateway and start it by hand with the data directory
|
|
186
|
+
of the plugin, so the log stays in its place:
|
|
187
|
+
|
|
188
|
+
```sh
|
|
189
|
+
pkill -f scripts/gateway.mjs
|
|
190
|
+
CLAUDE_PLUGIN_DATA=~/.claude/plugins/data/router-alexei-led-claude-router \
|
|
191
|
+
node <plugin>/scripts/gateway.mjs --force-tier high
|
|
192
|
+
```
|
|
193
|
+
|
|
194
|
+
## Data directory
|
|
195
|
+
|
|
196
|
+
| File | Content | Retention |
|
|
197
|
+
| ----------------- | ---------------------------------------------- | --------------------------------------- |
|
|
198
|
+
| `decisions.jsonl` | One line for each decision and each response. | Moves to `.1` above 20 MB. |
|
|
199
|
+
| `gateway.log` | Gateway events with timestamps. | Moves to `.1` above 20 MB, at gateway start. |
|
|
200
|
+
| `sessions/` | The routing memory of each session. | Removed after 30 days without use. |
|
|
@@ -0,0 +1,83 @@
|
|
|
1
|
+
# Evaluation
|
|
2
|
+
|
|
3
|
+
A snapshot of real work by one developer on one machine. The router ran in
|
|
4
|
+
Claude Code (claude-router) and in Pi (pi-model-router). Both use Jev.
|
|
5
|
+
|
|
6
|
+

|
|
7
|
+
|
|
8
|
+
## Claude Code
|
|
9
|
+
|
|
10
|
+
The window starts at the first request on the current models (Opus 5.5,
|
|
11
|
+
Sonnet 5, Haiku 4.5): 2026-09-23 09:05 UTC. It ends at 2026-09-24 06:00 UTC.
|
|
12
|
+
The data has 26 sessions and 1,611 requests. Jev advised 111 prompts.
|
|
13
|
+
|
|
14
|
+
| Tier | Model and effort | Requests | Share |
|
|
15
|
+
| -------- | ------------------ | -------- | ----- |
|
|
16
|
+
| `micro` | Haiku 4.5 | 154 | 9.6% |
|
|
17
|
+
| `low` | Sonnet 5 | 1,239 | 76.9% |
|
|
18
|
+
| `medium` | Opus 5.5, `high` | 73 | 4.5% |
|
|
19
|
+
| `high` | Opus 5.5, `xhigh` | 145 | 9.0% |
|
|
20
|
+
|
|
21
|
+
The cost of the same work at list prices:
|
|
22
|
+
|
|
23
|
+
| Scenario | Cost | Saving with the router |
|
|
24
|
+
| ------------------------- | ------- | ---------------------- |
|
|
25
|
+
| Always Opus at `xhigh` | $166.26 | 15.5% |
|
|
26
|
+
| Best model per session | $146.54 | 4.1% |
|
|
27
|
+
| jev-router | $140.55 | — |
|
|
28
|
+
|
|
29
|
+
"Best model per session" is a user who knows in advance the strongest tier
|
|
30
|
+
that Jev selected in each session, and uses it for the full session. The
|
|
31
|
+
result for each group of sessions:
|
|
32
|
+
|
|
33
|
+
| Strongest tier in the session | Sessions | Requests | Router | Baseline | Saving |
|
|
34
|
+
| ----------------------------- | -------- | -------- | ------- | -------- | ------ |
|
|
35
|
+
| `low` (Sonnet) | 23 | 573 | $39.74 | $39.74 | 0% |
|
|
36
|
+
| `medium` (Opus, `high`) | 2 | 847 | $76.89 | $84.57 | 9.1% |
|
|
37
|
+
| `high` (Opus, `xhigh`) | 1 | 191 | $23.92 | $22.23 | −7.6% |
|
|
38
|
+
|
|
39
|
+
## Output size
|
|
40
|
+
|
|
41
|
+
The baselines keep the output tokens of the router. A stronger model at a
|
|
42
|
+
higher effort writes more, so the real savings are larger. If the stronger
|
|
43
|
+
route writes more output on the requests that the router sent lower, the
|
|
44
|
+
savings are:
|
|
45
|
+
|
|
46
|
+
| Output of the stronger route | vs. always Opus at `xhigh` | vs. best model per session |
|
|
47
|
+
| ---------------------------- | -------------------------- | -------------------------- |
|
|
48
|
+
| Same (1×) | 15.5% | 4.1% |
|
|
49
|
+
| 1.5× | 21.1% | 7.6% |
|
|
50
|
+
| 2× | 26.0% | 10.8% |
|
|
51
|
+
|
|
52
|
+
## Pi
|
|
53
|
+
|
|
54
|
+
Pi used two OpenAI profiles from 2026-09-22 21:00 to 2026-09-24 06:00 UTC,
|
|
55
|
+
with 25 sessions and 1,964 requests. Jev advised 148 decisions. The profiles
|
|
56
|
+
put correctness first: `high` served 83.6% of the requests and `medium` served
|
|
57
|
+
16.4%. The router cost $347.90. The best model per session cost $348.37. With
|
|
58
|
+
this policy, routing is cost-neutral.
|
|
59
|
+
|
|
60
|
+
## Method
|
|
61
|
+
|
|
62
|
+
- **Data.** Claude Code: `decisions.jsonl`, one `observed` line with the
|
|
63
|
+
usage of each response. Pi: the session files, a router decision and the
|
|
64
|
+
usage of each message. Neither source has prompt text.
|
|
65
|
+
- **Prices.** List prices in USD per million tokens. Claude: the defaults in
|
|
66
|
+
`lib/config.mjs`. Pi: the Pi model registry, with long-context prices. For
|
|
67
|
+
subscription plans, the dollars show the relative cost only.
|
|
68
|
+
- **Cache writes.** In Claude Code, the input that is not a cache read counts
|
|
69
|
+
as a cache write at the granted TTL: 1.25× input for 5 minutes, 2× for 1
|
|
70
|
+
hour.
|
|
71
|
+
- **Baselines.** Each baseline uses the tokens and the cache reads of the
|
|
72
|
+
router. After each change of route in a conversation, the baseline also
|
|
73
|
+
reads the previous context from the cache, if the cache is still warm. Thus
|
|
74
|
+
the baseline does not pay for the cache writes that routing causes.
|
|
75
|
+
- **Tiers with one model.** Opus at `high` and at `xhigh` have the same price.
|
|
76
|
+
Subagents count toward the strongest tier of their session.
|
|
77
|
+
|
|
78
|
+
## Limits
|
|
79
|
+
|
|
80
|
+
- One developer and two days make a snapshot, not a benchmark.
|
|
81
|
+
- Answer quality was not measured.
|
|
82
|
+
- For Pi, the baseline assumes that the OpenAI cache stays warm for 10
|
|
83
|
+
minutes.
|
|
@@ -0,0 +1,73 @@
|
|
|
1
|
+
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 720 424" role="img" aria-labelledby="title desc" font-family="system-ui, -apple-system, 'Segoe UI', sans-serif">
|
|
2
|
+
<title id="title">jev-router in Claude Code: where the requests went, and what the work cost</title>
|
|
3
|
+
<desc id="desc">26 sessions and 1,611 requests. Haiku 9.6%, Sonnet 76.9%, Opus at high effort 4.5%, Opus at xhigh effort 9.0%. List-price cost: always Opus at xhigh $166, best model per session $147, jev-router $141, which is 15% less than always Opus.</desc>
|
|
4
|
+
<style>
|
|
5
|
+
.bg { fill: #fcfcfb; }
|
|
6
|
+
.t1 { fill: #0b0b0b; }
|
|
7
|
+
.t2 { fill: #52514e; }
|
|
8
|
+
.tm { fill: #898781; }
|
|
9
|
+
.axis { stroke: #c3c2b7; }
|
|
10
|
+
.r1 { fill: #86b6ef; }
|
|
11
|
+
.r2 { fill: #3987e5; }
|
|
12
|
+
.r3 { fill: #1c5cab; }
|
|
13
|
+
.r4 { fill: #0d366b; }
|
|
14
|
+
.grey { fill: #898781; }
|
|
15
|
+
.acc { fill: #2a78d6; }
|
|
16
|
+
.good { fill: #006300; }
|
|
17
|
+
@media (prefers-color-scheme: dark) {
|
|
18
|
+
.bg { fill: #1a1a19; }
|
|
19
|
+
.t1 { fill: #ffffff; }
|
|
20
|
+
.t2 { fill: #c3c2b7; }
|
|
21
|
+
.axis { stroke: #383835; }
|
|
22
|
+
.r1 { fill: #184f95; }
|
|
23
|
+
.r2 { fill: #2a78d6; }
|
|
24
|
+
.r3 { fill: #6da7ec; }
|
|
25
|
+
.r4 { fill: #b7d3f6; }
|
|
26
|
+
.grey { fill: #6b6a65; }
|
|
27
|
+
.acc { fill: #3987e5; }
|
|
28
|
+
.good { fill: #0ca30c; }
|
|
29
|
+
}
|
|
30
|
+
</style>
|
|
31
|
+
|
|
32
|
+
<rect class="bg" width="720" height="424" rx="8"/>
|
|
33
|
+
<text class="t1" x="24" y="36" font-size="18" font-weight="700">jev-router in Claude Code: one day of real work</text>
|
|
34
|
+
<text class="tm" x="24" y="58" font-size="13">26 sessions · 1,611 requests · 2026-09-23 to 2026-09-24</text>
|
|
35
|
+
|
|
36
|
+
<text class="t2" x="24" y="94" font-size="12" font-weight="600" letter-spacing="0.4">WHERE THE REQUESTS WENT</text>
|
|
37
|
+
<line class="axis" x1="232" y1="104" x2="232" y2="216" stroke-width="1"/>
|
|
38
|
+
|
|
39
|
+
<text class="t2" x="220" y="123" font-size="13" text-anchor="end">Haiku · micro</text>
|
|
40
|
+
<rect class="r1" x="232" y="108" width="40.3" height="20" rx="3"/>
|
|
41
|
+
<text class="t1" x="280" y="123" font-size="13" font-weight="600">9.6%</text>
|
|
42
|
+
|
|
43
|
+
<text class="t2" x="220" y="151" font-size="13" text-anchor="end">Sonnet · low</text>
|
|
44
|
+
<rect class="r2" x="232" y="136" width="323" height="20" rx="3"/>
|
|
45
|
+
<text class="t1" x="563" y="151" font-size="13" font-weight="600">76.9%</text>
|
|
46
|
+
|
|
47
|
+
<text class="t2" x="220" y="179" font-size="13" text-anchor="end">Opus, high effort · medium</text>
|
|
48
|
+
<rect class="r3" x="232" y="164" width="18.9" height="20" rx="3"/>
|
|
49
|
+
<text class="t1" x="259" y="179" font-size="13" font-weight="600">4.5%</text>
|
|
50
|
+
|
|
51
|
+
<text class="t2" x="220" y="207" font-size="13" text-anchor="end">Opus, xhigh effort · high</text>
|
|
52
|
+
<rect class="r4" x="232" y="192" width="37.8" height="20" rx="3"/>
|
|
53
|
+
<text class="t1" x="278" y="207" font-size="13" font-weight="600">9.0%</text>
|
|
54
|
+
|
|
55
|
+
<text class="t2" x="24" y="260" font-size="12" font-weight="600" letter-spacing="0.4">COST OF THE SAME WORK, AT LIST PRICES</text>
|
|
56
|
+
<line class="axis" x1="232" y1="270" x2="232" y2="354" stroke-width="1"/>
|
|
57
|
+
|
|
58
|
+
<text class="t2" x="220" y="289" font-size="13" text-anchor="end">Always Opus at xhigh</text>
|
|
59
|
+
<rect class="grey" x="232" y="274" width="410" height="20" rx="3"/>
|
|
60
|
+
<text class="t1" x="650" y="289" font-size="13" font-weight="600">$166</text>
|
|
61
|
+
|
|
62
|
+
<text class="t2" x="220" y="317" font-size="13" text-anchor="end">Best model per session</text>
|
|
63
|
+
<rect class="grey" x="232" y="302" width="361.4" height="20" rx="3"/>
|
|
64
|
+
<text class="t1" x="601" y="317" font-size="13" font-weight="600">$147</text>
|
|
65
|
+
|
|
66
|
+
<text class="t1" x="220" y="345" font-size="13" font-weight="700" text-anchor="end">jev-router</text>
|
|
67
|
+
<rect class="acc" x="232" y="330" width="346.6" height="20" rx="3"/>
|
|
68
|
+
<text class="t1" x="586" y="345" font-size="13" font-weight="600">$141</text>
|
|
69
|
+
<text class="good" x="624" y="345" font-size="13" font-weight="600">−15%</text>
|
|
70
|
+
|
|
71
|
+
<text class="tm" x="24" y="388" font-size="12">Lower bound: the baselines keep the output tokens of the router. A stronger route writes more.</text>
|
|
72
|
+
<text class="tm" x="24" y="406" font-size="12">Method, table and the Pi numbers: docs/evaluation.md</text>
|
|
73
|
+
</svg>
|