@alexeiled/claude-router 0.6.1 → 0.7.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/plugin.json +1 -1
- package/CHANGELOG.md +102 -0
- package/README.md +94 -78
- package/docs/architecture.md +416 -291
- package/docs/configuration.md +153 -162
- package/docs/evaluation.md +83 -0
- package/docs/tier-share.svg +73 -44
- package/docs/user-guide.md +176 -114
- package/lib/config.mjs +100 -13
- package/lib/facts.mjs +35 -6
- package/lib/gateway.mjs +33 -30
- package/lib/jev.mjs +7 -4
- package/lib/policy.mjs +7 -2
- package/lib/rewrite.mjs +93 -3
- package/lib/router.mjs +41 -8
- package/lib/runtime.mjs +72 -7
- package/lib/status.mjs +2 -1
- package/lib/store.mjs +2 -1
- package/package.json +3 -2
- package/scripts/ensure-gateway.mjs +34 -8
- package/scripts/gateway.mjs +12 -2
- package/scripts/status.mjs +16 -3
- package/scripts/statusline.mjs +6 -2
package/docs/architecture.md
CHANGED
|
@@ -1,312 +1,437 @@
|
|
|
1
1
|
# Architecture
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
[
|
|
28
|
-
[
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
3
|
+
claude-router selects a model and an effort level for each Claude Code turn.
|
|
4
|
+
A local gateway changes each request before it goes to Anthropic. TypeSafe Jev
|
|
5
|
+
advises a tier, and a local policy makes the decision.
|
|
6
|
+
|
|
7
|
+
- [System context](#system-context)
|
|
8
|
+
- [Tiers](#tiers)
|
|
9
|
+
- [Request flow](#request-flow)
|
|
10
|
+
- [Request classes](#request-classes)
|
|
11
|
+
- [Switching policy](#switching-policy)
|
|
12
|
+
- [Cache and cost model](#cache-and-cost-model)
|
|
13
|
+
- [Modules](#modules)
|
|
14
|
+
- [Gateway lifecycle](#gateway-lifecycle)
|
|
15
|
+
- [Failure handling](#failure-handling)
|
|
16
|
+
- [Security and privacy](#security-and-privacy)
|
|
17
|
+
- [Observability](#observability)
|
|
18
|
+
- [Release](#release)
|
|
19
|
+
- [Known limits](#known-limits)
|
|
20
|
+
|
|
21
|
+
## System context
|
|
22
|
+
|
|
23
|
+
```mermaid
|
|
24
|
+
flowchart TB
|
|
25
|
+
subgraph machine["Your machine"]
|
|
26
|
+
direction LR
|
|
27
|
+
CC["Claude Code<br/>model: jev-router"]
|
|
28
|
+
HOOK["Plugin hooks"]
|
|
29
|
+
UI["Status line<br/>/router:status"]
|
|
30
|
+
GW["Gateway<br/>127.0.0.1:43170"]
|
|
31
|
+
DATA[("Data directory<br/>sessions · decisions.jsonl")]
|
|
32
|
+
end
|
|
33
|
+
subgraph remote["Remote services"]
|
|
34
|
+
direction LR
|
|
35
|
+
API["api.anthropic.com"]
|
|
36
|
+
JEV["TypeSafe Jev"]
|
|
37
|
+
end
|
|
38
|
+
|
|
39
|
+
CC <-->|"Messages API"| GW
|
|
40
|
+
HOOK -.->|"start or replace"| GW
|
|
41
|
+
UI -.->|"GET /router/status"| GW
|
|
42
|
+
GW -->|"prompt and<br/>recent turns"| JEV
|
|
43
|
+
GW <-->|"changed request,<br/>unchanged response"| API
|
|
44
|
+
GW --> DATA
|
|
45
|
+
|
|
46
|
+
classDef ext fill:#eef2ff,stroke:#6366f1,color:#1e1b4b
|
|
47
|
+
classDef core fill:#ecfdf5,stroke:#059669,color:#064e3b
|
|
48
|
+
classDef store fill:#fff7ed,stroke:#ea580c,color:#431407
|
|
49
|
+
class JEV,API ext
|
|
50
|
+
class GW core
|
|
51
|
+
class DATA store
|
|
52
|
+
style machine fill:#f8fafc,stroke:#94a3b8,color:#0f172a
|
|
53
|
+
style remote fill:#f8fafc,stroke:#94a3b8,color:#0f172a
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
- `/router:setup` points `ANTHROPIC_BASE_URL` at the gateway and sets the
|
|
57
|
+
model to `jev-router[1m]`. One gateway serves all sessions on the machine.
|
|
58
|
+
- The gateway changes only requests for the alias. It sets `model`,
|
|
59
|
+
`output_config.effort` and `thinking`. For a small model, it also limits
|
|
60
|
+
`max_tokens` and removes the 1M context beta header and the thinking edits.
|
|
61
|
+
- Claude Code builds each request for the model that the alias behaves as
|
|
62
|
+
(Opus). A model without one of its [request features](configuration.md#models)
|
|
63
|
+
gets the request without it, the way Claude Code retries after a 400:
|
|
64
|
+
- no `mid-conversation-tool-changes` (Sonnet, Haiku): `tool_addition` and
|
|
65
|
+
`tool_removal` blocks go; their cache breakpoint moves to the block before;
|
|
66
|
+
- no `per-turn-control` (Sonnet, Haiku): the beta header and the
|
|
67
|
+
`output_config` on messages (the effort of each turn) go;
|
|
68
|
+
- no `mid-conversation-system` (Haiku): each `system` message becomes a
|
|
69
|
+
`<system-reminder>` text in the user message next to it.
|
|
70
|
+
The same history adapts the same way on every turn, so the cached prefix
|
|
71
|
+
holds.
|
|
72
|
+
- On a model with `per-turn-control`, the effort of each turn overrides the
|
|
73
|
+
request's effort. A route with its own effort (`medium`, `high`) therefore
|
|
74
|
+
sets it in every message's `output_config` too.
|
|
75
|
+
- The gateway never changes `system` or `tools`, and never changes `messages`
|
|
76
|
+
for a model that takes all the features. Prompt caching and a claude.ai
|
|
77
|
+
login work as with a direct connection.
|
|
78
|
+
- Responses go back unchanged. The gateway reads `usage` from them.
|
|
79
|
+
|
|
80
|
+
**Why a gateway.** Only a gateway can route every turn. The tier skills are
|
|
81
|
+
manual pins. Claude Code 2.1.x switches the model from their `model:`
|
|
82
|
+
frontmatter for `sonnet` and `opus`, and that turn passes through. For `haiku`
|
|
83
|
+
it sends the turn to the alias; the gateway reads the `/router:micro` command
|
|
84
|
+
in the prompt and serves the turn on `micro`, reason `pinned`. After a pin, the
|
|
85
|
+
next prompt is decided from the route before the pin.
|
|
54
86
|
|
|
55
87
|
## Tiers
|
|
56
88
|
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
89
|
+
Four tiers, from `micro` (Haiku) to `high` (Opus at `xhigh`), map to a model
|
|
90
|
+
and an effort. The [README](../README.md#how-jev-selects-a-tier) shows the
|
|
91
|
+
work that Jev selects each tier for. The [routes](configuration.md#routes)
|
|
92
|
+
set the mapping.
|
|
93
|
+
|
|
94
|
+
- `low` is the baseline.
|
|
95
|
+
- `medium` and `high` use the same model at two efforts.
|
|
96
|
+
- The gateway lowers an effort to a level that the model accepts. Haiku
|
|
97
|
+
accepts no effort, so it runs without thinking.
|
|
98
|
+
|
|
99
|
+
## Request flow
|
|
100
|
+
|
|
101
|
+
A new user turn:
|
|
102
|
+
|
|
103
|
+
```mermaid
|
|
104
|
+
sequenceDiagram
|
|
105
|
+
autonumber
|
|
106
|
+
participant CC as Claude Code
|
|
107
|
+
participant GW as Gateway
|
|
108
|
+
participant R as Router
|
|
109
|
+
participant J as Jev
|
|
110
|
+
participant A as Anthropic
|
|
111
|
+
|
|
112
|
+
CC->>GW: POST /v1/messages (jev-router)
|
|
113
|
+
GW->>R: route(body, session, hints)
|
|
114
|
+
Note over R: facts from body and session memory
|
|
115
|
+
R->>J: prompt and last 6 turns
|
|
116
|
+
J-->>R: tier probabilities, continuation
|
|
117
|
+
Note over R: switching policy selects the tier
|
|
118
|
+
R-->>GW: body with model, effort, thinking
|
|
119
|
+
GW->>A: request
|
|
120
|
+
A-->>GW: response stream
|
|
121
|
+
GW-->>CC: same bytes
|
|
122
|
+
GW->>R: usage: tokens, cache reads, TTL
|
|
123
|
+
Note over R: session memory, decisions.jsonl
|
|
124
|
+
```
|
|
63
125
|
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
no effort and no adaptive thinking. The ids are configuration.
|
|
126
|
+
A tool continuation and a resent request skip the Jev call. They keep the
|
|
127
|
+
route of their turn.
|
|
67
128
|
|
|
68
129
|
## Request classes
|
|
69
130
|
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
131
|
+
```mermaid
|
|
132
|
+
flowchart TD
|
|
133
|
+
IN(["Request"]) --> ALIAS{"model is<br/>jev-router?"}
|
|
134
|
+
ALIAS -- no --> PASS["Pass through unchanged"]
|
|
135
|
+
ALIAS -- yes --> MSG{"POST<br/>/v1/messages?"}
|
|
136
|
+
MSG -- no --> LAST["Last route of the session"]
|
|
137
|
+
MSG -- yes --> SIDE{"Side request?"}
|
|
138
|
+
SIDE -- yes --> AUX["auxiliaryTier<br/>memory unchanged"]
|
|
139
|
+
SIDE -- no --> BRK{"History break?"}
|
|
140
|
+
BRK -- yes --> RESET["Reset cache estimates,<br/>votes and hold"]
|
|
141
|
+
BRK -- no --> TOOL
|
|
142
|
+
RESET --> TOOL{"tool_result or<br/>resent request?"}
|
|
143
|
+
TOOL -- yes --> KEEP["Route of the turn"]
|
|
144
|
+
TOOL -- no --> POL["Jev and switching policy"]
|
|
145
|
+
KEEP --> FIT["Fit the context window"]
|
|
146
|
+
POL --> FIT
|
|
147
|
+
FIT --> OUT(["Change and forward"])
|
|
148
|
+
|
|
149
|
+
classDef stop fill:#f1f5f9,stroke:#64748b,color:#0f172a
|
|
150
|
+
classDef act fill:#ecfdf5,stroke:#059669,color:#064e3b
|
|
151
|
+
class PASS,LAST,AUX stop
|
|
152
|
+
class RESET,KEEP,POL,FIT act
|
|
153
|
+
```
|
|
89
154
|
|
|
90
|
-
|
|
155
|
+
| Check | How the gateway decides |
|
|
156
|
+
| --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
157
|
+
| Pass through | Any other model: `/model`, subagents with their own model. |
|
|
158
|
+
| Pin | The current message starts with the `/router:<tier>` command block. That tier serves the turn and its tool calls, reason `pinned`. No Jev call. |
|
|
159
|
+
| Not a turn | `count_tokens` and other endpoints get the last route of the session. |
|
|
160
|
+
| Side request | Header `x-claude-code-request-class` is `auxiliary` or `compaction`. Without the header: thinking disabled or an output format in the body. |
|
|
161
|
+
| History break | Fewer messages than the last main request (a compaction or a rewind), or the header `x-claude-code-context-compacted`. |
|
|
162
|
+
| Resent request | Same last user message at the same position. Claude Code resends after a 429, a 529, a dropped stream, or a 400 that it answers by dropping a feature. No Jev call, no vote. |
|
|
163
|
+
| System messages | Claude Code sends hook output and tool additions as `system` messages after the prompt. The turn is the last user or assistant message; Jev never receives `system` text. |
|
|
164
|
+
| Context fit | The route moves to a model whose window holds the next context at 80% fill. Reason `context-fit`. |
|
|
165
|
+
| Session memory | Key `x-claude-code-session-id`. A subagent adds its agent id: `<session>.<agent id>`. Its turns do not change the route of the main conversation. |
|
|
166
|
+
|
|
167
|
+
## Switching policy
|
|
168
|
+
|
|
169
|
+
```mermaid
|
|
170
|
+
flowchart TD
|
|
171
|
+
S(["New user turn"]) --> ERR{"Same error twice,<br/>edit between?"}
|
|
172
|
+
ERR -- yes --> ESC["escalation<br/>one tier up for 2 turns"]
|
|
173
|
+
ERR -- no --> HOLD{"Hold active?"}
|
|
174
|
+
HOLD -- yes --> R1["hold"]
|
|
175
|
+
HOLD -- no --> ADV{"Jev answered?"}
|
|
176
|
+
ADV -- no --> R2["no-advice"]
|
|
177
|
+
ADV -- yes --> CONT{"Continues<br/>the task?"}
|
|
178
|
+
CONT -- yes --> R3["continuation"]
|
|
179
|
+
CONT -- no --> DIR{"Jev choice vs<br/>current tier"}
|
|
180
|
+
DIR -- "same or uncertain" --> R4["same-tier<br/>uncertain"]
|
|
181
|
+
DIR -- higher --> GUARD{"Cold-write<br/>guard blocks?"}
|
|
182
|
+
GUARD -- yes --> CG["cash-gate<br/>strongest plan tier"]
|
|
183
|
+
GUARD -- no --> JMP{"2 tiers up,<br/>U ≥ 0.95?"}
|
|
184
|
+
JMP -- yes --> JUMP["jump"]
|
|
185
|
+
JMP -- no --> UPQ{"2 votes,<br/>U ≥ bar?"}
|
|
186
|
+
UPQ -- yes --> UP["upgrade"]
|
|
187
|
+
UPQ -- no --> R5["upgrade-pending"]
|
|
188
|
+
DIR -- lower --> DNQ{"2 votes,<br/>D ≥ bar?"}
|
|
189
|
+
DNQ -- yes --> DOWN["downgrade"]
|
|
190
|
+
DNQ -- no --> R6["downgrade-pending"]
|
|
191
|
+
|
|
192
|
+
classDef stay fill:#f1f5f9,stroke:#64748b,color:#0f172a
|
|
193
|
+
classDef up fill:#fef3c7,stroke:#d97706,color:#451a03
|
|
194
|
+
classDef down fill:#e0f2fe,stroke:#0284c7,color:#082f49
|
|
195
|
+
class R1,R2,R3,R4,R5,R6 stay
|
|
196
|
+
class ESC,CG,JUMP,UP up
|
|
197
|
+
class DOWN down
|
|
198
|
+
```
|
|
199
|
+
|
|
200
|
+
Each box is the `reason` that the log and the status line show. Grey boxes
|
|
201
|
+
keep the current route. The cold-write guard serves the strongest plan tier
|
|
202
|
+
only when that tier is above the current route.
|
|
203
|
+
|
|
204
|
+
- `U` is the Jev probability mass above the current tier. `D` is the mass at
|
|
205
|
+
or below the candidate. The `uncertain` mass supports neither.
|
|
206
|
+
- The upgrade bar grows with the switching tax:
|
|
207
|
+
`bar = 0.75 + 0.15 × tax / (tax + $0.50)`. It stays between 0.75 and 0.90.
|
|
208
|
+
- The downgrade bar grows the same way, from the other side: a candidate
|
|
209
|
+
colder than the incumbent pays a cache write, not just a smaller model.
|
|
210
|
+
`bar = 0.90 + 0.08 × tax / (tax + $0.50)`. It stays between 0.90 and 0.98.
|
|
211
|
+
- A prompt continues the task when the Jev continuation answer is 0.7 or
|
|
212
|
+
more. A continuing prompt never votes for a downgrade.
|
|
213
|
+
- Upgrades have no cooldown. Plan, then execute, then hard work again is a
|
|
214
|
+
valid sequence.
|
|
215
|
+
- A failure signal is two `tool_result` errors with the same text and an edit
|
|
216
|
+
between them, in the last 40 messages. Each signal escalates once.
|
|
217
|
+
- The thresholds are defaults in [configuration](configuration.md#policy).
|
|
218
|
+
|
|
219
|
+
## Cache and cost model
|
|
220
|
+
|
|
221
|
+
The gateway keeps one cache record for each model and effort, for example
|
|
222
|
+
`claude-opus-5-5@xhigh`. A change of effort replaces the cached messages, so
|
|
223
|
+
`medium` and `high` are two caches.
|
|
224
|
+
|
|
225
|
+
```mermaid
|
|
226
|
+
stateDiagram-v2
|
|
227
|
+
direction LR
|
|
228
|
+
[*] --> unknown
|
|
229
|
+
unknown --> warm: response recorded
|
|
230
|
+
warm --> expired: TTL minus 30 s passed
|
|
231
|
+
expired --> warm: response recorded
|
|
232
|
+
warm --> unknown: history break or context shrank
|
|
233
|
+
expired --> unknown: history break or context shrank
|
|
234
|
+
```
|
|
91
235
|
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
- Anthropic errors (429, 529, 5xx) pass through unchanged, with `Retry-After`.
|
|
96
|
-
Claude Code owns retries and backoff; a second retry layer in the gateway
|
|
97
|
-
would multiply attempts and cannot replay a stream that has started.
|
|
98
|
-
- The response is relayed with `pipeline()`. An upstream reset destroys the
|
|
99
|
-
client response, so Claude Code sees a reset and retries at once. A client
|
|
100
|
-
that leaves (Esc) destroys the upstream request, so the model stops
|
|
101
|
-
generating an answer that nobody reads.
|
|
102
|
-
- Disk errors while the gateway saves session memory or the decision log are
|
|
103
|
-
logged. The routing decision stands.
|
|
104
|
-
- Jev: one retry on a network error or a transient status, after
|
|
105
|
-
`Retry-After` when it fits the 1.5 s budget. After three failures in a row,
|
|
106
|
-
new turns skip Jev for a minute, then try once per pause.
|
|
107
|
-
- The daemon logs a stray exception instead of exiting. On `SIGTERM` it
|
|
108
|
-
releases the port at once and finishes open streams for up to 10 minutes.
|
|
109
|
-
A second daemon on a busy port exits quietly.
|
|
110
|
-
- The `SessionStart` and `UserPromptSubmit` hooks start the gateway when the
|
|
111
|
-
port does not answer, and replace a gateway older than the plugin. They never
|
|
112
|
-
replace a newer one.
|
|
113
|
-
- The gateway exits after `gateway.idleShutdownMs` (two hours) without
|
|
114
|
-
requests. Claude Code holds no connection open between requests, so the
|
|
115
|
-
gateway cannot tell a closed session from an idle one; time is the signal.
|
|
116
|
-
Two cases keep it running: a request in flight, and a turn whose last
|
|
117
|
-
response asked for a tool (`stop_reason: tool_use`), such as an unanswered
|
|
118
|
-
permission prompt. The answer to that prompt reaches the gateway without a
|
|
119
|
-
new prompt, so without the hook that would start it again. A session that
|
|
120
|
-
died mid-turn stops counting after a day. Two hours outlast `/loop` wakeups
|
|
121
|
-
and Monitor waits, which also arrive without a prompt.
|
|
122
|
-
- The gateway refuses requests with a non-loopback `Host` (DNS rebinding) or a
|
|
123
|
-
web `Origin` (cross-site requests from a browser).
|
|
124
|
-
|
|
125
|
-
## Cache and cost inputs
|
|
126
|
-
|
|
127
|
-
All inputs come from the traffic of the gateway. The gateway does not read
|
|
128
|
-
transcripts.
|
|
129
|
-
|
|
130
|
-
| Input | Source |
|
|
131
|
-
| ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
132
|
-
| Context of the last request, cache reads, output | `usage` in the response (`message_start` and `message_delta`) |
|
|
133
|
-
| Granted TTL | `usage.cache_creation.ephemeral_1h_input_tokens` or the `5m` field |
|
|
134
|
-
| Cache identity | The model id and the effort the gateway sent (`claude-opus-5-5@xhigh`). An effort change rewrites the messages cache, so each effort is its own cache. |
|
|
135
|
-
| Cache warmth | The time of the last response for that cache, plus the TTL, minus 30 s. `unknown` when no response since the session started or the history broke. |
|
|
136
|
-
| Reusable prefix | The context plus the output at the last response for that cache. Cleared by a history break, or when the context shrinks by more than 20% (context editing). |
|
|
137
|
-
| Failure signal | Two `tool_result` blocks with `is_error` and the same signature, with an edit tool call between them |
|
|
138
|
-
| Continuation | The last message contains a `tool_result`. For a new prompt, a Jev Noul answers "does this prompt continue the task". |
|
|
139
|
-
|
|
140
|
-
Prices are a list-price table in the configuration (see
|
|
141
|
-
[configuration](configuration.md#models)). `modelPricing` is a managed setting
|
|
142
|
-
and is not readable. The switching tax for a candidate `c` against the current
|
|
143
|
-
route `i` is:
|
|
236
|
+
The switching tax compares the input cost of the next request on the
|
|
237
|
+
candidate and on the current route:
|
|
144
238
|
|
|
239
|
+
```text
|
|
240
|
+
input_cost = P_read × W + P_write × (N − W)
|
|
241
|
+
tax = max(0, input_cost(candidate) − input_cost(current))
|
|
145
242
|
```
|
|
146
|
-
|
|
147
|
-
|
|
243
|
+
|
|
244
|
+
| Symbol | Source |
|
|
245
|
+
| --------- | ------------------------------------------------------------------------------------------------------ |
|
|
246
|
+
| `N` | Next context: tokens and output of the last response. |
|
|
247
|
+
| `W` | Warm prefix of the candidate cache. Zero when the cache is `expired` or `unknown`. |
|
|
248
|
+
| `P_read` | List price for cache reads. |
|
|
249
|
+
| `P_write` | Input list price × 1.25 (5-minute TTL) or × 2 (1-hour TTL). The TTL comes from `usage.cache_creation`. |
|
|
250
|
+
|
|
251
|
+
- **Upper bound.** An effort change counts the whole prefix as lost. On some
|
|
252
|
+
models, the tools and the system prompt stay in the cache.
|
|
253
|
+
- **Context editing.** A context that shrinks by more than 20% with no fewer
|
|
254
|
+
messages resets the cache records only. The votes stay.
|
|
255
|
+
- **Cold-write guard.** A model with `billing: "credits"` bills cash. When its
|
|
256
|
+
cache is not warm, `policy.cashCapUsd` limits the first cache write. It is
|
|
257
|
+
not a budget for the turn. No default model bills credits.
|
|
258
|
+
- **Prices.** List prices in USD per million tokens. For plan models, the
|
|
259
|
+
dollars only give an order between models. A test pins the defaults to
|
|
260
|
+
`test/fixtures/list-prices.json`, which names its source and date.
|
|
261
|
+
- **Shadow estimate.** When the Jev choice differs from the current route, the
|
|
262
|
+
log records the cost of the next turn, the cost of each later turn with
|
|
263
|
+
output, and the turns until a switch repays its cost. The policy does not
|
|
264
|
+
read it.
|
|
265
|
+
|
|
266
|
+
## Modules
|
|
267
|
+
|
|
268
|
+
```mermaid
|
|
269
|
+
flowchart TD
|
|
270
|
+
subgraph scripts["scripts/ · entry points"]
|
|
271
|
+
D["gateway.mjs<br/>daemon"]
|
|
272
|
+
E["ensure-gateway.mjs<br/>hook"]
|
|
273
|
+
SL["statusline.mjs<br/>status line"]
|
|
274
|
+
SC["status.mjs<br/>/router:status"]
|
|
275
|
+
end
|
|
276
|
+
subgraph lib["lib/"]
|
|
277
|
+
GW["gateway<br/>HTTP, relay"]
|
|
278
|
+
IDLE["idle"]
|
|
279
|
+
SSE["sse"]
|
|
280
|
+
RT["router<br/>orchestration"]
|
|
281
|
+
FA["facts"]
|
|
282
|
+
JEV["jev"]
|
|
283
|
+
POL["policy"]
|
|
284
|
+
COST["cost"]
|
|
285
|
+
RW["rewrite"]
|
|
286
|
+
STA["status"]
|
|
287
|
+
STORE[("store")]
|
|
288
|
+
RUN["runtime"]
|
|
289
|
+
end
|
|
290
|
+
|
|
291
|
+
D -->|"creates, injects router"| GW
|
|
292
|
+
D --> RT & RUN & IDLE & STA & STORE
|
|
293
|
+
GW --> IDLE & SSE & STA
|
|
294
|
+
GW -.->|"side-request test"| RT
|
|
295
|
+
RT --> FA & JEV & POL & COST & RW & STORE
|
|
296
|
+
POL --> COST --> RW
|
|
297
|
+
STA --> RW
|
|
298
|
+
E --> RUN & STA & STORE
|
|
299
|
+
SL --> RUN & STA
|
|
300
|
+
SC --> RUN & STA
|
|
301
|
+
RUN --> STORE
|
|
302
|
+
|
|
303
|
+
classDef pure fill:#ecfdf5,stroke:#059669,color:#064e3b
|
|
304
|
+
classDef io fill:#fff7ed,stroke:#ea580c,color:#431407
|
|
305
|
+
class FA,POL,COST,RW,SSE pure
|
|
306
|
+
class STORE,GW,JEV io
|
|
307
|
+
style scripts fill:#f8fafc,stroke:#94a3b8,color:#0f172a
|
|
308
|
+
style lib fill:#f8fafc,stroke:#94a3b8,color:#0f172a
|
|
148
309
|
```
|
|
149
310
|
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
7. Logs. The tier, the reason, the estimates, and the observed model, tokens
|
|
185
|
-
and cache reads of each routed response. The memory changes only from
|
|
186
|
-
responses that the gateway sent.
|
|
187
|
-
|
|
188
|
-
## Cache identity, history breaks and shadow economics (2026-09-23)
|
|
189
|
-
|
|
190
|
-
From a design review with the architect of pi-model-router, the Pi router
|
|
191
|
-
that uses the same Jev tiers.
|
|
192
|
-
|
|
193
|
-
- Cache identity is the model and the effort. A top-level effort change
|
|
194
|
-
invalidates the messages cache; the cache-preserving per-message effort is
|
|
195
|
-
not available on Opus 5.5. Before 0.6.1 the key was the model alone, so
|
|
196
|
-
`medium` ↔ `high` (Opus at `high` and `xhigh`) looked like a free switch
|
|
197
|
-
between warm caches.
|
|
198
|
-
- A history break (fewer messages, or the compaction header) drops the cached
|
|
199
|
-
prefixes, the votes and the escalation hold: they were about turns that are
|
|
200
|
-
gone. Context editing shrinks the context but keeps the messages; it drops
|
|
201
|
-
only the prefixes. The gateway has no branch id, so it does not restore state
|
|
202
|
-
from before a rewind; `unknown` is the honest cache state after one.
|
|
203
|
-
- `/router:status` and the status line show the reason; the report adds the
|
|
204
|
-
estimate. The dollars are list prices, a list-price equivalent for plan
|
|
205
|
-
models.
|
|
206
|
-
- Shadow economics in `decisions.jsonl`, not read by the policy: for a turn
|
|
207
|
-
where Jev's choice differs from the current route, the extra cost of the
|
|
208
|
-
next turn, the difference for each later turn (input and output), and the
|
|
209
|
-
turns until a cheaper route repays its cache write. The owner's first
|
|
210
|
-
request (2026-09-22) was to stay while the model works on a warm cache and
|
|
211
|
-
to step down once the thinking is done. Rule 2 covers the first half; the
|
|
212
|
-
shadow estimate measures the second before any rule acts on it.
|
|
213
|
-
- The default prices match `test/fixtures/list-prices.json`, which names its
|
|
214
|
-
source and date. A test pins how a tenfold cache-read error changes one
|
|
215
|
-
decision: the bar moves within `upgradeBase` and `upgradeBase +
|
|
216
|
-
upgradeSlope`, and a confident jump ignores it. That error happened once
|
|
217
|
-
(a838f7c).
|
|
218
|
-
|
|
219
|
-
Not taken: the agent type as a routing signal; a payback check on downgrades
|
|
220
|
-
before shadow data; removing the switching tax from the upgrade bar before a
|
|
221
|
-
replay of the logs. The Pi router rejects the tax-to-confidence formula
|
|
222
|
-
because it mixes dollars with an uncalibrated probability; the replay decides.
|
|
223
|
-
|
|
224
|
-
## Real-world evaluation (2026-09-23)
|
|
225
|
-
|
|
226
|
-
A day of dogfooding this repository on the installed plugin: 31 Claude Code
|
|
227
|
-
sessions, 1,578 routed requests, one machine. `decisions.jsonl` holds the
|
|
228
|
-
tier, the reason and the token counts for every request — no prompt text.
|
|
229
|
-
|
|
230
|
-

|
|
231
|
-
|
|
232
|
-
82% of turns never needed more than Sonnet, 10% stayed on Haiku, and 8% needed
|
|
233
|
-
Opus. `medium` (Opus at `high` effort) fired once: a confident vote jumps two
|
|
234
|
-
tiers straight to `high` instead of stopping at `medium` (switching policy,
|
|
235
|
-
rule 4).
|
|
236
|
-
|
|
237
|
-
Repricing that same traffic — same tokens, same observed cache reads — at
|
|
238
|
-
Opus's rates puts the input-token bill 18.8% above what the router actually
|
|
239
|
-
spent. That number covers input tokens only: the price table has no output
|
|
240
|
-
price (see [Cache and cost inputs](#cache-and-cost-inputs)), so it cannot say
|
|
241
|
-
how much of the real saving is left out — likely more, since Haiku and Sonnet
|
|
242
|
-
also bill less per output token than Opus. Answer quality isn't measured
|
|
243
|
-
here either.
|
|
244
|
-
|
|
245
|
-
One developer, one day: a dogfood snapshot, not a benchmark.
|
|
246
|
-
|
|
247
|
-
## Layout
|
|
311
|
+
Green modules are pure functions. Orange modules do I/O: `gateway` serves
|
|
312
|
+
HTTP, `jev` calls Jev, and in `lib/` `store` is the only module that writes
|
|
313
|
+
files. Outside `lib/`, the hook `ensure-gateway.mjs` also opens `gateway.log`
|
|
314
|
+
for the daemon that it starts. The Jev transport and the router are injected,
|
|
315
|
+
so tests run without a network. `gateway` imports one pure function from
|
|
316
|
+
`router`, the side-request test, not the router itself. Modules that need
|
|
317
|
+
settings or tier names import `config.mjs` (not drawn).
|
|
318
|
+
|
|
319
|
+
| Module | Responsibility |
|
|
320
|
+
| --------- | ---------------------------------------------------------------------------------------------- |
|
|
321
|
+
| `gateway` | HTTP server, loopback checks, relay with `pipeline()`, status endpoint. |
|
|
322
|
+
| `router` | One request: class, facts, advice, policy, rewrite, session memory, log. |
|
|
323
|
+
| `facts` | Prompt, recent turns, continuation, failure signal, resend key. |
|
|
324
|
+
| `jev` | One bounded Jev request with a Choice (tier) and a Noul (continuation). |
|
|
325
|
+
| `policy` | Switching policy, context fit, cold-write guard. |
|
|
326
|
+
| `cost` | Cache key, warmth, input cost, switching tax, shadow estimate. |
|
|
327
|
+
| `rewrite` | Model, effort, thinking and output limit for each model family. |
|
|
328
|
+
| `sse` | `usage` from SSE and JSON responses. |
|
|
329
|
+
| `idle` | When the daemon can exit. |
|
|
330
|
+
| `status` | Status snapshot, status line segment, `/router:status` report. |
|
|
331
|
+
| `store` | Session memory, `decisions.jsonl`, rotation. |
|
|
332
|
+
| `runtime` | Configuration path (anchored under `~/.claude`), data directory, flags, gateway process check. |
|
|
333
|
+
|
|
334
|
+
## Gateway lifecycle
|
|
335
|
+
|
|
336
|
+
```mermaid
|
|
337
|
+
stateDiagram-v2
|
|
338
|
+
direction LR
|
|
339
|
+
[*] --> Running: hook starts it, detached
|
|
340
|
+
Running --> Draining: SIGTERM
|
|
341
|
+
Draining --> Stopped: streams done, max 10 min
|
|
342
|
+
Running --> Stopped: 2 h idle
|
|
343
|
+
Stopped --> Running: next prompt
|
|
344
|
+
```
|
|
248
345
|
|
|
346
|
+
- The `SessionStart` and `UserPromptSubmit` hooks run `ensure-gateway.mjs`.
|
|
347
|
+
It starts a missing gateway and sends `SIGTERM` to an older version. It
|
|
348
|
+
never replaces a newer version.
|
|
349
|
+
- On `SIGTERM`, the gateway releases the port at once. A new gateway can
|
|
350
|
+
start while the old one finishes its streams.
|
|
351
|
+
- The idle exit waits for two hours without requests. It also waits while a
|
|
352
|
+
request is open or a turn waits for a tool result, for example a
|
|
353
|
+
permission prompt. A waiting turn stops counting after one day.
|
|
354
|
+
- A second daemon on a busy port exits quietly.
|
|
355
|
+
|
|
356
|
+
## Failure handling
|
|
357
|
+
|
|
358
|
+
| Failure | Behavior |
|
|
359
|
+
| ------------------------------ | ---------------------------------------------------------------- |
|
|
360
|
+
| Anthropic 429, 529, 5xx | Pass through with `Retry-After`. Claude Code owns retries. |
|
|
361
|
+
| Upstream reset during a stream | The client connection resets, and Claude Code retries at once. |
|
|
362
|
+
| Client leaves (Esc) | The gateway stops the upstream request. |
|
|
363
|
+
| Jev error or timeout | One retry inside the 1.5 s budget. Then the current route stays. |
|
|
364
|
+
| Three Jev failures in a row | New turns skip Jev for one minute, then try once. |
|
|
365
|
+
| Routing error | The baseline tier serves. The alias never goes to Anthropic. |
|
|
366
|
+
| Disk error | Logged. The routing decision stays. |
|
|
367
|
+
| Unexpected exception | Logged. The daemon continues. |
|
|
368
|
+
|
|
369
|
+
## Security and privacy
|
|
370
|
+
|
|
371
|
+
- The gateway listens on `127.0.0.1` only. It refuses a non-loopback `Host`
|
|
372
|
+
(DNS rebinding) and any request with an `Origin` header, also from a page on
|
|
373
|
+
another loopback port: Claude Code sends none.
|
|
374
|
+
- Jev receives the prompt and the text of the six turns before it, up to
|
|
375
|
+
1,200 characters each. A longer text keeps about 600 characters from its
|
|
376
|
+
start and from its end. Tool results and system reminders are not sent.
|
|
377
|
+
- The Jev key is in the macOS Keychain. The status endpoint shows only whether
|
|
378
|
+
a key is set.
|
|
379
|
+
- `decisions.jsonl` has no prompt text.
|
|
380
|
+
|
|
381
|
+
## Observability
|
|
382
|
+
|
|
383
|
+
- `GET /router/status?session=<id>` gives the routes and the last turn of a
|
|
384
|
+
session. The status line and `/router:status` read it.
|
|
385
|
+
- `decisions.jsonl` has one line for each decision (with the model and the
|
|
386
|
+
effort sent), each response, each failed routed turn (`failed`, with the
|
|
387
|
+
status) and each routing error (`reason: error`, without the message). It
|
|
388
|
+
has no prompt text.
|
|
389
|
+
- `gateway.log` has the daemon events.
|
|
390
|
+
|
|
391
|
+
The [user guide](user-guide.md#read-the-status-line) explains how to read
|
|
392
|
+
them. The [data directory](configuration.md#data-directory) gives their
|
|
393
|
+
retention.
|
|
394
|
+
|
|
395
|
+
## Release
|
|
396
|
+
|
|
397
|
+
```mermaid
|
|
398
|
+
flowchart LR
|
|
399
|
+
TAG["Signed tag<br/>vX.Y.Z on main"] --> VER["Check signature,<br/>tag = package.json"]
|
|
400
|
+
VER --> TEST["Lint, tests,<br/>pack dry run"]
|
|
401
|
+
TEST --> NPM["npm publish<br/>with provenance"]
|
|
402
|
+
NPM --> GH["GitHub release<br/>named by the tag"]
|
|
403
|
+
|
|
404
|
+
classDef step fill:#eef2ff,stroke:#6366f1,color:#1e1b4b
|
|
405
|
+
class TAG,VER,TEST,NPM,GH step
|
|
249
406
|
```
|
|
250
|
-
|
|
251
|
-
|
|
252
|
-
|
|
253
|
-
|
|
254
|
-
|
|
255
|
-
|
|
256
|
-
|
|
257
|
-
|
|
258
|
-
|
|
259
|
-
|
|
260
|
-
|
|
261
|
-
|
|
262
|
-
|
|
263
|
-
|
|
264
|
-
|
|
265
|
-
|
|
266
|
-
lib/router.mjs orchestration for one request, session memory
|
|
267
|
-
lib/gateway.mjs HTTP passthrough and rewrite
|
|
268
|
-
lib/store.mjs files: configuration, memory, decisions.jsonl
|
|
269
|
-
lib/status.mjs status snapshot, status line segment, report
|
|
270
|
-
skills/<tier>/SKILL.md manual pins (model and effort frontmatter)
|
|
271
|
-
skills/setup/SKILL.md writes model, base URL, picker row, status line
|
|
272
|
-
skills/status/SKILL.md /router:status
|
|
273
|
-
test/ node:test, builders in helpers.mjs
|
|
407
|
+
|
|
408
|
+
- The repository root is the plugin and the npm package
|
|
409
|
+
`@alexeiled/claude-router`.
|
|
410
|
+
- Claude Code installs the plugin from Git: the marketplace source is
|
|
411
|
+
`github`. An npm source fails with npm 12 (`EALLOWREMOTE`).
|
|
412
|
+
- `claude plugin update` compares the version in `.claude-plugin/plugin.json`.
|
|
413
|
+
Each release changes it together with `package.json`.
|
|
414
|
+
- `ci.yml` runs the checks and the tests for each push and pull request.
|
|
415
|
+
|
|
416
|
+
To work on the code:
|
|
417
|
+
|
|
418
|
+
```sh
|
|
419
|
+
npm install
|
|
420
|
+
git config --local core.hooksPath scripts/git-hooks # Biome, tests and Gitleaks before commit and push
|
|
421
|
+
npm test && npm run check # node:test, Biome
|
|
422
|
+
claude --plugin-dir . --model jev-router # with ANTHROPIC_BASE_URL and TYPESAFE_API_KEY
|
|
274
423
|
```
|
|
275
424
|
|
|
276
|
-
##
|
|
277
|
-
|
|
278
|
-
- The exit from plan mode is not visible in the request body. The policy has
|
|
279
|
-
no boundary rule.
|
|
280
|
-
- The thresholds are not tuned. The estimated and the observed cache reads in
|
|
281
|
-
`decisions.jsonl` are the input for the tuning.
|
|
282
|
-
- The gateway reads the configuration once. A reload without a restart is not
|
|
283
|
-
implemented.
|
|
284
|
-
- Without `CLAUDE_CODE_GATEWAY_HINT_HEADERS=1`, the gateway guesses side
|
|
285
|
-
requests from the body; the guesses miss some. A missed side request with
|
|
286
|
-
a short history also counts as a history break.
|
|
287
|
-
- `x-claude-code-agent-type` is logged, not used: no policy rule reads it yet.
|
|
288
|
-
- Does a downgrade that the shadow estimate says never repays deserve a rule,
|
|
289
|
-
and does the switching tax belong in the upgrade bar? A replay of
|
|
290
|
-
`decisions.jsonl` against the `shadow` and `observed` lines decides.
|
|
291
|
-
- Effort changes how much a model writes. The shadow estimate uses the last
|
|
292
|
-
output size for both routes.
|
|
293
|
-
- Whether tools and system survive an effort change is model-specific; the
|
|
294
|
-
gateway counts the whole prefix as lost, an upper bound.
|
|
425
|
+
## Known limits
|
|
295
426
|
|
|
296
|
-
|
|
427
|
+
- The request body does not show the exit from plan mode.
|
|
428
|
+
- The thresholds are start values. The `observed` and `shadow` lines in
|
|
429
|
+
`decisions.jsonl` are the input for tuning.
|
|
430
|
+
- The gateway reads its configuration once. A change needs a restart.
|
|
431
|
+
- Without the hint headers, the gateway guesses side requests. A missed side
|
|
432
|
+
request with a short history counts as a history break.
|
|
433
|
+
- `x-claude-code-agent-type` goes to the log only.
|
|
434
|
+
- The shadow estimate uses the last output size for both routes, but effort
|
|
435
|
+
changes how much a model writes.
|
|
297
436
|
|
|
298
|
-
|
|
299
|
-
The marketplace `alexei-led-claude-router` in `.claude-plugin/marketplace.json`
|
|
300
|
-
points at this GitHub repository (`"source": "github"`), so Claude Code installs the
|
|
301
|
-
plugin from git and never calls npm. An npm source
|
|
302
|
-
fails under npm 12: Claude Code fetches the tarball URL, and npm 12 refuses
|
|
303
|
-
remote tarballs by default (`EALLOWREMOTE`). `claude plugin update` compares
|
|
304
|
-
the `version` in `.claude-plugin/plugin.json`, so each release bumps it. Local
|
|
305
|
-
development uses `claude --plugin-dir .`.
|
|
306
|
-
|
|
307
|
-
A release is a signed annotated tag `v<version>` on `main`, where the version
|
|
308
|
-
matches `package.json`. The `release.yml` workflow makes sure that the tag is
|
|
309
|
-
signed and on `main`, runs the checks and the tests, publishes to npm with
|
|
310
|
-
trusted publishing (`npm publish --provenance`, no token), and creates the
|
|
311
|
-
GitHub release. The `ci.yml` workflow runs the checks and the tests for each
|
|
312
|
-
push and pull request on `main`.
|
|
437
|
+
A day of real usage is in [Evaluation](evaluation.md).
|