@alexeiled/claude-router 0.6.1 → 0.7.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,312 +1,437 @@
1
1
  # Architecture
2
2
 
3
- Each decision has the date when the owner took it.
4
-
5
- ## Goal
6
-
7
- Select a model and an effort level for each user turn, from the models that
8
- Claude Code offers, with one TypeSafe Jev Choice. Stay inside the Claude Code
9
- terminal interface.
10
-
11
- ## Mechanism: local gateway (2026-09-22)
12
-
13
- Claude Code runs with `--model jev-router`. `ANTHROPIC_BASE_URL` points at a
14
- gateway on `127.0.0.1`. The gateway sends each request to `api.anthropic.com`
15
- unchanged, except a request whose `model` is the alias. For that request, the
16
- gateway reads facts from the body, asks Jev once for each new user turn, runs
17
- the policy, and changes `model`, `output_config.effort` and `thinking`.
18
- Responses go through unchanged. The gateway reads `usage` from the response to
19
- get the context size, the cache reads and the cache TTL.
20
-
21
- For a model without thinking (Haiku), the gateway also removes the
22
- `clear_thinking_*` edits from `context_management`, because the API rejects
23
- them without thinking. The gateway never changes `system`, `tools` or
24
- `messages`. Thinking and prompt caching work as if Claude Code talked to
25
- Anthropic directly. Claude Code documents this gateway mode, including the
26
- OAuth value for a claude.ai login. See
27
- [llm-gateway](https://code.claude.com/docs/en/llm-gateway) and
28
- [protocol](https://code.claude.com/docs/en/llm-gateway-protocol).
29
-
30
- Module dependencies point in one direction: `gateway.mjs` (HTTP) →
31
- `router.mjs` (orchestration) → `facts`, `jev`, `policy` → `cost`, `rewrite`,
32
- `store`. The modules `facts`, `cost`, `rewrite`, `sse` and `policy` are pure.
33
- Only `store` writes files. The Jev transport is injected.
34
-
35
- The `SessionStart` hook of the plugin starts the gateway when the port does not
36
- answer. `/router:setup` writes `model`, `ANTHROPIC_BASE_URL` and the picker row to
37
- the user settings once. A plugin cannot set them by itself.
38
-
39
- The gateway serves `GET /router/status?session=<id>`: the routes and the last
40
- turn of the session, never the key. `/router:status` and the status line
41
- wrapper read it. Claude Code shows only the alias as the model, so the wrapper
42
- is the only place where the real model of the turn is visible.
43
-
44
- ### Why not the native skill path
45
-
46
- The first design used a `UserPromptSubmit` hook that asked Claude to call a
47
- tier skill, relying on the skill's frontmatter `model:` and `effort:` to serve
48
- the rest of the turn. On Claude Code 2.1.278 that only works for a skill the
49
- user types (`/router:medium …`); the same skill called by Claude through the
50
- Skill tool does not change the model — the transcript records
51
- `attributionSkill`, but the session model answers. So `/router:<tier>` skills
52
- stay as manual pins, and the gateway is the only path that routes turns Claude
53
- calls on its own.
3
+ claude-router selects a model and an effort level for each Claude Code turn.
4
+ A local gateway changes each request before it goes to Anthropic. TypeSafe Jev
5
+ advises a tier, and a local policy makes the decision.
6
+
7
+ - [System context](#system-context)
8
+ - [Tiers](#tiers)
9
+ - [Request flow](#request-flow)
10
+ - [Request classes](#request-classes)
11
+ - [Switching policy](#switching-policy)
12
+ - [Cache and cost model](#cache-and-cost-model)
13
+ - [Modules](#modules)
14
+ - [Gateway lifecycle](#gateway-lifecycle)
15
+ - [Failure handling](#failure-handling)
16
+ - [Security and privacy](#security-and-privacy)
17
+ - [Observability](#observability)
18
+ - [Release](#release)
19
+ - [Known limits](#known-limits)
20
+
21
+ ## System context
22
+
23
+ ```mermaid
24
+ flowchart TB
25
+ subgraph machine["Your machine"]
26
+ direction LR
27
+ CC["Claude Code<br/>model: jev-router"]
28
+ HOOK["Plugin hooks"]
29
+ UI["Status line<br/>/router:status"]
30
+ GW["Gateway<br/>127.0.0.1:43170"]
31
+ DATA[("Data directory<br/>sessions · decisions.jsonl")]
32
+ end
33
+ subgraph remote["Remote services"]
34
+ direction LR
35
+ API["api.anthropic.com"]
36
+ JEV["TypeSafe Jev"]
37
+ end
38
+
39
+ CC <-->|"Messages API"| GW
40
+ HOOK -.->|"start or replace"| GW
41
+ UI -.->|"GET /router/status"| GW
42
+ GW -->|"prompt and<br/>recent turns"| JEV
43
+ GW <-->|"changed request,<br/>unchanged response"| API
44
+ GW --> DATA
45
+
46
+ classDef ext fill:#eef2ff,stroke:#6366f1,color:#1e1b4b
47
+ classDef core fill:#ecfdf5,stroke:#059669,color:#064e3b
48
+ classDef store fill:#fff7ed,stroke:#ea580c,color:#431407
49
+ class JEV,API ext
50
+ class GW core
51
+ class DATA store
52
+ style machine fill:#f8fafc,stroke:#94a3b8,color:#0f172a
53
+ style remote fill:#f8fafc,stroke:#94a3b8,color:#0f172a
54
+ ```
55
+
56
+ - `/router:setup` points `ANTHROPIC_BASE_URL` at the gateway and sets the
57
+ model to `jev-router[1m]`. One gateway serves all sessions on the machine.
58
+ - The gateway changes only requests for the alias. It sets `model`,
59
+ `output_config.effort` and `thinking`. For a small model, it also limits
60
+ `max_tokens` and removes the 1M context beta header and the thinking edits.
61
+ - Claude Code builds each request for the model that the alias behaves as
62
+ (Opus). A model without one of its [request features](configuration.md#models)
63
+ gets the request without it, the way Claude Code retries after a 400:
64
+ - no `mid-conversation-tool-changes` (Sonnet, Haiku): `tool_addition` and
65
+ `tool_removal` blocks go; their cache breakpoint moves to the block before;
66
+ - no `per-turn-control` (Sonnet, Haiku): the beta header and the
67
+ `output_config` on messages (the effort of each turn) go;
68
+ - no `mid-conversation-system` (Haiku): each `system` message becomes a
69
+ `<system-reminder>` text in the user message next to it.
70
+ The same history adapts the same way on every turn, so the cached prefix
71
+ holds.
72
+ - On a model with `per-turn-control`, the effort of each turn overrides the
73
+ request's effort. A route with its own effort (`medium`, `high`) therefore
74
+ sets it in every message's `output_config` too.
75
+ - The gateway never changes `system` or `tools`, and never changes `messages`
76
+ for a model that takes all the features. Prompt caching and a claude.ai
77
+ login work as with a direct connection.
78
+ - Responses go back unchanged. The gateway reads `usage` from them.
79
+
80
+ **Why a gateway.** Only a gateway can route every turn. The tier skills are
81
+ manual pins. Claude Code 2.1.x switches the model from their `model:`
82
+ frontmatter for `sonnet` and `opus`, and that turn passes through. For `haiku`
83
+ it sends the turn to the alias; the gateway reads the `/router:micro` command
84
+ in the prompt and serves the turn on `micro`, reason `pinned`. After a pin, the
85
+ next prompt is decided from the route before the pin.
54
86
 
55
87
  ## Tiers
56
88
 
57
- | Tier | model | effort | id sent to Anthropic |
58
- | ------ | ------ | ------- | -------------------- |
59
- | high | opus | xhigh | claude-opus-5-5 |
60
- | medium | opus | high | claude-opus-5-5 |
61
- | low | sonnet | as sent | claude-sonnet-5 |
62
- | micro | haiku | none | claude-haiku-4-5 |
89
+ Four tiers, from `micro` (Haiku) to `high` (Opus at `xhigh`), map to a model
90
+ and an effort. The [README](../README.md#how-jev-selects-a-tier) shows the
91
+ work that Jev selects each tier for. The [routes](configuration.md#routes)
92
+ set the mapping.
93
+
94
+ - `low` is the baseline.
95
+ - `medium` and `high` use the same model at two efforts.
96
+ - The gateway lowers an effort to a level that the model accepts. Haiku
97
+ accepts no effort, so it runs without thinking.
98
+
99
+ ## Request flow
100
+
101
+ A new user turn:
102
+
103
+ ```mermaid
104
+ sequenceDiagram
105
+ autonumber
106
+ participant CC as Claude Code
107
+ participant GW as Gateway
108
+ participant R as Router
109
+ participant J as Jev
110
+ participant A as Anthropic
111
+
112
+ CC->>GW: POST /v1/messages (jev-router)
113
+ GW->>R: route(body, session, hints)
114
+ Note over R: facts from body and session memory
115
+ R->>J: prompt and last 6 turns
116
+ J-->>R: tier probabilities, continuation
117
+ Note over R: switching policy selects the tier
118
+ R-->>GW: body with model, effort, thinking
119
+ GW->>A: request
120
+ A-->>GW: response stream
121
+ GW-->>CC: same bytes
122
+ GW->>R: usage: tokens, cache reads, TTL
123
+ Note over R: session memory, decisions.jsonl
124
+ ```
63
125
 
64
- The gateway lowers the effort to a level the model's `efforts` list accepts
65
- (see [configuration](configuration.md#models)); Haiku accepts none, so it gets
66
- no effort and no adaptive thinking. The ids are configuration.
126
+ A tool continuation and a resent request skip the Jev call. They keep the
127
+ route of their turn.
67
128
 
68
129
  ## Request classes
69
130
 
70
- - New user turn (the last message has no `tool_result`): Jev and the policy.
71
- - Tool continuation: the route of the turn, without a Jev call.
72
- - The classes `auxiliary` and `compaction` from the header
73
- `x-claude-code-request-class`: `auxiliaryTier`, and the memory stays
74
- unchanged. `main`, `subagent` and `workflow` get routing. Without the header,
75
- a body with `thinking: disabled` and a `format` is a side request.
76
- - A subagent (header `x-claude-code-agent-id`): routing with its own memory,
77
- under the key `<session>.<agent id>`. The main conversation keeps its route.
78
- - Any other `model`: unchanged. This covers `/router:<tier>` pins,
79
- `/model` changes and subagents with their own model.
80
- - A resent request (the same history length and the same last message): the
81
- route of the turn, without a Jev call or a vote. Claude Code resends after a
82
- 429, a 529 or a dropped connection.
83
- - A side endpoint with the alias, such as `/v1/messages/count_tokens`: the
84
- model of the session's last route. Only `POST /v1/messages` is a turn.
85
- - A history break: a main request with fewer messages than the last one (a
86
- compaction or a rewind), or the header `x-claude-code-context-compacted`.
87
- The gateway drops the cached prefixes, the votes and the escalation hold,
88
- then routes the request as usual. The route stays until the next decision.
131
+ ```mermaid
132
+ flowchart TD
133
+ IN(["Request"]) --> ALIAS{"model is<br/>jev-router?"}
134
+ ALIAS -- no --> PASS["Pass through unchanged"]
135
+ ALIAS -- yes --> MSG{"POST<br/>/v1/messages?"}
136
+ MSG -- no --> LAST["Last route of the session"]
137
+ MSG -- yes --> SIDE{"Side request?"}
138
+ SIDE -- yes --> AUX["auxiliaryTier<br/>memory unchanged"]
139
+ SIDE -- no --> BRK{"History break?"}
140
+ BRK -- yes --> RESET["Reset cache estimates,<br/>votes and hold"]
141
+ BRK -- no --> TOOL
142
+ RESET --> TOOL{"tool_result or<br/>resent request?"}
143
+ TOOL -- yes --> KEEP["Route of the turn"]
144
+ TOOL -- no --> POL["Jev and switching policy"]
145
+ KEEP --> FIT["Fit the context window"]
146
+ POL --> FIT
147
+ FIT --> OUT(["Change and forward"])
148
+
149
+ classDef stop fill:#f1f5f9,stroke:#64748b,color:#0f172a
150
+ classDef act fill:#ecfdf5,stroke:#059669,color:#064e3b
151
+ class PASS,LAST,AUX stop
152
+ class RESET,KEEP,POL,FIT act
153
+ ```
89
154
 
90
- ## Failure handling
155
+ | Check | How the gateway decides |
156
+ | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
157
+ | Pass through | Any other model: `/model`, subagents with their own model. |
158
+ | Pin | The current message starts with the `/router:<tier>` command block. That tier serves the turn and its tool calls, reason `pinned`. No Jev call. |
159
+ | Not a turn | `count_tokens` and other endpoints get the last route of the session. |
160
+ | Side request | Header `x-claude-code-request-class` is `auxiliary` or `compaction`. Without the header: thinking disabled or an output format in the body. |
161
+ | History break | Fewer messages than the last main request (a compaction or a rewind), or the header `x-claude-code-context-compacted`. |
162
+ | Resent request | Same last user message at the same position. Claude Code resends after a 429, a 529, a dropped stream, or a 400 that it answers by dropping a feature. No Jev call, no vote. |
163
+ | System messages | Claude Code sends hook output and tool additions as `system` messages after the prompt. The turn is the last user or assistant message; Jev never receives `system` text. |
164
+ | Context fit | The route moves to a model whose window holds the next context at 80% fill. Reason `context-fit`. |
165
+ | Session memory | Key `x-claude-code-session-id`. A subagent adds its agent id: `<session>.<agent id>`. Its turns do not change the route of the main conversation. |
166
+
167
+ ## Switching policy
168
+
169
+ ```mermaid
170
+ flowchart TD
171
+ S(["New user turn"]) --> ERR{"Same error twice,<br/>edit between?"}
172
+ ERR -- yes --> ESC["escalation<br/>one tier up for 2 turns"]
173
+ ERR -- no --> HOLD{"Hold active?"}
174
+ HOLD -- yes --> R1["hold"]
175
+ HOLD -- no --> ADV{"Jev answered?"}
176
+ ADV -- no --> R2["no-advice"]
177
+ ADV -- yes --> CONT{"Continues<br/>the task?"}
178
+ CONT -- yes --> R3["continuation"]
179
+ CONT -- no --> DIR{"Jev choice vs<br/>current tier"}
180
+ DIR -- "same or uncertain" --> R4["same-tier<br/>uncertain"]
181
+ DIR -- higher --> GUARD{"Cold-write<br/>guard blocks?"}
182
+ GUARD -- yes --> CG["cash-gate<br/>strongest plan tier"]
183
+ GUARD -- no --> JMP{"2 tiers up,<br/>U ≥ 0.95?"}
184
+ JMP -- yes --> JUMP["jump"]
185
+ JMP -- no --> UPQ{"2 votes,<br/>U ≥ bar?"}
186
+ UPQ -- yes --> UP["upgrade"]
187
+ UPQ -- no --> R5["upgrade-pending"]
188
+ DIR -- lower --> DNQ{"2 votes,<br/>D ≥ bar?"}
189
+ DNQ -- yes --> DOWN["downgrade"]
190
+ DNQ -- no --> R6["downgrade-pending"]
191
+
192
+ classDef stay fill:#f1f5f9,stroke:#64748b,color:#0f172a
193
+ classDef up fill:#fef3c7,stroke:#d97706,color:#451a03
194
+ classDef down fill:#e0f2fe,stroke:#0284c7,color:#082f49
195
+ class R1,R2,R3,R4,R5,R6 stay
196
+ class ESC,CG,JUMP,UP up
197
+ class DOWN down
198
+ ```
199
+
200
+ Each box is the `reason` that the log and the status line show. Grey boxes
201
+ keep the current route. The cold-write guard serves the strongest plan tier
202
+ only when that tier is above the current route.
203
+
204
+ - `U` is the Jev probability mass above the current tier. `D` is the mass at
205
+ or below the candidate. The `uncertain` mass supports neither.
206
+ - The upgrade bar grows with the switching tax:
207
+ `bar = 0.75 + 0.15 × tax / (tax + $0.50)`. It stays between 0.75 and 0.90.
208
+ - The downgrade bar grows the same way, from the other side: a candidate
209
+ colder than the incumbent pays a cache write, not just a smaller model.
210
+ `bar = 0.90 + 0.08 × tax / (tax + $0.50)`. It stays between 0.90 and 0.98.
211
+ - A prompt continues the task when the Jev continuation answer is 0.7 or
212
+ more. A continuing prompt never votes for a downgrade.
213
+ - Upgrades have no cooldown. Plan, then execute, then hard work again is a
214
+ valid sequence.
215
+ - A failure signal is two `tool_result` errors with the same text and an edit
216
+ between them, in the last 40 messages. Each signal escalates once.
217
+ - The thresholds are defaults in [configuration](configuration.md#policy).
218
+
219
+ ## Cache and cost model
220
+
221
+ The gateway keeps one cache record for each model and effort, for example
222
+ `claude-opus-5-5@xhigh`. A change of effort replaces the cached messages, so
223
+ `medium` and `high` are two caches.
224
+
225
+ ```mermaid
226
+ stateDiagram-v2
227
+ direction LR
228
+ [*] --> unknown
229
+ unknown --> warm: response recorded
230
+ warm --> expired: TTL minus 30 s passed
231
+ expired --> warm: response recorded
232
+ warm --> unknown: history break or context shrank
233
+ expired --> unknown: history break or context shrank
234
+ ```
91
235
 
92
- One gateway serves every Claude Code session on the machine, so a failure in one
93
- request must not reach the others.
94
-
95
- - Anthropic errors (429, 529, 5xx) pass through unchanged, with `Retry-After`.
96
- Claude Code owns retries and backoff; a second retry layer in the gateway
97
- would multiply attempts and cannot replay a stream that has started.
98
- - The response is relayed with `pipeline()`. An upstream reset destroys the
99
- client response, so Claude Code sees a reset and retries at once. A client
100
- that leaves (Esc) destroys the upstream request, so the model stops
101
- generating an answer that nobody reads.
102
- - Disk errors while the gateway saves session memory or the decision log are
103
- logged. The routing decision stands.
104
- - Jev: one retry on a network error or a transient status, after
105
- `Retry-After` when it fits the 1.5 s budget. After three failures in a row,
106
- new turns skip Jev for a minute, then try once per pause.
107
- - The daemon logs a stray exception instead of exiting. On `SIGTERM` it
108
- releases the port at once and finishes open streams for up to 10 minutes.
109
- A second daemon on a busy port exits quietly.
110
- - The `SessionStart` and `UserPromptSubmit` hooks start the gateway when the
111
- port does not answer, and replace a gateway older than the plugin. They never
112
- replace a newer one.
113
- - The gateway exits after `gateway.idleShutdownMs` (two hours) without
114
- requests. Claude Code holds no connection open between requests, so the
115
- gateway cannot tell a closed session from an idle one; time is the signal.
116
- Two cases keep it running: a request in flight, and a turn whose last
117
- response asked for a tool (`stop_reason: tool_use`), such as an unanswered
118
- permission prompt. The answer to that prompt reaches the gateway without a
119
- new prompt, so without the hook that would start it again. A session that
120
- died mid-turn stops counting after a day. Two hours outlast `/loop` wakeups
121
- and Monitor waits, which also arrive without a prompt.
122
- - The gateway refuses requests with a non-loopback `Host` (DNS rebinding) or a
123
- web `Origin` (cross-site requests from a browser).
124
-
125
- ## Cache and cost inputs
126
-
127
- All inputs come from the traffic of the gateway. The gateway does not read
128
- transcripts.
129
-
130
- | Input | Source |
131
- | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
132
- | Context of the last request, cache reads, output | `usage` in the response (`message_start` and `message_delta`) |
133
- | Granted TTL | `usage.cache_creation.ephemeral_1h_input_tokens` or the `5m` field |
134
- | Cache identity | The model id and the effort the gateway sent (`claude-opus-5-5@xhigh`). An effort change rewrites the messages cache, so each effort is its own cache. |
135
- | Cache warmth | The time of the last response for that cache, plus the TTL, minus 30 s. `unknown` when no response since the session started or the history broke. |
136
- | Reusable prefix | The context plus the output at the last response for that cache. Cleared by a history break, or when the context shrinks by more than 20% (context editing). |
137
- | Failure signal | Two `tool_result` blocks with `is_error` and the same signature, with an edit tool call between them |
138
- | Continuation | The last message contains a `tool_result`. For a new prompt, a Jev Noul answers "does this prompt continue the task". |
139
-
140
- Prices are a list-price table in the configuration (see
141
- [configuration](configuration.md#models)). `modelPricing` is a managed setting
142
- and is not readable. The switching tax for a candidate `c` against the current
143
- route `i` is:
236
+ The switching tax compares the input cost of the next request on the
237
+ candidate and on the current route:
144
238
 
239
+ ```text
240
+ input_cost = P_read × W + P_write × (N − W)
241
+ tax = max(0, input_cost(candidate) − input_cost(current))
145
242
  ```
146
- input_cost(m) = P_read(m) * W_m + P_write(m) * (N - W_m)
147
- tax = max(0, input_cost(c) - input_cost(i))
243
+
244
+ | Symbol | Source |
245
+ | --------- | ------------------------------------------------------------------------------------------------------ |
246
+ | `N` | Next context: tokens and output of the last response. |
247
+ | `W` | Warm prefix of the candidate cache. Zero when the cache is `expired` or `unknown`. |
248
+ | `P_read` | List price for cache reads. |
249
+ | `P_write` | Input list price × 1.25 (5-minute TTL) or × 2 (1-hour TTL). The TTL comes from `usage.cache_creation`. |
250
+
251
+ - **Upper bound.** An effort change counts the whole prefix as lost. On some
252
+ models, the tools and the system prompt stay in the cache.
253
+ - **Context editing.** A context that shrinks by more than 20% with no fewer
254
+ messages resets the cache records only. The votes stay.
255
+ - **Cold-write guard.** A model with `billing: "credits"` bills cash. When its
256
+ cache is not warm, `policy.cashCapUsd` limits the first cache write. It is
257
+ not a budget for the turn. No default model bills credits.
258
+ - **Prices.** List prices in USD per million tokens. For plan models, the
259
+ dollars only give an order between models. A test pins the defaults to
260
+ `test/fixtures/list-prices.json`, which names its source and date.
261
+ - **Shadow estimate.** When the Jev choice differs from the current route, the
262
+ log records the cost of the next turn, the cost of each later turn with
263
+ output, and the turns until a switch repays its cost. The policy does not
264
+ read it.
265
+
266
+ ## Modules
267
+
268
+ ```mermaid
269
+ flowchart TD
270
+ subgraph scripts["scripts/ · entry points"]
271
+ D["gateway.mjs<br/>daemon"]
272
+ E["ensure-gateway.mjs<br/>hook"]
273
+ SL["statusline.mjs<br/>status line"]
274
+ SC["status.mjs<br/>/router:status"]
275
+ end
276
+ subgraph lib["lib/"]
277
+ GW["gateway<br/>HTTP, relay"]
278
+ IDLE["idle"]
279
+ SSE["sse"]
280
+ RT["router<br/>orchestration"]
281
+ FA["facts"]
282
+ JEV["jev"]
283
+ POL["policy"]
284
+ COST["cost"]
285
+ RW["rewrite"]
286
+ STA["status"]
287
+ STORE[("store")]
288
+ RUN["runtime"]
289
+ end
290
+
291
+ D -->|"creates, injects router"| GW
292
+ D --> RT & RUN & IDLE & STA & STORE
293
+ GW --> IDLE & SSE & STA
294
+ GW -.->|"side-request test"| RT
295
+ RT --> FA & JEV & POL & COST & RW & STORE
296
+ POL --> COST --> RW
297
+ STA --> RW
298
+ E --> RUN & STA & STORE
299
+ SL --> RUN & STA
300
+ SC --> RUN & STA
301
+ RUN --> STORE
302
+
303
+ classDef pure fill:#ecfdf5,stroke:#059669,color:#064e3b
304
+ classDef io fill:#fff7ed,stroke:#ea580c,color:#431407
305
+ class FA,POL,COST,RW,SSE pure
306
+ class STORE,GW,JEV io
307
+ style scripts fill:#f8fafc,stroke:#94a3b8,color:#0f172a
308
+ style lib fill:#f8fafc,stroke:#94a3b8,color:#0f172a
148
309
  ```
149
310
 
150
- The subscription economics are not symmetric. Every default model uses the plan
151
- limits, and dollars give the order between them. A model with `billing:
152
- "credits"` bills cash, on the 5m TTL, and behind the gateway without the
153
- consent prompt of Claude Code. `policy.cashCapUsd` is a cold-write guard: it
154
- caps the estimated first cache write of an automatic route to such a model
155
- when its cache is not warm. It is not a budget: a warm cache passes, and
156
- output is not counted. A Claude Code turn starts at about 100k tokens (system
157
- prompt and tool definitions), so the guard binds on the first switch, not
158
- later. No default model bills credits; the guard stays for configurations that
159
- add one.
160
-
161
- ## Switching policy v0
162
-
163
- Agreed with Codex on 2026-09-22. The thresholds are start values.
164
-
165
- 1. Pins win. A request for a real model id goes through unchanged.
166
- 2. To stay is a decision. The current route is the route that served the last
167
- turn. A prompt that continues the task keeps it and never votes for a
168
- downgrade.
169
- 3. Escalation floor. Two failed repairs of the same failure signature, with an
170
- edit between them, move the route one tier up. The route holds for two
171
- turns, once for each signature.
172
- 4. Votes keep the Jev probabilities. `U` is the mass above the current tier.
173
- `D` is the mass at or below the candidate. The `uncertain` mass supports
174
- neither. An upgrade needs two consecutive votes above the current tier and
175
- `U >= 0.75 + 0.15 * tax / (tax + 0.5)`. A jump of two tiers with
176
- `U >= 0.95` skips the delay. A downgrade needs `D >= 0.90` and two
177
- consecutive votes.
178
- 5. Cold-write guard (reason `cash-gate`). An automatic route to a `credits`
179
- model needs a warm cache, or a cold write below the cap. Otherwise the
180
- strongest `plan` tier serves.
181
- 6. No cooldown on upgrades. Plan, then execute, then hard again is sometimes
182
- the correct routing. The log separates reversals from real changes in the
183
- required capability.
184
- 7. Logs. The tier, the reason, the estimates, and the observed model, tokens
185
- and cache reads of each routed response. The memory changes only from
186
- responses that the gateway sent.
187
-
188
- ## Cache identity, history breaks and shadow economics (2026-09-23)
189
-
190
- From a design review with the architect of pi-model-router, the Pi router
191
- that uses the same Jev tiers.
192
-
193
- - Cache identity is the model and the effort. A top-level effort change
194
- invalidates the messages cache; the cache-preserving per-message effort is
195
- not available on Opus 5.5. Before 0.6.1 the key was the model alone, so
196
- `medium` ↔ `high` (Opus at `high` and `xhigh`) looked like a free switch
197
- between warm caches.
198
- - A history break (fewer messages, or the compaction header) drops the cached
199
- prefixes, the votes and the escalation hold: they were about turns that are
200
- gone. Context editing shrinks the context but keeps the messages; it drops
201
- only the prefixes. The gateway has no branch id, so it does not restore state
202
- from before a rewind; `unknown` is the honest cache state after one.
203
- - `/router:status` and the status line show the reason; the report adds the
204
- estimate. The dollars are list prices, a list-price equivalent for plan
205
- models.
206
- - Shadow economics in `decisions.jsonl`, not read by the policy: for a turn
207
- where Jev's choice differs from the current route, the extra cost of the
208
- next turn, the difference for each later turn (input and output), and the
209
- turns until a cheaper route repays its cache write. The owner's first
210
- request (2026-09-22) was to stay while the model works on a warm cache and
211
- to step down once the thinking is done. Rule 2 covers the first half; the
212
- shadow estimate measures the second before any rule acts on it.
213
- - The default prices match `test/fixtures/list-prices.json`, which names its
214
- source and date. A test pins how a tenfold cache-read error changes one
215
- decision: the bar moves within `upgradeBase` and `upgradeBase +
216
- upgradeSlope`, and a confident jump ignores it. That error happened once
217
- (a838f7c).
218
-
219
- Not taken: the agent type as a routing signal; a payback check on downgrades
220
- before shadow data; removing the switching tax from the upgrade bar before a
221
- replay of the logs. The Pi router rejects the tax-to-confidence formula
222
- because it mixes dollars with an uncalibrated probability; the replay decides.
223
-
224
- ## Real-world evaluation (2026-09-23)
225
-
226
- A day of dogfooding this repository on the installed plugin: 31 Claude Code
227
- sessions, 1,578 routed requests, one machine. `decisions.jsonl` holds the
228
- tier, the reason and the token counts for every request — no prompt text.
229
-
230
- ![Share of requests by tier, and the input-token cost of the same traffic repriced at Opus's rates](tier-share.svg)
231
-
232
- 82% of turns never needed more than Sonnet, 10% stayed on Haiku, and 8% needed
233
- Opus. `medium` (Opus at `high` effort) fired once: a confident vote jumps two
234
- tiers straight to `high` instead of stopping at `medium` (switching policy,
235
- rule 4).
236
-
237
- Repricing that same traffic — same tokens, same observed cache reads — at
238
- Opus's rates puts the input-token bill 18.8% above what the router actually
239
- spent. That number covers input tokens only: the price table has no output
240
- price (see [Cache and cost inputs](#cache-and-cost-inputs)), so it cannot say
241
- how much of the real saving is left out — likely more, since Haiku and Sonnet
242
- also bill less per output token than Opus. Answer quality isn't measured
243
- here either.
244
-
245
- One developer, one day: a dogfood snapshot, not a benchmark.
246
-
247
- ## Layout
311
+ Green modules are pure functions. Orange modules do I/O: `gateway` serves
312
+ HTTP, `jev` calls Jev, and in `lib/` `store` is the only module that writes
313
+ files. Outside `lib/`, the hook `ensure-gateway.mjs` also opens `gateway.log`
314
+ for the daemon that it starts. The Jev transport and the router are injected,
315
+ so tests run without a network. `gateway` imports one pure function from
316
+ `router`, the side-request test, not the router itself. Modules that need
317
+ settings or tier names import `config.mjs` (not drawn).
318
+
319
+ | Module | Responsibility |
320
+ | --------- | ---------------------------------------------------------------------------------------------- |
321
+ | `gateway` | HTTP server, loopback checks, relay with `pipeline()`, status endpoint. |
322
+ | `router` | One request: class, facts, advice, policy, rewrite, session memory, log. |
323
+ | `facts` | Prompt, recent turns, continuation, failure signal, resend key. |
324
+ | `jev` | One bounded Jev request with a Choice (tier) and a Noul (continuation). |
325
+ | `policy` | Switching policy, context fit, cold-write guard. |
326
+ | `cost` | Cache key, warmth, input cost, switching tax, shadow estimate. |
327
+ | `rewrite` | Model, effort, thinking and output limit for each model family. |
328
+ | `sse` | `usage` from SSE and JSON responses. |
329
+ | `idle` | When the daemon can exit. |
330
+ | `status` | Status snapshot, status line segment, `/router:status` report. |
331
+ | `store` | Session memory, `decisions.jsonl`, rotation. |
332
+ | `runtime` | Configuration path (anchored under `~/.claude`), data directory, flags, gateway process check. |
333
+
334
+ ## Gateway lifecycle
335
+
336
+ ```mermaid
337
+ stateDiagram-v2
338
+ direction LR
339
+ [*] --> Running: hook starts it, detached
340
+ Running --> Draining: SIGTERM
341
+ Draining --> Stopped: streams done, max 10 min
342
+ Running --> Stopped: 2 h idle
343
+ Stopped --> Running: next prompt
344
+ ```
248
345
 
346
+ - The `SessionStart` and `UserPromptSubmit` hooks run `ensure-gateway.mjs`.
347
+ It starts a missing gateway and sends `SIGTERM` to an older version. It
348
+ never replaces a newer version.
349
+ - On `SIGTERM`, the gateway releases the port at once. A new gateway can
350
+ start while the old one finishes its streams.
351
+ - The idle exit waits for two hours without requests. It also waits while a
352
+ request is open or a turn waits for a tool result, for example a
353
+ permission prompt. A waiting turn stops counting after one day.
354
+ - A second daemon on a busy port exits quietly.
355
+
356
+ ## Failure handling
357
+
358
+ | Failure | Behavior |
359
+ | ------------------------------ | ---------------------------------------------------------------- |
360
+ | Anthropic 429, 529, 5xx | Pass through with `Retry-After`. Claude Code owns retries. |
361
+ | Upstream reset during a stream | The client connection resets, and Claude Code retries at once. |
362
+ | Client leaves (Esc) | The gateway stops the upstream request. |
363
+ | Jev error or timeout | One retry inside the 1.5 s budget. Then the current route stays. |
364
+ | Three Jev failures in a row | New turns skip Jev for one minute, then try once. |
365
+ | Routing error | The baseline tier serves. The alias never goes to Anthropic. |
366
+ | Disk error | Logged. The routing decision stays. |
367
+ | Unexpected exception | Logged. The daemon continues. |
368
+
369
+ ## Security and privacy
370
+
371
+ - The gateway listens on `127.0.0.1` only. It refuses a non-loopback `Host`
372
+ (DNS rebinding) and any request with an `Origin` header, also from a page on
373
+ another loopback port: Claude Code sends none.
374
+ - Jev receives the prompt and the text of the six turns before it, up to
375
+ 1,200 characters each. A longer text keeps about 600 characters from its
376
+ start and from its end. Tool results and system reminders are not sent.
377
+ - The Jev key is in the macOS Keychain. The status endpoint shows only whether
378
+ a key is set.
379
+ - `decisions.jsonl` has no prompt text.
380
+
381
+ ## Observability
382
+
383
+ - `GET /router/status?session=<id>` gives the routes and the last turn of a
384
+ session. The status line and `/router:status` read it.
385
+ - `decisions.jsonl` has one line for each decision (with the model and the
386
+ effort sent), each response, each failed routed turn (`failed`, with the
387
+ status) and each routing error (`reason: error`, without the message). It
388
+ has no prompt text.
389
+ - `gateway.log` has the daemon events.
390
+
391
+ The [user guide](user-guide.md#read-the-status-line) explains how to read
392
+ them. The [data directory](configuration.md#data-directory) gives their
393
+ retention.
394
+
395
+ ## Release
396
+
397
+ ```mermaid
398
+ flowchart LR
399
+ TAG["Signed tag<br/>vX.Y.Z on main"] --> VER["Check signature,<br/>tag = package.json"]
400
+ VER --> TEST["Lint, tests,<br/>pack dry run"]
401
+ TEST --> NPM["npm publish<br/>with provenance"]
402
+ NPM --> GH["GitHub release<br/>named by the tag"]
403
+
404
+ classDef step fill:#eef2ff,stroke:#6366f1,color:#1e1b4b
405
+ class TAG,VER,TEST,NPM,GH step
249
406
  ```
250
- .claude-plugin/plugin.json userConfig.typesafe_api_key (Keychain)
251
- .claude-plugin/marketplace.json github source alexei-led/claude-router
252
- hooks/hooks.json SessionStart -> scripts/ensure-gateway.mjs
253
- scripts/gateway.mjs daemon entry
254
- scripts/ensure-gateway.mjs port probe, detached spawn
255
- scripts/statusline.mjs status line wrapper: wrapped command, then the route
256
- scripts/status.mjs report for /router:status
257
- scripts/transcript-models.sh model for each assistant line of a transcript
258
- lib/runtime.mjs configuration and data directory from the environment
259
- lib/config.mjs defaults, user file, validation
260
- lib/facts.mjs request body and memory -> facts (pure)
261
- lib/jev.mjs request, injected transport, parse
262
- lib/cost.mjs cache key, warmth, input cost, switching tax, shadow economics
263
- lib/policy.mjs switching policy v0
264
- lib/rewrite.mjs model, effort, thinking per model family
265
- lib/sse.mjs usage reader for SSE and JSON bodies
266
- lib/router.mjs orchestration for one request, session memory
267
- lib/gateway.mjs HTTP passthrough and rewrite
268
- lib/store.mjs files: configuration, memory, decisions.jsonl
269
- lib/status.mjs status snapshot, status line segment, report
270
- skills/<tier>/SKILL.md manual pins (model and effort frontmatter)
271
- skills/setup/SKILL.md writes model, base URL, picker row, status line
272
- skills/status/SKILL.md /router:status
273
- test/ node:test, builders in helpers.mjs
407
+
408
+ - The repository root is the plugin and the npm package
409
+ `@alexeiled/claude-router`.
410
+ - Claude Code installs the plugin from Git: the marketplace source is
411
+ `github`. An npm source fails with npm 12 (`EALLOWREMOTE`).
412
+ - `claude plugin update` compares the version in `.claude-plugin/plugin.json`.
413
+ Each release changes it together with `package.json`.
414
+ - `ci.yml` runs the checks and the tests for each push and pull request.
415
+
416
+ To work on the code:
417
+
418
+ ```sh
419
+ npm install
420
+ git config --local core.hooksPath scripts/git-hooks # Biome, tests and Gitleaks before commit and push
421
+ npm test && npm run check # node:test, Biome
422
+ claude --plugin-dir . --model jev-router # with ANTHROPIC_BASE_URL and TYPESAFE_API_KEY
274
423
  ```
275
424
 
276
- ## Open questions
277
-
278
- - The exit from plan mode is not visible in the request body. The policy has
279
- no boundary rule.
280
- - The thresholds are not tuned. The estimated and the observed cache reads in
281
- `decisions.jsonl` are the input for the tuning.
282
- - The gateway reads the configuration once. A reload without a restart is not
283
- implemented.
284
- - Without `CLAUDE_CODE_GATEWAY_HINT_HEADERS=1`, the gateway guesses side
285
- requests from the body; the guesses miss some. A missed side request with
286
- a short history also counts as a history break.
287
- - `x-claude-code-agent-type` is logged, not used: no policy rule reads it yet.
288
- - Does a downgrade that the shadow estimate says never repays deserve a rule,
289
- and does the switching tax belong in the upgrade bar? A replay of
290
- `decisions.jsonl` against the `shadow` and `observed` lines decides.
291
- - Effort changes how much a model writes. The shadow estimate uses the last
292
- output size for both routes.
293
- - Whether tools and system survive an effort change is model-specific; the
294
- gateway counts the whole prefix as lost, an upper bound.
425
+ ## Known limits
295
426
 
296
- ## Release
427
+ - The request body does not show the exit from plan mode.
428
+ - The thresholds are start values. The `observed` and `shadow` lines in
429
+ `decisions.jsonl` are the input for tuning.
430
+ - The gateway reads its configuration once. A change needs a restart.
431
+ - Without the hint headers, the gateway guesses side requests. A missed side
432
+ request with a short history counts as a history break.
433
+ - `x-claude-code-agent-type` goes to the log only.
434
+ - The shadow estimate uses the last output size for both routes, but effort
435
+ changes how much a model writes.
297
436
 
298
- The repository root is the plugin and the npm package `@alexeiled/claude-router`.
299
- The marketplace `alexei-led-claude-router` in `.claude-plugin/marketplace.json`
300
- points at this GitHub repository (`"source": "github"`), so Claude Code installs the
301
- plugin from git and never calls npm. An npm source
302
- fails under npm 12: Claude Code fetches the tarball URL, and npm 12 refuses
303
- remote tarballs by default (`EALLOWREMOTE`). `claude plugin update` compares
304
- the `version` in `.claude-plugin/plugin.json`, so each release bumps it. Local
305
- development uses `claude --plugin-dir .`.
306
-
307
- A release is a signed annotated tag `v<version>` on `main`, where the version
308
- matches `package.json`. The `release.yml` workflow makes sure that the tag is
309
- signed and on `main`, runs the checks and the tests, publishes to npm with
310
- trusted publishing (`npm publish --provenance`, no token), and creates the
311
- GitHub release. The `ci.yml` workflow runs the checks and the tests for each
312
- push and pull request on `main`.
437
+ A day of real usage is in [Evaluation](evaluation.md).