@alexeiled/claude-router 1.21.2 → 1.22.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/plugin.json +1 -1
- package/CHANGELOG.md +30 -0
- package/README.md +7 -7
- package/docs/activity-routing.md +56 -24
- package/docs/architecture.md +32 -20
- package/docs/configuration.md +4 -4
- package/docs/evaluation.md +21 -6
- package/docs/native-router.md +1 -1
- package/docs/user-guide.md +17 -15
- package/hooks/native-router.mjs +113 -8
- package/lib/activity-stats.mjs +5 -3
- package/lib/activity.mjs +29 -3
- package/lib/band.mjs +7 -2
- package/lib/classifier-apis.mjs +79 -24
- package/lib/classifier-client.mjs +16 -5
- package/lib/classifier-contract.mjs +47 -11
- package/lib/config.mjs +1 -1
- package/lib/decisions.mjs +51 -4
- package/lib/display.mjs +10 -6
- package/lib/facts.mjs +32 -1
- package/lib/panel.mjs +5 -3
- package/lib/route.mjs +32 -8
- package/lib/view.mjs +11 -1
- package/package.json +1 -1
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "router",
|
|
3
3
|
"displayName": "Router",
|
|
4
|
-
"version": "1.
|
|
4
|
+
"version": "1.22.1",
|
|
5
5
|
"description": "Native model and effort routing within your provider. Anthropic, DeepSeek, z.ai, MiniMax, MiMo and Kimi defaults, custom provider profiles, classifier advice and one model per subagent.",
|
|
6
6
|
"author": {
|
|
7
7
|
"name": "Alexei Ledenev",
|
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,35 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 1.22.1 (2026-11-13)
|
|
4
|
+
|
|
5
|
+
### Changed
|
|
6
|
+
|
|
7
|
+
- Separate the task goal from the current phase. Implementation stays `code` through verification, commit and reporting; a supported command-only phase can still use `ops` after the normal routing gates. Mixed tests and fixes are not execution-only.
|
|
8
|
+
- Keep a bounded goal in session memory only on explicit new-task/continuation evidence. History-prefix and request-generation fences reject replaced context and stale reviews. Unassociated notifications cannot justify downgrades.
|
|
9
|
+
- Show goal, phase and effective activity separately. Start a separate goal-contract counter series; older turn-label counters remain stored until **Delete history**. Tool compatibility is explicitly not accuracy.
|
|
10
|
+
- Use Jev `jev-1.14.0` by default. Explicit model overrides remain unchanged; Ollama remains goal-only without phase or continuity.
|
|
11
|
+
|
|
12
|
+
No configuration migration is required. Restart the plugin session after updating. Synthetic checks do not establish better real-task quality, calibration or savings.
|
|
13
|
+
|
|
14
|
+
## 1.22.0 (2026-11-11)
|
|
15
|
+
|
|
16
|
+
### Added
|
|
17
|
+
|
|
18
|
+
- A decision row keeps the probability mass the policy compared and the bar it compared it with (`mass`, `bar`), for a tier move and an activity move up, taken or held. `/router decisions` shows them after the reason: `(upgrade-pending 52% vs bar 79%)`. A held move can show a mass above its bar: it waits for a second vote or a checkpoint. A `jump`, a move on the turn's own tool calls and a context or availability override show neither. Rows saved before this change still read.
|
|
19
|
+
|
|
20
|
+
### Changed
|
|
21
|
+
|
|
22
|
+
- The docs say what transcripts show about an effort change on the same model: the cache was kept in 102 of 104 changes, which supports pricing an effort-only move as no cache write. On Claude Code 2.1.322 an effort changed inside a turn was not sent in 6 of 6 cases, so the band and the decision record can name an effort that did not run. Counts and limits: [What we do not know yet](docs/activity-routing.md#what-we-do-not-know-yet).
|
|
23
|
+
- The activity routing guide now says Sonnet 5.5 reads cached context at $0.10 per million tokens, half of Opus 5.5's $0.20, as it has since 1.21.0. Its 350K-token examples are recomputed: a move from Opus to Sonnet saves about $0.07 a request and takes about 22 requests to repay its cache write, and Sonnet to Haiku repays after about 9.
|
|
24
|
+
|
|
25
|
+
### Fixed
|
|
26
|
+
|
|
27
|
+
- Repeated tool failures no longer move a session to a weaker model when routing is turned on over a model no route names, such as Fable 5.2 on the defaults. That model was counted as the start tier's route, so failures on Fable at xhigh "escalated" to Haiku at high. The session's model is now the one in use: it stays unless a stronger route exists, and otherwise leaves for the start tier's route through the same checks as a routing-mode change. A model chosen with `/model` after routed turns still resumes the last routed route.
|
|
28
|
+
|
|
29
|
+
### Upgrade
|
|
30
|
+
|
|
31
|
+
Restart Claude Code after updating.
|
|
32
|
+
|
|
3
33
|
## 1.21.2 (2026-11-10)
|
|
4
34
|
|
|
5
35
|
### Added
|
package/README.md
CHANGED
|
@@ -35,18 +35,18 @@ Execution provider selection is separate from classifier selection. Observed usa
|
|
|
35
35
|
|
|
36
36
|
## Supported classifiers
|
|
37
37
|
|
|
38
|
-
The active classifier is asked at the start of each logical turn, for the tier
|
|
38
|
+
The active classifier is asked at the start of each logical turn, for the tier, task goal and current phase, and again in the background every 4 requests while the turn runs (`policy.reviewEveryRequests`). Pick one on the pane's **Classifier** tab or in the selected router file; only the active one receives prompt text. With subagent routing `on` (the default), it is also asked once per routed subagent, with the task's description and prompt; set `"subagentRouting": "off"` to keep that text local.
|
|
39
39
|
|
|
40
40
|
| Classifier | Service | Credential | Prompt text goes to | How it is asked for the activity |
|
|
41
41
|
| ---------- | ------------------------------------------ | ----------------------------------- | ------------------------- | --------------------------------------------------------------------- |
|
|
42
|
-
| Jev | [typesafe.ai](https://typesafe.ai) | Jev API key | typesafe.ai |
|
|
42
|
+
| Jev | [typesafe.ai](https://typesafe.ai) | Jev API key | typesafe.ai | Goal and phase `choice` questions in the same request |
|
|
43
43
|
| Clef | Cloudflare Workers AI, 27B | Cloudflare API token and account ID | Cloudflare | The same |
|
|
44
44
|
| Clef Flash | Cloudflare Workers AI, 9B | Cloudflare API token and account ID | Cloudflare | The same |
|
|
45
45
|
| Clef Omni | Cloudflare Workers AI, 30B MoE | Cloudflare API token and account ID | Cloudflare | The same |
|
|
46
|
-
| OpenAI | OpenAI, `gpt-6-luna`
|
|
47
|
-
| Ollama | A local [Ollama](https://ollama.com) model | None | Stays on this machine |
|
|
46
|
+
| OpenAI | OpenAI, `gpt-6-luna` | OpenAI API key | OpenAI (`api.openai.com`) | Goal and phase choices in `questions`, in the same request |
|
|
47
|
+
| Ollama | A local [Ollama](https://ollama.com) model | None | Stays on this machine | Goal only in a second request; no phase or continuation answer |
|
|
48
48
|
|
|
49
|
-
Jev is the default. [Configuration](docs/configuration.md#classifiers) gives the endpoints, timeouts, and how to add another service that speaks one of the supported protocols.
|
|
49
|
+
Jev 1.14 is the default; a configured model override stays in force. [Configuration](docs/configuration.md#classifiers) gives the endpoints, timeouts, and how to add another service that speaks one of the supported protocols.
|
|
50
50
|
|
|
51
51
|

|
|
52
52
|
|
|
@@ -81,7 +81,7 @@ sequenceDiagram
|
|
|
81
81
|
Mod-->>Dev: Status band and router pane
|
|
82
82
|
```
|
|
83
83
|
|
|
84
|
-
The active classifier labels each new logical turn, and the router checks the route again while the task runs: a turn labelled `ops` that edits code moves up to the `code` route at once, and a background classifier review every 4 requests can move the tier or the activity either way. A cheaper route on the same task waits for a dwell and a checkpoint its work can show: a passed test, build or lint run, or for commands, docs and read-only work the success the router can see. A
|
|
84
|
+
The active classifier labels each new logical turn, and the router checks the route again while the task runs: a turn labelled `ops` that edits code moves up to the `code` route at once, and a background classifier review every 4 requests can move the tier or the activity either way. A cheaper route on the same task waits for a dwell and a checkpoint its work can show: a passed test, build or lint run, or for commands, docs and read-only work the success the router can see. A notification is treated as progress, but cannot justify a downgrade because it may belong to an older task. `/router decisions` lists why each turn ran where it did. See [Changes during a task](docs/user-guide.md#changes-during-a-task). A local policy applies vote, context, failure, and cache-cost rules before the Mod changes the next step. Claude Code owns request construction, model credentials, tools, streaming, and the API cost ledger.
|
|
85
85
|
|
|
86
86
|
With the Anthropic profile, the base routes are:
|
|
87
87
|
|
|
@@ -94,7 +94,7 @@ With the Anthropic profile, the base routes are:
|
|
|
94
94
|
|
|
95
95
|
These are routing defaults. They do not claim equal model quality. [Configuration](docs/configuration.md#built-in-defaults) gives the published results behind them and the `router.json` lines that restore the 1.3 Sonnet ladder.
|
|
96
96
|
|
|
97
|
-
Activity routing
|
|
97
|
+
Activity routing classifies the task's goal (`code`, `debug`, `explore`, `plan`, `review`, `ops`, or `docs`) separately from the current phase. Bounded verification or execution can use the `ops` route without relabeling the goal. A route is then picked by tier and effective activity. It is on by default since 1.7. A `·` cell uses the tier's own route, the base route:
|
|
98
98
|
|
|
99
99
|
The Anthropic profile uses these activity defaults. Other profiles have [their own defaults](docs/providers.md#activity-overrides).
|
|
100
100
|
|
package/docs/activity-routing.md
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
# Activity routing
|
|
2
2
|
|
|
3
|
-
The router
|
|
4
|
-
|
|
5
|
-
|
|
3
|
+
The router uses **tier** for difficulty, **goal** for the task's final deliverable, and **phase** for the current step.
|
|
4
|
+
The route uses the goal's activity unless a supported verification-only or execution-only phase qualifies for `ops`.
|
|
5
|
+
Activity routing is `on` by default. This page shows the default choices, why each one is there, and how to
|
|
6
6
|
change them.
|
|
7
7
|
|
|
8
8
|
The numbers below come from Anthropic's system cards and from a replay of one developer's sessions at list prices.
|
|
@@ -10,19 +10,36 @@ They do not measure this router's answer quality or your bill.
|
|
|
10
10
|
|
|
11
11
|
## What it does
|
|
12
12
|
|
|
13
|
-
For each new turn
|
|
14
|
-
|
|
13
|
+
For each new turn, Jev, Clef and OpenAI answer tier, continuation, goal and phase questions together. The existing
|
|
14
|
+
`activity` field and route matrix name the goal; the phase can select the `ops` cell for bounded command-only work.
|
|
15
|
+
Ollama asks tier and goal in two calls and does not provide a phase, so it uses the goal route. A route is a model and an effort.
|
|
15
16
|
|
|
16
|
-
|
|
|
17
|
+
| Goal | The task produces |
|
|
17
18
|
| --------- | --------------------------------------------------------------- |
|
|
18
|
-
| `code` | Changed code or config, including tests
|
|
19
|
+
| `code` | Changed code or config, including tests and their verification |
|
|
19
20
|
| `debug` | A found cause: failing tests, stack traces, "why does X..." |
|
|
20
21
|
| `explore` | An answer: explain code, find where something happens, research |
|
|
21
22
|
| `plan` | A decision or plan: design, task breakdown, trade-offs |
|
|
22
23
|
| `review` | Findings about existing code: a PR, a diff, an audit |
|
|
23
|
-
| `ops` |
|
|
24
|
+
| `ops` | Specified operations as the task's final deliverable |
|
|
24
25
|
| `docs` | Prose for people: README, docs, comments, commit messages |
|
|
25
26
|
|
|
27
|
+
The phases are `produce`, `reason`, `verify`, `execute`, `report` and `uncertain`. A request to implement
|
|
28
|
+
an endpoint and run tests has goal `code`; “run the tests” can have phase `verify` while the goal stays `code`.
|
|
29
|
+
Tests **and fixes** are not verification-only. A terminal command by itself does not make the task an `ops` goal.
|
|
30
|
+
|
|
31
|
+
When both the goal and a `verify` or `execute` phase meet `policy.activityMass`, the router considers the
|
|
32
|
+
`ops` cell. It uses the lower of their supports for the existing activity-switch gates; it does not multiply them
|
|
33
|
+
or claim they are independently calibrated. A missing, malformed, uncertain or weak phase keeps the goal route.
|
|
34
|
+
The usual cache, dwell, escalation, model availability, context and credits checks still apply. A background phase answer expires when newer messages have arrived since its snapshot; stale bounded-command evidence cannot justify a cheaper step. A successful
|
|
35
|
+
classifier decision displays goal, phase and the route activity separately. Pins ignore activity routing.
|
|
36
|
+
|
|
37
|
+
The router keeps a bounded goal in memory for this session. It sends it as untrusted classifier context; a positively
|
|
38
|
+
identified continuation keeps it, a new task replaces it, and an ambiguous continuation drops it. A progress
|
|
39
|
+
notification uses the active goal but does not replace it or authorize a downgrade, because its originating task is not reliably known. The goal clears on session end, compaction or a detected
|
|
40
|
+
history rewind. A late notification may concern an older task; it is not proof of the current goal.
|
|
41
|
+
Goal memory is created only on positive new-task evidence; an unanchored continuation does not become a new goal. Ollama has no continuity answer and does not retain the anchor. No goal text is written to the activity or decision records. Classifier providers receive the goal and recent text.
|
|
42
|
+
|
|
26
43
|
Most cells use the tier's own route, the **base route**. A few cells have an **override**. When the activity is
|
|
27
44
|
unclear, or the classifier gives it less than 60% probability, the router moves back to the base route if the move
|
|
28
45
|
passes the cache check below. When the classifier gives no activity answer at all, the running activity stays. A pinned
|
|
@@ -32,7 +49,7 @@ The router compares routes, not labels. If the new activity resolves to the rout
|
|
|
32
49
|
A move to a cheaper route must pay back its cache write first. A move to a stronger route needs the classifier's
|
|
33
50
|
support. The [user guide](user-guide.md#route-by-activity) has the full rules.
|
|
34
51
|
|
|
35
|
-
The
|
|
52
|
+
The goal and phase come from the prompt, so they can be wrong. A prompt such as "run the worker brief" reads as `ops` and the
|
|
36
53
|
turn may then spend 80 requests writing code. The router therefore also watches the turn's own tool calls. Once the
|
|
37
54
|
turn edits a file, it moves to the `code` route of its tier (or `docs` for doc files) if that route is stronger. This
|
|
38
55
|
move only goes up. The cache cost does not hold it back; only the cash cap on a credits-billed model can. A later
|
|
@@ -64,7 +81,8 @@ Why each row:
|
|
|
64
81
|
|
|
65
82
|
The matrix uses five routes on three models. Opus 5.5, Sonnet 5.5 and Haiku 5.5 keep one cache across efforts, so the
|
|
66
83
|
five routes use three caches, one more (Sonnet) than the base routes alone. On a provider that caches each effort apart,
|
|
67
|
-
each (model, effort) pair is its own cache.
|
|
84
|
+
each (model, effort) pair is its own cache. [What we do not know yet](#what-we-do-not-know-yet) has what transcripts
|
|
85
|
+
show about that shared cache, and its limits.
|
|
68
86
|
|
|
69
87
|
## Which model fits which work
|
|
70
88
|
|
|
@@ -76,7 +94,7 @@ each (model, effort) pair is its own cache.
|
|
|
76
94
|
- Avoid `xhigh` in multi-turn chats. Anthropic reports that Haiku then sometimes puts the whole answer in its thinking
|
|
77
95
|
and shows no reply.
|
|
78
96
|
- Above 100,000 tokens of context, Anthropic charges 5 times Haiku's list price. Even then it costs at most a quarter
|
|
79
|
-
of what
|
|
97
|
+
of what Opus costs per token, and against Sonnet a quarter for input and output and half for cache reads.
|
|
80
98
|
|
|
81
99
|
**Sonnet 5.5**: small, well-defined code changes.
|
|
82
100
|
|
|
@@ -84,8 +102,8 @@ each (model, effort) pair is its own cache.
|
|
|
84
102
|
on FrontierCode Main.
|
|
85
103
|
- At `medium` it scored about 37% and 39.2%, below Haiku at `high` on FrontierCode Main. That is why the defaults use
|
|
86
104
|
`high`.
|
|
87
|
-
-
|
|
88
|
-
Opus.
|
|
105
|
+
- Every Sonnet rate is half of Opus's: it reads cached context at $0.10 per million tokens against $0.20. In long
|
|
106
|
+
sessions a request costs about half of one on Opus.
|
|
89
107
|
|
|
90
108
|
**Opus 5.5**: ordinary and hard engineering, design, review, and research answers.
|
|
91
109
|
|
|
@@ -101,18 +119,19 @@ These are Anthropic's results, mostly at `max` effort and in other harnesses. Th
|
|
|
101
119
|
Each model keeps its own cache of the conversation. A model change writes the whole conversation into the new model's
|
|
102
120
|
cache. In the sample below Claude Code always wrote a one-hour cache, which costs twice the input price.
|
|
103
121
|
|
|
104
|
-
At 350,000 tokens of context, the typical size in that sample, at list prices
|
|
122
|
+
At 350,000 tokens of context, the typical size in that sample, at list prices, for a request that reads the whole
|
|
123
|
+
context from the cache and writes about 3,000 output tokens:
|
|
105
124
|
|
|
106
125
|
| Cost | Haiku 5.5 | Sonnet 5.5 | Opus 5.5 |
|
|
107
126
|
| ---------------------------- | ----------- | ----------- | ----------- |
|
|
108
127
|
| Write the whole context once | $0.35 | $1.40 | $2.80 |
|
|
109
|
-
| One request on a warm cache | about $0.03 | about $0.
|
|
128
|
+
| One request on a warm cache | about $0.03 | about $0.07 | about $0.13 |
|
|
110
129
|
|
|
111
|
-
- A move from Opus to Haiku pays back after about 4 requests, from Sonnet after about
|
|
112
|
-
requests at the median, so the `ops` move pays back. Question-and-answer runs lasted 1 to 3,
|
|
113
|
-
does not.
|
|
114
|
-
- A move from Opus to Sonnet saves about $0.
|
|
115
|
-
the defaults do not move `medium` or `high` work to Sonnet.
|
|
130
|
+
- A move from Opus to Haiku pays back after about 4 requests, from Sonnet after about 9. Command runs lasted 6 to 11
|
|
131
|
+
requests at the median, so the `ops` move at `medium`, from Opus, pays back. Question-and-answer runs lasted 1 to 3,
|
|
132
|
+
so moving `explore` does not.
|
|
133
|
+
- A move from Opus to Sonnet saves about $0.07 per request and costs $1.40 to start: about 22 requests to pay back.
|
|
134
|
+
It rarely does. That is why the defaults do not move `medium` or `high` work to Sonnet.
|
|
116
135
|
|
|
117
136
|
The router does this sum for every move with the prices in `router.json`. A move to a cheaper route inside a tier is
|
|
118
137
|
taken only when it pays back within the payback horizon. A move down a tier weighs the same cost against the
|
|
@@ -197,7 +216,7 @@ Two other directions, with what they trade:
|
|
|
197
216
|
|
|
198
217
|
**Activity threshold.** An activity applies only when the classifier gives it at least this probability. Below it, a
|
|
199
218
|
new task goes back to the base route if that move passes the usual checks; the running task follows
|
|
200
|
-
[Changes during a task](user-guide.md#changes-during-a-task). The default is 60% (`policy.activityMass: 0.6`). Change it on the Policy tab under **Activities**,
|
|
219
|
+
[Changes during a task](user-guide.md#changes-during-a-task). The default is 60% (`policy.activityMass: 0.6`); the same threshold applies independently to the phase. Change it on the Policy tab under **Activities**,
|
|
201
220
|
or in `router.json`. There is no data to tune it on yet: the probe results record whether the
|
|
202
221
|
label was right, not the classifier's probabilities.
|
|
203
222
|
|
|
@@ -246,10 +265,23 @@ Limits:
|
|
|
246
265
|
|
|
247
266
|
- The answer quality of routed sessions.
|
|
248
267
|
- How well the classifier labels real turns. On 70 synthetic prompts, Jev and OpenAI labelled all 70 right; see
|
|
249
|
-
[Evaluation](evaluation.md#activity-probe-set). The **
|
|
268
|
+
[Evaluation](evaluation.md#activity-probe-set). The **Tool compatibility** lines, on the
|
|
250
269
|
Classifier tab for this session and across sessions, check your own turns against the tools they used.
|
|
251
|
-
- Whether an effort change on the same model keeps the cache.
|
|
252
|
-
|
|
270
|
+
- Whether an effort change on the same model always keeps the cache. Claude Code's prompt-caching docs say it does on
|
|
271
|
+
the 5.5 models and Fable 5.1 with an API key or a subscription, and the router prices it so. On 2026-11-11 we counted
|
|
272
|
+
104 such changes with under 270 seconds between requests in 30 days of Claude Code transcripts, main thread only, up
|
|
273
|
+
or down, inside a turn or at a turn start. The request after the change still read the cache in 102 of 104. The
|
|
274
|
+
median share of the prompt written to the cache after a change was 0.4 to 0.5% on Opus and Haiku and 0.1 to 0.7% on
|
|
275
|
+
Sonnet; the same models without a change wrote 0.2 to 0.4%. Limits: these are transcripts, not a controlled test.
|
|
276
|
+
Opus had 29 changes up and 33 down and Haiku 18 up and 15 down, but Sonnet had 7 up and 1 down, and Fable 1 up,
|
|
277
|
+
which is too small to count. Two changes lost the cache, and we did not look into why.
|
|
278
|
+
- Whether Claude Code sends an effort that the router changes inside a turn. The router changes it in the same
|
|
279
|
+
`turn.step` hook that changes the model. On Claude Code 2.1.322, in transcripts of 2026-11-10 and 2026-11-11, the
|
|
280
|
+
router did so in 6 cases: all 6 requests still carried the turn's first effort, while the changed model was sent in
|
|
281
|
+
every case. At the start of a turn the router's effort was sent in 145 of 145 decisions. Transcripts from older
|
|
282
|
+
versions, 2.1.296 to 2.1.319, show the effort changing between tool results inside a turn. We have not reproduced this
|
|
283
|
+
in isolation. So the band and the decision record can name an effort that did not run; the request parameters in the
|
|
284
|
+
transcript show what ran.
|
|
253
285
|
- How many requests a turn needs on each model.
|
|
254
286
|
|
|
255
287
|
## Sources
|
package/docs/architecture.md
CHANGED
|
@@ -30,19 +30,19 @@ sequenceDiagram
|
|
|
30
30
|
|
|
31
31
|
The Mod does not construct an Anthropic request or proxy its response. Claude Code owns authentication, request fields, streaming, tools, model fallback, and its native usage ledger. No local gateway process or port is part of the runtime. The Ollama classifier calls an Ollama server that you run; the Mod starts none.
|
|
32
32
|
|
|
33
|
-
The active classifier receives
|
|
33
|
+
The active classifier receives the current prompt, up to six recent user/assistant text messages, and optional bounded active-goal and current-step text. Each text is capped at 1,200 characters. System messages, tool inputs, and tool results are not included. Only the active classifier is asked: Jev sends that text to typesafe.ai, Clef, Clef Flash and Clef Omni to Cloudflare Workers AI, OpenAI to api.openai.com. Ollama runs on this machine, so the text stays here. The Jev, Cloudflare, and OpenAI keys are sensitive plugin options; the Cloudflare account ID is a plain one that fills the endpoint. Ollama takes no key. The Mod does not receive Claude's Anthropic credentials.
|
|
34
34
|
|
|
35
|
-
Each classifier names its wire protocol, `api`, and `classifier-apis.mjs` holds one adapter per protocol. `system-one` (Jev, Clef, Clef Flash, Clef Omni) sends the typed `state` and `questions` request and reads `answers.route`; Cloudflare's REST API wraps that answer in `{ result, success, errors }`, which the parser unwraps. `openai-decisions` (OpenAI) sends the state as `input` text and reads a `choice` answer plus a `predicate` for the continuation. `ollama` asks for one letter with thinking off and reads the log-probability of each letter from the top 20 candidates. A letter's probability is its share of the letters found there; a letter missing from the list counts as zero. The Ollama adapter asks no continuation question, so `continuation` is null there. Every adapter returns the
|
|
35
|
+
Each classifier names its wire protocol, `api`, and `classifier-apis.mjs` holds one adapter per protocol. `system-one` (Jev, Clef, Clef Flash, Clef Omni) sends the typed `state` and `questions` request and reads `answers.route`; Cloudflare's REST API wraps that answer in `{ result, success, errors }`, which the parser unwraps. `openai-decisions` (OpenAI) sends the state as `input` text and reads a `choice` answer plus a `predicate` for the continuation. `ollama` asks for one letter with thinking off and reads the log-probability of each letter from the top 20 candidates. A letter's probability is its share of the letters found there; a letter missing from the list counts as zero. The Ollama adapter asks no continuation question, so `continuation` is null there. Every adapter returns the common advice shape. Ollama has neither phase nor continuity; it uses the goal's route and does not retain a cross-turn goal anchor.
|
|
36
36
|
|
|
37
|
-
|
|
37
|
+
Advice carries `activity` (the task goal) and optional `phase`, each with a choice and probabilities. The effective routing activity can differ from the goal for bounded command phases. Protocols ask them independently:
|
|
38
38
|
|
|
39
39
|
| Protocol | How the activity is asked |
|
|
40
40
|
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
41
|
-
| `system-one` |
|
|
42
|
-
| `openai-decisions` |
|
|
41
|
+
| `system-one` | Independent `activity` (goal) and `phase` choice questions in the same request. |
|
|
42
|
+
| `openai-decisions` | Independent goal and phase choices in `questions`. |
|
|
43
43
|
| `ollama` | A second request after the route answer, with the same system message and state prefix so Ollama can reuse its cache. Letters A to H, one token. |
|
|
44
44
|
|
|
45
|
-
The activity is optional at every layer. An adapter reads a missing, malformed, or unknown activity answer as `activity: null` and keeps the strict parsing of the route answer. With `activityRouting: off` no adapter builds the activity question, so the request bodies are the ones 1.5 sent; a fixture test checks this. The criteria and instructions live in `classifier-contract.mjs` beside the tier ones, and
|
|
45
|
+
The activity is optional at every layer. An adapter reads a missing, malformed, or unknown activity answer as `activity: null` and keeps the strict parsing of the route answer. With `activityRouting: off` no adapter builds the activity question, so the request bodies are the ones 1.5 sent; a fixture test checks this. The criteria and instructions live in `classifier-contract.mjs` beside the tier ones, and describe the final task deliverable separately from the current step. Unknown phase answers never invent a command-only route. The classifier criteria keep showing the base route of each tier, not the activity overrides: the tier question is about difficulty.
|
|
46
46
|
|
|
47
47
|
Claude Code builds each request for the model the Mod selects, so no request rewriting is needed: Haiku 4.5 got its own thinking mode and a 32K output cap. On Claude Code 2.1.289, a billed chain of pinned turns (Opus, Haiku 4.5, Sonnet at `xhigh`, Sonnet, Haiku 4.5) used a tool on every turn over the previous models' history, and no request was rejected. Sonnet and Opus accepted histories above 589K input tokens. These checks cover tested requests. They do not prove every Claude Code feature combination.
|
|
48
48
|
|
|
@@ -91,16 +91,16 @@ The router decides on a _cell_: a tier and an activity. The route is `{ ...route
|
|
|
91
91
|
`chooseRoute` decides at a turn's first step, and again on each review answered inside the turn:
|
|
92
92
|
|
|
93
93
|
1. **Pin.** A pinned tier wins, with no activity: the pin's tier route.
|
|
94
|
-
2. **Activity.** Only in `on`; `off` uses none, and ignores the activity a turn in `on` ran with before the mode changed. With no advice, or advice without an activity answer, the incumbent's activity stays. On the same task (`sameTaskOf`;
|
|
94
|
+
2. **Activity.** Only in `on`; derive the route activity from the goal and a supported phase, with bounded verify/execute using `ops`. Missing phase keeps goal routing. Progress origins cannot justify downgrades. `off` uses none, and ignores the activity a turn in `on` ran with before the mode changed. With no advice, or advice without an activity answer, the incumbent's activity stays. On the same task (`sameTaskOf`; reviews and progress origins count as same-task even without a continuation answer, including Ollama) a label whose route at the incumbent tier is stronger applies at once, and a weaker one only with at least `downgradeMass` probability once `facts.downAllowed` holds; an unclear label keeps the running activity. Otherwise the activity with probability at least `activityMass` applies; below that, or for `uncertain`, there is none and the tier's base route is used.
|
|
95
95
|
3. **Tier.** The tier policy runs as before, with every candidate tier priced at its route for the activity and the incumbent at the route it runs. Repeated tool errors move to the lowest tier above whose route is stronger than the running one, or sends a request that cannot be ordered against it. With no activity that is one tier up, as in 1.5. If no tier above helps, nothing escalates and the failure signature is kept.
|
|
96
|
-
4. **Lateral move.** If the tier stays and the route would change, `lateralMove` decides. Routes order by explicit model strength when both models declare it, otherwise by output price, then effort; a session effort counts as the effort Claude Code sends. Two routes that send the same request do not switch. A pair that cannot be ordered (an unpriced model, a numeric effort, two models at one price) counts as a move up, so the base route is always reachable. A stronger route needs the activity's probability to clear the upgrade bar for its switching tax (a base route with no activity counts as probability 1). A cheaper route needs its downgrade tax to be paid back within `downgradeHorizonTurns`, except before the first measured reply of a history, when there is no cache to protect. The cash gate applies to both. The result is `activity-up`, `activity-down`, `activity-pending` (stay on the running route and its activity), `cash-gate`, or `hold`. While an escalation hold lasts, only a stronger route may be taken. One vote is enough
|
|
96
|
+
4. **Lateral move.** If the tier stays and the route would change, `lateralMove` decides. Routes order by explicit model strength when both models declare it, otherwise by output price, then effort; a session effort counts as the effort Claude Code sends. Two routes that send the same request do not switch. A pair that cannot be ordered (an unpriced model, a numeric effort, two models at one price) counts as a move up, so the base route is always reachable. A stronger route needs the activity's probability to clear the upgrade bar for its switching tax (a base route with no activity counts as probability 1). A cheaper route needs its downgrade tax to be paid back within `downgradeHorizonTurns`, except before the first measured reply of a history, when there is no cache to protect. The cash gate applies to both. The result is `activity-up`, `activity-down`, `activity-pending` (stay on the running route and its activity), `cash-gate`, or `hold`. While an escalation hold lasts, only a stronger route may be taken. One vote is enough for a proposed activity cell, but the labels remain model estimates, not observed facts. If the tier changes, the tier gates have already priced the new route, and no lateral rule applies.
|
|
97
97
|
5. **Fit.** Context fit and model availability resolve through `route(tier, activity)`. A fallback tries the last cell, the native model's cell, then each tier with the activity. The last and the native cell count only when they carry no activity or the turn's; otherwise, or when nothing fits, the native model runs at the session effort.
|
|
98
98
|
|
|
99
99
|
**Observed move.** On each later step of the turn, `observedMove` reads the turn's tool calls since its prompt, or since the transcript snapshot of the last review applied inside the turn (`loop.observedFrom`), since the review already judged the earlier ones, (`observedActivity`: `code` after an edit to a non-doc file, `docs` after edits to doc files only, else `ops`, `read` or `talk`). When that activity's route at the decision's tier is stronger than the running one, the remaining steps take it, with reason `activity-up` and `lastActivity` set, so the next turn starts there. The tier is fitted to the context first, so the gates price the route that will run. It goes through `lateralMove` with probability 1: the tools are the evidence, so the switching tax never holds it back (the upgrade bar tops out near 0.9), and only the cash gate can. The turn's reply label (`turnActivities.label`) follows the move; the classifier's answer stays as recorded. It never moves down, and skips pinned turns and `off`. `canMoveUp` skips the transcript read when no activity's route at the tier is stronger, so a step on the strongest route costs nothing. A move to a smaller window waits for a measured context, as in `routeTurn`. Context fit and model availability then apply as for any continuation.
|
|
100
100
|
|
|
101
|
-
**Review inside a turn.** Every `reviewEveryRequests` requests after the last classifier answer (`loop.reviewedAt`, counted in `loop.requests`), `reviewMidTurn` reads the transcript and asks the active classifier in the background, with the turn's prompt and recent dialogue and `facts.midTurn` set. The classifier reads text only: the dialogue holds user and assistant text, not tool calls, results or edits. The policy reads those itself (`observedMove`, repeated failures, `completedSince`). The step does not wait. The first step after the answer arrives runs `chooseRoute` on it, before the observed move, with only the advice taken from the review: the checkpoint, context and cache facts are read at that step. Tier and activity move by the same rules as a continuation prompt: up on votes, down only past `downAllowed`. A pinned turn, a failed read or ask, an answer without advice, and an answer about a replaced history (another `generation`) change nothing; turn completion and session end abort a review in flight. `observeResponse` counts each request and records the request and message count at which the
|
|
101
|
+
**Review inside a turn.** Every `reviewEveryRequests` requests after the last classifier answer (`loop.reviewedAt`, counted in `loop.requests`), `reviewMidTurn` reads the transcript and asks the active classifier in the background, with the turn's prompt and recent dialogue and `facts.midTurn` set. The classifier reads text only: the dialogue holds user and assistant text, not tool calls, results or edits. The policy reads those itself (`observedMove`, repeated failures, `completedSince`). The step does not wait. The first step after the answer arrives runs `chooseRoute` on it, before the observed move, with only the advice taken from the review: the checkpoint, context and cache facts are read at that step. Tier and activity move by the same rules as a continuation prompt: up on votes, down only past `downAllowed`. A pinned turn, a failed read or ask, an answer without advice, and an answer about a replaced history (another `generation`) change nothing; turn completion and session end abort a review in flight. `observeResponse` counts each request and records the request and message count at which the requested (model, effort) last changed (`switchedAt`), which the dwell and the verification window start from. A failed transcript read keeps the running route. A verification is a Bash command whose status is the check's: a checker (`pytest`, `tsc`, `eslint`, …) or a launcher whose subcommand names a check (`npm test`, `npm run lint`, `go test`, `make lint`, `uv run pytest`), alone or in an `&&` list. A pipeline, `;`, `||` or background job does not count, since its status can hide a failed check; neither do a `--version` or `--help` run, an option before the subcommand (`npm --prefix test install`), package names, paths and file names, nor a command Claude Code sent to the background, by request or after a timeout. The first response a loop serves also sets `switchedAt`, so checks from before the router chose a route do not count. When compaction or a rewind shortens the history, `prepareLoop` moves `switchedAt.messageCount` and `observedFrom` back to its end and keeps the request-based dwell.
|
|
102
102
|
|
|
103
|
-
At session start the incumbent is the session's model. `cellForModel` looks for it in the base routes, baseline first, and only with `activityRouting: on` in the activity overrides too, so a session that starts on an override route keeps its cell until the classifier says otherwise. The start mode uses the same lookup: with `on`, a start model that only an override uses (Sonnet 5.5 on the defaults) starts with routing on.
|
|
103
|
+
At session start the incumbent is the session's model. `cellForModel` looks for it in the base routes, baseline first, and only with `activityRouting: on` in the activity overrides too, so a session that starts on an override route keeps its cell until the classifier says otherwise. The start mode uses the same lookup: with `on`, a start model that only an override uses (Sonnet 5.5 on the defaults) starts with routing on. A session model with no cell (Fable 5.2 on the defaults, turned on with `/router auto`) starts at `baselineTier`, but `emptyLoop` records the model itself, at the session's effort, as the route in use (`ownRoute`). So escalation, the switching tax and cache state compare against what ran, and the session leaves it through a gate: failure escalation to a stronger route, a tier switch the classifier supports (a jump or the upgrade votes, which carry no dwell), or `lateralMove` for a cheaper route, which waits for the dwell and a checkpoint on the same task. Until the first routed decision, a manual reply replaces that record with the model that served it (a model with a cell records none), so `/model` before `/router auto` starts from the model in use. After routed turns, `/model` then `/router auto` resumes the last routed route, as for any model.
|
|
104
104
|
|
|
105
105
|
## Routing modes
|
|
106
106
|
|
|
@@ -114,7 +114,7 @@ The turn-start decision (`decideTurn`) writes `router decision {mode, modeSource
|
|
|
114
114
|
|
|
115
115
|
**Prompt origin.** `turn.start` carries the prompt text but not its origin; `prompt.submit` carries `origin` (`PromptOrigin`), stamped by the engine. The hook keeps the kind (`originKind` in `lib/decisions.mjs`). An idle submission starts its turn inside `next(e)`, so that turn takes it; a submission queued behind a running turn waits (`queueOrigin`, at most 16 for at most 10 minutes) and names the later turn that starts with its text (`takeOrigin`), or expires when it was delivered into the running turn. Unmatched turns are `unknown` and route as before. A progress origin (`task-notification`, `observer`, `observer-activity`, `auto-continuation`) sets `facts.progress`, so the turn is the same task (`sameTaskOf`), and `progressInput` asks the classifier about the session's active goal, with the notification as the newest dialogue line. The goal is the last prompt from a goal origin (`composer`, `peer`, `plugin`, …), held in memory for the session only: after a resume or a reload a notification is asked about as it reads. `unclassified` and kinds a later engine adds route as before and leave the goal.
|
|
116
116
|
|
|
117
|
-
**Decision record.** Each routed turn collects rows (`decisionRow`): its start, each applied review, each observed move, at most 10 per turn (`addTurnRow`), and at `turn.complete` an end row (`endRow`) with the turn's tool bucket, routed requests, priced requests and unpriced ones by cause. `flushDecisions` adds them to the session's own `decisions:v1:<session id>` record (newest 60 rows) under the reset marker, like the usage records. Session start deletes other sessions' records past 30 days or beyond the 20 newest (`prunedDecisions`), each read again before its delete. Rows hold labels, shares and counts, never prompt, tool or path text. `/router decisions [n]` prints them; **Delete history** deletes them. `experiments/mod-router/scripts/trace-eval.mjs` replays the gateway trace once per mode at tier level; it cannot replay moves inside a tier.
|
|
117
|
+
**Decision record.** Each routed turn collects rows (`decisionRow`): its start, each applied review, each observed move, at most 10 per turn (`addTurnRow`), and at `turn.complete` an end row (`endRow`) with the turn's tool bucket, routed requests, priced requests and unpriced ones by cause. `flushDecisions` adds them to the session's own `decisions:v1:<session id>` record (newest 60 rows) under the reset marker, like the usage records. Session start deletes other sessions' records past 30 days or beyond the 20 newest (`prunedDecisions`), each read again before its delete. A start or review row whose reason is `upgrade`, `upgrade-pending`, `downgrade`, `downgrade-pending`, `downgrade-held`, `activity-up` or `activity-pending` also keeps the estimate's mass and threshold as `mass` and `bar`, rounded to two decimals, when the estimate has both. The mass is the one above the incumbent for a move up; for a move down the one at or below the target, and on `downgrade-pending` at or below the classifier's choice (`decide` reports that estimate, though a tier between the choice and the incumbent can be closer to its bar); for an activity move up the label's share, or 1 when the move needs no label (back to the tier's route, a route the config changed). A pending or held row can show a mass at or above its bar: the values are rounded, and the move can wait for its vote streak or its checkpoint. Other rows keep neither: a `jump` is weighed against `jumpConfidence`, a context or availability override keeps the estimate of the decision it replaced, an observed move takes no classifier mass, and a hold, a cash gate and a cheaper activity route judged on its cache write compare none. A lateral outcome replaces the reason and the estimate of a tier stay (`decideCell`), so a tier move held on a turn that also weighed an activity route shows the activity's reason and numbers, not its own. Rows hold labels, shares and counts, never prompt, tool or path text. A row names the route the router chose, not the request that ran: on Claude Code 2.1.322 an effort changed inside a turn was not sent in the 6 cases found (see [What we do not know yet](activity-routing.md#what-we-do-not-know-yet)). `/router decisions [n]` prints them; **Delete history** deletes them. `experiments/mod-router/scripts/trace-eval.mjs` replays the gateway trace once per mode at tier level; it cannot replay moves inside a tier.
|
|
118
118
|
|
|
119
119
|
## State and controls
|
|
120
120
|
|
|
@@ -124,14 +124,26 @@ Clear, branch, rewind, and committed compaction clear cache evidence and votes.
|
|
|
124
124
|
|
|
125
125
|
### Observed activity and stats
|
|
126
126
|
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
- **Session**: `activityStats` in the view
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
127
|
+
Completed turns have tool buckets (`code`, `docs`, `shell`, `ops`, `check`, `read`, `talk`).
|
|
128
|
+
They describe tools, not the semantic goal. Shell-written code, delegated work and unknown tools limit this oracle.
|
|
129
|
+
[Evaluation](evaluation.md#tool-agreement) describes the many-to-many compatibility check.
|
|
130
|
+
|
|
131
|
+
- **Session**: `activityStats` in the view counts turns, requests, tokens, priced replies/routes, switches and
|
|
132
|
+
tool compatibility. Its activity breakdown is the effective route activity, not necessarily the task goal.
|
|
133
|
+
The `activityContract` marker starts a fresh session counter series when the label contract changes.
|
|
134
|
+
- **Across sessions**: `activity:goals:v1:<session id>` stores the raw goal-vs-tool confusion counts,
|
|
135
|
+
bounded run-length buckets, lateral moves and `since`. Records under the old
|
|
136
|
+
`activity:stats:v1` namespace remain stored but are not merged into this new series. They are removed only
|
|
137
|
+
by **Delete history**. Goal and old turn-label percentages are not one accuracy series.
|
|
138
|
+
- **Metrics**: `activity:metrics:v1:<session id>` retains the existing latency histograms, cheaper moves and
|
|
139
|
+
following escalations, and unpriced replies by cause. These are operational readings, not semantic accuracy.
|
|
140
|
+
|
|
141
|
+
Each key has one session writer. The view's `activityStore` and `activityMetrics` are read at session start
|
|
142
|
+
and when the pane opens, then updated after a turn writes its record. The shared reset marker prevents an
|
|
143
|
+
older held record from undoing another session's reset, subject to the existing millisecond/clock limits.
|
|
144
|
+
**Delete history** clears old and new counters, metrics, usage and subagent records. No prompt text is stored.
|
|
145
|
+
A failed transcript/store read loses that turn's counts and never fails generation. Activity routing off records
|
|
146
|
+
no activity counters.
|
|
135
147
|
|
|
136
148
|
### Routing vs no routing
|
|
137
149
|
|
|
@@ -172,7 +184,7 @@ Execution profile selection is fixed at session start by `CLAUDE_ROUTER_PROVIDER
|
|
|
172
184
|
|
|
173
185
|
The explicit-cache rules below describe Anthropic. Automatic prefix-cache providers instead use observed hit/miss input rates, no write premium and no guaranteed TTL. Their future price bounds allow a full miss; counterfactual cache comparisons are omitted unless the observed route is identical. Same-model effort downgrades skip input-only payback, which cannot value generated thinking-token changes, but retain the vote, dwell and cold-input guards. Their cache observations remain effort-specific until sharing is established. `disabled` cache mode uses the uncached input rate throughout.
|
|
174
186
|
|
|
175
|
-
Cache identity uses the response model ID with the `[1m]` context hint removed, plus the effective effort on models that keep one cache per effort. Opus 5.5, Sonnet 5.5, Haiku 5.5, Fable 5.1 and Fable 5.2 keep one cache across efforts with an API key or a Claude subscription (code.claude.com/docs/en/prompt-caching, "Changing effort level"), so an effort-only move costs no cache write there. The hook gives every effort its own cache when the session runs on Amazon Bedrock, Google Cloud, Microsoft Foundry, a custom `ANTHROPIC_BASE_URL`, or with `CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS` (`effortSplitsCache`). A HIPAA configuration is not visible to a Mod; there the router prices effort moves as free. The Haiku 4.5 dated snapshot has an explicit mapping; Haiku 5.5 has no dated ID. An unrecognized substituted model gets no cache credit for the requested route.
|
|
187
|
+
Cache identity uses the response model ID with the `[1m]` context hint removed, plus the effective effort on models that keep one cache per effort. Opus 5.5, Sonnet 5.5, Haiku 5.5, Fable 5.1 and Fable 5.2 keep one cache across efforts with an API key or a Claude subscription (code.claude.com/docs/en/prompt-caching, "Changing effort level"), so an effort-only move costs no cache write there. Transcripts show the cache kept in 102 of 104 same-model effort changes, with limits; see [What we do not know yet](activity-routing.md#what-we-do-not-know-yet). The hook gives every effort its own cache when the session runs on Amazon Bedrock, Google Cloud, Microsoft Foundry, a custom `ANTHROPIC_BASE_URL`, or with `CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS` (`effortSplitsCache`). A HIPAA configuration is not visible to a Mod; there the router prices effort moves as free. The Haiku 4.5 dated snapshot has an explicit mapping; Haiku 5.5 has no dated ID. An unrecognized substituted model gets no cache credit for the requested route.
|
|
176
188
|
|
|
177
189
|
The cached prefix is observed `cache_read + cache_creation`. Output tokens count only toward the next context size. Freshness uses the configured five-minute lifetime with its safety margin. Claude's usage counters do not prove one-hour cache lifetime.
|
|
178
190
|
|
package/docs/configuration.md
CHANGED
|
@@ -153,7 +153,7 @@ Prices are configured list-price inputs for estimates. They are not a subscripti
|
|
|
153
153
|
|
|
154
154
|
## Activity routing
|
|
155
155
|
|
|
156
|
-
The classifier
|
|
156
|
+
The classifier names the task goal separately from its current phase. The existing `activities` matrix still selects routes; a supported command-only phase can select `ops` without changing the goal. `activityRouting` decides what the router does with it. [Activity routing](activity-routing.md) explains the defaults for end users.
|
|
157
157
|
|
|
158
158
|
| `activityRouting` | Behavior |
|
|
159
159
|
| ----------------- | ------------------------------------------------------------------------------------------ |
|
|
@@ -221,7 +221,7 @@ Switch the mode with `/router activities off|on`, or with the **Activity routing
|
|
|
221
221
|
|
|
222
222
|
| `classifier` | Endpoint | Request `model` | Key option | `timeoutMs` |
|
|
223
223
|
| ------------ | -------------------------------------------------------------------------------------------------------- | --------------- | ---------------------- | ----------: |
|
|
224
|
-
| `jev` | `https://api.typesafe.ai/v1/systemone` | `jev-1.
|
|
224
|
+
| `jev` | `https://api.typesafe.ai/v1/systemone` | `jev-1.14.0` | `typesafe_api_key` | `1500` |
|
|
225
225
|
| `clef` | `https://api.cloudflare.com/client/v4/accounts/{cloudflare_account_id}/ai/run/@cf/cloudflare/clef` | `clef` | `cloudflare_api_token` | `3000` |
|
|
226
226
|
| `clef-flash` | `https://api.cloudflare.com/client/v4/accounts/{cloudflare_account_id}/ai/run/@cf/cloudflare/clef-flash` | `clef-flash` | `cloudflare_api_token` | `3000` |
|
|
227
227
|
| `clef-omni` | `https://api.cloudflare.com/client/v4/accounts/{cloudflare_account_id}/ai/run/@cf/cloudflare/clef-omni` | `clef-omni` | `cloudflare_api_token` | `3000` |
|
|
@@ -245,7 +245,7 @@ You can add an entry for another service that speaks one of the three protocols.
|
|
|
245
245
|
"jev-proxy": {
|
|
246
246
|
"label": "Jev",
|
|
247
247
|
"endpoint": "https://jev-proxy.example.internal/v1/systemone",
|
|
248
|
-
"model": "jev-1.
|
|
248
|
+
"model": "jev-1.14.0",
|
|
249
249
|
"keyOption": "typesafe_api_key",
|
|
250
250
|
"timeoutMs": 1500
|
|
251
251
|
}
|
|
@@ -377,7 +377,7 @@ The pane writes a subset of these settings. The **Routes** and **Policy** tabs s
|
|
|
377
377
|
|
|
378
378
|
A project can set the routing mode, and only that, in `.claude/router.json` in the directory Claude Code runs in: `{ "mode": "cost" }`. Any other content is ignored with a notice; it never makes routing unavailable. A session's `/router mode` beats it, and it beats the profile's `mode`. See [Routing modes](routing-modes.md).
|
|
379
379
|
|
|
380
|
-
Opus 5.5, Sonnet 5.5, Haiku 5.5, Fable 5.1 and Fable 5.2 keep one prompt cache across effort levels, so the policy prices an effort-only move, such as Opus at `medium` to Opus at `high`, as no cache write. On Amazon Bedrock, Google Cloud, Microsoft Foundry, a custom `ANTHROPIC_BASE_URL`, or with `CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS`, each effort has its own cache and the move pays its write. A HIPAA configuration cannot be detected; the router then prices those moves as free.
|
|
380
|
+
Opus 5.5, Sonnet 5.5, Haiku 5.5, Fable 5.1 and Fable 5.2 keep one prompt cache across effort levels, so the policy prices an effort-only move, such as Opus at `medium` to Opus at `high`, as no cache write. Transcripts support that price, with limits; see [What we do not know yet](activity-routing.md#what-we-do-not-know-yet). On Amazon Bedrock, Google Cloud, Microsoft Foundry, a custom `ANTHROPIC_BASE_URL`, or with `CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS`, each effort has its own cache and the move pays its write. A HIPAA configuration cannot be detected; the router then prices those moves as free.
|
|
381
381
|
|
|
382
382
|
Native cache freshness is unknown after 270 seconds or after a history reset. Claude usage does not report the cache TTL. The policy evaluates price bounds for five-minute and one-hour writes, but the one-hour case is not observed fact. See the [architecture](architecture.md#cache-and-cost) for the estimate rules.
|
|
383
383
|
|
package/docs/evaluation.md
CHANGED
|
@@ -92,9 +92,9 @@ The script writes aggregate counts, reason histograms, route transitions, and sh
|
|
|
92
92
|
|
|
93
93
|
## Activity routing
|
|
94
94
|
|
|
95
|
-
|
|
95
|
+
Goal/phase routing has no measured answer-quality or savings result. The activity contract changed from a turn's result to a task goal, so older goal/phase and tool-compatibility counts cannot be compared as one accuracy series. The results below are historical turn-label evidence, not evaluation of the new contract. The replay above predates it and covers tiers only. It is `on` by default since 1.7, with defaults chosen from Anthropic's published benchmarks and a list-price replay of one developer's sessions; [Activity routing](activity-routing.md#what-the-estimates-mean) gives that replay and its limits. The pane counts turns, requests, switches, and the turns `on` would route differently, with a configured list-price estimate of those turns' next requests, not a measured saving. Two checks look at the classifier's activity label.
|
|
96
96
|
|
|
97
|
-
The figures plan §11 tracks (tool agreement of at least 80% on `code` against `ops` and `explore`, Jev p95 latency of at most 450 ms, at most 1 tool-error escalation in 10 cheaper activity moves, the number of activity switches) are read from the Classifier tab's **ACROSS SESSIONS** block, which shows the counts kept since the last reset: **Code/ops/explore** for the tool agreement on turns the classifier answered `code`, `ops` or `explore`, **
|
|
97
|
+
The figures plan §11 tracks (tool agreement of at least 80% on `code` against `ops` and `explore`, Jev p95 latency of at most 450 ms, at most 1 tool-error escalation in 10 cheaper activity moves, the number of activity switches) are read from the Classifier tab's **ACROSS SESSIONS** block, which shows the counts kept since the last reset: **Code/ops/explore** for the tool agreement on turns the classifier answered `code`, `ops` or `explore`, **Tool compatibility** and **Mismatch** for all of them, **Activity moves** for moves inside a tier taken and refused (a refusal by a hold or the credits cap included), **Escalations** for the tool-error escalations that followed a cheaper activity move in `on` while the turns stayed on its route in `on` (turning routing off, a turn with routing off, or a turn in `off` ends the watch, a pin in between does not, and an escalation the credits cap held counts), **Unpriced** for the routed replies the usage record left out, by cause: no usage in the reply, a model without a configured price, an unknown cache lifetime, or a failed comparison. Before 1.16 those replies were dropped without a count, so usage covered only the priced part of the routed traffic, and a session whose replies were all unpriced left no usage record at all. The active classifier's p95 wait is on its **Deadline** line on the same tab, above that block. Latency is kept in fixed buckets, so the p95 reads as a bound (`≤ 350 ms`); it is the wait for the classifier's answer, transient retries included, without the routing after it, so it compares with the probe's p95 below, and is not recorded with activity routing `off`. `/router` without a UI surface prints both agreements. The underlying counts are the per-session `activity:goals:v1:<session id>` and `activity:metrics:v1:<session id>` records described in [Architecture](architecture.md).
|
|
98
98
|
|
|
99
99
|
### Tool agreement
|
|
100
100
|
|
|
@@ -126,7 +126,22 @@ The classifier's activity agrees with the turn when the bucket is one it allows:
|
|
|
126
126
|
|
|
127
127
|
Before 1.16 a shell write, a check run and any other command were all `ops`. A `code` turn that edited through the shell therefore read as a miss, and so did a review that ran the tests. The finer buckets keep that apart. Because a shell write cannot be told from routine work, it agrees with every activity that may produce it, so it cannot catch a cheap route on shell-written code. Agreement from before 1.16 and after it is not comparable, and a store holding the new buckets reads as empty in an older version. Routing reads `shell` and `check` as `ops`, as before.
|
|
128
128
|
|
|
129
|
-
|
|
129
|
+
Tool compatibility is a weak check, not classification accuracy or a quality score. A task goal can remain `code` while a step only runs tests, and the tool bucket cannot establish the goal. Shell edits, delegated work and unknown tools create additional blind spots. It cannot reliably identify every cheap route on a coding task. The Classifier tab shows the session's count as `Tool compatibility` under **THIS SESSION**, over the turns where the classifier gave an activity at or above `activityMass`, in `on` too when the route kept another activity. The counts kept across sessions hold the classifier's raw answer, `uncertain` and no answer included, against the bucket. The pane's **ACROSS SESSIONS** block counts agreement over the answers that name an activity; `uncertain` and no answer appear only in **Labelled**.
|
|
130
|
+
|
|
131
|
+
### Goal/phase development checks
|
|
132
|
+
|
|
133
|
+
An earlier 24-case synthetic development pilot separated schema changes from goal-anchor context and compared Jev
|
|
134
|
+
1.13 with 1.14. A work-account Sonnet label check agreed with the 24 supplied goal/phase references, without seeing
|
|
135
|
+
those labels. This is model agreement under a contract, not human ground truth.
|
|
136
|
+
|
|
137
|
+
A later eight-case check through the production Jev 1.14 adapter returned eight usable goal/phase answers and matched
|
|
138
|
+
seven author reference pairs. The missing-brief case returned goal `uncertain`, phase `reason`, where the reference
|
|
139
|
+
phase was `uncertain`. This development result is retained, not relabeled as a success. Subsequent parser, rubric and
|
|
140
|
+
lifecycle changes mean it is not a benchmark of the final release commit.
|
|
141
|
+
|
|
142
|
+
Run `node --env-file=/path/to/private.env test/goal-phase-live.mjs --live` for a billed synthetic check.
|
|
143
|
+
It is not run in CI and sends no private conversation text. Offline regression tests cover the actual request,
|
|
144
|
+
parser, effective route and hook lifecycle. No paired real-task completion, calibration or savings study was run.
|
|
130
145
|
|
|
131
146
|
### Activity probe set
|
|
132
147
|
|
|
@@ -138,7 +153,7 @@ node scripts/probe-activity.mjs [classifier]
|
|
|
138
153
|
|
|
139
154
|
The script sends each prompt to one classifier as the router would ask it, with activity routing forced to `on`. It reads credentials as `probe-classifier.mjs` does and needs a classifier you can reach. The run is billed on a paid classifier, and it is run by hand, not in CI. It writes accuracy, an expected-by-answered confusion matrix, p50 and p95 latency against the classifier's deadline, and error counts to `experiments/mod-router/results/activity-probe-<classifier>.json`. For Ollama the latency covers both requests. It never prints a key.
|
|
140
155
|
|
|
141
|
-
Five probe results are checked in, run on 2026-10-09 and 2026-10-10 (UTC) with the synthetic set above: Jev against the live typesafe.ai API, OpenAI against the live Decisions API, Clef and Clef Flash against the live Cloudflare API, Ollama with `qwen3.5:9b` on an Apple M2 Pro laptop:
|
|
156
|
+
Five historical turn-activity probe results are checked in, run on 2026-10-09 and 2026-10-10 (UTC) with the synthetic set above: Jev against the live typesafe.ai API, OpenAI against the live Decisions API, Clef and Clef Flash against the live Cloudflare API, Ollama with `qwen3.5:9b` on an Apple M2 Pro laptop:
|
|
142
157
|
|
|
143
158
|
| Classifier | Correct | p50 latency | p95 latency | Result file |
|
|
144
159
|
| ---------- | -------: | ----------: | ----------: | ---------------------------------------------------------------------------------------------------- |
|
|
@@ -148,7 +163,7 @@ Five probe results are checked in, run on 2026-10-09 and 2026-10-10 (UTC) with t
|
|
|
148
163
|
| Clef Flash | 69 of 70 | 470 ms | 874 ms | [`activity-probe-clef-flash.json`](../experiments/mod-router/results/activity-probe-clef-flash.json) |
|
|
149
164
|
| Ollama | 66 of 70 | 2986 ms | 3483 ms | [`activity-probe-ollama.json`](../experiments/mod-router/results/activity-probe-ollama.json) |
|
|
150
165
|
|
|
151
|
-
Ollama's latency covers both requests and stays inside its 5000 ms deadline; on slower hardware the activity step is skipped when less than 300 ms remain. They
|
|
166
|
+
Ollama's latency covers both requests and stays inside its 5000 ms deadline; on slower hardware the activity step is skipped when less than 300 ms remain. They measured the old turn-activity contract on synthetic prompts, not the new goal/phase contract, routing quality or savings. The activity question costs little time: 30 alternating pairs of the same prompts, with and without it, gave Jev p50 259 ms both ways and p95 300 ms without, 306 ms with; OpenAI p50 163 ms and 162 ms, p95 296 ms and 349 ms. OpenAI's first request of a run took about 2.3 s of its 3000 ms deadline. The Classifier tab shows each result under its classifier, such as `activity probe 70/70 · p95 299 ms · Oct 10` (UTC date), when the classifier runs the probed model. The script regenerates `lib/probe-results.mjs` from the result files after each run; `npm run probe:results` does it alone, and a test fails when the two disagree. A run without a single activity answer (an activity or `uncertain`) or without a p95 latency, such as one with a revoked key or a model whose answers all lack the activity, exits with an error and writes neither file. `policy.activityMass` was not tuned on any classifier; run the probe before you trust it with a new one. Clef Omni has no probe result yet, so its Classifier tab row shows none.
|
|
152
167
|
|
|
153
168
|
## Routing vs no routing
|
|
154
169
|
|
|
@@ -172,7 +187,7 @@ Limits:
|
|
|
172
187
|
- Output length is held the same. A stronger model may write less, or finish in fewer tool steps; a cheaper one may need more. The readout cannot see either.
|
|
173
188
|
- Answer quality is not measured. A saving bought with worse answers would read the same.
|
|
174
189
|
- Your model's cache is simulated, and a five-minute session that the lifetime rule takes for one hour (a plan subscriber billed as overage) would read too much cache on your side.
|
|
175
|
-
- A route that changes only the effort costs the same per token. On Opus 5.5, Sonnet 5.5, Haiku 5.5, Fable 5.1 and Fable 5.2 it shares your cache and reads $0.00; on a provider that caches each effort apart, it shows its cache writes.
|
|
190
|
+
- A route that changes only the effort costs the same per token. On Opus 5.5, Sonnet 5.5, Haiku 5.5, Fable 5.1 and Fable 5.2 it shares your cache and reads $0.00; on a provider that caches each effort apart, it shows its cache writes. [Activity routing](activity-routing.md#what-we-do-not-know-yet) has what transcripts show about the shared cache.
|
|
176
191
|
- Classifier charges, replies with routing off, and the side calls the router does not see (titles, compaction, server-side tools) are not counted; Claude Code's `/cost` counts them. On a Claude plan the dollars stand for quota, not cash, and the pane marks them `≈`.
|
|
177
192
|
- The usage records keep one record per session, by local calendar day, so sessions finishing turns at the same moment both count. A turn's switch counts on the day it ended, its replies on the days they arrived. Claude Code's store has no atomic update, which leaves these gaps: the same session open in two Claude Code processes at once can lose one's turn; a turn finishing in another process during **Delete history** can be lost; and a session's total from before a reset comes back if its first saved reply has the same millisecond as the reset, or if the system clock moved back across the reset. The router reads every session's record at session start and when the pane opens, so that read grows with the number of sessions kept. A session drops its days older than 400 days when it writes its record; on that read, another session's record with no day in the last 400 days is deleted, and past 1,000 records of other sessions those with the oldest last day go too. A record that does not read whole, such as one a later version wrote, counts toward the 1,000 but is never deleted. Totals kept before the dated usage records have no dates: All adds them, and the other periods leave them out.
|
|
178
193
|
|
package/docs/native-router.md
CHANGED
|
@@ -22,7 +22,7 @@ The activity is what the turn produces: `code`, `debug`, `explore`, `plan`, `rev
|
|
|
22
22
|
| `/router pin <tier>` | Pin the next turn and tool continuations. Routing on must already be enabled. |
|
|
23
23
|
| `/router activities <off\|on>` | Set the activity routing mode. Written to `router.json` at once; **Undo** in the pane reverts it. |
|
|
24
24
|
| `/router mode <cost\|balanced\|quality\|default>` | Set or clear this session's [routing mode](routing-modes.md), from the next turn. Kept for a resume in `$.store` `routing-mode:<session id>`. Never aborts a turn. |
|
|
25
|
-
| `/router decisions [n]` | Print this session's last `n` routing decisions (default 20, at most 60): origin, classifier answer, task, tier, route, reason, and each turn's end. No text. |
|
|
25
|
+
| `/router decisions [n]` | Print this session's last `n` routing decisions (default 20, at most 60): origin, classifier answer, task, tier, route, reason with the mass and bar it compared, and each turn's end. No text. |
|
|
26
26
|
| `/model <name>` | Select a model and turn routing off. `/router auto` turns it back on. |
|
|
27
27
|
|
|
28
28
|
The pane's **Set up** and **Credentials → Edit** buttons open Claude Code's secure plugin configuration. Claude Code refuses `$.command.run` from inside a `command.run` hook, so the Mod starts that dialog from a timer after the press returns.
|