@alexeiled/claude-router 1.6.0 → 1.7.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -57,58 +57,70 @@ Default model settings:
57
57
  | `sonnet` | 2 / 10 / 0.2 | 1,000,000 | `plan` | `low`, `medium`, `high`, `xhigh`, `max` |
58
58
  | `haiku` | 0.1 / 0.5 / 0.01 | 1,000,000 | `plan` | `low`, `medium`, `high`, `xhigh`, `max` |
59
59
 
60
- Prices are configured list-price inputs for estimates. They are not a subscription bill or a claim of savings. Haiku 5.5 prices are for prompts up to 100,000 tokens; above that Anthropic charges more, so estimates run low there. A model with `"efforts": []` receives no effort field. A route in `router.json` without an `effort` takes the effort of that tier's default route. To keep the session effort, clamped to what the model supports, set `"effort": null`, for example `"routes": { "high": { "model": "opus", "effort": null } }`. A 1.3 file whose `low` route had no `effort` kept the session effort; in 1.4 it runs at `high` until you add `"effort": null`.
60
+ Prices are configured list-price inputs for estimates. They are not a subscription bill or a claim of savings. Haiku 5.5 lists these prices for prompts up to 100,000 tokens and five times each of them, cache reads and writes included, above that. Its default `longContext` is `{ "above": 100000, "multiplier": 5 }`: a request whose prompt, counting input, cache reads and cache writes, is over `above` tokens is priced at `multiplier` times every rate. Every estimate prices through it. Set `"longContext": null` to remove it, for example when you point the `haiku` alias at another model with `id`. A model with `"efforts": []` receives no effort field. A route in `router.json` without an `effort` takes the effort of that tier's default route. To keep the session effort, clamped to what the model supports, set `"effort": null`, for example `"routes": { "high": { "model": "opus", "effort": null } }`. A 1.3 file whose `low` route had no `effort` kept the session effort; in 1.4 it runs at `high` until you add `"effort": null`.
61
61
 
62
62
  ## Activity routing
63
63
 
64
- The classifier also names the activity of a turn: what the turn produces. `activityRouting` decides what the router does with it.
64
+ The classifier also names the activity of a turn: what the turn produces. `activityRouting` decides what the router does with it. [Activity routing](activity-routing.md) explains the defaults for end users.
65
65
 
66
- | `activityRouting` | Behavior |
67
- | ----------------- | ------------------------------------------------------------------------------------------------------------------ |
68
- | `off` | Asks nothing new. The classifier requests are the ones 1.5 sent, and routes are exactly the tier routes. |
69
- | `shadow` | The default. Asks, shows the activity, records stats, and computes what `on` would run. The tier routes still run. |
70
- | `on` | Applies the `activities` overrides below. |
66
+ | `activityRouting` | Behavior |
67
+ | ----------------- | ----------------------------------------------------------------------------------------------------- |
68
+ | `off` | Does not ask the activity: the classifier is asked for the tier only. The base routes run. |
69
+ | `shadow` | Asks, shows the activity, records stats, and computes what `on` would run. The base routes still run. |
70
+ | `on` | The default since 1.7. Applies the `activities` overrides below. |
71
71
 
72
- The seven activities are `code`, `debug`, `explore`, `plan`, `review`, `ops`, and `docs`. The classifier may also answer `uncertain`. In `on`, an activity applies only when its probability is at least `policy.activityMass`. Below that, or on `uncertain`, the turn goes back to its tier's route, if the move passes the cache checks. When the classifier gives no usable activity answer, the running activity stays and the tier answer still applies. When it gives no answer at all, the running route stays. In `off` and `shadow`, the tier route runs.
72
+ The seven activities are `code`, `debug`, `explore`, `plan`, `review`, `ops`, and `docs`. The classifier may also answer `uncertain`. In `on`, an activity applies only when its probability is at least `policy.activityMass`. Below that, or on `uncertain`, a new task goes back to its base route if the move passes the cache checks; a continuation of the running task keeps its activity, or moves to the classifier's activity when that route is stronger. When the classifier gives no usable activity answer, the running activity stays and the tier answer still applies. When it gives no answer at all, the running route stays. In `off` and `shadow`, the base route runs.
73
73
 
74
- `activities` maps an activity to a tier to a route override. An override may set `model`, `effort`, or both. A field it leaves out comes from the tier's route:
74
+ `activities` maps an activity to a tier to a route override. An override may set `model`, `effort`, or both. A field it leaves out comes from the built-in override for that cell, or from the tier's own route, the **base route**, where there is none:
75
75
 
76
76
  ```
77
77
  route(tier, activity) = { ...routes[tier], ...activities[activity][tier] }
78
78
  ```
79
79
 
80
- The built-in overrides ship with 1.6 and run only with `activityRouting: "on"`. Cells marked `·` have no override:
80
+ The built-in overrides run only with `activityRouting: "on"`. A `·` cell uses the base route:
81
81
 
82
- | Activity | `micro` | `low` | `medium` | `high` |
83
- | --------------------------------- | ------- | --------------- | --------------- | ------ |
84
- | `code`, `debug`, `plan`, `review` | `·` | Sonnet · medium | `·` | `·` |
85
- | `docs` | `·` | Sonnet · medium | Sonnet · medium | `·` |
86
- | `ops`, `explore` | `·` | `·` | Sonnet · medium | `·` |
82
+ | Activity | `micro` | `low` | `medium` | `high` |
83
+ | --------------------------------- | -------------- | ----------------- | ---------------- | ------------ |
84
+ | Base route | Haiku · medium | Haiku · high | Opus · medium | Opus · xhigh |
85
+ | `code`, `debug`, `plan`, `review` | · | **Sonnet · high** | · | · |
86
+ | `ops` | · | · | **Haiku · high** | · |
87
+ | `explore`, `docs` | · | · | · | · |
87
88
 
88
- At `low`, `ops` and `explore` keep Haiku at `high`: they are the cheap cases the `low` route is for. Coding is where Haiku 5.5 trails most (Terminal-Bench 4.0: Haiku 39.2%, Sonnet 70.6%). On knowledge and terminal tasks Sonnet 5.5 matched Opus 5.5 (GDPval-AA 1844 against 1846; Terminal-Bench 70.6% against 66.4%). These are vendor benchmarks at max or xhigh effort, not measurements of this router, and they are not a claim of equal quality. Every override uses Sonnet at `medium`, the one pair the four base routes do not use, so activity routing adds one cache.
89
+ - `code`, `debug`, `plan`, `review` at `low` run on Sonnet at `high`. Coding is where Haiku 5.5 trails most (Terminal-Bench 4.0: Haiku 39.2%, Sonnet 70.6%). Sonnet at `medium` scored below Haiku at `high` on FrontierCode Main (about 37% against 42%); at `high` it scored about 49%.
90
+ - `ops` at `medium` runs on Haiku at `high`. Command runs last long enough for Haiku to repay its cache write: about 4 requests from Opus at 350,000 tokens of context, against a median run of 6 to 11 requests in one developer's sessions. Haiku is weaker on multi-step tool work (OSWorld 2.1: about 61% against 79% for Opus at `medium`), so `ops` at `high` stays on Opus, and repeated tool errors move an `ops` turn to the `high` route.
91
+ - `explore` and `docs` keep the base routes: moving them saved nothing in that replay, and Haiku scores lower on research benchmarks (Humanity's Last Exam with tools: 50.1% against 63.0% for Opus at `medium`).
89
92
 
90
- A user file merges over these overrides one field at a time, as `routes` does. This sets only the effort of the built-in `code` override at `low`, so the cell runs Sonnet at `high`, and adds a new override at `high`:
93
+ These are vendor benchmarks and a list-price replay, not measurements of this router, and they are not a claim of equal quality. The overrides add one (model, effort) pair the base routes do not use, Sonnet at `high`. The router counts each pair as its own cache, so activity routing adds one. [Activity routing](activity-routing.md) has the evidence per cell and its limits.
94
+
95
+ A user file merges over these overrides one field at a time, as `routes` does. This sets only the effort of the built-in `code` override at `low`, so the cell runs Sonnet at `xhigh`, and adds a new override at `high`:
91
96
 
92
97
  ```json
93
98
  {
94
- "activityRouting": "on",
95
99
  "activities": {
96
- "code": { "low": { "effort": "high" }, "high": { "effort": "max" } }
100
+ "code": { "low": { "effort": "xhigh" }, "high": { "effort": "max" } }
97
101
  }
98
102
  }
99
103
  ```
100
104
 
105
+ An override for a cell with no built-in names both fields, or inherits the missing one from the base route. This runs `docs` at `medium` on Sonnet at the base route's `medium` effort:
106
+
107
+ ```json
108
+ { "activities": { "docs": { "medium": { "model": "sonnet" } } } }
109
+ ```
110
+
101
111
  - `effort: null` keeps the session effort, as in `routes`.
102
- - A cell cannot be `null`. To drop a built-in override, set it to the tier's route, for example `"ops": { "medium": { "model": "opus", "effort": "medium" } }`. The pane writes this for you when you remove an override.
112
+ - A cell cannot be `null`. To drop a built-in override, set it to the base route, for example `"ops": { "medium": { "model": "opus", "effort": "medium" } }`. The pane writes this for you when you remove an override. The cell keeps that route if you later change the tier's route; remove it again then.
103
113
  - An override applies to whatever the tier's route is. If you move `routes.low` to Opus, the built-in `code` override at `low` still runs Sonnet. Set it to match, or remove it.
104
114
  - Unknown activities and tiers, unknown model aliases, and invalid efforts make the file invalid, like any other invalid setting. The pane refuses to save one and names it, such as `activities.code.low.model is not in models`.
105
- - Each distinct (model, effort) pair is its own cache. The Routing tab counts them.
115
+ - The router counts each distinct (model, effort) pair as its own cache. The Routing tab counts them.
116
+
117
+ `policy.activityMass` (default `0.6`) is the lowest probability at which an activity applies. The Routing tab sets it as **Activity threshold** under POLICY, from 50% to 80%. It has not been tuned on any classifier, and the probe results do not record probabilities, so there is no data to tune it on yet.
106
118
 
107
- `policy.activityMass` (default `0.6`) is the lowest probability at which an activity applies. It has not been tuned on any classifier. Probabilities differ between classifiers, so recalibrate it from the [probe set](evaluation.md#activity-probe-set) if you change classifier.
119
+ Switch the mode with `/router activities off|shadow|on`, or with the **Activity routing** selector at the top of the Routing tab's ACTIVITIES section. The command writes `router.json` at once; the selector joins the Routing draft and writes on **Save**. Choosing `on`, the default, removes the key from the file.
108
120
 
109
- Switch the mode with `/router activities off|shadow|on`, or with the selector at the top of the Routing tab's OVERRIDES section. The command writes `router.json` at once; the selector joins the Routing draft and writes on **Save**. Choosing `shadow`, the default, removes the key from the file.
121
+ **Upgrade from 1.6.** 1.6 defaulted to `shadow` with Sonnet at `medium` in every override. A file without `activityRouting` now runs `on`; a file that sets it keeps its mode. Choosing `shadow` in 1.6 removed the key, because `shadow` was the default then: if you chose `shadow` in 1.6, run `/router activities shadow` again after upgrading. A session started on Sonnet 5.5 now starts with routing on, because Sonnet runs the `code` cell at `low`; to stay on Sonnet in that session, choose it with `/model` or run `/router off`, and run `/router activities shadow` to make future Sonnet sessions start with routing off. Cells you saved stay as written. A cell that sets one field takes the other from 1.7's built-in for that cell, or from the base route where 1.7 has none (`explore` and `docs` at `medium`, `docs` at `low`). 1.6's overrides in those cells and in `ops` at `medium` were Sonnet · medium, so such a cell can now run another model or effort; write both fields to keep 1.6's route. A 1.6 removal of a built-in that 1.7 no longer has, such as `docs` at `low`, stays in the file as the tier's route; the pane lists it as "same as base: no effect", and **remove** drops it.
110
122
 
111
- **Rollback.** Router 1.5 rejects unknown keys and makes routing unavailable. Before you install 1.5 again, remove `activities`, `activityRouting`, and `policy.activityMass` from `router.json`. The pane writes them when you set the mode to `off` or `on`, or edit or remove an override.
123
+ **Rollback.** To go back to 1.6 behavior without downgrading, set `"activityRouting": "shadow"`, or `off`. Router 1.6 reads the same keys; under 1.6 a file without `activityRouting` runs `shadow` with 1.6's overrides. Router 1.5 rejects unknown keys and makes routing unavailable. Before you install 1.5 again, remove `activities`, `activityRouting`, and `policy.activityMass` from `router.json`. The pane writes them when you set the mode to `off` or `shadow`, change the activity threshold, or edit or remove an override.
112
124
 
113
125
  ## Classifiers
114
126
 
@@ -176,9 +188,9 @@ The Mod validates the whole file. Unknown keys and invalid values make routing u
176
188
  | ------------------------------ | ------------------------------------------------------------------------------------ | -------------------------------------------------------- |
177
189
  | `baselineTier` | `micro`, `low`, `medium`, `high` | `low` |
178
190
  | `routes.<tier>` | `model`, optional `effort` (a level, or `null` for the session effort) | As in the route table above |
179
- | `activityRouting` | `off`, `shadow`, `on` | `shadow` |
191
+ | `activityRouting` | `off`, `shadow`, `on` | `on` |
180
192
  | `activities.<activity>.<tier>` | optional `model` and `effort`, as in `routes` | The overrides in [Activity routing](#activity-routing) |
181
- | `models.<alias>` | `id`, `input`, optional `output`, `cacheRead`, `contextWindow`, `billing`, `efforts` | Three defaults above |
193
+ | `models.<alias>` | `id`, `input`, optional `output`, `cacheRead`, optional `longContext` (`above`, `multiplier`, or `null`), `contextWindow`, `billing`, `efforts` | Three defaults above |
182
194
  | `cache` | `writeMultiplier`, `ttlMs`, `warmMarginMs` | `5m: 1.25`, `1h: 2`; `300000` / `3600000` ms; `30000` ms |
183
195
  | `policy` | Fields below | Values below |
184
196
  | `classifier` | An id in `classifiers` | `jev` |
@@ -187,24 +199,24 @@ The Mod validates the whole file. Unknown keys and invalid values make routing u
187
199
 
188
200
  Policy defaults:
189
201
 
190
- | Field | Default | Meaning |
191
- | ----------------------- | ------: | --------------------------------------------------------------------------------------------------------- |
192
- | `upgradeVotes` | `2` | Consecutive supporting votes before an upgrade; one before a history's first measured reply. |
193
- | `upgradeBase` | `0.75` | Minimum probability mass required for an upgrade. |
194
- | `upgradeSlope` | `0.15` | Maximum increase to the upgrade bar from estimated switching cost. |
195
- | `upgradePivotUsd` | `0.5` | Cost scale used by the upgrade bar. Must be positive. |
196
- | `jumpConfidence` | `0.95` | Support for a two-tier jump without waiting for votes. |
197
- | `downgradeVotes` | `2` | Consecutive supporting votes before a downgrade; one before a history's first measured reply. |
198
- | `downgradeMass` | `0.9` | Minimum probability mass required for a downgrade. |
199
- | `downgradeSlope` | `0.08` | Maximum increase to the downgrade bar from estimated switching cost. |
200
- | `downgradePivotUsd` | `0.5` | Cost scale used by the downgrade bar. Must be positive. |
201
- | `downgradeHorizonTurns` | `5` | Later turns included in a downgrade payback estimate. |
202
- | `continuationMass` | `0.7` | Advice probability that keeps the current route for a continuation. |
203
- | `escalationHoldTurns` | `2` | Turns held after a repeated tool error escalates the route. |
204
- | `cashCapUsd` | `2` | Maximum estimated cold cache write for a `credits` model. It does not cap output or session spend. |
205
- | `activityMass` | `0.6` | Minimum probability for the classifier's activity to apply. Not tuned on any classifier. Not in the pane. |
206
-
207
- The pane writes a subset of these settings. The **Routing** tab sets `routes`, `baselineTier`, `activities`, `activityRouting`, `policy.downgradeVotes`, `policy.downgradeHorizonTurns`, and `policy.cashCapUsd` in one save: it writes only values that differ from the defaults and removes the rest. `/router activities <mode>` writes `activityRouting` at once, with **Undo**. The **Classifier** tab sets `classifier` and the active classifier's `timeoutMs` at once. **Undo** reverts the settings the last pane write changed. Each save validates the whole file, keeps other keys, and applies from the next turn. A failed check names the setting and leaves the file unchanged. The pane refuses to write through a symlink. Model aliases and the other settings require a direct edit.
202
+ | Field | Default | Meaning |
203
+ | ----------------------- | ------: | -------------------------------------------------------------------------------------------------- |
204
+ | `upgradeVotes` | `2` | Consecutive supporting votes before an upgrade; one before a history's first measured reply. |
205
+ | `upgradeBase` | `0.75` | Minimum probability mass required for an upgrade. |
206
+ | `upgradeSlope` | `0.15` | Maximum increase to the upgrade bar from estimated switching cost. |
207
+ | `upgradePivotUsd` | `0.5` | Cost scale used by the upgrade bar. Must be positive. |
208
+ | `jumpConfidence` | `0.95` | Support for a two-tier jump without waiting for votes. |
209
+ | `downgradeVotes` | `2` | Consecutive supporting votes before a downgrade; one before a history's first measured reply. |
210
+ | `downgradeMass` | `0.9` | Minimum probability mass required for a downgrade. |
211
+ | `downgradeSlope` | `0.08` | Maximum increase to the downgrade bar from estimated switching cost. |
212
+ | `downgradePivotUsd` | `0.5` | Cost scale used by the downgrade bar. Must be positive. |
213
+ | `downgradeHorizonTurns` | `5` | Later turns included in a downgrade payback estimate. |
214
+ | `continuationMass` | `0.7` | Advice probability that keeps the current route for a continuation. |
215
+ | `escalationHoldTurns` | `2` | Turns held after a repeated tool error escalates the route. |
216
+ | `cashCapUsd` | `2` | Maximum estimated cold cache write for a `credits` model. It does not cap output or session spend. |
217
+ | `activityMass` | `0.6` | Minimum probability for the classifier's activity to apply. Not tuned on any classifier. |
218
+
219
+ The pane writes a subset of these settings. The **Routing** tab sets `routes`, `baselineTier`, `activities`, `activityRouting`, `policy.downgradeVotes`, `policy.downgradeHorizonTurns`, `policy.cashCapUsd`, and `policy.activityMass` in one save: it writes only values that differ from the defaults and removes the rest. `/router activities <mode>` writes `activityRouting` at once, with **Undo**. The **Classifier** tab sets `classifier` and the active classifier's `timeoutMs` at once. **Undo** reverts the settings the last pane write changed. Each save validates the whole file, keeps other keys, and applies from the next turn. A failed check names the setting and leaves the file unchanged. The pane refuses to write through a symlink. Model aliases and the other settings require a direct edit.
208
220
 
209
221
  Native cache freshness is unknown after 270 seconds or after a history reset. Claude usage does not report the cache TTL. The policy evaluates price bounds for five-minute and one-hour writes, but the one-hour case is not observed fact. See the [architecture](architecture.md#cache-and-cost) for the estimate rules.
210
222
 
@@ -1,6 +1,6 @@
1
1
  # Evaluation
2
2
 
3
- The native Mod has no measured savings result. The pane reports Claude's own usage and configured-price scenarios. It does not maintain a counterfactual bill or a cumulative savings total.
3
+ The native Mod has no measured savings result. The pane reports Claude's own usage and configured-price scenarios. Its one counterfactual is [routing vs your model](#routing-vs-your-model), an estimate at configured list prices, not a bill or a measured saving.
4
4
 
5
5
  This page separates an older gateway experiment from a shadow replay of native policy. Neither result measures answer quality or proves lower real spend. [Activity routing](#activity-routing) adds two checks of the activity label, with no savings result either.
6
6
 
@@ -54,7 +54,9 @@ The script writes aggregate counts, reason histograms, route transitions, and sh
54
54
 
55
55
  ## Activity routing
56
56
 
57
- Activity routing has no measured result. The replay above predates it and covers tiers only. The pane counts turns, requests, switches, and the turns `on` would route differently, and it shows no dollar figure for them. Two checks look at the classifier's activity label.
57
+ Activity routing has no measured result. The replay above predates it and covers tiers only. It is `on` by default since 1.7, with defaults chosen from Anthropic's published benchmarks and a list-price replay of one developer's sessions; [Activity routing](activity-routing.md#what-the-estimates-mean) gives that replay and its limits. The pane counts turns, requests, switches, and the turns `on` would route differently, with a configured list-price estimate of those turns' next requests, not a measured saving. Two checks look at the classifier's activity label.
58
+
59
+ The figures plan §11 tracks (tool agreement of at least 80% on `code` against `ops` and `explore`, a negative would-route estimate in `shadow`, the number of activity switches) are read from the Usage tab's **ACROSS SESSIONS** block, which shows the counts kept since the last reset: **Code/ops/explore** for the tool agreement on turns the classifier answered `code`, `ops` or `explore`, **Agreement** and **Mismatch** for all of them, **Shadow** for the would-route count and estimate, **Activity moves** for activity switches inside a tier. `/router` without a UI surface prints both agreements. The underlying counts are the `activity:stats:v1` store described in [Architecture](architecture.md).
58
60
 
59
61
  ### Tool agreement
60
62
 
@@ -79,10 +81,10 @@ The classifier's activity agrees with the turn when the bucket is one it allows:
79
81
  | `explore` | `read`, `talk` |
80
82
  | `plan` | `talk`, `read`, `docs` |
81
83
  | `review` | `read`, `talk` |
82
- | `ops` | `ops` |
84
+ | `ops` | `ops`, `read` |
83
85
  | `docs` | `docs` |
84
86
 
85
- Agreement is a weak check. It tells `ops` from `code` from `explore` well. It cannot tell `plan` from `review`, and it says nothing about whether the route was good enough. It does catch the costly mistake, a cheap route on a coding turn. The Usage tab shows the session's count as `Agreement`, over the turns where the classifier gave an activity at or above `activityMass`, in `on` too when the route kept another activity. The counts kept across sessions hold the classifier's raw answer, `uncertain` and no answer included, against the bucket. They are not shown in the pane.
87
+ Agreement is a weak check. It tells `code` from `ops` and `explore` well, but not `ops` from `explore` when the commands only read. It cannot tell `plan` from `review`, and it says nothing about whether the route was good enough. It does catch the costly mistake, a cheap route on a coding turn. The Usage tab shows the session's count as `Agreement`, over the turns where the classifier gave an activity at or above `activityMass`, in `on` too when the route kept another activity. The counts kept across sessions hold the classifier's raw answer, `uncertain` and no answer included, against the bucket. The pane's **ACROSS SESSIONS** block counts agreement over the answers that name an activity; `uncertain` and no answer appear only in **Labelled**.
86
88
 
87
89
  ### Activity probe set
88
90
 
@@ -94,14 +96,43 @@ node scripts/probe-activity.mjs [classifier]
94
96
 
95
97
  The script sends each prompt to one classifier as the router would ask it, with activity routing forced to `shadow`. It reads credentials as `probe-classifier.mjs` does and needs a classifier you can reach. The run is billed on a paid classifier, and it is run by hand, not in CI. It writes accuracy, an expected-by-answered confusion matrix, p50 and p95 latency against the classifier's deadline, and error counts to `experiments/mod-router/results/activity-probe-<classifier>.json`. For Ollama the latency covers both requests. It never prints a key.
96
98
 
97
- Two probe results are checked in, both run on 2026-10-10 against the live Cloudflare API with the synthetic set above:
99
+ Five probe results are checked in, all run on 2026-10-10 with the synthetic set above: Jev against the live typesafe.ai API, OpenAI against the live Decisions API, Clef and Clef Flash against the live Cloudflare API, Ollama with `qwen3.5:9b` on an Apple M2 Pro laptop:
98
100
 
99
101
  | Classifier | Correct | p50 latency | p95 latency | Result file |
100
102
  | ---------- | -------: | ----------: | ----------: | ---------------------------------------------------------------------------------------------------- |
103
+ | Jev | 70 of 70 | 260 ms | 299 ms | [`activity-probe-jev.json`](../experiments/mod-router/results/activity-probe-jev.json) |
104
+ | OpenAI | 70 of 70 | 161 ms | 286 ms | [`activity-probe-openai.json`](../experiments/mod-router/results/activity-probe-openai.json) |
101
105
  | Clef | 69 of 70 | 444 ms | 883 ms | [`activity-probe-clef.json`](../experiments/mod-router/results/activity-probe-clef.json) |
102
106
  | Clef Flash | 69 of 70 | 470 ms | 874 ms | [`activity-probe-clef-flash.json`](../experiments/mod-router/results/activity-probe-clef-flash.json) |
107
+ | Ollama | 66 of 70 | 2986 ms | 3483 ms | [`activity-probe-ollama.json`](../experiments/mod-router/results/activity-probe-ollama.json) |
108
+
109
+ Ollama's latency covers both requests and stays inside its 5000 ms deadline; on slower hardware the activity step is skipped when less than 300 ms remain. They measure the label on synthetic prompts, not routing quality or savings. The activity question costs little time: 30 alternating pairs of the same prompts, with and without it, gave Jev p50 259 ms both ways and p95 300 ms without, 306 ms with; OpenAI p50 163 ms and 162 ms, p95 296 ms and 349 ms. OpenAI's first request of a run took about 2.3 s of its 3000 ms deadline. The pane shows no probe accuracy. `policy.activityMass` was not tuned on any classifier; run the probe before you trust it with a new one.
110
+
111
+ ## Routing vs your model
112
+
113
+ The Usage tab compares what routed replies cost with what the same tokens would cost on your model, the session's model and effort before routing. It answers "at list prices, did routing spend more or less than my own model for this work". It does not answer whether the answers were as good.
114
+
115
+ Method, per main reply that routing chose:
116
+
117
+ - **Routed** prices the usage Claude reported (uncached input, cache read, cache write, output) on the model that served it.
118
+ - **Yours** prices the same tokens on your model. While this reply and the one before it ran on your route, it is the reply's own usage, so a session on your model differs by exactly $0.00. Otherwise your model keeps one cache that never switched: it reads the previous prompt, up to this one less its uncached input, inside the cache lifetime, unless a compaction shrank the history or your model or effort changed since, and never less than the API actually read; it writes the rest. Output tokens are the same.
119
+ - **Parts**: the served model on your cache against yours is the model price, under cheaper models when negative and stronger than yours when positive; routed against the served model on your cache is the cache the switches cost, never negative. They sum to the difference. The pane rounds to cents and gives the remainder only to a part that is not zero: a surplus to the switch part, else the stronger part, else the cheaper part toward $0.00; a shortfall comes off the stronger part, down to $0.00, then the cheaper part. The shown parts add up, none changes sign, and none appears from rounding alone.
120
+ - **Prices**: `router.json` list prices through `ratesAt`, so Haiku 5.5 above 100,000 prompt tokens is priced at 5x. Claude Code reports no 5m/1h split to a Mod, so cache writes use the lifetime Claude Code 2.1.296 gives the main conversation by its own rule: one hour on a Claude plan unless an environment variable or the `promptCacheTtl` setting says otherwise, five minutes with an API key.
121
+
122
+ Checks:
123
+
124
+ - Unit tests (`test/savings.test.mjs`) cover a session that saves, one that costs more with Haiku as your model, equal routes at exactly 0, cache expiry at 5m and 1h, a compaction, a change of your model or effort, the 100K surcharge, an unpriced model, and a corrupted store. `test/hook-routing.test.mjs` covers two sessions finishing turns at the same moment.
125
+ - A one-off replay, not checked in, of this machine's main-conversation transcripts from 2026-10-01 to 2026-10-10 (nine sessions with 10 or more replies) gave $0.00 for every session that ran only on the model it was compared with: three on Opus 5.5, one on Haiku 5.5. Every main-conversation cache write in them was a one-hour write, which is what the lifetime rule gives on this Max plan.
126
+ - A live Claude Code 2.1.296 session on 2026-10-10, Opus 5.5 at `high` with turns pinned to Haiku 5.5, showed **Routed replies $0.48** beside **Cost $0.480 reported by Claude** for the same 16 replies; its last Haiku reply had a 114K-token prompt, which agrees only at the 5x rate. A Haiku 5.5 session with a pinned Opus turn showed $0.45 routed against $0.446 reported.
127
+
128
+ Limits:
103
129
 
104
- They measure the label on synthetic prompts, not routing quality or savings. Jev, OpenAI and Ollama have no probe result yet. The pane shows no probe accuracy. `policy.activityMass` was not tuned on any classifier; run the probe before you trust it with a new one.
130
+ - Output length is held the same. A stronger model may write less, or finish in fewer tool steps; a cheaper one may need more. The readout cannot see either.
131
+ - Answer quality is not measured. A saving bought with worse answers would read the same.
132
+ - Your model's cache is simulated, and a five-minute session that the lifetime rule takes for one hour (a plan subscriber billed as overage) would read too much cache on your side.
133
+ - A route that changes only the effort costs the same per token; it shows only its cache writes.
134
+ - Subagents, classifier charges and replies with routing off are not counted. On a Claude plan the dollars stand for quota, not cash.
135
+ - The saved column keeps one record per session, so sessions finishing turns at the same moment both count. Claude Code's store has no atomic update, which leaves these gaps: the same session open in two Claude Code processes at once can lose one's turn; a turn finishing in another process during **Reset stats** can be lost; a session's total from before a reset comes back if its first saved reply has the same millisecond as the reset, or if the system clock moved back across the reset; and the activity counts across sessions are still one shared record, where two sessions finishing turns at the same moment can lose one turn's counts. The router reads every session's record at session start and when the pane opens, so that read grows with the number of sessions since the last reset.
105
136
 
106
137
  ## Historical method and limits
107
138
 
@@ -10,7 +10,7 @@ The classifier receives the current prompt and at most six preceding user or ass
10
10
 
11
11
  The default tiers are defined in [Configuration](configuration.md#built-in-defaults). Policy uses vote hysteresis, repeated tool errors, context fit, model availability, and switching-cost estimates. The [architecture](architecture.md#request-flow) explains the event boundary and failure handling.
12
12
 
13
- The activity is what the turn produces: `code`, `debug`, `explore`, `plan`, `review`, `ops`, or `docs`. A route is the tier's route with the activity's override on top. In `shadow`, the default, the router shows and records the activity and what `on` would route, and keeps the tier route. In `on` it applies the overrides. The activity question reuses the same prompt and dialogue; no other text is sent. With `off` the request is the one 1.5 sent. [Route by activity](user-guide.md#route-by-activity) explains the modes, and [Configuration](configuration.md#activity-routing) the matrix.
13
+ The activity is what the turn produces: `code`, `debug`, `explore`, `plan`, `review`, `ops`, or `docs`. A route is the tier's own route, the base route, with the activity's override on top. In `on`, the default since 1.7, it applies the overrides. In `shadow` the router shows and records the activity and what `on` would route, and keeps the base route. The activity question reuses the same prompt and dialogue; no other text is sent. With `off` the classifier is asked for the tier only. [Route by activity](user-guide.md#route-by-activity) explains the modes, [Activity routing](activity-routing.md) the default matrix and its evidence, and [Configuration](configuration.md#activity-routing) the schema.
14
14
 
15
15
  ## Controls and session modes
16
16
 
@@ -29,7 +29,7 @@ Mode belongs to a Claude Code session. Clear starts a new session with routing o
29
29
 
30
30
  ## Router pane
31
31
 
32
- The status band is one line above the prompt: a tier meter, the activity, the route, a short reason, classifier support, and context and cache use. Segments drop by priority when the band is narrow: classifier support first, then context and cache, then the activity, then the reason. A **Routing off** button sits on the band while routing is on, and hovering reveals pins and a two-row view. A fresh session on a model no tier routes to shows `<model> is not a routing tier` on the band until the mode changes. The band yields to surveys and does not describe subagent transcripts. A model change between turns raises a toast. A pin, or a step that changes only the effort, such as `medium` to `high` on the defaults, raises none. While the classifier runs, the turn's spinner says `Choosing model`. In Manual mode or when routing is unavailable, the prompt footer carries `routing off` or `routing unavailable`. The **Router** button opens the pane. The pane has four tabs, Now, Routing, Classifier, and Usage, with Routing on and Routing off on every tab. The [user guide](user-guide.md#open-the-router-pane) describes each tab.
32
+ The status band is one line above the prompt: a tier meter, the activity, the route, a short reason, classifier support, and context and cache use. Segments drop by priority when the band is narrow: classifier support first, then context and cache, then the activity, then the reason. Hovering reveals pins, **Turn off**, and a two-row view; while routing is off, **Turn on** sits on the band itself, so the band carries one routing control at a time. A fresh session that starts with routing off because of its model shows `<model> is not a routing tier` on the band until the mode changes. The band yields to surveys and does not describe subagent transcripts. A model change between turns raises a toast. A pin, or a step that changes only the effort, such as `medium` to `high` on the defaults, raises none. While the classifier runs, the turn's spinner says `Choosing model`. In Manual mode or when routing is unavailable, the prompt footer carries `routing off` or `routing unavailable`. The **Router** button opens the pane. The pane has four tabs, Now, Routing, Classifier, and Usage, with the routing on/off pair at the top of every tab. The [user guide](user-guide.md#open-the-router-pane) describes each tab.
33
33
 
34
34
  | Reading | Meaning and limits |
35
35
  | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
@@ -38,7 +38,9 @@ The status band is one line above the prompt: a tier meter, the activity, the ro
38
38
  | Activity | The classifier's activity choice with its share and the two next-best answers. Empty with routing off, a pin, or no answer. |
39
39
  | Route (activity) | With `on`: the route for the tier and activity, marked `(override)` with the base route beside it, or `(base)`. With `shadow`: what `on` would use, and the route in use (`shadow; same route` when they agree). |
40
40
  | Replies | The tier that served each of the last 30 main-conversation replies; a dot where the router did not choose. A second row of letters names each reply's activity: `c` code, `d` debug, `e` explore, `p` plan, `r` review, `o` ops, `w` docs. |
41
- | Activity stats (Usage) | Counts for this session: turns, requests and share per activity; switches by tier and by activity; tool agreement; in `shadow`, turns `on` would route differently. Counts only, no dollar figures. |
41
+ | Routing vs your model (Usage) | Routed replies at configured list prices against the same tokens on your model (the session's model and effort), this session and since the last reset (each session against its own model); the difference split into cheaper models, stronger than yours, and cache writes from switches. `too early` below 10 replies. Output length is held the same; answer quality is not measured. |
42
+ | Activity stats (Usage) | Counts for this session: turns, requests and share per activity; switches by tier and by activity; tool agreement; in `shadow`, turns `on` would route differently with their summed next-request estimate at configured list prices. |
43
+ | Across sessions (Usage) | The counts kept since the last reset: labelled turns, tool agreement on the classifier's raw answer, overall and for `code`, `ops` and `explore`, the two largest mismatches, activity moves inside a tier taken and refused, and the shadow readout. `/router` text shows the agreement. |
42
44
  | Cost | Claude Code's native usage value. A plan's list-price estimates are not subscription cash charges. |
43
45
  | Context bar | Maximum available observed and local estimated context, compared with the routed model's window. The fit rule reserves 20%. |
44
46
  | Observed / estimated input | Claude's local context reading and the router's best current context estimate. `unknown` blocks a move to a smaller window. |
@@ -50,20 +52,21 @@ The status band is one line above the prompt: a tier meter, the activity, the ro
50
52
  | Conservative payback | Conditional later-turn estimate based on configured cache prices, future reads, and the last output size. It is not measured savings. |
51
53
  | Cache read benefit | Estimated price difference for observed cache reads before cache writes. It is not net savings. |
52
54
 
53
- There is no router-owned cost ledger or cumulative savings counter. Classifier charges are not included. See [Evaluation](evaluation.md) for what the available trace can support.
55
+ The router keeps no bill. Routing vs your model is a configured-price estimate, not a measured saving, and leaves out classifier charges. See [Evaluation](evaluation.md) for what the available trace can support.
54
56
 
55
57
  ## Routing and classifier settings
56
58
 
57
- The Routing tab edits each tier's model alias and effort, the baseline tier, the activity overrides and mode, and three policy controls as one draft. The draft does not affect the active turn, survives a tab change or closing the pane, and is dropped at a new session. **Save** validates the whole file with the same loader the Mod starts with, writes `router.json` for future turns, and keeps unrelated keys. A route, activity override, or policy value equal to the built-in default is removed from the file, so later default changes still reach it. The Classifier tab writes the classifier choice and its deadline at once. **Undo** reverts the settings the last pane write changed, from any tab, until the next write or a new session.
59
+ The Routing tab edits each tier's model alias and effort, the baseline tier, the activity overrides and mode, and four policy controls as one draft. The draft does not affect the active turn, survives a tab change or closing the pane, and is dropped at a new session. **Save** validates the whole file with the same loader the Mod starts with, writes `router.json` for future turns, and keeps unrelated keys. A route, activity override, or policy value equal to the built-in default is removed from the file, so later default changes still reach it. The Classifier tab writes the classifier choice and its deadline at once. **Undo** reverts the settings the last pane write changed, from any tab, until the next write or a new session.
58
60
 
59
- | Control | Values in the pane | Effect |
60
- | ---------------- | ---------------------------------- | ---------------------------------------------------------------------------------------- |
61
- | Deadline | 500, 1,000, 1,500, 3,000, 5,000 ms | The active classifier's total advice time, including any transient retry. Saved at once. |
62
- | Votes to go down | 1, 2, 3 | Consecutive votes for a downgrade; one before a history's first measured reply. |
63
- | Payback horizon | 1, 3, 5, 10 turns | Later turns included in downgrade economics. |
64
- | Credits cap | $0.50, $1, $2, $5 | Largest estimated cold write on a `credits` model. |
61
+ | Control | Values in the pane | Effect |
62
+ | ------------------ | ---------------------------------- | ---------------------------------------------------------------------------------------- |
63
+ | Deadline | 500, 1,000, 1,500, 3,000, 5,000 ms | The active classifier's total advice time, including any transient retry. Saved at once. |
64
+ | Votes to go down | 1, 2, 3 | Consecutive votes for a downgrade; one before a history's first measured reply. |
65
+ | Payback horizon | 1, 3, 5, 10 turns | Later turns included in downgrade economics. |
66
+ | Credits cap | $0.50, $1, $2, $5 | Largest estimated cold write on a `credits` model. |
67
+ | Activity threshold | 50%, 60%, 70%, 80% | `policy.activityMass`: the least probability at which an activity applies. |
65
68
 
66
- Under the routes, **ACTIVITIES** shows the effective matrix of the draft: one row per distinct set of cells, `·` where a cell uses the tier's route, and the count of distinct routes, each its own cache. Until the mode is `on` the count carries `once activity routing is on`. **OVERRIDES** starts with the `off · shadow · on` selector. Each override is a row with a model, an effort, and **remove**, and a selector adds one for a free activity and tier. A removed built-in override is saved as the tier's own route, because a file that only omitted it would get the built-in back. Yellow notes flag an override equal to its tier's route, one stronger than the tier above, and one that adds a cache. **Reset routes to defaults** resets the routes only; pending override and mode edits stay in the draft. The Usage tab's **Reset activity stats** clears the session counts and the counts kept across sessions.
69
+ Under the routes, **ACTIVITIES** starts with the **Activity routing** selector, `off · shadow · on`, and a hint for the mode. Below it is the effective matrix of the draft: one row per distinct set of cells, `·` where a cell uses the base route, routes abbreviated such as `S·high` for Sonnet · high, and the count of distinct routes, each counted as its own cache. Until the mode is `on` the count carries `once activity routing is on`. **OVERRIDES** reads `in use`, or in yellow `not in use:` with the reason: activity routing is `shadow` or `off`, routing is off, or routing is unavailable. Each override is a row with a model, an effort, and **remove**, and a selector adds one for a free activity and tier. A removed built-in override is saved as the tier's own route, because a file that only omitted it would get the built-in back. Yellow notes flag an override equal to its base route, one stronger than the tier above, and the only cell or tier on its (model, effort) pair, which adds a cache. **Reset routes to defaults** resets the routes only; pending override and mode edits stay in the draft. The Usage tab's **Reset stats** clears the activity counts and routing vs your model, for this session and across sessions.
67
70
 
68
71
  The classifier section lists every configured classifier as a row with its service host and whether its key and endpoint settings are complete. Selecting a row writes `classifier` to `router.json` at once, or removes it for the default, through the same validation, and resets the new classifier's failure count. A turn already being classified finishes with its own classifier; its readings stay off the pane. **Undo** returns to the previous classifier. The deadline saves at once too, and a built-in classifier's default deadline is written as no override. **Sends** names the host that receives prompt text.
69
72
 
@@ -72,10 +75,11 @@ The pane refuses to write through a symlink. A failed validation names the setti
72
75
  ## Safety and failure states
73
76
 
74
77
  - A missing key or account ID, a classifier failure, or a policy refusal keeps the current model. Network refusal is never bypassed with a helper process.
75
- - A missing, malformed, or timed-out activity answer keeps the running activity in `on`; the tier answer still applies. In `shadow` the tier route runs as always. It does not count as a classifier failure. If the whole classifier call fails, the running route stays. Activity statistics are written inside a guard: a failed store read or write loses that turn's counts and never fails a turn.
78
+ - A missing, malformed, or timed-out activity answer keeps the running activity in `on`; the tier answer still applies. In `shadow` the base route runs as always. It does not count as a classifier failure. If the whole classifier call fails, the running route stays. Activity statistics are written inside a guard: a failed store read or write loses that turn's counts and never fails a turn.
76
79
  - Three launched classifier failures open a 60-second pause for that classifier. One in-flight request is allowed per classifier in each active Mod instance. Pending host HTTP work blocks another request until it settles.
77
80
  - A `router.json` with v0.8 gateway keys or the 1.1 `jev` section marks the router unavailable until the [migration command](configuration.md#convert-an-older-routerjson) converts it.
78
81
  - Unsupported Claude Code versions and leftover v0.8 gateway settings (the `jev-router` model or the `127.0.0.1:43170` base URL) mark the router unavailable. `/router` lists the settings to remove. A local gateway does not run as part of this Mod.
82
+ - While the router is unavailable, the pane header reads `ROUTING unavailable Claude’s model is kept` with the reason, and offers no on/off pair or pins. `/router` without a surface starts with `Routing unavailable: <reason>. Claude’s model is kept.`, and `/router auto`, `off`, `pin`, and `activities` reply the same and change nothing.
79
83
  - A context-window error makes that model ineligible until the next history reset. The Mod does not retry an Anthropic request after a stream begins.
80
84
  - Unknown context retains the current model. There is no characters-to-tokens fallback.
81
85
  - Unknown model substitutions receive no cache credit for the requested model.