@alexeiled/claude-router 1.23.1 → 1.24.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/plugin.json +1 -1
- package/CHANGELOG.md +18 -1
- package/docs/auto-engine.md +152 -0
- package/docs/configuration.md +36 -1
- package/docs/evaluation.md +219 -0
- package/docs/providers.md +28 -0
- package/docs/routing-modes.md +8 -1
- package/docs/user-guide.md +9 -2
- package/hooks/native-router.mjs +877 -151
- package/lib/auto-engine.mjs +407 -0
- package/lib/auto-route.mjs +247 -0
- package/lib/config.mjs +42 -8
- package/lib/decisions.mjs +71 -0
- package/lib/facts.mjs +27 -0
- package/lib/panel.mjs +4 -0
- package/lib/providers.mjs +7 -0
- package/lib/replay-trace.mjs +547 -0
- package/lib/replay.mjs +712 -0
- package/lib/route.mjs +14 -5
- package/package.json +1 -1
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "router",
|
|
3
3
|
"displayName": "Router",
|
|
4
|
-
"version": "1.
|
|
4
|
+
"version": "1.24.0",
|
|
5
5
|
"description": "Native model and effort routing within your provider. Anthropic, DeepSeek, z.ai, MiniMax, MiMo and Kimi defaults, custom provider profiles, classifier advice and one model per subagent.",
|
|
6
6
|
"author": {
|
|
7
7
|
"name": "Alexei Ledenev",
|
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,22 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 1.24.0 (2026-11-17)
|
|
4
|
+
|
|
5
|
+
### Added
|
|
6
|
+
|
|
7
|
+
- Opt in to stateful main-turn routing with `routingEngine: "auto"`. The default `table` behavior is unchanged; opt-in `shadow` records proposals while the table routes. Auto evaluation and default cutover remain pending. See [configuration](docs/configuration.md#routing-engine).
|
|
8
|
+
- Per-effort capability declarations guide auto and shadow. Anthropic seeds four balanced base-route pairs: Haiku 5.5 medium 0 and high 1; Opus 5.5 medium 2 and xhigh 3. These are user-derived policy judgments, not quality calibration. Other pairs and provider presets remain unknown. A supplied map replaces the preset; changing a model ID clears inherited capability. See [providers](docs/providers.md#capability-evidence).
|
|
9
|
+
- Shadow collects bounded, text-free replay inputs. Traces exclude prompts, tool results, filesystem-path fields, classifier endpoints and credentials, but retain routing aliases and model IDs, including path-shaped IDs. `/router trace export` merges retained traces into JSON, overwriting the previous export and rolling over near 3 MiB. **Delete history** does not delete exported files. See [trace limits and privacy](docs/evaluation.md#replay-trace-recorder).
|
|
10
|
+
- Offline replay from a checkout checks table fidelity before simulating auto. Coverage and exposure gates remain unavailable or not established; costs are conditional estimates, not savings or quality evidence. The recorded benchmark smoke checks harness mechanics on installed Router 1.23.1 only, not this checkout or an engine comparison. See [evaluation](docs/evaluation.md#offline-automatic-engine-replay).
|
|
11
|
+
|
|
12
|
+
### Changed
|
|
13
|
+
|
|
14
|
+
- Equal-cost auto candidates favor the incumbent, then the lower sufficient declared capability. Estimates do not model unseen effort-dependent thinking tokens; mode margin supplies headroom. See [selection](docs/auto-engine.md#demand-target-and-move).
|
|
15
|
+
|
|
16
|
+
### Upgrade
|
|
17
|
+
|
|
18
|
+
Restart Claude Code after updating. Auto and shadow require explicit opt-in.
|
|
19
|
+
|
|
3
20
|
## 1.23.1 (2026-11-15)
|
|
4
21
|
|
|
5
22
|
### Fixed
|
|
@@ -323,7 +340,7 @@ The Now tab shows this session and today instead of the last 30 replies, a windo
|
|
|
323
340
|
|
|
324
341
|
## 1.12.0 (2026-10-13)
|
|
325
342
|
|
|
326
|
-
The Usage tab now shows any period, not only this session and the time since the last reset. See [Compare routing with
|
|
343
|
+
The Usage tab now shows any period, not only this session and the time since the last reset. See [Compare routing with no routing](docs/user-guide.md#compare-routing-with-no-routing).
|
|
327
344
|
|
|
328
345
|
### Added
|
|
329
346
|
|
|
@@ -0,0 +1,152 @@
|
|
|
1
|
+
# Pure automatic routing engine
|
|
2
|
+
|
|
3
|
+
`lib/auto-engine.mjs` implements the [automatic routing design](plans/2026-11-16-auto-routing-design.md).
|
|
4
|
+
It has no I/O. Opt-in `routingEngine: "auto"` uses the stateful adapter `lib/auto-route.mjs` in the main-turn hook; the default table engine is unchanged. Shadow remains a one-step proposal and offline replay has its own state. Evaluation and default cutover remain pending.
|
|
5
|
+
|
|
6
|
+
## Inputs
|
|
7
|
+
|
|
8
|
+
All functions take one object. Pass the original validated config, including its catalog, cache settings and
|
|
9
|
+
base `policy`. Do not pass `applyMode(config, mode)`: per-mode policy patches belong to the table engine only.
|
|
10
|
+
The automatic mode changes only the capability requirement.
|
|
11
|
+
|
|
12
|
+
A route is `{ model: alias, effort: string | number | null }`. Generated candidates use only supported strings,
|
|
13
|
+
or `null` for a model with `efforts: []`. A real native incumbent with an unresolved null or finite numeric effort
|
|
14
|
+
is valid but has unknown capability. An unsupported explicit string is not silently clamped.
|
|
15
|
+
Capability comes only from `models[alias].capability[effort]`, or `capability.default` on a no-effort model.
|
|
16
|
+
No declaration means unknown, not zero. Anthropic includes user-derived balanced base-route policy seeds, not
|
|
17
|
+
measured quality; all other pairs remain unknown. Users can replace them as described in
|
|
18
|
+
[Providers](providers.md#capability-evidence).
|
|
19
|
+
|
|
20
|
+
The facts object uses the existing controller shapes:
|
|
21
|
+
|
|
22
|
+
- `contextTokens`, `historyMeasured`, `hasImages`;
|
|
23
|
+
- `models`: cache key → `{ prefixTokens, lastAt, ... }`;
|
|
24
|
+
- `effort`: native session effort, and `lastRequest: { tokens, outputTokens, ... }`;
|
|
25
|
+
- `ineligible`: model IDs excluded for this history;
|
|
26
|
+
- `dwellAllowed`: a separate boolean for physical downgrades, required even on a new task. Set it to
|
|
27
|
+
`requests - switchedAt.requests >= config.policy.minDwellRequests`, using the actual route's last switch and
|
|
28
|
+
request counter. At session start, before any request/switch exists, set it to true. Missing means false.
|
|
29
|
+
- `downAllowed`: the existing combined dwell/checkpoint fact, required in addition to `dwellAllowed` only on the
|
|
30
|
+
same task; `midTurn`, `progress` and advice continuation identify that task. Off the same task no checkpoint
|
|
31
|
+
is required. Upgrades, forced refits, escalation and explicit stronger mode/catalog moves bypass dwell.
|
|
32
|
+
- `failure: { signature, index }` or null. The same signature at a new index is a new occurrence.
|
|
33
|
+
Re-reading the same occurrence does not escalate again. History reset must reset engine state.
|
|
34
|
+
|
|
35
|
+
Advice uses the existing `{ choice, probabilities, continuation, ... }` classifier shape, or null.
|
|
36
|
+
`now` is the caller's timestamp in milliseconds. `availableModels` and `nativeModel` use the existing
|
|
37
|
+
`isModelAllowed` semantics. Pass the active provider's cache configuration, including effort-cache separation.
|
|
38
|
+
If the host has resolved a configured model ID to a response ID, project those IDs into the supplied catalog
|
|
39
|
+
before assessing cache state, as the table controller's economic config does.
|
|
40
|
+
|
|
41
|
+
## Demand, target and move
|
|
42
|
+
|
|
43
|
+
| Export | Input | Output |
|
|
44
|
+
| ----------------------- | ---------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------- |
|
|
45
|
+
| `initialAutoState` | `{ demand = 2, routedDemand = demand, cause = 'classifier' }` | Initial state |
|
|
46
|
+
| `initialIncumbentState` | `{ config, incumbent, facts }` | Baseline demand with actual known incumbent capability as routed demand |
|
|
47
|
+
| `advanceDemand` | `{ config, state, advice = null, facts, pin = null, boundedExecute = false }` | `{ raw, level, reason, headroom, escalated, state }` |
|
|
48
|
+
| `assessCandidates` | `{ config, facts, now, incumbent, contextKnown = Number.isFinite(facts.contextTokens), availableModels, nativeModel }` | Candidate facts array |
|
|
49
|
+
| `selectTarget` | `{ candidates, demand, mode = 'balanced', headroom = false, escalation = false }` | `{ outcome, route, capability, requirement, estimate, reason, headroom }` |
|
|
50
|
+
| `decideMove` | `{ config, state, demandResult, target, candidates, advice = null, facts, now, changeCause = null, startup = false }` | `{ move, route, reason, evidence, estimate, state }` |
|
|
51
|
+
| `proposeAutoRoute` | Combined inputs above, using `incumbent` and optional `state` | `{ demand, candidates, target, move, state }` |
|
|
52
|
+
|
|
53
|
+
The facade initializes missing state with `initialIncumbentState`. Raw demand starts at `config.baselineTier`;
|
|
54
|
+
routed demand starts at the actual incumbent's declared capability for its supported/resolved native effort.
|
|
55
|
+
Unknown capability, an unlisted model, or unresolved/unsupported effort conservatively keeps baseline routed
|
|
56
|
+
demand; it is never inferred from a route table, mode offset or a guessed effort. Runtime and replay use the
|
|
57
|
+
same initializer. `initialAutoState` remains available for callers supplying explicit state.
|
|
58
|
+
State includes integer `demand`, `turn`, `votes`, `holdUntilTurn`, `escalatedSignature`, `escalatedIndex`,
|
|
59
|
+
`headroom`, `demandCause`, `routedDemand` and `routeCause`.
|
|
60
|
+
`raw` is integer demand; `level` is effective demand after the noncumulative half-step bounded-execute discount.
|
|
61
|
+
The caller supplies `boundedExecute: true` only for an already accepted bounded execute/ops phase, never
|
|
62
|
+
for a goal label alone or verify/report phases. An escalation hold protects the raised demand.
|
|
63
|
+
|
|
64
|
+
Each candidate has `route`, `requestKey` (canonical physical identity), `capability` (number or null), `eligible`, `reasons`, `incumbent`, `selectable`,
|
|
65
|
+
`cache`, `coldWriteUsd` and `estimate`. The incumbent is assessed even if it left the catalog.
|
|
66
|
+
A native unresolved effort is included in normal target comparison so it wins equal estimates, but cannot be
|
|
67
|
+
nominated as a new non-incumbent route. Declared-below-demand candidates still precede an unknown incumbent.
|
|
68
|
+
On a no-effort model, numeric native effort and the generated null-effort candidate have the same request key.
|
|
69
|
+
|
|
70
|
+
All availability, context, image, effort and cold-write cash constraints run before fallback.
|
|
71
|
+
An invalid incumbent and no eligible candidate produce explicit `no-route`, never an invented native route.
|
|
72
|
+
Target outcomes are `meets-demand`, `declared-below-demand`, `unknown-only`, `incumbent` and `no-route`.
|
|
73
|
+
|
|
74
|
+
Estimates are absolute next-turn `{ minUsd, maxUsd }` ranges: `cost.inputBounds` plus the last reply's output
|
|
75
|
+
length at `cost.ratesAt` prices. This estimate is effort-blind; it does not model additional thinking tokens.
|
|
76
|
+
Selection uses the conservative `maxUsd` bound, then the incumbent, lower sufficient declared capability and
|
|
77
|
+
config order. The mode margin supplies headroom through the requirement, not a tie preference. Automatic prefix
|
|
78
|
+
caches have no assumed TTL or warm-cache exemption.
|
|
79
|
+
Switch gates use the existing route-to-route `switchingTaxUsd` and `downgradeTaxUsd` math.
|
|
80
|
+
These are conditional estimates, not expected costs or measured savings.
|
|
81
|
+
|
|
82
|
+
## State ownership
|
|
83
|
+
|
|
84
|
+
Shadow needs only `advanceDemand → assessCandidates → selectTarget`. Retain the returned demand state and
|
|
85
|
+
always assess the real incumbent/cache; do not retain a hypothetical target as route history.
|
|
86
|
+
Pass `escalation: demand.escalated` and `headroom: demand.headroom` to `selectTarget`. After either escalation or
|
|
87
|
+
pending-headroom selection, copy `target.headroom` back to the returned demand state's `headroom` field.
|
|
88
|
+
It is true only on exhausted escalation, permitting retry if a stronger candidate becomes eligible.
|
|
89
|
+
Target reasons are null, `escalation`, `escalation-unknown` (unknown capability) and `escalation-exhausted`.
|
|
90
|
+
A declared stronger eligible request precedes unknown alternatives; neither search can return the same physical
|
|
91
|
+
request via another alias. The facade performs this headroom feedback automatically.
|
|
92
|
+
|
|
93
|
+
Stateful replay/auto compares effective demand with `state.routedDemand`, not the previous turn's demand.
|
|
94
|
+
A physical move updates routed demand and route cause. A `same-route` selection instead accepts the current
|
|
95
|
+
effective demand (including zero) as routed demand without a switch: `move: false`, route cause, hold, votes
|
|
96
|
+
and failure markers remain unchanged. Canonically identical physical requests under different aliases count
|
|
97
|
+
as same-route. Pins, no-route, exhausted escalation and blocked moves do not adopt demand.
|
|
98
|
+
The facade returns accepted state in `result.state` and `result.move.state`; its demand-only state remains in
|
|
99
|
+
`result.demand.state`. A held proposal leaves routed demand and cause unchanged, so it remains pending.
|
|
100
|
+
Runtime and offline replay use the same `decideMove` transition.
|
|
101
|
+
`changeCause` is null, `mode` or `catalog`; it cannot relabel a demand change as an explicit configuration move.
|
|
102
|
+
The separate `startup` input defaults to false and must come from genuine per-history routing evidence,
|
|
103
|
+
never `state.turn === 0` or the physical request count. A first tier pin consumes eligibility even though its
|
|
104
|
+
demand state remains unchanged. Manual/off native requests do not consume it; their cache and dwell observations
|
|
105
|
+
still count normally. The loop persists `hasRoutedDecision` with the same state write as the routed decision,
|
|
106
|
+
including table decisions and pins. Reload, off/on, engine changes and compaction preserve it; a new session or
|
|
107
|
+
`/clear` new history starts false. Legacy unmarked histories with request, decision or auto-state evidence
|
|
108
|
+
conservatively count as consumed. This adds no configuration flag or state version.
|
|
109
|
+
Startup qualification uses the selector's existing `target.requirement`: effective demand after any accepted
|
|
110
|
+
bounded-execute discount plus the mode margin. It does not cancel that discount or invent an undiscounted floor.
|
|
111
|
+
Only a declared known incumbent below that requirement and a declared target meeting it can qualify; unknown
|
|
112
|
+
capability never grants startup evidence. After unchanged demand voting, qualifying startup movement takes
|
|
113
|
+
priority over classifier upgrade/downgrade movement and uses mass 1 through the same base-policy explicit-up
|
|
114
|
+
switch-tax gate. All candidate hard constraints and the cash cap still apply; this is not a forced move.
|
|
115
|
+
A blocked first decision consumes the chance, so there is no second-turn startup retry. For example, native
|
|
116
|
+
capability 0, baseline 1 and first low probability 0.57 can move with startup mass 1; the same probability on an
|
|
117
|
+
ordinary later demand upgrade still needs the unchanged classifier mass gate. Fresh micro advice that reduces
|
|
118
|
+
effective demand to 0 does not authorize startup when capability 0 already meets the resulting requirement.
|
|
119
|
+
Startup move reason and route cause are `startup`, not `mode`. Replay compares accepted automatic mode/catalog
|
|
120
|
+
evidence rather than treating table-only pin config snapshots as catalog edits; genuine edits survive pins.
|
|
121
|
+
The live adapter and replay fingerprint the actual allowed configured aliases, including native `default` resolution.
|
|
122
|
+
An allowlist expansion is catalog evidence; reordering it, cache warmth and routine response-ID resolution are not.
|
|
123
|
+
|
|
124
|
+
Tier pins accept a tier string or integer 0..3. They are one-turn overrides: no mode margin and no move gates,
|
|
125
|
+
but normal candidate constraints, outcome precedence and incumbent ties. The facade returns the pinned route
|
|
126
|
+
with `move: false`, reason `pinned` and unchanged state. The caller resumes the pre-pin incumbent and state on
|
|
127
|
+
the next unpinned turn. With the lower-level API, pass `mode: 'balanced'` to `selectTarget` for a pinned demand.
|
|
128
|
+
Exact model/effort pins remain the controller's separate responsibility.
|
|
129
|
+
|
|
130
|
+
An observed native fallback records `incumbentOrigin: 'external-fallback'` and rebases the physical incumbent without
|
|
131
|
+
resetting demand, votes, hold or routed demand. Substituted effort remains unknown. The current turn stays native;
|
|
132
|
+
on the next turn, `decideMove` takes `incumbentOrigin` and permits `reentry` to a different eligible target without
|
|
133
|
+
mass, dwell or payback gates. Hard constraints and the cash cap still apply. Re-entry sets routed demand and cause
|
|
134
|
+
`reentry`, retaining the escalation hold; an equal physical target accepts current effective demand and clears
|
|
135
|
+
origin without changing route cause or counting a switch. Pinned fallbacks retain normal state and
|
|
136
|
+
apply the physical fallback provenance after restoring the pin. Replay counts external requests separately.
|
|
137
|
+
|
|
138
|
+
## Reloads
|
|
139
|
+
|
|
140
|
+
Before routing after a Mod reload without `session.start`, the hook lazily loads and validates the selected
|
|
141
|
+
provider profile and project mode. Constructor defaults never choose a main-turn route. A failed load marks
|
|
142
|
+
routing unavailable and passes the native request unchanged, without resetting persisted auto state to table.
|
|
143
|
+
A real repeated `session.start` invalidates old turn-config and classifier/review jobs before using the reloaded
|
|
144
|
+
profile. Compatible persisted automatic state survives; an actual engine change resets incompatible engine state.
|
|
145
|
+
Lazy profile loading is not a fresh-session signal: shadow recording remains explicitly incomplete mid-session.
|
|
146
|
+
|
|
147
|
+
## Checks
|
|
148
|
+
|
|
149
|
+
Run `node --test test/auto-engine.test.mjs` for the design's §5.4 invariants and §3.3 move-evidence cases.
|
|
150
|
+
The tests also cover failure occurrence deduplication, raw/effective demand, pins, native efforts, cache semantics,
|
|
151
|
+
range/cash/context boundaries and a capability-monotonicity matrix across sparse catalogs and differing prices.
|
|
152
|
+
No paid API calls or classifier calls are needed.
|
package/docs/configuration.md
CHANGED
|
@@ -37,6 +37,25 @@ See [Providers](providers.md) for selection examples, built-in defaults and comp
|
|
|
37
37
|
The same schema applies to every router file. This reference uses `router.json` as shorthand for the selected file.
|
|
38
38
|
A non-Anthropic file must exist and validate. It never falls back to `router.json` or inherits settings from it.
|
|
39
39
|
|
|
40
|
+
### Model capability declarations
|
|
41
|
+
|
|
42
|
+
`models.<alias>.capability` is an optional map of supported efforts to adequacy levels: 0 (micro),
|
|
43
|
+
1 (low), 2 (medium), or 3 (high), with half-steps allowed. For example:
|
|
44
|
+
|
|
45
|
+
```json
|
|
46
|
+
{ "models": { "sonnet": { "capability": { "medium": 1.5, "high": 2 } } } }
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
These example values illustrate the shape, not measured model performance. Anthropic has user-derived built-in
|
|
50
|
+
policy seeds; see [Providers](providers.md#capability-evidence). Users may replace or clear them. Missing maps and entries are
|
|
51
|
+
unknown, not 0. For a model with `efforts: []`, use only `"default"` for its no-effort candidate. Other keys,
|
|
52
|
+
unsupported efforts, non-numbers and values outside 0–3 or between half-steps are invalid.
|
|
53
|
+
|
|
54
|
+
A user-supplied map replaces the preset map in full; `{}` clears it. Replacing a model ID clears inherited
|
|
55
|
+
capability, even if the same alias is kept; the `[1m]` context hint alone does not change identity. Changing
|
|
56
|
+
`efforts` prunes unsupported inherited entries; unsupported explicitly supplied entries are rejected. Capability
|
|
57
|
+
is separate from `strength`, which still controls the current table routing. Capability declarations affect `auto` and `shadow`; the default `table` uses strength. Built-in declarations and their evidence limits are listed in [Providers](providers.md#capability-evidence).
|
|
58
|
+
|
|
40
59
|
### Provider prices
|
|
41
60
|
|
|
42
61
|
All non-Anthropic built-in model windows are at least 1M tokens.
|
|
@@ -106,6 +125,22 @@ The router never substitutes zero savings or invents another route's cache hit r
|
|
|
106
125
|
Models with `efforts: []`, including MiMo, ignore session effort in the comparison identity.
|
|
107
126
|
Missing counters remain unknown. See [provider acceptance checks](evaluation.md#provider-profiles).
|
|
108
127
|
|
|
128
|
+
## Routing engine
|
|
129
|
+
|
|
130
|
+
Set `"routingEngine": "shadow"` in the selected profile file to record one-step automatic-routing proposals alongside
|
|
131
|
+
the table's decisions. The default, `"table"`, does not run the automatic engine. Shadow uses the real incumbent and
|
|
132
|
+
cache for each proposal; the table still chooses every request. The Now pane and `/router decisions` show demand,
|
|
133
|
+
target, capability (or unknown), outcome and a next-turn cost estimate. Shadow requests no extra classifier advice.
|
|
134
|
+
The proposal is not a prediction of a stateful automatic-routing session.
|
|
135
|
+
|
|
136
|
+
Set `"routingEngine": "auto"` to opt in to stateful automatic routing. It selects from the configured model and
|
|
137
|
+
effort catalog using demand, a mode margin, hard constraints and observed cache economics. It does not use route
|
|
138
|
+
tables or mode policy patches. Tier pins override demand for one turn without a margin, but still obey hard constraints.
|
|
139
|
+
If no candidate fits, the original native request runs with reason `no-route-native`. Auto quality evaluation and
|
|
140
|
+
default cutover remain pending; see [evaluation](evaluation.md). After an observed fallback, the rest of its turn
|
|
141
|
+
stays native. The next turn can re-enter an eligible target without confidence, dwell or payback gates; hard constraints
|
|
142
|
+
and the cash cap still apply, and escalation holds survive. The three values are case-sensitive.
|
|
143
|
+
|
|
109
144
|
## Built-in defaults
|
|
110
145
|
|
|
111
146
|
The default baseline is tier `low`. Each route refers to an alias in `models`.
|
|
@@ -351,7 +386,7 @@ The Mod validates the whole file. Unknown keys and invalid values make routing u
|
|
|
351
386
|
| `mode` | `cost`, `balanced`, `quality` ([Routing modes](routing-modes.md)) | `balanced` |
|
|
352
387
|
| `modes.<cost\|quality>.policy` | any `policy` field, used while that mode is active | [Routing modes](routing-modes.md#policy-per-mode); not tuned |
|
|
353
388
|
| `activities.<activity>.<tier>` | optional `model` and `effort`, as in `routes` | The overrides in [Activity routing](#activity-routing) |
|
|
354
|
-
| `models.<alias>` | `id`, `input`, optional `output`, `cacheRead`, optional `longContext` (`above`, `multiplier`, or `null`), optional `offPeak` (`multiplier`, `peakDays` as UTC weekdays 0–6 from Sunday, `peakHoursUtc` as `[from, to)` hour ranges, or `null`), `contextWindow`, `billing`, `efforts`, optional `strength`, `imageInput` | Three defaults above |
|
|
389
|
+
| `models.<alias>` | `id`, `input`, optional `output`, `cacheRead`, optional `longContext` (`above`, `multiplier`, or `null`), optional `offPeak` (`multiplier`, `peakDays` as UTC weekdays 0–6 from Sunday, `peakHoursUtc` as `[from, to)` hour ranges, or `null`), `contextWindow`, `billing`, `efforts`, optional `strength`, optional `capability`, `imageInput` | Three defaults above |
|
|
355
390
|
| `cache` | `mode`, `effortScope`, `writeMultiplier`, `ttlMs`, `warmMarginMs` | `5m: 1.25`, `1h: 2`; `300000` / `3600000` ms; `30000` ms |
|
|
356
391
|
| `policy` | Fields below | Values below |
|
|
357
392
|
| `classifier` | An id in `classifiers` | `jev` |
|
package/docs/evaluation.md
CHANGED
|
@@ -4,6 +4,225 @@ The native Mod has no measured savings result. The pane reports Claude's own usa
|
|
|
4
4
|
|
|
5
5
|
This page separates an older gateway experiment from a shadow replay of native policy. Neither result measures answer quality or proves lower real spend. [Activity routing](#activity-routing) adds two checks of the activity label, with no savings result either.
|
|
6
6
|
|
|
7
|
+
## Automatic-routing shadow proposals
|
|
8
|
+
|
|
9
|
+
Opt-in `routingEngine: "shadow"` records a demand and target proposal against the real table incumbent for each
|
|
10
|
+
applicable decision. It does not run the proposed route or maintain a simulated route history. These rows do not
|
|
11
|
+
establish savings or answer quality; a next-turn cost estimate is not an observed cost or a counterfactual saving.
|
|
12
|
+
The bounded decision records are not replay traces. See [Configuration](configuration.md#routing-engine).
|
|
13
|
+
|
|
14
|
+
## Replay trace recorder
|
|
15
|
+
|
|
16
|
+
In `routingEngine: "shadow"`, the main-turn hooks also keep text-free replay inputs separately from decision rows:
|
|
17
|
+
the initial configured and mode-transformed routing policy, provider/cache semantics and model prices; classifier
|
|
18
|
+
tier/activity/phase vectors; failure fingerprints and occurrence indexes; pins, modes, resets, context/availability
|
|
19
|
+
facts and observed moves; and correlated request/reply model, effort and token counters. Checkpoint evidence uses
|
|
20
|
+
message indexes and success flags, not commands, tool results or filesystem-path fields, so a later replay can use
|
|
21
|
+
its own switch cursor. Validated routing aliases and model IDs are preserved exactly, including path-shaped IDs;
|
|
22
|
+
classifier endpoints, credentials and the config file's nativePath are not copied.
|
|
23
|
+
|
|
24
|
+
Records use `trace:v1:<session id>`: 30-day retention, 128 KiB per session, 1 MiB total trace budget and a measured
|
|
25
|
+
3 MiB guard on the shared plugin store. Pending turns are bounded too. A cap discards the entire pending turn,
|
|
26
|
+
keeps the earlier contiguous prefix usable and counts excluded turns/events. Late starts, resume/reload, gaps,
|
|
27
|
+
conflicts and lost writes are incomplete. A failed write taints memory and persists that taint on the next success;
|
|
28
|
+
an unavailable store cannot guarantee a durable failure marker. A clean session close is not required.
|
|
29
|
+
Freshness requires the first main request (index 0), no saved loop or trace, and readable normalized request
|
|
30
|
+
history with no assistant messages. Injected startup reminders are user rows and do not make a trace incomplete;
|
|
31
|
+
`classic.SessionStart` is not required. Unreadable or malformed history, reload and late enable are conservative.
|
|
32
|
+
Compaction preserves an existing trace with an explicit reset; `/clear` permits fresh history under a new session ID.
|
|
33
|
+
|
|
34
|
+
Run `/router trace export` to merge retained traces into `router-traces.json` beside the selected profile's
|
|
35
|
+
router config. The command also works after switching back to `table`; ordinary table turns do no trace I/O.
|
|
36
|
+
Exports overwrite their previous JSON after merging; near 3 MiB, a new dated `router-traces-*.json` file receives
|
|
37
|
+
the retained snapshot instead. Keep all export files for offline replay. Known taint is never erased by a longer
|
|
38
|
+
or newer clean prefix. **Delete history** clears stored traces in any mode, but exported files remain user-owned:
|
|
39
|
+
the host filesystem API has no unlink operation.
|
|
40
|
+
|
|
41
|
+
Recent active sessions have a one-hour last-write lease against budget eviction; stale activity cannot defeat
|
|
42
|
+
30-day retention. Store scans, quota checks and deletion are best effort across processes, not atomic reservations.
|
|
43
|
+
There is no live exposure, routing-quality result or savings claim from this recorder yet.
|
|
44
|
+
|
|
45
|
+
## Offline automatic-engine replay
|
|
46
|
+
|
|
47
|
+
Run the tool from a checkout with **all** incremental exports (shell-expanded file arguments, no implicit profile
|
|
48
|
+
access). It reads only those files and writes anonymous JSON to stdout; it never changes a router or provider profile.
|
|
49
|
+
|
|
50
|
+
```sh
|
|
51
|
+
node scripts/replay-traces.mjs /exports/router-traces.json /exports/router-traces-*.json > replay-report.json
|
|
52
|
+
node scripts/replay-traces.mjs --reversal-window 3 test/fixtures/replay-traces.json
|
|
53
|
+
node --test test/replay.test.mjs
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
Omit the dated-file argument when there are no rollover files. The second command uses only synthetic data.
|
|
57
|
+
The implementation follows [design §5.1 and §6 step 5](plans/2026-11-16-auto-routing-design.md#51-replay-traces-and-offline-replay).
|
|
58
|
+
|
|
59
|
+
1. Merge sessions with the recorder's `mergeTraceExport`: preserve lost-write taint, overlap conflicts and capped
|
|
60
|
+
counts. Keep usable capped contiguous prefixes; report their omitted turns/events separately.
|
|
61
|
+
2. Replay the table chain from **only** its initial header, applying config, controls, history preparation/resets,
|
|
62
|
+
choose/observed/continue operations and responses. Compare every recorded decision and request model/effort.
|
|
63
|
+
An unexplained mismatch on any otherwise usable prefix returns `status: "blocked"`, exits 1 and prints **no**
|
|
64
|
+
auto/session totals. It remains in the fidelity denominator. Taint/gaps/incompleteness are explained exclusions.
|
|
65
|
+
3. Only after fidelity passes, validate independent-replay evidence. Missing checkpoints, required boolean facts,
|
|
66
|
+
unconditional continuation modality or activity indexes produce `auto-input-incomplete` and exit 1, without
|
|
67
|
+
silently excluding a usable fidelity session. Earlier exports can lack these fields.
|
|
68
|
+
4. Simulate auto with its own demand/votes, incumbent, cache, routed-demand/provenance and switch/dwell cursors.
|
|
69
|
+
Classifier decisions advance votes; continuations reevaluate constraints and pending moves without adding votes.
|
|
70
|
+
Checkpoints and observed activity are reconstructed from text-free indexes against its own cursor. Only an
|
|
71
|
+
accepted bounded execute phase discounts demand; observed edits can remove it. No activity route table is used.
|
|
72
|
+
|
|
73
|
+
**Mode meaning:** fidelity and both comparisons follow the recorded mode-change history. Output is partitioned by
|
|
74
|
+
the mode active on each request, not three artificial reruns. Table uses the recorded transformed config; auto
|
|
75
|
+
uses the recorded base policy, with mode only as the capability margin. There is no forced-mode option.
|
|
76
|
+
|
|
77
|
+
The report has `fidelity`, `coverage`, `exposureDiagnostic` and anonymous numbered `sessions[].modes`. Each mode
|
|
78
|
+
contains both engines' conditional cost range, priced/unpriced request counts, effort mix, switches and disagreement
|
|
79
|
+
counts grouped by effective demand/activity. Unknown catalog capabilities are counted explicitly. Anthropic seeds four balanced base-route pairs as
|
|
80
|
+
user-derived policy judgments; other pairs and provider presets remain unknown. These seeds and their replay are not
|
|
81
|
+
quality-calibration evidence.
|
|
82
|
+
|
|
83
|
+
Switches are computed on the full session before mode partitioning and attributed to the **destination** mode.
|
|
84
|
+
A reversal closes A→B→A within the explicit `--reversal-window N` main turns (default 3); `reversals` counts closing
|
|
85
|
+
legs, `reversalMoves` counts participating legs, and `oneWay` counts unpaired moves. Rates are per 100 captured turns
|
|
86
|
+
in that mode. Pins do not count as switches and restore the pre-pin route/demand/dwell baseline; every pinned
|
|
87
|
+
continuation still checks eligibility. Manual intervals, history resets and external native interventions break the
|
|
88
|
+
policy-switch chain, including reversal pairing.
|
|
89
|
+
|
|
90
|
+
### Coverage and estimate limits
|
|
91
|
+
|
|
92
|
+
- `coverage.exportCohort` ratios describe **only captured/exported sessions**, a lower-bound cohort. Exports do not
|
|
93
|
+
count evicted or untraced sessions; `coverage.allSessions` denominators and its 80% gate stay unavailable.
|
|
94
|
+
Sufficient future metadata would include capture-eligible session starts (including untraced/evicted sessions),
|
|
95
|
+
total main turns and per-mode eligible routed turns over the same capture window. This tool invents none of it.
|
|
96
|
+
- Exposure thresholds (30 complete sessions, 1,000 main turns, 200 per evaluated mode) are diagnostics only.
|
|
97
|
+
Captured main turns are separate from eligible unpinned classifier-decision turns; manual-only sessions/turns
|
|
98
|
+
do not count as engine exposure. Freshness is unverified and the exposure gate remains `not-established`.
|
|
99
|
+
- Cost freezes observed total prompt/output lengths, timestamps, failures and tool calls. Auto simulates its own
|
|
100
|
+
full-prefix cache scenario, never the table's cache-hit tokens. Explicit cache warmth uses the existing conservative
|
|
101
|
+
five-minute helper; automatic caches retain hit/miss bounds, with no invented TTL guarantee. Effort-specific,
|
|
102
|
+
disabled, long-context and off-peak pricing use the existing pure cost helpers. These are **conditional estimates**,
|
|
103
|
+
not counterfactual savings, actual bills or paired quality results.
|
|
104
|
+
- Missing usage/prices and substituted replies make the affected total range `null`, not free. An explicitly named
|
|
105
|
+
`knownPricedRange` is only the priced subtotal. No transcript-length/quality change from a hypothetical request
|
|
106
|
+
is modeled.
|
|
107
|
+
- Fidelity distinguishes table echoes from genuine third-model/native fallbacks. Auto freezes actual external
|
|
108
|
+
interventions, response substitutions and routed null-stop suspension, rather than misreading a table echo as
|
|
109
|
+
its own fallback. It cannot predict a provider's fallback for a different hypothetical auto request.
|
|
110
|
+
- Default output includes no session IDs, prompts, tool text, model identifiers or paths. Input errors name the
|
|
111
|
+
numbered file/structural field, not its path. The CLI cannot discover omitted exports or verify their provenance.
|
|
112
|
+
|
|
113
|
+
## Paired benchmark proposal
|
|
114
|
+
|
|
115
|
+
The [automatic-routing design §5.3](plans/2026-11-16-auto-routing-design.md#53-paired-benchmark-smoke-gate-for-quality)
|
|
116
|
+
requires paired `table`/`auto` runs, three modes and two repeats, with per-task checks and discordances. The
|
|
117
|
+
24 synthetic expected-output cases in `test/fixtures/benchmark-routing.json` are harness fixtures, not calibrated
|
|
118
|
+
engineering tasks or a quality result. Their activity labels are fixture metadata, not reference classifications.
|
|
119
|
+
Eight are marked held out in the fixture split. The separate engineering proposal below needs fixtures and review
|
|
120
|
+
before any evaluation; shadow-disagreement sources are pending.
|
|
121
|
+
|
|
122
|
+
```sh
|
|
123
|
+
node scripts/benchmark-routing.mjs --dry-run
|
|
124
|
+
node scripts/benchmark-routing.mjs --dry-run --smoke
|
|
125
|
+
node --test test/benchmark-routing.test.mjs
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
The dry run prints a SHA-256, run counts, serialized token/rate assumptions and a configured-price scenario.
|
|
129
|
+
Dry-run planning uses the built-in catalog maximum prices through `ratesAt`; a live smoke preflight reads the selected
|
|
130
|
+
work profile's catalog without changing it. Cold-write multipliers apply only to explicit caches. The 10,000-input
|
|
131
|
+
assumption is not a limit; live smoke sets `CLAUDE_CODE_MAX_OUTPUT_TOKENS=512`, one agentic turn and a $0.40
|
|
132
|
+
CLI budget per call. These are not a proxy-price quotation or a bill. A run estimates before launching and reserves
|
|
133
|
+
$1.20 for three calls. An exclusive reservation file under `.pi/` refuses automatic reruns; further spending needs
|
|
134
|
+
review, not a hidden retry. The 288 planned task runs are for the toy mechanics fixtures, not an engineering benchmark cost estimate.
|
|
135
|
+
An engineering task run may use multiple inference calls, tools and retries; it needs new token, request, effort
|
|
136
|
+
and provider budgets plus lead approval before any run. No full matrix is authorized or run.
|
|
137
|
+
|
|
138
|
+
`--live --smoke` is limited to the three development tasks, `table`/`balanced`, one repeat, using the existing
|
|
139
|
+
installed release through `zsh -ic` and `claude-mp-work -p`. It loads no checkout plugin. The selected profile must
|
|
140
|
+
still be `table`; `--setting-sources user`, strict empty MCP and no tools limit the surface. Init metadata must
|
|
141
|
+
identify exactly one router at the installed user path. Profile contents are hashed before/after each call. The
|
|
142
|
+
wrapper's normal MCP housekeeping is permitted; the original active-environment bytes/existence are restored and
|
|
143
|
+
verified after success, failure, timeout or handled interrupt. SIGKILL, power loss and independently detached
|
|
144
|
+
processes are not covered. Process-group cleanup is best effort; Windows cannot kill a descendant tree via Node
|
|
145
|
+
alone. The smoke establishes harness mechanics only, not evaluation of this checkout or either engine.
|
|
146
|
+
|
|
147
|
+
Offline system-boundary tests exercise independent directories, output checks, deadlines, output limits, budgets,
|
|
148
|
+
unknown costs and paired discordance reporting. The harness checks its own and the manifest's hashes around child
|
|
149
|
+
execution; this detects persistent changes, not transient same-user writes or an OS sandbox. The checks run in the
|
|
150
|
+
parent outside the child's working directory. Unknown/zero total or per-model CLI prices stop further calls rather
|
|
151
|
+
than implying free usage. Reports sum every supplied attempt, including failures, and leave missing/unpriced
|
|
152
|
+
totals unknown. Duplicate attempts within a paired cell are reported as ambiguous, never silently replaced by a
|
|
153
|
+
passing rerun.
|
|
154
|
+
CLI-reported USD can use rates different from the selected catalog and may omit proxy retries, classifier spend or
|
|
155
|
+
upstream failed calls: actual billing remains unknown.
|
|
156
|
+
No comparison of quality, savings or measured routing performance follows from these checks.
|
|
157
|
+
|
|
158
|
+
The checked-in [installed-release smoke record](../experiments/mod-router/results/benchmark-harness-smoke.json)
|
|
159
|
+
contains three passing external checks, 19,298 ms summed subprocess wall time and $0.0098816 CLI-reported price
|
|
160
|
+
estimates. It records the manifest/config hashes and successful restoration/provenance checks without prompts,
|
|
161
|
+
profile paths or credentials. CLI usage metadata names DeepSeek Flash for two calls and GLM Flash for one; backend
|
|
162
|
+
identity is not independently verified.
|
|
163
|
+
These are harness-validation observations on installed Router 1.23.1, not a paired result for this checkout.
|
|
164
|
+
|
|
165
|
+
### Engineering task set: draft only
|
|
166
|
+
|
|
167
|
+
The following 24 synthetic engineering tasks are a proposed **separate** evaluation set, not implemented fixtures.
|
|
168
|
+
Difficulty and activity are author hypotheses, not classifier references. Each row needs a self-contained repository
|
|
169
|
+
fixture and an independently held-out automatic oracle outside model control. Freeze the fixture/manifest hash and
|
|
170
|
+
split before capability or margin tuning; do not run or inspect held-out outcomes during tuning. All rows are authored
|
|
171
|
+
examples: shadow-disagreement sourcing is **pending** and must be added or documented before the set is finalized.
|
|
172
|
+
|
|
173
|
+
| ID | Split | Activity / demand hypothesis | Task | Independent automatic oracle |
|
|
174
|
+
| --- | --- | --- | --- | --- |
|
|
175
|
+
| E01 | dev | code / low | Add integer clamping to a small library | Boundary and type tests |
|
|
176
|
+
| E02 | dev | debug / medium | Repair a parser that mishandles escaped delimiters | Existing tests plus hidden escaped and empty cases |
|
|
177
|
+
| E03 | dev | code / medium | Add pagination to a small service across handler and storage modules | Fixture API and deterministic pagination tests |
|
|
178
|
+
| E04 | heldout | code / medium | Implement stable deduplication across a CLI and helper | Output and duplicate-order tests |
|
|
179
|
+
| E05 | dev | security / high | Reject traversal and symlink escapes in an archive extractor | Traversal, symlink and valid-path tests |
|
|
180
|
+
| E06 | dev | debug / high | Fix a cross-module stale cache invalidation defect | Concurrent update and expiry regression tests |
|
|
181
|
+
| E07 | heldout | debug / medium | Fix time boundary logic for expiry at exact deadline | Fake-clock boundary tests |
|
|
182
|
+
| E08 | dev | ops / medium | Repair CI script so test and lint failures reach the exit status | Controlled subprocess exit-code matrix |
|
|
183
|
+
| E09 | dev | review / medium | Identify seeded unsafe defaults in a short configuration | Exact structured findings against injected defects |
|
|
184
|
+
| E10 | heldout | debug / high | Repair nondeterministic retry handling after partial network failures | Deterministic mocked-service request and retry tests |
|
|
185
|
+
| E11 | dev | code / high | Add idempotent transaction handling across API and persistence | Duplicate/concurrent request and rollback tests |
|
|
186
|
+
| E12 | dev | debug / medium | Diagnose and fix an off-by-one token budget after a tool continuation | Transcript fixture and boundary tests |
|
|
187
|
+
| E13 | heldout | code / low | Add case-insensitive header lookup preserving original values | Mixed-case and missing-header tests |
|
|
188
|
+
| E14 | dev | code / medium | Migrate a versioned configuration while preserving unknown fields | Old/new config round-trip and rejection tests |
|
|
189
|
+
| E15 | dev | debug / high | Fix matching failure signatures after an edit and repeated tool failure | Scripted transcript with two failures and edit tests |
|
|
190
|
+
| E16 | heldout | security / high | Correct authorization checks across two endpoints sharing a store | Cross-user read/write and allowed-access tests |
|
|
191
|
+
| E17 | dev | refactor / medium | Replace duplicate parsing in two modules without changing behavior | Characterization and randomized input tests |
|
|
192
|
+
| E18 | dev | ops / low | Correct shell quoting for filenames containing spaces | Subprocess tests on spaces and metacharacters |
|
|
193
|
+
| E19 | heldout | code / high | Implement bounded parallel workers with cancellation and cleanup | Deterministic concurrency and child-cleanup tests |
|
|
194
|
+
| E20 | dev | debug / medium | Fix a multi-file regression after two failed test attempts and an edit | Scripted tool history plus external final tests |
|
|
195
|
+
| E21 | dev | docs / low | Correct a documented CLI example for a real edge-case flag | Execute example against fixture CLI |
|
|
196
|
+
| E22 | heldout | ops / medium | Repair a build pipeline that masks errors in a chained command | Failure-injection exit-code tests |
|
|
197
|
+
| E23 | dev | review / high | Find seeded race and missing transaction rollback | Machine-readable findings with location and defect checks |
|
|
198
|
+
| E24 | heldout | debug / medium | Fix Unicode normalization inconsistency across reader and index | Composed/decomposed matching and preservation tests |
|
|
199
|
+
|
|
200
|
+
These are not yet the §5.3 benchmark: the tasks, initial repositories, test oracles, tool-history presentation,
|
|
201
|
+
permissions and expected output need independent implementation and review. No engineering benchmark or held-out
|
|
202
|
+
outcome has been collected. Difficulty does not establish a capability value.
|
|
203
|
+
|
|
204
|
+
### Full paired-run isolation: proposal only
|
|
205
|
+
|
|
206
|
+
Auto routing is available by opt-in in 1.24.0. The full paired benchmark adapter remains disabled pending checkout/profile
|
|
207
|
+
isolation and approval; a full live matrix always refuses. A native pass-through or table
|
|
208
|
+
run is never relabeled `auto`. The current hook reads `routingEngine` only from the selected profile file; there is
|
|
209
|
+
**no** run-scoped environment/settings override. Project `.claude/router.json` can set mode only.
|
|
210
|
+
|
|
211
|
+
Before the real paired evaluation, choose and implement one code-provenance path:
|
|
212
|
+
|
|
213
|
+
- Add an explicit supplied-router option to `ce`'s model-proxy path, matching its normal launch behavior. This can
|
|
214
|
+
preserve existing implicit authentication but needs wrapper maintenance and must disable the marketplace copy.
|
|
215
|
+
- Launch `claude-proxy` directly with per-run settings that disable the marketplace router and a checkout
|
|
216
|
+
`--plugin-dir`. This gives precise code provenance but must reuse an approved authentication interface, never
|
|
217
|
+
copy credentials or silently bypass account policy.
|
|
218
|
+
- Run after a release that includes the evaluated implementation. This avoids two plugin copies but does not
|
|
219
|
+
itself isolate the engine selection or establish checkout provenance.
|
|
220
|
+
|
|
221
|
+
Every path **also** needs a separately approved run-scoped engine override or safely isolated profile configuration
|
|
222
|
+
that keeps existing implicit authentication without copying secrets. Only then can each child select its own
|
|
223
|
+
`table` or `auto` without editing the user's real router file. No wrapper modification, new engine override,
|
|
224
|
+
authentication mechanism, live auto wiring or cutover is implemented here.
|
|
225
|
+
|
|
7
226
|
## Provider profiles
|
|
8
227
|
|
|
9
228
|
[Provider defaults](providers.md#built-in-defaults) are starting policies, not measured per-activity quality or savings results.
|
package/docs/providers.md
CHANGED
|
@@ -9,6 +9,34 @@ Custom profiles need their own model settings. They do not inherit Claude models
|
|
|
9
9
|
|
|
10
10
|
This guide assumes Claude Code already works with your provider. Endpoint and credential setup are outside its scope.
|
|
11
11
|
|
|
12
|
+
## Capability evidence
|
|
13
|
+
|
|
14
|
+
The Anthropic preset seeds `claude-haiku-5-5@medium` at 0, `claude-haiku-5-5@high` at 1,
|
|
15
|
+
`claude-opus-5-5@medium` at 2, and `claude-opus-5-5@xhigh` at 3. These are user-derived policy judgments
|
|
16
|
+
from the built-in balanced base routes, not model-card benchmarks or quality guarantees. All other pairs, including
|
|
17
|
+
Haiku at low and Opus at high, remain unknown, not zero. User-router PR #79 seed values are profile overrides layered
|
|
18
|
+
over these built-ins. Other built-in provider presets have no seeds. No rating is inferred from price, ordinal strength, benchmark rank,
|
|
19
|
+
or an effort name.
|
|
20
|
+
This metadata does not affect the current table router. See [configuration](configuration.md#model-capability-declarations)
|
|
21
|
+
for declarations and overrides.
|
|
22
|
+
|
|
23
|
+
The model-card IDs below identify the reviewed records, not verified served backends. The sources do not give
|
|
24
|
+
numeric values on this router's demand scale.
|
|
25
|
+
|
|
26
|
+
| Preset model / model-card ID | Evidence and gap |
|
|
27
|
+
| --- | --- |
|
|
28
|
+
| Fable 5.2 / `anthropic-claude-fable-5-2` | [Prompting guide](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-fable-5-2): effort guidance, not demand-tier calibration. |
|
|
29
|
+
| Opus 5.5 / `anthropic-claude-opus-5-5` | [Prompting guide](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-5-5): effort recommendations, not tier adequacy. |
|
|
30
|
+
| Sonnet 5.5 / `anthropic-claude-sonnet-5-5` | [Prompting guide](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-sonnet-5-5): well-specified agentic coding at medium does not establish all medium-tier work. |
|
|
31
|
+
| Haiku 5.5 / `anthropic-claude-haiku-5-5` | [Prompting guide](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-haiku-5-5): simple work at low does not calibrate reliability to micro demand. |
|
|
32
|
+
| DeepSeek Flash / `deepseek-deepseek-v4.1-flash-1017` | [Provider evaluation](evaluation.md#provider-profiles): earlier benchmark results do not establish the hosted snapshot's per-effort adequacy. |
|
|
33
|
+
| GLM-5.3 / `zai-glm-5.3`; GLM-5.3-Flash / `zai-glm-5.3-flash` | [Provider evaluation](evaluation.md#provider-profiles): max-effort results do not establish low/high performance or tier adequacy. |
|
|
34
|
+
| MiniMax M3.1 Flash / `minimax-minimax-m3.1-flash`; Kimi K3.1 Pro / `moonshot-kimi-k3.1-pro` | [Provider experiments](provider-experiments.md): small synthetic checks do not calibrate repository tasks per effort. |
|
|
35
|
+
| MiMo V2.6 Flash and Pro / no model-card record | [Provider experiments](provider-experiments.md): no granular effort setting; small synthetic checks do not establish demand-tier adequacy. |
|
|
36
|
+
|
|
37
|
+
The auto engine selects the cheapest eligible unknown when no eligible declaration is known. Neither the seeds nor
|
|
38
|
+
unknown-only selections establish quality or escalation-headroom calibration; users can configure their own maps.
|
|
39
|
+
|
|
12
40
|
## Select a provider
|
|
13
41
|
|
|
14
42
|
For DeepSeek, create `deepseek-router.json` in your Claude Code profile directory with this content:
|
package/docs/routing-modes.md
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
# Routing modes
|
|
2
2
|
|
|
3
3
|
One setting trades cost against answer quality: `cost`, `balanced` (the default) or `quality`. The classifier and
|
|
4
|
-
your routes stay as they are.
|
|
4
|
+
your routes stay as they are. With the default `table` engine, the mode moves the route each tier runs by one fixed step.
|
|
5
5
|
|
|
6
6
|
| Mode | What changes |
|
|
7
7
|
| ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
@@ -19,6 +19,11 @@ In every mode:
|
|
|
19
19
|
on a model listing only those three. A route with a `null` effort (the session's) or a model without efforts has no effort step.
|
|
20
20
|
- [MiniMax, MiMo and Kimi tables](providers.md#new-provider-modes) show every new preset's mode/activity combination.
|
|
21
21
|
|
|
22
|
+
Optional [automatic routing](configuration.md#routing-engine) uses the mode only as a capability margin: `cost`
|
|
23
|
+
subtracts 0.5 from demand, `balanced` adds nothing and `quality` adds 0.5, clamped to 0–3. It uses base policy,
|
|
24
|
+
not mode policy patches. Tier pins use exact demand without a margin; candidates still must fit. Shadow records a
|
|
25
|
+
proposal while the table routes. Auto is opt-in; quality evaluation and cutover are pending.
|
|
26
|
+
|
|
22
27
|
## Choose a mode
|
|
23
28
|
|
|
24
29
|
The mode comes from the first of these that sets one:
|
|
@@ -44,6 +49,8 @@ then the band says why the route stays. Pin a tier to run its configured route n
|
|
|
44
49
|
|
|
45
50
|
## Policy per mode
|
|
46
51
|
|
|
52
|
+
The policy patches in this section apply to `table` and `shadow`, not opt-in `auto`.
|
|
53
|
+
|
|
47
54
|
Each mode except `balanced` replaces some `policy` fields while it is active. The built-in values started as
|
|
48
55
|
codex-router's and are **not tuned** on outcomes. Since 1.16, `cost` upgrades on balanced's bar (2 votes from 0.75):
|
|
49
56
|
with 3 votes from 0.85, the cache write of a cold stronger model raised the bar to about 0.92 from 60k tokens and 0.97
|
package/docs/user-guide.md
CHANGED
|
@@ -283,6 +283,12 @@ and unpriced causes, not semantic success or task acceptance. A caller must eval
|
|
|
283
283
|
|
|
284
284
|
### Changes during a task
|
|
285
285
|
|
|
286
|
+
The route-table behavior below applies to the default `table` engine and to `shadow`. With opt-in `auto`, each
|
|
287
|
+
continuation reassesses context, images, availability and the cash cap; classifier reviews update demand, and observed
|
|
288
|
+
code or docs can end a bounded-execute discount. It does not use activity route-table moves. A tier pin retains its
|
|
289
|
+
demand for the turn but can be refitted when constraints change. If nothing fits, the native request runs with
|
|
290
|
+
`no-route-native`. See [automatic routing](configuration.md#routing-engine). Quality evaluation and cutover are pending.
|
|
291
|
+
|
|
286
292
|
The classifier labels a turn from its prompt, and the work can change after that: a brief that reads as `ops` may turn into an hour of coding, and a hard fix may end in routine commands. The router therefore checks the route again while a task runs, for the tier and the activity, in both directions.
|
|
287
293
|
|
|
288
294
|
- **On every later step of a turn** it reads the turn's tool calls. A turn labelled `ops` or `explore` that edits a code file moves at once to the tier's `code` route (`docs` when it edits only Markdown or `docs/` files). The tools are the evidence, so this needs no classifier support, and the cache cost does not hold it back; only the cash cap on a credits-billed model can. This move only goes up. The toast and the band show it as `↗ coding`.
|
|
@@ -334,13 +340,14 @@ On the **Classifier** tab, **ACROSS SESSIONS · since the last reset** shows the
|
|
|
334
340
|
| **on** / **off** on the pane (`o` / `f`) | The same as `/router auto` and `/router off`. |
|
|
335
341
|
| **pin** on the Now tab | Pin that tier for the next turn. Routing on must already be enabled. |
|
|
336
342
|
|
|
337
|
-
|
|
343
|
+
With `table` or `shadow`, a routing mode moves each tier's route one step toward cost or quality; pins and `micro` turns are not moved. With opt-in `auto`, the mode changes the demand margin instead, and a tier pin uses exact demand without a margin. The band and the Now tab show the mode and where it comes from, such as `cost (session)`. When the mode kept the cell's model and changed only its effort, as `cost` does for Opus at `high`, the band adds `effort only`: the cost estimates price the same tokens on both routes, so that change shows no difference there, though real output length can differ. A change applies from the next turn, and the route in use moves only through the usual checks, so on a running task a cheaper route can wait a few requests and for a checkpoint. **Mode** on the Policy tab sets the profile's mode. See [Routing modes](routing-modes.md).
|
|
338
344
|
|
|
339
345
|
### Routing decisions
|
|
340
346
|
|
|
341
347
|
Each routed turn adds rows to the session's decision record in `$.store` (`decisions:v1:<session id>`): one when the turn starts, one per classifier review or tool-based move inside it, and one when it ends. A row holds the prompt's origin (`composer`, `task-notification`, …, or `unknown`), the classifier's tier and activity with their shares, whether it counted as the running task, the tier, activity, route and reason chosen, and at the end the turn's tool bucket and how many replies were priced. It holds no prompt, tool or file text. The route is the one chosen, which is not always the one sent; see [Changes during a task](#changes-during-a-task). `/router decisions [n]` prints them. For a tier move or an activity move up, taken or held, the row also holds the probability mass the policy compared and the bar it compared it with, printed after the reason: `(upgrade-pending 52% vs bar 79%)`. A held move can show a mass above its bar: it then waits for a second vote or for a checkpoint. On `downgrade-pending` the mass is the support for the classifier's own choice; a tier between it and the running one can be closer to its bar. A `jump`, a move on the turn's own tool calls and a context or availability override show neither number. Each session keeps its newest 60 rows; records of the 20 most recent other sessions stay, for at most 30 days. **Delete history** deletes them all.
|
|
342
348
|
|
|
343
|
-
|
|
349
|
+
For `auto`, decision rows also show demand, target, outcome, move reason and a conditional next-turn cost estimate,
|
|
350
|
+
not measured savings. A tier pin does not change the next turn after the pinned turn finishes. With `table` or `shadow`, a fresh session starts with routing on for a model a tier routes to, or, with activity routing `on`, a model an activity override uses (Sonnet 5.5 on the defaults). With `auto`, any model in the configured catalog starts with routing on, even if no table route names it. A fresh session on any other model starts with routing off. `/clear` starts the new session with routing on. Resuming a saved session restores its saved routing on or off.
|
|
344
351
|
|
|
345
352
|
## Edit routes and policy
|
|
346
353
|
|