@alexeiled/claude-router 1.23.0 → 1.24.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "router",
3
3
  "displayName": "Router",
4
- "version": "1.23.0",
4
+ "version": "1.24.0",
5
5
  "description": "Native model and effort routing within your provider. Anthropic, DeepSeek, z.ai, MiniMax, MiMo and Kimi defaults, custom provider profiles, classifier advice and one model per subagent.",
6
6
  "author": {
7
7
  "name": "Alexei Ledenev",
package/CHANGELOG.md CHANGED
@@ -1,5 +1,33 @@
1
1
  # Changelog
2
2
 
3
+ ## 1.24.0 (2026-11-17)
4
+
5
+ ### Added
6
+
7
+ - Opt in to stateful main-turn routing with `routingEngine: "auto"`. The default `table` behavior is unchanged; opt-in `shadow` records proposals while the table routes. Auto evaluation and default cutover remain pending. See [configuration](docs/configuration.md#routing-engine).
8
+ - Per-effort capability declarations guide auto and shadow. Anthropic seeds four balanced base-route pairs: Haiku 5.5 medium 0 and high 1; Opus 5.5 medium 2 and xhigh 3. These are user-derived policy judgments, not quality calibration. Other pairs and provider presets remain unknown. A supplied map replaces the preset; changing a model ID clears inherited capability. See [providers](docs/providers.md#capability-evidence).
9
+ - Shadow collects bounded, text-free replay inputs. Traces exclude prompts, tool results, filesystem-path fields, classifier endpoints and credentials, but retain routing aliases and model IDs, including path-shaped IDs. `/router trace export` merges retained traces into JSON, overwriting the previous export and rolling over near 3 MiB. **Delete history** does not delete exported files. See [trace limits and privacy](docs/evaluation.md#replay-trace-recorder).
10
+ - Offline replay from a checkout checks table fidelity before simulating auto. Coverage and exposure gates remain unavailable or not established; costs are conditional estimates, not savings or quality evidence. The recorded benchmark smoke checks harness mechanics on installed Router 1.23.1 only, not this checkout or an engine comparison. See [evaluation](docs/evaluation.md#offline-automatic-engine-replay).
11
+
12
+ ### Changed
13
+
14
+ - Equal-cost auto candidates favor the incumbent, then the lower sufficient declared capability. Estimates do not model unseen effort-dependent thinking tokens; mode margin supplies headroom. See [selection](docs/auto-engine.md#demand-target-and-move).
15
+
16
+ ### Upgrade
17
+
18
+ Restart Claude Code after updating. Auto and shadow require explicit opt-in.
19
+
20
+ ## 1.23.1 (2026-11-15)
21
+
22
+ ### Fixed
23
+
24
+ - `cost` mode takes the tier below only when its model is cheaper per turn: a lower cache-read price, or an equal one and a lower input price. It compared input prices only, so in a mixed profile it could move to a model with a cheaper input but a dearer cache read (for example GLM-5.3-Flash below DeepSeek V4.1 Flash), which costs more on long sessions where almost all input is cache reads.
25
+ - A same-task move inside a tier waits for the dwell and a checkpoint by the same order, so the hold and `cost` mode agree on which model is cheaper.
26
+
27
+ ### Upgrade
28
+
29
+ Restart Claude Code after updating. In a mixed profile, a mid-task move to a model that is cheaper per turn now waits for the dwell and a checkpoint, as a tier downgrade does. No configuration is changed automatically.
30
+
3
31
  ## 1.23.0 (2026-11-15)
4
32
 
5
33
  ### Changed
@@ -312,7 +340,7 @@ The Now tab shows this session and today instead of the last 30 replies, a windo
312
340
 
313
341
  ## 1.12.0 (2026-10-13)
314
342
 
315
- The Usage tab now shows any period, not only this session and the time since the last reset. See [Compare routing with your model](docs/user-guide.md#compare-routing-with-your-model).
343
+ The Usage tab now shows any period, not only this session and the time since the last reset. See [Compare routing with no routing](docs/user-guide.md#compare-routing-with-no-routing).
316
344
 
317
345
  ### Added
318
346
 
@@ -115,7 +115,7 @@ At session start the incumbent is the session's model. `cellForModel` looks for
115
115
 
116
116
  ## Routing modes
117
117
 
118
- The routing mode (`lib/modes.mjs`) is a fixed transform of the configured routes, applied before policy. At `turn.start` the hook resolves the mode (session override in `$.store` `routing-mode:<session id>`, then the project's `.claude/router.json`, then `mode` in the profile `router.json`, then `balanced`) and stores `applyMode(router.config, mode)` as the turn's config: every tier's and activity's route as `modeRoute` gives it, and `modes[mode].policy` over `policy`. Every later read of the turn (policy, escalation, cache keys, activity order, the request) uses that one table. `modeRoute` always starts from the configured routes, so a turn never steps from a route a mode already moved. `cost` takes the route of the tier below (at `low`, the `low` route without its activity override; never `micro`) when its model has a lower input price, else one listed effort level less, and `high` always keeps its model; `quality` one listed effort level more, else the tier above's route. `micro` and pins keep the configured route.
118
+ The routing mode (`lib/modes.mjs`) is a fixed transform of the configured routes, applied before policy. At `turn.start` the hook resolves the mode (session override in `$.store` `routing-mode:<session id>`, then the project's `.claude/router.json`, then `mode` in the profile `router.json`, then `balanced`) and stores `applyMode(router.config, mode)` as the turn's config: every tier's and activity's route as `modeRoute` gives it, and `modes[mode].policy` over `policy`. Every later read of the turn (policy, escalation, cache keys, activity order, the request) uses that one table. `modeRoute` always starts from the configured routes, so a turn never steps from a route a mode already moved. `cost` takes the route of the tier below (at `low`, the `low` route without its activity override; never `micro`) when its model is cheaper per turn (`cheaperPerTurn` in `lib/cost.mjs`: the cache-read price, then the input price; the same order holds a same-task lateral move in `lib/activity.mjs`), else one listed effort level less, and `high` always keeps its model; `quality` one listed effort level more, else the tier above's route. `micro` and pins keep the configured route.
119
119
 
120
120
  **Route in use.** The loop keeps the route the last decision ran (`loop.route`, models key and effort) instead of rebuilding it from config. The incumbent's price, cache key and order come from it, and a decision that stays on the incumbent's cell keeps running it. So a mode change, or a route edit, reaches the request only through a gate: a tier switch, a taken lateral move, or a pin. A loop saved before 1.15 has no `loop.route` and rebuilds it once.
121
121
 
@@ -0,0 +1,152 @@
1
+ # Pure automatic routing engine
2
+
3
+ `lib/auto-engine.mjs` implements the [automatic routing design](plans/2026-11-16-auto-routing-design.md).
4
+ It has no I/O. Opt-in `routingEngine: "auto"` uses the stateful adapter `lib/auto-route.mjs` in the main-turn hook; the default table engine is unchanged. Shadow remains a one-step proposal and offline replay has its own state. Evaluation and default cutover remain pending.
5
+
6
+ ## Inputs
7
+
8
+ All functions take one object. Pass the original validated config, including its catalog, cache settings and
9
+ base `policy`. Do not pass `applyMode(config, mode)`: per-mode policy patches belong to the table engine only.
10
+ The automatic mode changes only the capability requirement.
11
+
12
+ A route is `{ model: alias, effort: string | number | null }`. Generated candidates use only supported strings,
13
+ or `null` for a model with `efforts: []`. A real native incumbent with an unresolved null or finite numeric effort
14
+ is valid but has unknown capability. An unsupported explicit string is not silently clamped.
15
+ Capability comes only from `models[alias].capability[effort]`, or `capability.default` on a no-effort model.
16
+ No declaration means unknown, not zero. Anthropic includes user-derived balanced base-route policy seeds, not
17
+ measured quality; all other pairs remain unknown. Users can replace them as described in
18
+ [Providers](providers.md#capability-evidence).
19
+
20
+ The facts object uses the existing controller shapes:
21
+
22
+ - `contextTokens`, `historyMeasured`, `hasImages`;
23
+ - `models`: cache key → `{ prefixTokens, lastAt, ... }`;
24
+ - `effort`: native session effort, and `lastRequest: { tokens, outputTokens, ... }`;
25
+ - `ineligible`: model IDs excluded for this history;
26
+ - `dwellAllowed`: a separate boolean for physical downgrades, required even on a new task. Set it to
27
+ `requests - switchedAt.requests >= config.policy.minDwellRequests`, using the actual route's last switch and
28
+ request counter. At session start, before any request/switch exists, set it to true. Missing means false.
29
+ - `downAllowed`: the existing combined dwell/checkpoint fact, required in addition to `dwellAllowed` only on the
30
+ same task; `midTurn`, `progress` and advice continuation identify that task. Off the same task no checkpoint
31
+ is required. Upgrades, forced refits, escalation and explicit stronger mode/catalog moves bypass dwell.
32
+ - `failure: { signature, index }` or null. The same signature at a new index is a new occurrence.
33
+ Re-reading the same occurrence does not escalate again. History reset must reset engine state.
34
+
35
+ Advice uses the existing `{ choice, probabilities, continuation, ... }` classifier shape, or null.
36
+ `now` is the caller's timestamp in milliseconds. `availableModels` and `nativeModel` use the existing
37
+ `isModelAllowed` semantics. Pass the active provider's cache configuration, including effort-cache separation.
38
+ If the host has resolved a configured model ID to a response ID, project those IDs into the supplied catalog
39
+ before assessing cache state, as the table controller's economic config does.
40
+
41
+ ## Demand, target and move
42
+
43
+ | Export | Input | Output |
44
+ | ----------------------- | ---------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------- |
45
+ | `initialAutoState` | `{ demand = 2, routedDemand = demand, cause = 'classifier' }` | Initial state |
46
+ | `initialIncumbentState` | `{ config, incumbent, facts }` | Baseline demand with actual known incumbent capability as routed demand |
47
+ | `advanceDemand` | `{ config, state, advice = null, facts, pin = null, boundedExecute = false }` | `{ raw, level, reason, headroom, escalated, state }` |
48
+ | `assessCandidates` | `{ config, facts, now, incumbent, contextKnown = Number.isFinite(facts.contextTokens), availableModels, nativeModel }` | Candidate facts array |
49
+ | `selectTarget` | `{ candidates, demand, mode = 'balanced', headroom = false, escalation = false }` | `{ outcome, route, capability, requirement, estimate, reason, headroom }` |
50
+ | `decideMove` | `{ config, state, demandResult, target, candidates, advice = null, facts, now, changeCause = null, startup = false }` | `{ move, route, reason, evidence, estimate, state }` |
51
+ | `proposeAutoRoute` | Combined inputs above, using `incumbent` and optional `state` | `{ demand, candidates, target, move, state }` |
52
+
53
+ The facade initializes missing state with `initialIncumbentState`. Raw demand starts at `config.baselineTier`;
54
+ routed demand starts at the actual incumbent's declared capability for its supported/resolved native effort.
55
+ Unknown capability, an unlisted model, or unresolved/unsupported effort conservatively keeps baseline routed
56
+ demand; it is never inferred from a route table, mode offset or a guessed effort. Runtime and replay use the
57
+ same initializer. `initialAutoState` remains available for callers supplying explicit state.
58
+ State includes integer `demand`, `turn`, `votes`, `holdUntilTurn`, `escalatedSignature`, `escalatedIndex`,
59
+ `headroom`, `demandCause`, `routedDemand` and `routeCause`.
60
+ `raw` is integer demand; `level` is effective demand after the noncumulative half-step bounded-execute discount.
61
+ The caller supplies `boundedExecute: true` only for an already accepted bounded execute/ops phase, never
62
+ for a goal label alone or verify/report phases. An escalation hold protects the raised demand.
63
+
64
+ Each candidate has `route`, `requestKey` (canonical physical identity), `capability` (number or null), `eligible`, `reasons`, `incumbent`, `selectable`,
65
+ `cache`, `coldWriteUsd` and `estimate`. The incumbent is assessed even if it left the catalog.
66
+ A native unresolved effort is included in normal target comparison so it wins equal estimates, but cannot be
67
+ nominated as a new non-incumbent route. Declared-below-demand candidates still precede an unknown incumbent.
68
+ On a no-effort model, numeric native effort and the generated null-effort candidate have the same request key.
69
+
70
+ All availability, context, image, effort and cold-write cash constraints run before fallback.
71
+ An invalid incumbent and no eligible candidate produce explicit `no-route`, never an invented native route.
72
+ Target outcomes are `meets-demand`, `declared-below-demand`, `unknown-only`, `incumbent` and `no-route`.
73
+
74
+ Estimates are absolute next-turn `{ minUsd, maxUsd }` ranges: `cost.inputBounds` plus the last reply's output
75
+ length at `cost.ratesAt` prices. This estimate is effort-blind; it does not model additional thinking tokens.
76
+ Selection uses the conservative `maxUsd` bound, then the incumbent, lower sufficient declared capability and
77
+ config order. The mode margin supplies headroom through the requirement, not a tie preference. Automatic prefix
78
+ caches have no assumed TTL or warm-cache exemption.
79
+ Switch gates use the existing route-to-route `switchingTaxUsd` and `downgradeTaxUsd` math.
80
+ These are conditional estimates, not expected costs or measured savings.
81
+
82
+ ## State ownership
83
+
84
+ Shadow needs only `advanceDemand → assessCandidates → selectTarget`. Retain the returned demand state and
85
+ always assess the real incumbent/cache; do not retain a hypothetical target as route history.
86
+ Pass `escalation: demand.escalated` and `headroom: demand.headroom` to `selectTarget`. After either escalation or
87
+ pending-headroom selection, copy `target.headroom` back to the returned demand state's `headroom` field.
88
+ It is true only on exhausted escalation, permitting retry if a stronger candidate becomes eligible.
89
+ Target reasons are null, `escalation`, `escalation-unknown` (unknown capability) and `escalation-exhausted`.
90
+ A declared stronger eligible request precedes unknown alternatives; neither search can return the same physical
91
+ request via another alias. The facade performs this headroom feedback automatically.
92
+
93
+ Stateful replay/auto compares effective demand with `state.routedDemand`, not the previous turn's demand.
94
+ A physical move updates routed demand and route cause. A `same-route` selection instead accepts the current
95
+ effective demand (including zero) as routed demand without a switch: `move: false`, route cause, hold, votes
96
+ and failure markers remain unchanged. Canonically identical physical requests under different aliases count
97
+ as same-route. Pins, no-route, exhausted escalation and blocked moves do not adopt demand.
98
+ The facade returns accepted state in `result.state` and `result.move.state`; its demand-only state remains in
99
+ `result.demand.state`. A held proposal leaves routed demand and cause unchanged, so it remains pending.
100
+ Runtime and offline replay use the same `decideMove` transition.
101
+ `changeCause` is null, `mode` or `catalog`; it cannot relabel a demand change as an explicit configuration move.
102
+ The separate `startup` input defaults to false and must come from genuine per-history routing evidence,
103
+ never `state.turn === 0` or the physical request count. A first tier pin consumes eligibility even though its
104
+ demand state remains unchanged. Manual/off native requests do not consume it; their cache and dwell observations
105
+ still count normally. The loop persists `hasRoutedDecision` with the same state write as the routed decision,
106
+ including table decisions and pins. Reload, off/on, engine changes and compaction preserve it; a new session or
107
+ `/clear` new history starts false. Legacy unmarked histories with request, decision or auto-state evidence
108
+ conservatively count as consumed. This adds no configuration flag or state version.
109
+ Startup qualification uses the selector's existing `target.requirement`: effective demand after any accepted
110
+ bounded-execute discount plus the mode margin. It does not cancel that discount or invent an undiscounted floor.
111
+ Only a declared known incumbent below that requirement and a declared target meeting it can qualify; unknown
112
+ capability never grants startup evidence. After unchanged demand voting, qualifying startup movement takes
113
+ priority over classifier upgrade/downgrade movement and uses mass 1 through the same base-policy explicit-up
114
+ switch-tax gate. All candidate hard constraints and the cash cap still apply; this is not a forced move.
115
+ A blocked first decision consumes the chance, so there is no second-turn startup retry. For example, native
116
+ capability 0, baseline 1 and first low probability 0.57 can move with startup mass 1; the same probability on an
117
+ ordinary later demand upgrade still needs the unchanged classifier mass gate. Fresh micro advice that reduces
118
+ effective demand to 0 does not authorize startup when capability 0 already meets the resulting requirement.
119
+ Startup move reason and route cause are `startup`, not `mode`. Replay compares accepted automatic mode/catalog
120
+ evidence rather than treating table-only pin config snapshots as catalog edits; genuine edits survive pins.
121
+ The live adapter and replay fingerprint the actual allowed configured aliases, including native `default` resolution.
122
+ An allowlist expansion is catalog evidence; reordering it, cache warmth and routine response-ID resolution are not.
123
+
124
+ Tier pins accept a tier string or integer 0..3. They are one-turn overrides: no mode margin and no move gates,
125
+ but normal candidate constraints, outcome precedence and incumbent ties. The facade returns the pinned route
126
+ with `move: false`, reason `pinned` and unchanged state. The caller resumes the pre-pin incumbent and state on
127
+ the next unpinned turn. With the lower-level API, pass `mode: 'balanced'` to `selectTarget` for a pinned demand.
128
+ Exact model/effort pins remain the controller's separate responsibility.
129
+
130
+ An observed native fallback records `incumbentOrigin: 'external-fallback'` and rebases the physical incumbent without
131
+ resetting demand, votes, hold or routed demand. Substituted effort remains unknown. The current turn stays native;
132
+ on the next turn, `decideMove` takes `incumbentOrigin` and permits `reentry` to a different eligible target without
133
+ mass, dwell or payback gates. Hard constraints and the cash cap still apply. Re-entry sets routed demand and cause
134
+ `reentry`, retaining the escalation hold; an equal physical target accepts current effective demand and clears
135
+ origin without changing route cause or counting a switch. Pinned fallbacks retain normal state and
136
+ apply the physical fallback provenance after restoring the pin. Replay counts external requests separately.
137
+
138
+ ## Reloads
139
+
140
+ Before routing after a Mod reload without `session.start`, the hook lazily loads and validates the selected
141
+ provider profile and project mode. Constructor defaults never choose a main-turn route. A failed load marks
142
+ routing unavailable and passes the native request unchanged, without resetting persisted auto state to table.
143
+ A real repeated `session.start` invalidates old turn-config and classifier/review jobs before using the reloaded
144
+ profile. Compatible persisted automatic state survives; an actual engine change resets incompatible engine state.
145
+ Lazy profile loading is not a fresh-session signal: shadow recording remains explicitly incomplete mid-session.
146
+
147
+ ## Checks
148
+
149
+ Run `node --test test/auto-engine.test.mjs` for the design's §5.4 invariants and §3.3 move-evidence cases.
150
+ The tests also cover failure occurrence deduplication, raw/effective demand, pins, native efforts, cache semantics,
151
+ range/cash/context boundaries and a capability-monotonicity matrix across sparse catalogs and differing prices.
152
+ No paid API calls or classifier calls are needed.
@@ -37,6 +37,25 @@ See [Providers](providers.md) for selection examples, built-in defaults and comp
37
37
  The same schema applies to every router file. This reference uses `router.json` as shorthand for the selected file.
38
38
  A non-Anthropic file must exist and validate. It never falls back to `router.json` or inherits settings from it.
39
39
 
40
+ ### Model capability declarations
41
+
42
+ `models.<alias>.capability` is an optional map of supported efforts to adequacy levels: 0 (micro),
43
+ 1 (low), 2 (medium), or 3 (high), with half-steps allowed. For example:
44
+
45
+ ```json
46
+ { "models": { "sonnet": { "capability": { "medium": 1.5, "high": 2 } } } }
47
+ ```
48
+
49
+ These example values illustrate the shape, not measured model performance. Anthropic has user-derived built-in
50
+ policy seeds; see [Providers](providers.md#capability-evidence). Users may replace or clear them. Missing maps and entries are
51
+ unknown, not 0. For a model with `efforts: []`, use only `"default"` for its no-effort candidate. Other keys,
52
+ unsupported efforts, non-numbers and values outside 0–3 or between half-steps are invalid.
53
+
54
+ A user-supplied map replaces the preset map in full; `{}` clears it. Replacing a model ID clears inherited
55
+ capability, even if the same alias is kept; the `[1m]` context hint alone does not change identity. Changing
56
+ `efforts` prunes unsupported inherited entries; unsupported explicitly supplied entries are rejected. Capability
57
+ is separate from `strength`, which still controls the current table routing. Capability declarations affect `auto` and `shadow`; the default `table` uses strength. Built-in declarations and their evidence limits are listed in [Providers](providers.md#capability-evidence).
58
+
40
59
  ### Provider prices
41
60
 
42
61
  All non-Anthropic built-in model windows are at least 1M tokens.
@@ -106,6 +125,22 @@ The router never substitutes zero savings or invents another route's cache hit r
106
125
  Models with `efforts: []`, including MiMo, ignore session effort in the comparison identity.
107
126
  Missing counters remain unknown. See [provider acceptance checks](evaluation.md#provider-profiles).
108
127
 
128
+ ## Routing engine
129
+
130
+ Set `"routingEngine": "shadow"` in the selected profile file to record one-step automatic-routing proposals alongside
131
+ the table's decisions. The default, `"table"`, does not run the automatic engine. Shadow uses the real incumbent and
132
+ cache for each proposal; the table still chooses every request. The Now pane and `/router decisions` show demand,
133
+ target, capability (or unknown), outcome and a next-turn cost estimate. Shadow requests no extra classifier advice.
134
+ The proposal is not a prediction of a stateful automatic-routing session.
135
+
136
+ Set `"routingEngine": "auto"` to opt in to stateful automatic routing. It selects from the configured model and
137
+ effort catalog using demand, a mode margin, hard constraints and observed cache economics. It does not use route
138
+ tables or mode policy patches. Tier pins override demand for one turn without a margin, but still obey hard constraints.
139
+ If no candidate fits, the original native request runs with reason `no-route-native`. Auto quality evaluation and
140
+ default cutover remain pending; see [evaluation](evaluation.md). After an observed fallback, the rest of its turn
141
+ stays native. The next turn can re-enter an eligible target without confidence, dwell or payback gates; hard constraints
142
+ and the cash cap still apply, and escalation holds survive. The three values are case-sensitive.
143
+
109
144
  ## Built-in defaults
110
145
 
111
146
  The default baseline is tier `low`. Each route refers to an alias in `models`.
@@ -351,7 +386,7 @@ The Mod validates the whole file. Unknown keys and invalid values make routing u
351
386
  | `mode` | `cost`, `balanced`, `quality` ([Routing modes](routing-modes.md)) | `balanced` |
352
387
  | `modes.<cost\|quality>.policy` | any `policy` field, used while that mode is active | [Routing modes](routing-modes.md#policy-per-mode); not tuned |
353
388
  | `activities.<activity>.<tier>` | optional `model` and `effort`, as in `routes` | The overrides in [Activity routing](#activity-routing) |
354
- | `models.<alias>` | `id`, `input`, optional `output`, `cacheRead`, optional `longContext` (`above`, `multiplier`, or `null`), optional `offPeak` (`multiplier`, `peakDays` as UTC weekdays 0–6 from Sunday, `peakHoursUtc` as `[from, to)` hour ranges, or `null`), `contextWindow`, `billing`, `efforts`, optional `strength`, `imageInput` | Three defaults above |
389
+ | `models.<alias>` | `id`, `input`, optional `output`, `cacheRead`, optional `longContext` (`above`, `multiplier`, or `null`), optional `offPeak` (`multiplier`, `peakDays` as UTC weekdays 0–6 from Sunday, `peakHoursUtc` as `[from, to)` hour ranges, or `null`), `contextWindow`, `billing`, `efforts`, optional `strength`, optional `capability`, `imageInput` | Three defaults above |
355
390
  | `cache` | `mode`, `effortScope`, `writeMultiplier`, `ttlMs`, `warmMarginMs` | `5m: 1.25`, `1h: 2`; `300000` / `3600000` ms; `30000` ms |
356
391
  | `policy` | Fields below | Values below |
357
392
  | `classifier` | An id in `classifiers` | `jev` |
@@ -4,6 +4,225 @@ The native Mod has no measured savings result. The pane reports Claude's own usa
4
4
 
5
5
  This page separates an older gateway experiment from a shadow replay of native policy. Neither result measures answer quality or proves lower real spend. [Activity routing](#activity-routing) adds two checks of the activity label, with no savings result either.
6
6
 
7
+ ## Automatic-routing shadow proposals
8
+
9
+ Opt-in `routingEngine: "shadow"` records a demand and target proposal against the real table incumbent for each
10
+ applicable decision. It does not run the proposed route or maintain a simulated route history. These rows do not
11
+ establish savings or answer quality; a next-turn cost estimate is not an observed cost or a counterfactual saving.
12
+ The bounded decision records are not replay traces. See [Configuration](configuration.md#routing-engine).
13
+
14
+ ## Replay trace recorder
15
+
16
+ In `routingEngine: "shadow"`, the main-turn hooks also keep text-free replay inputs separately from decision rows:
17
+ the initial configured and mode-transformed routing policy, provider/cache semantics and model prices; classifier
18
+ tier/activity/phase vectors; failure fingerprints and occurrence indexes; pins, modes, resets, context/availability
19
+ facts and observed moves; and correlated request/reply model, effort and token counters. Checkpoint evidence uses
20
+ message indexes and success flags, not commands, tool results or filesystem-path fields, so a later replay can use
21
+ its own switch cursor. Validated routing aliases and model IDs are preserved exactly, including path-shaped IDs;
22
+ classifier endpoints, credentials and the config file's nativePath are not copied.
23
+
24
+ Records use `trace:v1:<session id>`: 30-day retention, 128 KiB per session, 1 MiB total trace budget and a measured
25
+ 3 MiB guard on the shared plugin store. Pending turns are bounded too. A cap discards the entire pending turn,
26
+ keeps the earlier contiguous prefix usable and counts excluded turns/events. Late starts, resume/reload, gaps,
27
+ conflicts and lost writes are incomplete. A failed write taints memory and persists that taint on the next success;
28
+ an unavailable store cannot guarantee a durable failure marker. A clean session close is not required.
29
+ Freshness requires the first main request (index 0), no saved loop or trace, and readable normalized request
30
+ history with no assistant messages. Injected startup reminders are user rows and do not make a trace incomplete;
31
+ `classic.SessionStart` is not required. Unreadable or malformed history, reload and late enable are conservative.
32
+ Compaction preserves an existing trace with an explicit reset; `/clear` permits fresh history under a new session ID.
33
+
34
+ Run `/router trace export` to merge retained traces into `router-traces.json` beside the selected profile's
35
+ router config. The command also works after switching back to `table`; ordinary table turns do no trace I/O.
36
+ Exports overwrite their previous JSON after merging; near 3 MiB, a new dated `router-traces-*.json` file receives
37
+ the retained snapshot instead. Keep all export files for offline replay. Known taint is never erased by a longer
38
+ or newer clean prefix. **Delete history** clears stored traces in any mode, but exported files remain user-owned:
39
+ the host filesystem API has no unlink operation.
40
+
41
+ Recent active sessions have a one-hour last-write lease against budget eviction; stale activity cannot defeat
42
+ 30-day retention. Store scans, quota checks and deletion are best effort across processes, not atomic reservations.
43
+ There is no live exposure, routing-quality result or savings claim from this recorder yet.
44
+
45
+ ## Offline automatic-engine replay
46
+
47
+ Run the tool from a checkout with **all** incremental exports (shell-expanded file arguments, no implicit profile
48
+ access). It reads only those files and writes anonymous JSON to stdout; it never changes a router or provider profile.
49
+
50
+ ```sh
51
+ node scripts/replay-traces.mjs /exports/router-traces.json /exports/router-traces-*.json > replay-report.json
52
+ node scripts/replay-traces.mjs --reversal-window 3 test/fixtures/replay-traces.json
53
+ node --test test/replay.test.mjs
54
+ ```
55
+
56
+ Omit the dated-file argument when there are no rollover files. The second command uses only synthetic data.
57
+ The implementation follows [design §5.1 and §6 step 5](plans/2026-11-16-auto-routing-design.md#51-replay-traces-and-offline-replay).
58
+
59
+ 1. Merge sessions with the recorder's `mergeTraceExport`: preserve lost-write taint, overlap conflicts and capped
60
+ counts. Keep usable capped contiguous prefixes; report their omitted turns/events separately.
61
+ 2. Replay the table chain from **only** its initial header, applying config, controls, history preparation/resets,
62
+ choose/observed/continue operations and responses. Compare every recorded decision and request model/effort.
63
+ An unexplained mismatch on any otherwise usable prefix returns `status: "blocked"`, exits 1 and prints **no**
64
+ auto/session totals. It remains in the fidelity denominator. Taint/gaps/incompleteness are explained exclusions.
65
+ 3. Only after fidelity passes, validate independent-replay evidence. Missing checkpoints, required boolean facts,
66
+ unconditional continuation modality or activity indexes produce `auto-input-incomplete` and exit 1, without
67
+ silently excluding a usable fidelity session. Earlier exports can lack these fields.
68
+ 4. Simulate auto with its own demand/votes, incumbent, cache, routed-demand/provenance and switch/dwell cursors.
69
+ Classifier decisions advance votes; continuations reevaluate constraints and pending moves without adding votes.
70
+ Checkpoints and observed activity are reconstructed from text-free indexes against its own cursor. Only an
71
+ accepted bounded execute phase discounts demand; observed edits can remove it. No activity route table is used.
72
+
73
+ **Mode meaning:** fidelity and both comparisons follow the recorded mode-change history. Output is partitioned by
74
+ the mode active on each request, not three artificial reruns. Table uses the recorded transformed config; auto
75
+ uses the recorded base policy, with mode only as the capability margin. There is no forced-mode option.
76
+
77
+ The report has `fidelity`, `coverage`, `exposureDiagnostic` and anonymous numbered `sessions[].modes`. Each mode
78
+ contains both engines' conditional cost range, priced/unpriced request counts, effort mix, switches and disagreement
79
+ counts grouped by effective demand/activity. Unknown catalog capabilities are counted explicitly. Anthropic seeds four balanced base-route pairs as
80
+ user-derived policy judgments; other pairs and provider presets remain unknown. These seeds and their replay are not
81
+ quality-calibration evidence.
82
+
83
+ Switches are computed on the full session before mode partitioning and attributed to the **destination** mode.
84
+ A reversal closes A→B→A within the explicit `--reversal-window N` main turns (default 3); `reversals` counts closing
85
+ legs, `reversalMoves` counts participating legs, and `oneWay` counts unpaired moves. Rates are per 100 captured turns
86
+ in that mode. Pins do not count as switches and restore the pre-pin route/demand/dwell baseline; every pinned
87
+ continuation still checks eligibility. Manual intervals, history resets and external native interventions break the
88
+ policy-switch chain, including reversal pairing.
89
+
90
+ ### Coverage and estimate limits
91
+
92
+ - `coverage.exportCohort` ratios describe **only captured/exported sessions**, a lower-bound cohort. Exports do not
93
+ count evicted or untraced sessions; `coverage.allSessions` denominators and its 80% gate stay unavailable.
94
+ Sufficient future metadata would include capture-eligible session starts (including untraced/evicted sessions),
95
+ total main turns and per-mode eligible routed turns over the same capture window. This tool invents none of it.
96
+ - Exposure thresholds (30 complete sessions, 1,000 main turns, 200 per evaluated mode) are diagnostics only.
97
+ Captured main turns are separate from eligible unpinned classifier-decision turns; manual-only sessions/turns
98
+ do not count as engine exposure. Freshness is unverified and the exposure gate remains `not-established`.
99
+ - Cost freezes observed total prompt/output lengths, timestamps, failures and tool calls. Auto simulates its own
100
+ full-prefix cache scenario, never the table's cache-hit tokens. Explicit cache warmth uses the existing conservative
101
+ five-minute helper; automatic caches retain hit/miss bounds, with no invented TTL guarantee. Effort-specific,
102
+ disabled, long-context and off-peak pricing use the existing pure cost helpers. These are **conditional estimates**,
103
+ not counterfactual savings, actual bills or paired quality results.
104
+ - Missing usage/prices and substituted replies make the affected total range `null`, not free. An explicitly named
105
+ `knownPricedRange` is only the priced subtotal. No transcript-length/quality change from a hypothetical request
106
+ is modeled.
107
+ - Fidelity distinguishes table echoes from genuine third-model/native fallbacks. Auto freezes actual external
108
+ interventions, response substitutions and routed null-stop suspension, rather than misreading a table echo as
109
+ its own fallback. It cannot predict a provider's fallback for a different hypothetical auto request.
110
+ - Default output includes no session IDs, prompts, tool text, model identifiers or paths. Input errors name the
111
+ numbered file/structural field, not its path. The CLI cannot discover omitted exports or verify their provenance.
112
+
113
+ ## Paired benchmark proposal
114
+
115
+ The [automatic-routing design §5.3](plans/2026-11-16-auto-routing-design.md#53-paired-benchmark-smoke-gate-for-quality)
116
+ requires paired `table`/`auto` runs, three modes and two repeats, with per-task checks and discordances. The
117
+ 24 synthetic expected-output cases in `test/fixtures/benchmark-routing.json` are harness fixtures, not calibrated
118
+ engineering tasks or a quality result. Their activity labels are fixture metadata, not reference classifications.
119
+ Eight are marked held out in the fixture split. The separate engineering proposal below needs fixtures and review
120
+ before any evaluation; shadow-disagreement sources are pending.
121
+
122
+ ```sh
123
+ node scripts/benchmark-routing.mjs --dry-run
124
+ node scripts/benchmark-routing.mjs --dry-run --smoke
125
+ node --test test/benchmark-routing.test.mjs
126
+ ```
127
+
128
+ The dry run prints a SHA-256, run counts, serialized token/rate assumptions and a configured-price scenario.
129
+ Dry-run planning uses the built-in catalog maximum prices through `ratesAt`; a live smoke preflight reads the selected
130
+ work profile's catalog without changing it. Cold-write multipliers apply only to explicit caches. The 10,000-input
131
+ assumption is not a limit; live smoke sets `CLAUDE_CODE_MAX_OUTPUT_TOKENS=512`, one agentic turn and a $0.40
132
+ CLI budget per call. These are not a proxy-price quotation or a bill. A run estimates before launching and reserves
133
+ $1.20 for three calls. An exclusive reservation file under `.pi/` refuses automatic reruns; further spending needs
134
+ review, not a hidden retry. The 288 planned task runs are for the toy mechanics fixtures, not an engineering benchmark cost estimate.
135
+ An engineering task run may use multiple inference calls, tools and retries; it needs new token, request, effort
136
+ and provider budgets plus lead approval before any run. No full matrix is authorized or run.
137
+
138
+ `--live --smoke` is limited to the three development tasks, `table`/`balanced`, one repeat, using the existing
139
+ installed release through `zsh -ic` and `claude-mp-work -p`. It loads no checkout plugin. The selected profile must
140
+ still be `table`; `--setting-sources user`, strict empty MCP and no tools limit the surface. Init metadata must
141
+ identify exactly one router at the installed user path. Profile contents are hashed before/after each call. The
142
+ wrapper's normal MCP housekeeping is permitted; the original active-environment bytes/existence are restored and
143
+ verified after success, failure, timeout or handled interrupt. SIGKILL, power loss and independently detached
144
+ processes are not covered. Process-group cleanup is best effort; Windows cannot kill a descendant tree via Node
145
+ alone. The smoke establishes harness mechanics only, not evaluation of this checkout or either engine.
146
+
147
+ Offline system-boundary tests exercise independent directories, output checks, deadlines, output limits, budgets,
148
+ unknown costs and paired discordance reporting. The harness checks its own and the manifest's hashes around child
149
+ execution; this detects persistent changes, not transient same-user writes or an OS sandbox. The checks run in the
150
+ parent outside the child's working directory. Unknown/zero total or per-model CLI prices stop further calls rather
151
+ than implying free usage. Reports sum every supplied attempt, including failures, and leave missing/unpriced
152
+ totals unknown. Duplicate attempts within a paired cell are reported as ambiguous, never silently replaced by a
153
+ passing rerun.
154
+ CLI-reported USD can use rates different from the selected catalog and may omit proxy retries, classifier spend or
155
+ upstream failed calls: actual billing remains unknown.
156
+ No comparison of quality, savings or measured routing performance follows from these checks.
157
+
158
+ The checked-in [installed-release smoke record](../experiments/mod-router/results/benchmark-harness-smoke.json)
159
+ contains three passing external checks, 19,298 ms summed subprocess wall time and $0.0098816 CLI-reported price
160
+ estimates. It records the manifest/config hashes and successful restoration/provenance checks without prompts,
161
+ profile paths or credentials. CLI usage metadata names DeepSeek Flash for two calls and GLM Flash for one; backend
162
+ identity is not independently verified.
163
+ These are harness-validation observations on installed Router 1.23.1, not a paired result for this checkout.
164
+
165
+ ### Engineering task set: draft only
166
+
167
+ The following 24 synthetic engineering tasks are a proposed **separate** evaluation set, not implemented fixtures.
168
+ Difficulty and activity are author hypotheses, not classifier references. Each row needs a self-contained repository
169
+ fixture and an independently held-out automatic oracle outside model control. Freeze the fixture/manifest hash and
170
+ split before capability or margin tuning; do not run or inspect held-out outcomes during tuning. All rows are authored
171
+ examples: shadow-disagreement sourcing is **pending** and must be added or documented before the set is finalized.
172
+
173
+ | ID | Split | Activity / demand hypothesis | Task | Independent automatic oracle |
174
+ | --- | --- | --- | --- | --- |
175
+ | E01 | dev | code / low | Add integer clamping to a small library | Boundary and type tests |
176
+ | E02 | dev | debug / medium | Repair a parser that mishandles escaped delimiters | Existing tests plus hidden escaped and empty cases |
177
+ | E03 | dev | code / medium | Add pagination to a small service across handler and storage modules | Fixture API and deterministic pagination tests |
178
+ | E04 | heldout | code / medium | Implement stable deduplication across a CLI and helper | Output and duplicate-order tests |
179
+ | E05 | dev | security / high | Reject traversal and symlink escapes in an archive extractor | Traversal, symlink and valid-path tests |
180
+ | E06 | dev | debug / high | Fix a cross-module stale cache invalidation defect | Concurrent update and expiry regression tests |
181
+ | E07 | heldout | debug / medium | Fix time boundary logic for expiry at exact deadline | Fake-clock boundary tests |
182
+ | E08 | dev | ops / medium | Repair CI script so test and lint failures reach the exit status | Controlled subprocess exit-code matrix |
183
+ | E09 | dev | review / medium | Identify seeded unsafe defaults in a short configuration | Exact structured findings against injected defects |
184
+ | E10 | heldout | debug / high | Repair nondeterministic retry handling after partial network failures | Deterministic mocked-service request and retry tests |
185
+ | E11 | dev | code / high | Add idempotent transaction handling across API and persistence | Duplicate/concurrent request and rollback tests |
186
+ | E12 | dev | debug / medium | Diagnose and fix an off-by-one token budget after a tool continuation | Transcript fixture and boundary tests |
187
+ | E13 | heldout | code / low | Add case-insensitive header lookup preserving original values | Mixed-case and missing-header tests |
188
+ | E14 | dev | code / medium | Migrate a versioned configuration while preserving unknown fields | Old/new config round-trip and rejection tests |
189
+ | E15 | dev | debug / high | Fix matching failure signatures after an edit and repeated tool failure | Scripted transcript with two failures and edit tests |
190
+ | E16 | heldout | security / high | Correct authorization checks across two endpoints sharing a store | Cross-user read/write and allowed-access tests |
191
+ | E17 | dev | refactor / medium | Replace duplicate parsing in two modules without changing behavior | Characterization and randomized input tests |
192
+ | E18 | dev | ops / low | Correct shell quoting for filenames containing spaces | Subprocess tests on spaces and metacharacters |
193
+ | E19 | heldout | code / high | Implement bounded parallel workers with cancellation and cleanup | Deterministic concurrency and child-cleanup tests |
194
+ | E20 | dev | debug / medium | Fix a multi-file regression after two failed test attempts and an edit | Scripted tool history plus external final tests |
195
+ | E21 | dev | docs / low | Correct a documented CLI example for a real edge-case flag | Execute example against fixture CLI |
196
+ | E22 | heldout | ops / medium | Repair a build pipeline that masks errors in a chained command | Failure-injection exit-code tests |
197
+ | E23 | dev | review / high | Find seeded race and missing transaction rollback | Machine-readable findings with location and defect checks |
198
+ | E24 | heldout | debug / medium | Fix Unicode normalization inconsistency across reader and index | Composed/decomposed matching and preservation tests |
199
+
200
+ These are not yet the §5.3 benchmark: the tasks, initial repositories, test oracles, tool-history presentation,
201
+ permissions and expected output need independent implementation and review. No engineering benchmark or held-out
202
+ outcome has been collected. Difficulty does not establish a capability value.
203
+
204
+ ### Full paired-run isolation: proposal only
205
+
206
+ Auto routing is available by opt-in in 1.24.0. The full paired benchmark adapter remains disabled pending checkout/profile
207
+ isolation and approval; a full live matrix always refuses. A native pass-through or table
208
+ run is never relabeled `auto`. The current hook reads `routingEngine` only from the selected profile file; there is
209
+ **no** run-scoped environment/settings override. Project `.claude/router.json` can set mode only.
210
+
211
+ Before the real paired evaluation, choose and implement one code-provenance path:
212
+
213
+ - Add an explicit supplied-router option to `ce`'s model-proxy path, matching its normal launch behavior. This can
214
+ preserve existing implicit authentication but needs wrapper maintenance and must disable the marketplace copy.
215
+ - Launch `claude-proxy` directly with per-run settings that disable the marketplace router and a checkout
216
+ `--plugin-dir`. This gives precise code provenance but must reuse an approved authentication interface, never
217
+ copy credentials or silently bypass account policy.
218
+ - Run after a release that includes the evaluated implementation. This avoids two plugin copies but does not
219
+ itself isolate the engine selection or establish checkout provenance.
220
+
221
+ Every path **also** needs a separately approved run-scoped engine override or safely isolated profile configuration
222
+ that keeps existing implicit authentication without copying secrets. Only then can each child select its own
223
+ `table` or `auto` without editing the user's real router file. No wrapper modification, new engine override,
224
+ authentication mechanism, live auto wiring or cutover is implemented here.
225
+
7
226
  ## Provider profiles
8
227
 
9
228
  [Provider defaults](providers.md#built-in-defaults) are starting policies, not measured per-activity quality or savings results.
package/docs/providers.md CHANGED
@@ -9,6 +9,34 @@ Custom profiles need their own model settings. They do not inherit Claude models
9
9
 
10
10
  This guide assumes Claude Code already works with your provider. Endpoint and credential setup are outside its scope.
11
11
 
12
+ ## Capability evidence
13
+
14
+ The Anthropic preset seeds `claude-haiku-5-5@medium` at 0, `claude-haiku-5-5@high` at 1,
15
+ `claude-opus-5-5@medium` at 2, and `claude-opus-5-5@xhigh` at 3. These are user-derived policy judgments
16
+ from the built-in balanced base routes, not model-card benchmarks or quality guarantees. All other pairs, including
17
+ Haiku at low and Opus at high, remain unknown, not zero. User-router PR #79 seed values are profile overrides layered
18
+ over these built-ins. Other built-in provider presets have no seeds. No rating is inferred from price, ordinal strength, benchmark rank,
19
+ or an effort name.
20
+ This metadata does not affect the current table router. See [configuration](configuration.md#model-capability-declarations)
21
+ for declarations and overrides.
22
+
23
+ The model-card IDs below identify the reviewed records, not verified served backends. The sources do not give
24
+ numeric values on this router's demand scale.
25
+
26
+ | Preset model / model-card ID | Evidence and gap |
27
+ | --- | --- |
28
+ | Fable 5.2 / `anthropic-claude-fable-5-2` | [Prompting guide](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-fable-5-2): effort guidance, not demand-tier calibration. |
29
+ | Opus 5.5 / `anthropic-claude-opus-5-5` | [Prompting guide](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-5-5): effort recommendations, not tier adequacy. |
30
+ | Sonnet 5.5 / `anthropic-claude-sonnet-5-5` | [Prompting guide](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-sonnet-5-5): well-specified agentic coding at medium does not establish all medium-tier work. |
31
+ | Haiku 5.5 / `anthropic-claude-haiku-5-5` | [Prompting guide](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-haiku-5-5): simple work at low does not calibrate reliability to micro demand. |
32
+ | DeepSeek Flash / `deepseek-deepseek-v4.1-flash-1017` | [Provider evaluation](evaluation.md#provider-profiles): earlier benchmark results do not establish the hosted snapshot's per-effort adequacy. |
33
+ | GLM-5.3 / `zai-glm-5.3`; GLM-5.3-Flash / `zai-glm-5.3-flash` | [Provider evaluation](evaluation.md#provider-profiles): max-effort results do not establish low/high performance or tier adequacy. |
34
+ | MiniMax M3.1 Flash / `minimax-minimax-m3.1-flash`; Kimi K3.1 Pro / `moonshot-kimi-k3.1-pro` | [Provider experiments](provider-experiments.md): small synthetic checks do not calibrate repository tasks per effort. |
35
+ | MiMo V2.6 Flash and Pro / no model-card record | [Provider experiments](provider-experiments.md): no granular effort setting; small synthetic checks do not establish demand-tier adequacy. |
36
+
37
+ The auto engine selects the cheapest eligible unknown when no eligible declaration is known. Neither the seeds nor
38
+ unknown-only selections establish quality or escalation-headroom calibration; users can configure their own maps.
39
+
12
40
  ## Select a provider
13
41
 
14
42
  For DeepSeek, create `deepseek-router.json` in your Claude Code profile directory with this content:
@@ -1,11 +1,11 @@
1
1
  # Routing modes
2
2
 
3
3
  One setting trades cost against answer quality: `cost`, `balanced` (the default) or `quality`. The classifier and
4
- your routes stay as they are. The mode moves the route each tier runs by one fixed step.
4
+ your routes stay as they are. With the default `table` engine, the mode moves the route each tier runs by one fixed step.
5
5
 
6
6
  | Mode | What changes |
7
7
  | ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
8
- | `cost` | A cheaper route where the turn allows it: the tier below when that tier runs a model with a lower input price (at `low`, the `low` route without its activity override), else one effort level less on the same model. `high` keeps its model and drops one effort level. Never below `low`. |
8
+ | `cost` | A cheaper route where the turn allows it: the tier below when that tier runs a model that is cheaper per turn: a lower cache-read price, or an equal one and a lower input price (at `low`, the `low` route without its activity override), else one effort level less on the same model. `high` keeps its model and drops one effort level. Never below `low`. |
9
9
  | `balanced` | Nothing: every tier runs its route as configured. |
10
10
  | `quality` | One effort level more on the same model, else the route of the tier above. Haiku 5.5 automatically prefers High, except medium bounded ops may use Xhigh. Explicit higher efforts stay supported. |
11
11
 
@@ -19,6 +19,11 @@ In every mode:
19
19
  on a model listing only those three. A route with a `null` effort (the session's) or a model without efforts has no effort step.
20
20
  - [MiniMax, MiMo and Kimi tables](providers.md#new-provider-modes) show every new preset's mode/activity combination.
21
21
 
22
+ Optional [automatic routing](configuration.md#routing-engine) uses the mode only as a capability margin: `cost`
23
+ subtracts 0.5 from demand, `balanced` adds nothing and `quality` adds 0.5, clamped to 0–3. It uses base policy,
24
+ not mode policy patches. Tier pins use exact demand without a margin; candidates still must fit. Shadow records a
25
+ proposal while the table routes. Auto is opt-in; quality evaluation and cutover are pending.
26
+
22
27
  ## Choose a mode
23
28
 
24
29
  The mode comes from the first of these that sets one:
@@ -44,6 +49,8 @@ then the band says why the route stays. Pin a tier to run its configured route n
44
49
 
45
50
  ## Policy per mode
46
51
 
52
+ The policy patches in this section apply to `table` and `shadow`, not opt-in `auto`.
53
+
47
54
  Each mode except `balanced` replaces some `policy` fields while it is active. The built-in values started as
48
55
  codex-router's and are **not tuned** on outcomes. Since 1.16, `cost` upgrades on balanced's bar (2 votes from 0.75):
49
56
  with 3 votes from 0.85, the cache write of a cold stronger model raised the bar to about 0.92 from 60k tokens and 0.97