@alexeiled/claude-router 1.23.1 → 1.25.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/plugin.json +1 -1
- package/CHANGELOG.md +62 -1
- package/README.md +39 -7
- package/docs/architecture.md +27 -4
- package/docs/auto-engine.md +152 -0
- package/docs/configuration.md +47 -3
- package/docs/evaluation.md +222 -3
- package/docs/native-router.md +11 -3
- package/docs/providers.md +28 -0
- package/docs/router-band.svg +124 -133
- package/docs/router-pane-classifier.svg +24 -23
- package/docs/router-pane-now.svg +9 -7
- package/docs/router-pane-policy.svg +4 -4
- package/docs/router-pane-routes.svg +8 -8
- package/docs/router-pane-usage-whatif.svg +14 -14
- package/docs/router-pane-usage.svg +20 -20
- package/docs/routing-modes.md +13 -4
- package/docs/user-guide.md +79 -19
- package/hooks/native-router.mjs +1194 -215
- package/lib/activity-stats.mjs +23 -4
- package/lib/activity.mjs +7 -5
- package/lib/auto-engine.mjs +414 -0
- package/lib/auto-route.mjs +253 -0
- package/lib/band.mjs +35 -440
- package/lib/classifier-client.mjs +12 -8
- package/lib/config-file.mjs +7 -2
- package/lib/config.mjs +64 -15
- package/lib/cost.mjs +5 -1
- package/lib/costs.mjs +4 -2
- package/lib/decisions.mjs +80 -5
- package/lib/display.mjs +69 -20
- package/lib/facts.mjs +30 -0
- package/lib/modes.mjs +8 -4
- package/lib/panel.mjs +75 -2163
- package/lib/policy.mjs +7 -6
- package/lib/providers.mjs +7 -0
- package/lib/replay-trace.mjs +550 -0
- package/lib/replay.mjs +718 -0
- package/lib/route-line.mjs +46 -0
- package/lib/route.mjs +35 -21
- package/lib/savings.mjs +14 -3
- package/lib/subagents.mjs +41 -11
- package/lib/subcommands.mjs +60 -0
- package/lib/ui/fallback.mjs +22 -0
- package/lib/ui/graphical/band.mjs +256 -0
- package/lib/ui/graphical/charts.mjs +163 -0
- package/lib/ui/graphical/classifier.mjs +153 -0
- package/lib/ui/graphical/frame.mjs +227 -0
- package/lib/ui/graphical/kit.mjs +156 -0
- package/lib/ui/graphical/meter.mjs +25 -0
- package/lib/ui/graphical/now.mjs +353 -0
- package/lib/ui/graphical/policy.mjs +56 -0
- package/lib/ui/graphical/routes.mjs +264 -0
- package/lib/ui/graphical/squares.mjs +29 -0
- package/lib/ui/graphical/usage.mjs +376 -0
- package/lib/ui/model/band.mjs +439 -0
- package/lib/ui/model/classifier.mjs +308 -0
- package/lib/ui/model/draft.mjs +62 -0
- package/lib/ui/model/format.mjs +41 -0
- package/lib/ui/model/frame.mjs +359 -0
- package/lib/ui/model/now.mjs +390 -0
- package/lib/ui/model/policy.mjs +194 -0
- package/lib/ui/model/routes.mjs +392 -0
- package/lib/ui/model/usage-agents.mjs +311 -0
- package/lib/ui/model/usage-breakdown.mjs +237 -0
- package/lib/ui/model/usage-shared.mjs +58 -0
- package/lib/ui/model/usage.mjs +330 -0
- package/lib/ui/terminal/band.mjs +220 -0
- package/lib/ui/terminal/classifier.mjs +142 -0
- package/lib/ui/terminal/frame.mjs +169 -0
- package/lib/ui/terminal/kit.mjs +133 -0
- package/lib/ui/terminal/now.mjs +300 -0
- package/lib/ui/terminal/policy.mjs +52 -0
- package/lib/ui/terminal/routes.mjs +167 -0
- package/lib/ui/terminal/usage.mjs +363 -0
- package/lib/ui/terminal/wrap.mjs +37 -0
- package/lib/ui/tokens.mjs +106 -0
- package/lib/usage.mjs +5 -5
- package/lib/view.mjs +5 -2
- package/package.json +3 -2
- package/types/index.d.ts +15 -0
package/docs/evaluation.md
CHANGED
|
@@ -4,6 +4,225 @@ The native Mod has no measured savings result. The pane reports Claude's own usa
|
|
|
4
4
|
|
|
5
5
|
This page separates an older gateway experiment from a shadow replay of native policy. Neither result measures answer quality or proves lower real spend. [Activity routing](#activity-routing) adds two checks of the activity label, with no savings result either.
|
|
6
6
|
|
|
7
|
+
## Automatic-routing shadow proposals
|
|
8
|
+
|
|
9
|
+
Opt-in `routingEngine: "shadow"` records a demand and target proposal against the real table incumbent for each
|
|
10
|
+
applicable decision. It does not run the proposed route or maintain a simulated route history. These rows do not
|
|
11
|
+
establish savings or answer quality; a next-turn cost estimate is not an observed cost or a counterfactual saving.
|
|
12
|
+
The bounded decision records are not replay traces. See [Configuration](configuration.md#routing-engine).
|
|
13
|
+
|
|
14
|
+
## Replay trace recorder
|
|
15
|
+
|
|
16
|
+
In `routingEngine: "shadow"`, the main-turn hooks also keep text-free replay inputs separately from decision rows:
|
|
17
|
+
the initial configured and mode-transformed routing policy, provider/cache semantics and model prices; classifier
|
|
18
|
+
tier/activity/phase vectors; failure fingerprints and occurrence indexes; pins, modes, resets, context/availability
|
|
19
|
+
facts and observed moves; and correlated request/reply model, effort and token counters. Checkpoint evidence uses
|
|
20
|
+
message indexes and success flags, not commands, tool results or filesystem-path fields, so a later replay can use
|
|
21
|
+
its own switch cursor. Validated routing aliases and model IDs are preserved exactly, including path-shaped IDs;
|
|
22
|
+
classifier endpoints, credentials and the config file's nativePath are not copied.
|
|
23
|
+
|
|
24
|
+
Records use `trace:v1:<session id>`: 30-day retention, 128 KiB per session, 1 MiB total trace budget and a measured
|
|
25
|
+
3 MiB guard on the shared plugin store. Pending turns are bounded too. A cap discards the entire pending turn,
|
|
26
|
+
keeps the earlier contiguous prefix usable and counts excluded turns/events. Late starts, resume/reload, gaps,
|
|
27
|
+
conflicts and lost writes are incomplete. A failed write taints memory and persists that taint on the next success;
|
|
28
|
+
an unavailable store cannot guarantee a durable failure marker. A clean session close is not required.
|
|
29
|
+
Freshness requires the first main request (index 0), no saved loop or trace, and readable normalized request
|
|
30
|
+
history with no assistant messages. Injected startup reminders are user rows and do not make a trace incomplete;
|
|
31
|
+
`classic.SessionStart` is not required. Unreadable or malformed history, reload and late enable are conservative.
|
|
32
|
+
Compaction preserves an existing trace with an explicit reset; `/clear` permits fresh history under a new session ID.
|
|
33
|
+
|
|
34
|
+
Run `/router trace export` to merge retained traces into `router-traces.json` beside the selected profile's
|
|
35
|
+
router config. The command also works after switching back to `table`; ordinary table turns do no trace I/O.
|
|
36
|
+
Exports overwrite their previous JSON after merging; near 3 MiB, a new dated `router-traces-*.json` file receives
|
|
37
|
+
the retained snapshot instead. Keep all export files for offline replay. Known taint is never erased by a longer
|
|
38
|
+
or newer clean prefix. **Delete history** clears stored traces in any mode, but exported files remain user-owned:
|
|
39
|
+
the host filesystem API has no unlink operation.
|
|
40
|
+
|
|
41
|
+
Recent active sessions have a one-hour last-write lease against budget eviction; stale activity cannot defeat
|
|
42
|
+
30-day retention. Store scans, quota checks and deletion are best effort across processes, not atomic reservations.
|
|
43
|
+
There is no live exposure, routing-quality result or savings claim from this recorder yet.
|
|
44
|
+
|
|
45
|
+
## Offline automatic-engine replay
|
|
46
|
+
|
|
47
|
+
Run the tool from a checkout with **all** incremental exports (shell-expanded file arguments, no implicit profile
|
|
48
|
+
access). It reads only those files and writes anonymous JSON to stdout; it never changes a router or provider profile.
|
|
49
|
+
|
|
50
|
+
```sh
|
|
51
|
+
node scripts/replay-traces.mjs /exports/router-traces.json /exports/router-traces-*.json > replay-report.json
|
|
52
|
+
node scripts/replay-traces.mjs --reversal-window 3 test/fixtures/replay-traces.json
|
|
53
|
+
node --test test/replay.test.mjs
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
Omit the dated-file argument when there are no rollover files. The second command uses only synthetic data.
|
|
57
|
+
The implementation follows [design §5.1 and §6 step 5](plans/2026-11-16-auto-routing-design.md#51-replay-traces-and-offline-replay).
|
|
58
|
+
|
|
59
|
+
1. Merge sessions with the recorder's `mergeTraceExport`: preserve lost-write taint, overlap conflicts and capped
|
|
60
|
+
counts. Keep usable capped contiguous prefixes; report their omitted turns/events separately.
|
|
61
|
+
2. Replay the table chain from **only** its initial header, applying config, controls, history preparation/resets,
|
|
62
|
+
choose/observed/continue operations and responses. Compare every recorded decision and request model/effort.
|
|
63
|
+
An unexplained mismatch on any otherwise usable prefix returns `status: "blocked"`, exits 1 and prints **no**
|
|
64
|
+
auto/session totals. It remains in the fidelity denominator. Taint/gaps/incompleteness are explained exclusions.
|
|
65
|
+
3. Only after fidelity passes, validate independent-replay evidence. Missing checkpoints, required boolean facts,
|
|
66
|
+
unconditional continuation modality or activity indexes produce `auto-input-incomplete` and exit 1, without
|
|
67
|
+
silently excluding a usable fidelity session. Earlier exports can lack these fields.
|
|
68
|
+
4. Simulate auto with its own demand/votes, incumbent, cache, routed-demand/provenance and switch/dwell cursors.
|
|
69
|
+
Classifier decisions advance votes; continuations reevaluate constraints and pending moves without adding votes.
|
|
70
|
+
Checkpoints and observed activity are reconstructed from text-free indexes against its own cursor. Only an
|
|
71
|
+
accepted bounded execute phase discounts demand; observed edits can remove it. No activity route table is used.
|
|
72
|
+
|
|
73
|
+
**Mode meaning:** fidelity and both comparisons follow the recorded mode-change history. Output is partitioned by
|
|
74
|
+
the mode active on each request, not three artificial reruns. Table uses the recorded transformed config; auto
|
|
75
|
+
uses the recorded base policy, with mode only as the capability margin. There is no forced-mode option.
|
|
76
|
+
|
|
77
|
+
The report has `fidelity`, `coverage`, `exposureDiagnostic` and anonymous numbered `sessions[].modes`. Each mode
|
|
78
|
+
contains both engines' conditional cost range, priced/unpriced request counts, effort mix, switches and disagreement
|
|
79
|
+
counts grouped by effective demand/activity. Unknown catalog capabilities are counted explicitly. Anthropic seeds four balanced base-route pairs as
|
|
80
|
+
user-derived policy judgments; other pairs and provider presets remain unknown. These seeds and their replay are not
|
|
81
|
+
quality-calibration evidence.
|
|
82
|
+
|
|
83
|
+
Switches are computed on the full session before mode partitioning and attributed to the **destination** mode.
|
|
84
|
+
A reversal closes A→B→A within the explicit `--reversal-window N` main turns (default 3); `reversals` counts closing
|
|
85
|
+
legs, `reversalMoves` counts participating legs, and `oneWay` counts unpaired moves. Rates are per 100 captured turns
|
|
86
|
+
in that mode. Pins do not count as switches and restore the pre-pin route/demand/dwell baseline; every pinned
|
|
87
|
+
continuation still checks eligibility. Manual intervals, history resets and external native interventions break the
|
|
88
|
+
policy-switch chain, including reversal pairing.
|
|
89
|
+
|
|
90
|
+
### Coverage and estimate limits
|
|
91
|
+
|
|
92
|
+
- `coverage.exportCohort` ratios describe **only captured/exported sessions**, a lower-bound cohort. Exports do not
|
|
93
|
+
count evicted or untraced sessions; `coverage.allSessions` denominators and its 80% gate stay unavailable.
|
|
94
|
+
Sufficient future metadata would include capture-eligible session starts (including untraced/evicted sessions),
|
|
95
|
+
total main turns and per-mode eligible routed turns over the same capture window. This tool invents none of it.
|
|
96
|
+
- Exposure thresholds (30 complete sessions, 1,000 main turns, 200 per evaluated mode) are diagnostics only.
|
|
97
|
+
Captured main turns are separate from eligible unpinned classifier-decision turns; manual-only sessions/turns
|
|
98
|
+
do not count as engine exposure. Freshness is unverified and the exposure gate remains `not-established`.
|
|
99
|
+
- Cost freezes observed total prompt/output lengths, timestamps, failures and tool calls. Auto simulates its own
|
|
100
|
+
full-prefix cache scenario, never the table's cache-hit tokens. Explicit cache warmth uses the existing conservative
|
|
101
|
+
five-minute helper; automatic caches retain hit/miss bounds, with no invented TTL guarantee. Effort-specific,
|
|
102
|
+
disabled, long-context and off-peak pricing use the existing pure cost helpers. These are **conditional estimates**,
|
|
103
|
+
not counterfactual savings, actual bills or paired quality results.
|
|
104
|
+
- Missing usage/prices and substituted replies make the affected total range `null`, not free. An explicitly named
|
|
105
|
+
`knownPricedRange` is only the priced subtotal. No transcript-length/quality change from a hypothetical request
|
|
106
|
+
is modeled.
|
|
107
|
+
- Fidelity distinguishes table echoes from genuine third-model/native fallbacks. Auto freezes actual external
|
|
108
|
+
interventions, response substitutions and routed null-stop suspension, rather than misreading a table echo as
|
|
109
|
+
its own fallback. It cannot predict a provider's fallback for a different hypothetical auto request.
|
|
110
|
+
- Default output includes no session IDs, prompts, tool text, model identifiers or paths. Input errors name the
|
|
111
|
+
numbered file/structural field, not its path. The CLI cannot discover omitted exports or verify their provenance.
|
|
112
|
+
|
|
113
|
+
## Paired benchmark proposal
|
|
114
|
+
|
|
115
|
+
The [automatic-routing design §5.3](plans/2026-11-16-auto-routing-design.md#53-paired-benchmark-smoke-gate-for-quality)
|
|
116
|
+
requires paired `table`/`auto` runs, three modes and two repeats, with per-task checks and discordances. The
|
|
117
|
+
24 synthetic expected-output cases in `test/fixtures/benchmark-routing.json` are harness fixtures, not calibrated
|
|
118
|
+
engineering tasks or a quality result. Their activity labels are fixture metadata, not reference classifications.
|
|
119
|
+
Eight are marked held out in the fixture split. The separate engineering proposal below needs fixtures and review
|
|
120
|
+
before any evaluation; shadow-disagreement sources are pending.
|
|
121
|
+
|
|
122
|
+
```sh
|
|
123
|
+
node scripts/benchmark-routing.mjs --dry-run
|
|
124
|
+
node scripts/benchmark-routing.mjs --dry-run --smoke
|
|
125
|
+
node --test test/benchmark-routing.test.mjs
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
The dry run prints a SHA-256, run counts, serialized token/rate assumptions and a configured-price scenario.
|
|
129
|
+
Dry-run planning uses the built-in catalog maximum prices through `ratesAt`; a live smoke preflight reads the selected
|
|
130
|
+
work profile's catalog without changing it. Cold-write multipliers apply only to explicit caches. The 10,000-input
|
|
131
|
+
assumption is not a limit; live smoke sets `CLAUDE_CODE_MAX_OUTPUT_TOKENS=512`, one agentic turn and a $0.40
|
|
132
|
+
CLI budget per call. These are not a proxy-price quotation or a bill. A run estimates before launching and reserves
|
|
133
|
+
$1.20 for three calls. An exclusive reservation file under `.pi/` refuses automatic reruns; further spending needs
|
|
134
|
+
review, not a hidden retry. The 288 planned task runs are for the toy mechanics fixtures, not an engineering benchmark cost estimate.
|
|
135
|
+
An engineering task run may use multiple inference calls, tools and retries; it needs new token, request, effort
|
|
136
|
+
and provider budgets plus lead approval before any run. No full matrix is authorized or run.
|
|
137
|
+
|
|
138
|
+
`--live --smoke` is limited to the three development tasks, `table`/`balanced`, one repeat, using the existing
|
|
139
|
+
installed release through `zsh -ic` and `claude-mp-work -p`. It loads no checkout plugin. The selected profile must
|
|
140
|
+
still be `table`; `--setting-sources user`, strict empty MCP and no tools limit the surface. Init metadata must
|
|
141
|
+
identify exactly one router at the installed user path. Profile contents are hashed before/after each call. The
|
|
142
|
+
wrapper's normal MCP housekeeping is permitted; the original active-environment bytes/existence are restored and
|
|
143
|
+
verified after success, failure, timeout or handled interrupt. SIGKILL, power loss and independently detached
|
|
144
|
+
processes are not covered. Process-group cleanup is best effort; Windows cannot kill a descendant tree via Node
|
|
145
|
+
alone. The smoke establishes harness mechanics only, not evaluation of this checkout or either engine.
|
|
146
|
+
|
|
147
|
+
Offline system-boundary tests exercise independent directories, output checks, deadlines, output limits, budgets,
|
|
148
|
+
unknown costs and paired discordance reporting. The harness checks its own and the manifest's hashes around child
|
|
149
|
+
execution; this detects persistent changes, not transient same-user writes or an OS sandbox. The checks run in the
|
|
150
|
+
parent outside the child's working directory. Unknown/zero total or per-model CLI prices stop further calls rather
|
|
151
|
+
than implying free usage. Reports sum every supplied attempt, including failures, and leave missing/unpriced
|
|
152
|
+
totals unknown. Duplicate attempts within a paired cell are reported as ambiguous, never silently replaced by a
|
|
153
|
+
passing rerun.
|
|
154
|
+
CLI-reported USD can use rates different from the selected catalog and may omit proxy retries, classifier spend or
|
|
155
|
+
upstream failed calls: actual billing remains unknown.
|
|
156
|
+
No comparison of quality, savings or measured routing performance follows from these checks.
|
|
157
|
+
|
|
158
|
+
The checked-in [installed-release smoke record](../experiments/mod-router/results/benchmark-harness-smoke.json)
|
|
159
|
+
contains three passing external checks, 19,298 ms summed subprocess wall time and $0.0098816 CLI-reported price
|
|
160
|
+
estimates. It records the manifest/config hashes and successful restoration/provenance checks without prompts,
|
|
161
|
+
profile paths or credentials. CLI usage metadata names DeepSeek Flash for two calls and GLM Flash for one; backend
|
|
162
|
+
identity is not independently verified.
|
|
163
|
+
These are harness-validation observations on installed Router 1.23.1, not a paired result for this checkout.
|
|
164
|
+
|
|
165
|
+
### Engineering task set: draft only
|
|
166
|
+
|
|
167
|
+
The following 24 synthetic engineering tasks are a proposed **separate** evaluation set, not implemented fixtures.
|
|
168
|
+
Difficulty and activity are author hypotheses, not classifier references. Each row needs a self-contained repository
|
|
169
|
+
fixture and an independently held-out automatic oracle outside model control. Freeze the fixture/manifest hash and
|
|
170
|
+
split before capability or margin tuning; do not run or inspect held-out outcomes during tuning. All rows are authored
|
|
171
|
+
examples: shadow-disagreement sourcing is **pending** and must be added or documented before the set is finalized.
|
|
172
|
+
|
|
173
|
+
| ID | Split | Activity / demand hypothesis | Task | Independent automatic oracle |
|
|
174
|
+
| --- | --- | --- | --- | --- |
|
|
175
|
+
| E01 | dev | code / low | Add integer clamping to a small library | Boundary and type tests |
|
|
176
|
+
| E02 | dev | debug / medium | Repair a parser that mishandles escaped delimiters | Existing tests plus hidden escaped and empty cases |
|
|
177
|
+
| E03 | dev | code / medium | Add pagination to a small service across handler and storage modules | Fixture API and deterministic pagination tests |
|
|
178
|
+
| E04 | heldout | code / medium | Implement stable deduplication across a CLI and helper | Output and duplicate-order tests |
|
|
179
|
+
| E05 | dev | security / high | Reject traversal and symlink escapes in an archive extractor | Traversal, symlink and valid-path tests |
|
|
180
|
+
| E06 | dev | debug / high | Fix a cross-module stale cache invalidation defect | Concurrent update and expiry regression tests |
|
|
181
|
+
| E07 | heldout | debug / medium | Fix time boundary logic for expiry at exact deadline | Fake-clock boundary tests |
|
|
182
|
+
| E08 | dev | ops / medium | Repair CI script so test and lint failures reach the exit status | Controlled subprocess exit-code matrix |
|
|
183
|
+
| E09 | dev | review / medium | Identify seeded unsafe defaults in a short configuration | Exact structured findings against injected defects |
|
|
184
|
+
| E10 | heldout | debug / high | Repair nondeterministic retry handling after partial network failures | Deterministic mocked-service request and retry tests |
|
|
185
|
+
| E11 | dev | code / high | Add idempotent transaction handling across API and persistence | Duplicate/concurrent request and rollback tests |
|
|
186
|
+
| E12 | dev | debug / medium | Diagnose and fix an off-by-one token budget after a tool continuation | Transcript fixture and boundary tests |
|
|
187
|
+
| E13 | heldout | code / low | Add case-insensitive header lookup preserving original values | Mixed-case and missing-header tests |
|
|
188
|
+
| E14 | dev | code / medium | Migrate a versioned configuration while preserving unknown fields | Old/new config round-trip and rejection tests |
|
|
189
|
+
| E15 | dev | debug / high | Fix matching failure signatures after an edit and repeated tool failure | Scripted transcript with two failures and edit tests |
|
|
190
|
+
| E16 | heldout | security / high | Correct authorization checks across two endpoints sharing a store | Cross-user read/write and allowed-access tests |
|
|
191
|
+
| E17 | dev | refactor / medium | Replace duplicate parsing in two modules without changing behavior | Characterization and randomized input tests |
|
|
192
|
+
| E18 | dev | ops / low | Correct shell quoting for filenames containing spaces | Subprocess tests on spaces and metacharacters |
|
|
193
|
+
| E19 | heldout | code / high | Implement bounded parallel workers with cancellation and cleanup | Deterministic concurrency and child-cleanup tests |
|
|
194
|
+
| E20 | dev | debug / medium | Fix a multi-file regression after two failed test attempts and an edit | Scripted tool history plus external final tests |
|
|
195
|
+
| E21 | dev | docs / low | Correct a documented CLI example for a real edge-case flag | Execute example against fixture CLI |
|
|
196
|
+
| E22 | heldout | ops / medium | Repair a build pipeline that masks errors in a chained command | Failure-injection exit-code tests |
|
|
197
|
+
| E23 | dev | review / high | Find seeded race and missing transaction rollback | Machine-readable findings with location and defect checks |
|
|
198
|
+
| E24 | heldout | debug / medium | Fix Unicode normalization inconsistency across reader and index | Composed/decomposed matching and preservation tests |
|
|
199
|
+
|
|
200
|
+
These are not yet the §5.3 benchmark: the tasks, initial repositories, test oracles, tool-history presentation,
|
|
201
|
+
permissions and expected output need independent implementation and review. No engineering benchmark or held-out
|
|
202
|
+
outcome has been collected. Difficulty does not establish a capability value.
|
|
203
|
+
|
|
204
|
+
### Full paired-run isolation: proposal only
|
|
205
|
+
|
|
206
|
+
Auto routing is available by opt-in in 1.24.0. The full paired benchmark adapter remains disabled pending checkout/profile
|
|
207
|
+
isolation and approval; a full live matrix always refuses. A native pass-through or table
|
|
208
|
+
run is never relabeled `auto`. The current hook reads `routingEngine` only from the selected profile file; there is
|
|
209
|
+
**no** run-scoped environment/settings override. Project `.claude/router.json` can set mode only.
|
|
210
|
+
|
|
211
|
+
Before the real paired evaluation, choose and implement one code-provenance path:
|
|
212
|
+
|
|
213
|
+
- Add an explicit supplied-router option to `ce`'s model-proxy path, matching its normal launch behavior. This can
|
|
214
|
+
preserve existing implicit authentication but needs wrapper maintenance and must disable the marketplace copy.
|
|
215
|
+
- Launch `claude-proxy` directly with per-run settings that disable the marketplace router and a checkout
|
|
216
|
+
`--plugin-dir`. This gives precise code provenance but must reuse an approved authentication interface, never
|
|
217
|
+
copy credentials or silently bypass account policy.
|
|
218
|
+
- Run after a release that includes the evaluated implementation. This avoids two plugin copies but does not
|
|
219
|
+
itself isolate the engine selection or establish checkout provenance.
|
|
220
|
+
|
|
221
|
+
Every path **also** needs a separately approved run-scoped engine override or safely isolated profile configuration
|
|
222
|
+
that keeps existing implicit authentication without copying secrets. Only then can each child select its own
|
|
223
|
+
`table` or `auto` without editing the user's real router file. No wrapper modification, new engine override,
|
|
224
|
+
authentication mechanism, live auto wiring or cutover is implemented here.
|
|
225
|
+
|
|
7
226
|
## Provider profiles
|
|
8
227
|
|
|
9
228
|
[Provider defaults](providers.md#built-in-defaults) are starting policies, not measured per-activity quality or savings results.
|
|
@@ -178,7 +397,7 @@ Method, per main reply that routing chose:
|
|
|
178
397
|
|
|
179
398
|
Checks:
|
|
180
399
|
|
|
181
|
-
- Unit tests (`test/savings.test.mjs`) cover a session that saves, one that costs more with Haiku as your model, equal routes at exactly 0, cache expiry at 5m and 1h, a compaction, a change of your model or effort, the 100K surcharge, an unpriced model, and a corrupted store. `test/hook-
|
|
400
|
+
- Unit tests (`test/savings.test.mjs`) cover a session that saves, one that costs more with Haiku as your model, equal routes at exactly 0, cache expiry at 5m and 1h, a compaction, a change of your model or effort, the 100K surcharge, an unpriced model, and a corrupted store. `test/hook-usage.test.mjs` covers two sessions finishing turns at the same moment, a `/model` change mid-session that keeps each reply against its own model, and a running turn's replies counted once before and after the turn is recorded. `test/usage.test.mjs` covers calendar days across month ends and daylight saving changes, the periods, the record cap and expiry, the trend buckets and what is worth a look; `test/usage-view.test.mjs` covers the Usage tab.
|
|
182
401
|
- A one-off replay, not checked in, of this machine's main-conversation transcripts from 2026-10-01 to 2026-10-10 (nine sessions with 10 or more replies) gave $0.00 for every session that ran only on the model it was compared with: three on Opus 5.5, one on Haiku 5.5. Every main-conversation cache write in them was a one-hour write, which is what the lifetime rule gives on this Max plan.
|
|
183
402
|
- A live Claude Code 2.1.296 session on 2026-10-10, Opus 5.5 at `high` with turns pinned to Haiku 5.5, showed **Routed replies $0.48** beside **Cost $0.480 reported by Claude** for the same 16 replies; its last Haiku reply had a 114K-token prompt, which agrees only at the 5x rate. A Haiku 5.5 session with a pinned Opus turn showed $0.45 routed against $0.446 reported.
|
|
184
403
|
|
|
@@ -208,7 +427,7 @@ Method:
|
|
|
208
427
|
Checks:
|
|
209
428
|
|
|
210
429
|
- `test/subagents.test.mjs` covers each pass reason, the pin rule, the classify mapping, core's model estimate, the router's move, the subagent lifetime, Haiku's long-context price, and the store's days, merge, reset, bad values and records from before 1.14.0. `test/hook-subagents.test.mjs` covers `on` (the routed model), a fork, an explicit model, a teammate, a workflow agent and a pinned project agent (each passes), a classifier failure (core's model), parallel spawns, a spawn-path pause that leaves main routing alone, respawns, session end, Delete history, and the Usage and Now tabs. `test/usage-view.test.mjs` covers the block by period, an open row, a narrow pane and undated records; `test/hook-band.test.mjs` the band in a subagent's transcript.
|
|
211
|
-
- `experiments/mod-router/results/agent-spawn-probe.json` records what Claude Code 2.1.296 does with a spawn's model, frontmatter and workflow agents.
|
|
430
|
+
- `experiments/mod-router/results/agent-spawn-probe.json` records what Claude Code 2.1.296 does with a spawn's model, frontmatter and workflow agents. On 2.1.321 a live probe showed that a model rewrite applies to workflow agents, and that a resumed subagent raises `agent.spawn` with `e.resume` and keeps its first model.
|
|
212
431
|
- `experiments/mod-router/results/subagent-routing-live.json` is a live `on` run on 2026-10-10 with Sonnet as the parent. Explore was sent Haiku and ran on it. general-purpose was classified `standard`. A pinned project agent and an explicit `opus` call passed through. The store counted each one.
|
|
213
432
|
|
|
214
433
|
When to turn it `on`:
|
|
@@ -222,7 +441,7 @@ Keep it if requests per agent and respawns per type stay no higher than before y
|
|
|
222
441
|
Limits:
|
|
223
442
|
|
|
224
443
|
- Answer quality is not measured.
|
|
225
|
-
- Workflow agents
|
|
444
|
+
- Workflow agents are classified by default (`subagents.types['workflow-subagent']`). In the maintainer's sessions they were 22 of 24 Opus subagents (`workflow-subagent-models.json`), so this reaches most of the inherited Opus spend. The routed effect is not measured yet, and answer quality is not either.
|
|
226
445
|
- `on` estimates core's model. It does not apply the family-alias rule or the `availableModels` substitution.
|
|
227
446
|
- Respawns are a proxy. A parent may spawn the same type twice on purpose.
|
|
228
447
|
- Records follow the savings records' storage, with the same gaps.
|
package/docs/native-router.md
CHANGED
|
@@ -1,11 +1,13 @@
|
|
|
1
1
|
# Native router: behavior and evidence
|
|
2
2
|
|
|
3
|
-
This guide describes the Mod shipped in the plugin. Claude Code 2.1.
|
|
3
|
+
This guide describes the Mod shipped in the plugin. Claude Code 2.1.321 is the minimum tested version. The router changes `model` and `effort` on main-conversation steps. Claude Code handles the Anthropic request, credentials, stream, tool execution, and cost ledger. Subagent steps keep their model and effort; the router chooses an agent's model once, at spawn, unless its definition pins one ([configuration](configuration.md#subagent-routing)).
|
|
4
4
|
|
|
5
5
|
## Turn behavior
|
|
6
6
|
|
|
7
7
|
At the first main step of a logical turn, the Mod reads recent message text and asks the active classifier (Jev, Clef, Clef Flash, Clef Omni, OpenAI, or Ollama) for a tier, and for the turn's activity unless `activityRouting` is `off`. It applies local policy and saves the route. Tool continuations reuse that route, except that a turn whose tool calls edit code moves up to the `code` route, and every `policy.reviewEveryRequests` requests a background classifier review may move it either way ([Changes during a task](user-guide.md#changes-during-a-task)). Before each continuation, context fit is checked again. A large tool result can move the step to a model with a larger window.
|
|
8
8
|
|
|
9
|
+
Claude Code sends a model change on every step, but an effort change on a tool-result step only when `CLAUDE_CODE_MODEL_CAPABILITIES=per_turn_effort` is set in its environment (the router does not set it). Without it a turn keeps the effort of its first request, so the band and the decision record can name an effort that did not run; see [what we do not know yet](activity-routing.md#what-we-do-not-know-yet).
|
|
10
|
+
|
|
9
11
|
The classifier receives the current prompt and at most six preceding user or assistant text messages. Each item is limited to 1,200 characters. It does not receive system messages, tool inputs, or tool results. Claude Code's Anthropic credentials never enter the classifier request. Jev, Clef, Clef Flash, Clef Omni, and OpenAI receive the text on their services; Ollama receives it on this machine. [Configuration](configuration.md#classifiers) lists each host.
|
|
10
12
|
|
|
11
13
|
The default tiers are defined in [Configuration](configuration.md#built-in-defaults). Policy uses vote hysteresis, repeated tool errors, context fit, model availability, and switching-cost estimates. The [architecture](architecture.md#request-flow) explains the event boundary and failure handling.
|
|
@@ -23,15 +25,20 @@ The activity is what the turn produces: `code`, `debug`, `explore`, `plan`, `rev
|
|
|
23
25
|
| `/router activities <off\|on>` | Set the activity routing mode. Written to `router.json` at once; **Undo** in the pane reverts it. |
|
|
24
26
|
| `/router mode <cost\|balanced\|quality\|default>` | Set or clear this session's [routing mode](routing-modes.md), from the next turn. Kept for a resume in `$.store` `routing-mode:<session id>`. Never aborts a turn. |
|
|
25
27
|
| `/router decisions [n]` | Print this session's last `n` routing decisions (default 20, at most 60): origin, classifier answer, task, tier, route, reason with the mass and bar it compared, and each turn's end. No text. |
|
|
28
|
+
| `/router line <off\|moves\|all>` | Set when a turn leaves a route line (default `moves`: only when its route moved). Kept in `$.store` as `route:line`, across sessions. |
|
|
26
29
|
| `/model <name>` | Select a model and turn routing off. `/router auto` turns it back on. |
|
|
27
30
|
|
|
28
|
-
The
|
|
31
|
+
The prompt's typeahead (`prompt.autocomplete`) offers these words after `/router ` and their values after `pin `, `mode `, `activities ` and `line `. It starts with the first typed character, because the engine passes a token that is never empty. The command, the typeahead and the argument hint all read one table, `lib/subcommands.mjs`, and a test runs the command with every word and value in it.
|
|
32
|
+
|
|
33
|
+
The route line is one `$.ui.place` row per routed turn, drawn by a `ui.render` hook on `Placed`. At `turn.complete` of the main loop, skipping an aborted turn, a subagent's turn and a turn with routing off, the router writes the route that ran (model, effort, tier, reason, activity and the policy's estimate, never text) to `$.state` under `router.lines` with the row's id (`rl-` and the turn id, made safe for `$.ui.place`), then places the row. The hook draws only rows whose `plugin` is the router's and leaves the rest to `next(e)`. A surface that cannot draw the row at that moment (`isPlaced: false`) is logged to the debug log and the turn goes on. Claude Code does not bring rows back on `--resume` and ends them at a compaction, so the records are not read again.
|
|
34
|
+
|
|
35
|
+
In a terminal, the pane's **Set up** and **Credentials → Edit** buttons open Claude Code's secure plugin configuration. The desktop app, VS Code and mobile cannot open that dialog, so there both buttons show a card with the command to run in a terminal (the band's **Set up** does the same). Claude Code refuses `$.command.run` from inside a `command.run` hook, so the Mod starts that dialog from a timer after the press returns.
|
|
29
36
|
|
|
30
37
|
Mode belongs to a Claude Code session. Clear starts a new session with routing on. Resume restores the saved mode for that session. A new or forked session has a new ID: it starts with routing on for a model that one of the tiers routes to and with routing off for any other model, as a fresh launch does. With activity routing `on`, a model that only an activity override uses also starts the session with routing on. A pin applies to one logical turn, then the prior route resumes. A history reset clears votes and cache evidence; the current turn keeps its route, pin and any native fallback.
|
|
31
38
|
|
|
32
39
|
## Router pane
|
|
33
40
|
|
|
34
|
-
|
|
41
|
+
In the terminal, the status band is one line above the prompt: a tier meter, the model, the activity, a short reason, classifier support and the switch cost, and context and cache use. Segments drop by priority when the band is narrow: classifier support and the switch cost first, then context and cache, then the activity, then the reason. Hovering shows, on the same row, a pin per tier, **Turn off**, and a two-row view whose second row shows the plan windows on a Claude plan and the session's cost against no routing; while routing is off, **Turn on** sits on the band itself, so the band carries one routing control at a time. A fresh session that starts with routing off because of its model shows `not a routing tier` on the band until the mode changes. The band yields to surveys. While subagents that the router routed are running, a row under the band lists them (`agents 2 ▸ ■ Haiku 5.5 Explore · retry usages ●`, at most three and then `+N`, never a teammate); pressing one opens that agent's transcript through `$.ui.view`, which Claude Code allows only as an answer to the person's own input, and a refusal is shown as a toast. The row is read from `$.agent.list()` joined with the router's spawn records, and redraws when the set of running agents changes (`subagentIds` in the view). In an agent's transcript the band starts with `◂ main`, which returns to the main conversation. In a subagent's transcript it names the agent's model and who picked it, such as `agent ■ Haiku 5.5 ↓ from Opus 5.5 (Claude Code’s pick)`; without a record of the agent it reads `agent · model chosen at spawn`, or `agent · keeps its own model (subagent routing off)`. A model change between turns raises a toast. A pin, or a step that changes only the effort, such as `medium` to `high` on the defaults, raises none. While the classifier runs, the turn's spinner says `Choosing model`. In Manual mode or when routing is unavailable, the prompt footer carries `routing off` or `routing unavailable`. The **Router** button opens the pane. In the terminal the pane has five numbered tabs, Now, Routes, Policy, Classifier, and Usage, under one `ROUTER` line with the routing on/off pair and the classifier state. The desktop app, VS Code and mobile draw a fixed one-row band, with nothing revealed on hover, and tabs without digits; see [the user guide](user-guide.md#in-the-desktop-app-vs-code-and-mobile). The [user guide](user-guide.md#open-the-router-pane) describes each tab.
|
|
35
42
|
|
|
36
43
|
| Reading | Meaning and limits |
|
|
37
44
|
| ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
@@ -84,6 +91,7 @@ The pane refuses to write through a symlink. A failed validation names the setti
|
|
|
84
91
|
- Three launched classifier failures open a 60-second pause for that classifier. One in-flight request is allowed per classifier in each active Mod instance. Pending host HTTP work blocks another request until it settles.
|
|
85
92
|
- A `router.json` with v0.8 gateway keys or the 1.1 `jev` section marks the router unavailable until the [migration command](configuration.md#convert-an-older-routerjson) converts it.
|
|
86
93
|
- Unsupported Claude Code versions and leftover v0.8 gateway settings (the `jev-router` model or the `127.0.0.1:43170` base URL) mark the router unavailable. `/router` lists the settings to remove. A local gateway does not run as part of this Mod.
|
|
94
|
+
- A session that starts with the router unavailable shows one toast with the reason (`$.ui.toast`, not `$.ui.notify`, which is a native terminal notification). It repeats at most once a day per cause: the time is kept in `$.store` as `notice:unavailable:<version|gateway|config>`. A `claude -p` run shows none.
|
|
87
95
|
- While the router is unavailable, the pane header reads `ROUTER unavailable Claude’s model is kept` with the reason, and offers no on/off pair or pins. `/router` without a surface starts with `Routing unavailable: <reason>. Claude’s model is kept.`, and `/router auto`, `off`, `pin`, and `activities` reply the same and change nothing.
|
|
88
96
|
- A context-window error makes that model ineligible until the next history reset. The Mod does not retry an Anthropic request after a stream begins.
|
|
89
97
|
- Unknown context retains the current model. There is no characters-to-tokens fallback.
|
package/docs/providers.md
CHANGED
|
@@ -9,6 +9,34 @@ Custom profiles need their own model settings. They do not inherit Claude models
|
|
|
9
9
|
|
|
10
10
|
This guide assumes Claude Code already works with your provider. Endpoint and credential setup are outside its scope.
|
|
11
11
|
|
|
12
|
+
## Capability evidence
|
|
13
|
+
|
|
14
|
+
The Anthropic preset seeds `claude-haiku-5-5@medium` at 0, `claude-haiku-5-5@high` at 1,
|
|
15
|
+
`claude-opus-5-5@medium` at 2, and `claude-opus-5-5@xhigh` at 3. These are user-derived policy judgments
|
|
16
|
+
from the built-in balanced base routes, not model-card benchmarks or quality guarantees. All other pairs, including
|
|
17
|
+
Haiku at low and Opus at high, remain unknown, not zero. User-router PR #79 seed values are profile overrides layered
|
|
18
|
+
over these built-ins. Other built-in provider presets have no seeds. No rating is inferred from price, ordinal strength, benchmark rank,
|
|
19
|
+
or an effort name.
|
|
20
|
+
This metadata does not affect the current table router. See [configuration](configuration.md#model-capability-declarations)
|
|
21
|
+
for declarations and overrides.
|
|
22
|
+
|
|
23
|
+
The model-card IDs below identify the reviewed records, not verified served backends. The sources do not give
|
|
24
|
+
numeric values on this router's demand scale.
|
|
25
|
+
|
|
26
|
+
| Preset model / model-card ID | Evidence and gap |
|
|
27
|
+
| --- | --- |
|
|
28
|
+
| Fable 5.2 / `anthropic-claude-fable-5-2` | [Prompting guide](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-fable-5-2): effort guidance, not demand-tier calibration. |
|
|
29
|
+
| Opus 5.5 / `anthropic-claude-opus-5-5` | [Prompting guide](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-5-5): effort recommendations, not tier adequacy. |
|
|
30
|
+
| Sonnet 5.5 / `anthropic-claude-sonnet-5-5` | [Prompting guide](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-sonnet-5-5): well-specified agentic coding at medium does not establish all medium-tier work. |
|
|
31
|
+
| Haiku 5.5 / `anthropic-claude-haiku-5-5` | [Prompting guide](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-haiku-5-5): simple work at low does not calibrate reliability to micro demand. |
|
|
32
|
+
| DeepSeek Flash / `deepseek-deepseek-v4.1-flash-1017` | [Provider evaluation](evaluation.md#provider-profiles): earlier benchmark results do not establish the hosted snapshot's per-effort adequacy. |
|
|
33
|
+
| GLM-5.3 / `zai-glm-5.3`; GLM-5.3-Flash / `zai-glm-5.3-flash` | [Provider evaluation](evaluation.md#provider-profiles): max-effort results do not establish low/high performance or tier adequacy. |
|
|
34
|
+
| MiniMax M3.1 Flash / `minimax-minimax-m3.1-flash`; Kimi K3.1 Pro / `moonshot-kimi-k3.1-pro` | [Provider experiments](provider-experiments.md): small synthetic checks do not calibrate repository tasks per effort. |
|
|
35
|
+
| MiMo V2.6 Flash and Pro / no model-card record | [Provider experiments](provider-experiments.md): no granular effort setting; small synthetic checks do not establish demand-tier adequacy. |
|
|
36
|
+
|
|
37
|
+
The auto engine selects the cheapest eligible unknown when no eligible declaration is known. Neither the seeds nor
|
|
38
|
+
unknown-only selections establish quality or escalation-headroom calibration; users can configure their own maps.
|
|
39
|
+
|
|
12
40
|
## Select a provider
|
|
13
41
|
|
|
14
42
|
For DeepSeek, create `deepseek-router.json` in your Claude Code profile directory with this content:
|