@alexeiled/claude-router 1.6.0 → 1.6.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "router",
3
3
  "displayName": "Router",
4
- "version": "1.6.0",
4
+ "version": "1.6.1",
5
5
  "description": "Model and effort routing for the main conversation by task difficulty and activity, advised by a prompt classifier (Jev, Clef, Clef Flash, OpenAI or local Ollama). Preserves subagent model choices.",
6
6
  "author": {
7
7
  "name": "Alexei Ledenev",
package/CHANGELOG.md CHANGED
@@ -1,5 +1,20 @@
1
1
  # Changelog
2
2
 
3
+ ## 1.6.1 (2026-10-10)
4
+
5
+ ### Added
6
+
7
+ - The **Shadow** line on the Usage tab now estimates what the turns `on` would route differently would cost on their next request, as a range at the configured list prices in `router.json`, such as `est. −$0.300 … −$0.120 at list prices`. It is an estimate, not a measured saving.
8
+ - **ACROSS SESSIONS** on the Usage tab shows the activity counts kept since the last reset: labelled turns, tool agreement overall and on `code`, `ops` and `explore` turns, the two most frequent mismatches, lateral switches taken and refused, and the shadow readout with its estimate. `/router` without a UI surface prints the agreement across sessions. **Reset activity stats** clears it too.
9
+
10
+ ### Changed
11
+
12
+ - Tool agreement counts a turn that only ran read-only commands, such as `git status`, as agreeing with `ops`. It used to count as a mismatch, which understated agreement on `ops`. Counts already stored are not recomputed.
13
+
14
+ ### Upgrade notes
15
+
16
+ - Downgrading to 1.6.0 resets the activity counts kept across sessions: 1.6.0 does not read the shadow estimate 1.6.1 stores.
17
+
3
18
  ## 1.6.0 (2026-10-10)
4
19
 
5
20
  The classifier now also names the turn's **activity**: what the turn produces (`code`, `debug`, `explore`, `plan`, `review`, `ops`, `docs`, or `uncertain`). The route can then depend on tier and activity, so a `low` coding turn can run on Sonnet 5.5 while a `low` git turn stays on Haiku 5.5. Routing does not change until you opt in.
@@ -109,7 +109,7 @@ When a turn completes, the Mod labels it from the tool calls the assistant made
109
109
  Two stores hold counts:
110
110
 
111
111
  - **Session**: `activityStats` in the view, reset when the session ends. It has turns, requests and tokens per activity, switches by tier and by activity, tool agreement, and the shadow readout. The label counted per activity is the one the turn used: the applied activity in `on`, the answer that cleared `activityMass` in `shadow`. Tool agreement always compares the answer that cleared `activityMass`, so it measures the classifier in both modes. The view also keeps an `activities` array aligned with the last 30 replies.
112
- - **Across sessions**: one `$.store` key, `activity:stats:v1`. It holds a confusion matrix of the classifier's raw answer (including `uncertain` and `none`) against the observed bucket, run lengths per activity in five buckets, lateral switches taken and refused, and shadow turns and differences. The key set is fixed, so the size is bounded. A stored value with any other shape is read as empty. The open run is written when the session ends. **Reset activity stats** clears both stores.
112
+ - **Across sessions**: one `$.store` key, `activity:stats:v1`. It holds a confusion matrix of the classifier's raw answer (including `uncertain` and `none`) against the observed bucket, run lengths per activity in five buckets, lateral switches taken and refused, and shadow turns, differences, and the summed next-request estimate of the differing turns with the count it covers. The key set is fixed, so the size is bounded. A 1.6.0 value without the estimate loads with zeros; a stored value with any other shape is read as empty. The view keeps the last value read or written as `activityStore`: read at session start and when the pane opens, replaced after each turn's write and after a reset. The open run is written when the session ends. **Reset activity stats** clears both stores.
113
113
 
114
114
  Nothing is recorded with `activityRouting: off`. Stats run inside a guard: a failed transcript read or store access loses that turn's counts and never fails the turn. A turn from a session that has since changed is dropped.
115
115
 
@@ -54,7 +54,9 @@ The script writes aggregate counts, reason histograms, route transitions, and sh
54
54
 
55
55
  ## Activity routing
56
56
 
57
- Activity routing has no measured result. The replay above predates it and covers tiers only. The pane counts turns, requests, switches, and the turns `on` would route differently, and it shows no dollar figure for them. Two checks look at the classifier's activity label.
57
+ Activity routing has no measured result. The replay above predates it and covers tiers only. The pane counts turns, requests, switches, and the turns `on` would route differently, with a configured list-price estimate of those turns' next requests, not a measured saving. Two checks look at the classifier's activity label.
58
+
59
+ The figures for turning activity routing on by default (plan §11: tool agreement of at least 80% on `code` against `ops` and `explore`, a negative would-route estimate, the number of activity switches) are read from the Usage tab's **ACROSS SESSIONS** block, which shows the counts kept since the last reset: **Code/ops/explore** for the tool agreement on turns the classifier answered `code`, `ops` or `explore`, **Agreement** and **Mismatch** for all of them, **Shadow** for the would-route count and estimate, **Lateral** for activity switches. `/router` without a UI surface prints both agreements. The underlying counts are the `activity:stats:v1` store described in [Architecture](architecture.md).
58
60
 
59
61
  ### Tool agreement
60
62
 
@@ -79,10 +81,10 @@ The classifier's activity agrees with the turn when the bucket is one it allows:
79
81
  | `explore` | `read`, `talk` |
80
82
  | `plan` | `talk`, `read`, `docs` |
81
83
  | `review` | `read`, `talk` |
82
- | `ops` | `ops` |
84
+ | `ops` | `ops`, `read` |
83
85
  | `docs` | `docs` |
84
86
 
85
- Agreement is a weak check. It tells `ops` from `code` from `explore` well. It cannot tell `plan` from `review`, and it says nothing about whether the route was good enough. It does catch the costly mistake, a cheap route on a coding turn. The Usage tab shows the session's count as `Agreement`, over the turns where the classifier gave an activity at or above `activityMass`, in `on` too when the route kept another activity. The counts kept across sessions hold the classifier's raw answer, `uncertain` and no answer included, against the bucket. They are not shown in the pane.
87
+ Agreement is a weak check. It tells `code` from `ops` and `explore` well, but not `ops` from `explore` when the commands only read. It cannot tell `plan` from `review`, and it says nothing about whether the route was good enough. It does catch the costly mistake, a cheap route on a coding turn. The Usage tab shows the session's count as `Agreement`, over the turns where the classifier gave an activity at or above `activityMass`, in `on` too when the route kept another activity. The counts kept across sessions hold the classifier's raw answer, `uncertain` and no answer included, against the bucket. The pane's **ACROSS SESSIONS** block counts agreement over the answers that name an activity; `uncertain` and no answer appear only in **Labelled**.
86
88
 
87
89
  ### Activity probe set
88
90
 
@@ -94,14 +96,16 @@ node scripts/probe-activity.mjs [classifier]
94
96
 
95
97
  The script sends each prompt to one classifier as the router would ask it, with activity routing forced to `shadow`. It reads credentials as `probe-classifier.mjs` does and needs a classifier you can reach. The run is billed on a paid classifier, and it is run by hand, not in CI. It writes accuracy, an expected-by-answered confusion matrix, p50 and p95 latency against the classifier's deadline, and error counts to `experiments/mod-router/results/activity-probe-<classifier>.json`. For Ollama the latency covers both requests. It never prints a key.
96
98
 
97
- Two probe results are checked in, both run on 2026-10-10 against the live Cloudflare API with the synthetic set above:
99
+ Four probe results are checked in, all run on 2026-10-10 with the synthetic set above: Jev against the live typesafe.ai API, Clef and Clef Flash against the live Cloudflare API, Ollama with `qwen3.5:9b` on an Apple M2 Pro laptop:
98
100
 
99
101
  | Classifier | Correct | p50 latency | p95 latency | Result file |
100
102
  | ---------- | -------: | ----------: | ----------: | ---------------------------------------------------------------------------------------------------- |
103
+ | Jev | 70 of 70 | 260 ms | 299 ms | [`activity-probe-jev.json`](../experiments/mod-router/results/activity-probe-jev.json) |
101
104
  | Clef | 69 of 70 | 444 ms | 883 ms | [`activity-probe-clef.json`](../experiments/mod-router/results/activity-probe-clef.json) |
102
105
  | Clef Flash | 69 of 70 | 470 ms | 874 ms | [`activity-probe-clef-flash.json`](../experiments/mod-router/results/activity-probe-clef-flash.json) |
106
+ | Ollama | 66 of 70 | 2986 ms | 3483 ms | [`activity-probe-ollama.json`](../experiments/mod-router/results/activity-probe-ollama.json) |
103
107
 
104
- They measure the label on synthetic prompts, not routing quality or savings. Jev, OpenAI and Ollama have no probe result yet. The pane shows no probe accuracy. `policy.activityMass` was not tuned on any classifier; run the probe before you trust it with a new one.
108
+ Ollama's latency covers both requests and stays inside its 5000 ms deadline; on slower hardware the activity step is skipped when less than 300 ms remain. They measure the label on synthetic prompts, not routing quality or savings. OpenAI has no probe result yet. The activity question costs Jev almost no time: 30 alternating pairs of the same prompts, with and without it, gave p50 259 ms both ways and p95 300 ms without, 306 ms with. The pane shows no probe accuracy. `policy.activityMass` was not tuned on any classifier; run the probe before you trust it with a new one.
105
109
 
106
110
  ## Historical method and limits
107
111
 
@@ -38,7 +38,8 @@ The status band is one line above the prompt: a tier meter, the activity, the ro
38
38
  | Activity | The classifier's activity choice with its share and the two next-best answers. Empty with routing off, a pin, or no answer. |
39
39
  | Route (activity) | With `on`: the route for the tier and activity, marked `(override)` with the base route beside it, or `(base)`. With `shadow`: what `on` would use, and the route in use (`shadow; same route` when they agree). |
40
40
  | Replies | The tier that served each of the last 30 main-conversation replies; a dot where the router did not choose. A second row of letters names each reply's activity: `c` code, `d` debug, `e` explore, `p` plan, `r` review, `o` ops, `w` docs. |
41
- | Activity stats (Usage) | Counts for this session: turns, requests and share per activity; switches by tier and by activity; tool agreement; in `shadow`, turns `on` would route differently. Counts only, no dollar figures. |
41
+ | Activity stats (Usage) | Counts for this session: turns, requests and share per activity; switches by tier and by activity; tool agreement; in `shadow`, turns `on` would route differently with their summed next-request estimate at configured list prices. |
42
+ | Across sessions (Usage) | The counts kept since the last reset: labelled turns, tool agreement on the classifier's raw answer, overall and for `code`, `ops` and `explore`, the two largest mismatches, lateral switches, and the shadow readout. `/router` text shows the agreement. |
42
43
  | Cost | Claude Code's native usage value. A plan's list-price estimates are not subscription cash charges. |
43
44
  | Context bar | Maximum available observed and local estimated context, compared with the routed model's window. The fit rule reserves 20%. |
44
45
  | Observed / estimated input | Claude's local context reading and the router's best current context estimate. `unknown` blocks a move to a smaller window. |
@@ -77,7 +77,7 @@ The **Routing** tab edits the routes, the activity overrides, and the policy. In
77
77
  | 1 Now | The current tier, model, activity, and reason; the classifier's support for each tier with a **pin** button; the last 30 replies by tier and activity; context and cache. |
78
78
  | 2 Routing | The model and effort for each tier, the baseline tier, the cache cost of each step up; the activity matrix and overrides; downgrade votes, payback horizon, and credits cap; the `router.json` path. |
79
79
  | 3 Classifier | The classifier rows with their credentials state, the active classifier's deadline and health, and the credentials each classifier needs. |
80
- | 4 Usage | Claude-reported cost, context and cache detail, input per reply, quota, activity counts for this session, and configured-price estimates. |
80
+ | 4 Usage | Claude-reported cost, context and cache detail, input per reply, quota, activity counts for this session and across sessions, and configured-price estimates. |
81
81
 
82
82
  **Routing on** and **Routing off** stay at the top of every tab. **?** shows what the estimates leave out.
83
83
 
@@ -145,9 +145,11 @@ The band's activity label appears only when the activity applies, so a weak or `
145
145
 
146
146
  - **Now**: the **Activity** line names the classifier's choice with its share and the two next-best answers, such as `code 84% debug 8% · plan 5%`. The **Route** line shows the route the cell resolves to. **Replies** has a letter row under the strip.
147
147
  - **Routing**: the **ACTIVITIES** matrix shows the effective routes per tier, with `·` where a cell uses the tier's route, and counts the distinct routes, because each is its own cache. **OVERRIDES** lists each override with a model, an effort, and **remove**, plus a selector to add one and the mode selector. Edits join the routing draft: **Save** and **Discard** apply as for routes. Yellow notes flag an override with no effect (`same as base`), one stronger than the tier above, and one that adds a cache.
148
- - **Usage**: **ACTIVITY · this session** lists turns, requests, and share per activity, the switches split by tier and by activity, and **Agreement**, how often the tools a turn used fit the classifier's activity (see [Evaluation](evaluation.md#tool-agreement)). In `shadow` the **Shadow** line counts how many turns `on` would have routed differently. **Reset activity stats** clears the session counts and the counts kept across sessions. These are counts, not dollar figures.
148
+ - **Usage**: **ACTIVITY · this session** lists turns, requests, and share per activity, the switches split by tier and by activity, and **Agreement**, how often the tools a turn used fit the classifier's activity (see [Evaluation](evaluation.md#tool-agreement)). In `shadow` the **Shadow** line counts how many turns `on` would have routed differently, such as `on would route 5 of 20 turns differently · est. −$0.300 … −$0.120 at list prices`. The estimate sums, over those turns, what the next request on the route `on` would use would cost against the route that ran, as a low and a high scenario at the configured list prices in `router.json`. Minus means cheaper. It covers only turns after a measured reply, and adds `for 3` when only three of the turns have one. It is not a measured saving.
149
149
 
150
- `/router` without a UI surface adds an `Activity: ops (81%)` line and the resolved route.
150
+ **ACROSS SESSIONS · since the last reset** shows the counts kept across sessions: the turns labelled from their tools, **Agreement** over them, **Code/ops/explore**, the same agreement over the turns the classifier answered `code`, `ops` or `explore` (omitted when there are none), the two most frequent mismatches as `predicted → observed count` (`code → read 3`), lateral switches taken and refused, and the **Shadow** line summed over all shadow turns. This agreement compares the classifier's raw answer, so it can differ from the session's. The pane reads these counts when the session starts and when it opens, and after each turn. **Reset activity stats** clears the session counts and the counts kept across sessions.
151
+
152
+ `/router` without a UI surface adds an `Activity: ops (81%)` line, the resolved route, and `Agreement across sessions: 19 of 22 turns (86%) · code/ops/explore 9 of 10 (90%)`.
151
153
 
152
154
  ## Change routing
153
155
 
@@ -38,6 +38,7 @@ import {
38
38
  GATEWAY_SETTINGS,
39
39
  missingCredentials,
40
40
  NOT_A_TIER_REASON,
41
+ storeAgreementLine,
41
42
  } from '../lib/display.mjs';
42
43
  import { clip, observedActivity, promptIndex } from '../lib/facts.mjs';
43
44
  import { renderPanel, routeDraftOf, routingChanges } from '../lib/panel.mjs';
@@ -146,10 +147,20 @@ async function updateView($, router, patch) {
146
147
  await $.state.set(VIEW, router.view);
147
148
  }
148
149
 
150
+ // The counts kept across sessions for the view, read when the session starts and the pane opens: another session
151
+ // may have added to them. A failed read keeps what the view has; it never breaks a turn or a render.
152
+ async function storeView($) {
153
+ try {
154
+ return { activityStore: readStore(await $.store.get(STORE_KEY)) };
155
+ } catch {
156
+ return {};
157
+ }
158
+ }
159
+
149
160
  // A notice answers the last press in the pane, so the pane opens without one; the last write keeps its Undo in the
150
161
  // status bar.
151
162
  async function openPane($, router) {
152
- await updateView($, router, { notice: null });
163
+ await updateView($, router, { notice: null, ...(await storeView($)) });
153
164
  await $.ui.open({ id: PANE, title: PANE_TITLE, focus: true, closeOnEscape: true });
154
165
  }
155
166
 
@@ -289,6 +300,7 @@ function detailText(config, view) {
289
300
  `Observed: ${view.actualModel ?? 'no response yet'}`,
290
301
  `Reason: ${view.reason}`,
291
302
  ...activityDetailLines(config, view),
303
+ storeAgreementLine(view),
292
304
  view.error || missing ? classifierStatus(config, view.error ?? missing) : `${activeClassifier(config).label} ready`,
293
305
  `Context: ${view.contextKnown ? `${view.contextTokens} tokens (estimate)` : 'unknown'}`,
294
306
  `Observed cache: ${view.cacheRead ?? 'unknown'} read, ${view.cacheWrite ?? 'unknown'} written tokens`,
@@ -479,6 +491,7 @@ async function recordActivity($, router, turn) {
479
491
  tierSwitches: turn.switched === 'tier' ? 1 : 0,
480
492
  activitySwitches: turn.switched === 'activity' ? 1 : 0,
481
493
  wouldDiffer: turn.wouldDiffer,
494
+ wouldUsd: turn.wouldUsd,
482
495
  shadow: turn.mode === 'shadow',
483
496
  }),
484
497
  });
@@ -486,16 +499,17 @@ async function recordActivity($, router, turn) {
486
499
  const store = readStore(await $.store.get(STORE_KEY));
487
500
  // Writing counts read before a reset would bring them back; this turn's are lost instead.
488
501
  if (router.statsResets !== resets) return;
489
- await $.store.set(
490
- STORE_KEY,
491
- recordStore(store, {
492
- predicted: turn.predicted,
493
- observed,
494
- runEnded: ended,
495
- lateral: turn.lateral,
496
- wouldDiffer: turn.wouldDiffer,
497
- }),
498
- );
502
+ const written = recordStore(store, {
503
+ predicted: turn.predicted,
504
+ observed,
505
+ runEnded: ended,
506
+ lateral: turn.lateral,
507
+ wouldDiffer: turn.wouldDiffer,
508
+ wouldUsd: turn.wouldUsd,
509
+ });
510
+ await $.store.set(STORE_KEY, written);
511
+ // A reset during the write emptied the view; these counts are older than it.
512
+ if (router.statsResets === resets) await updateView($, router, { activityStore: written });
499
513
  } catch {}
500
514
  }
501
515
 
@@ -503,7 +517,7 @@ async function recordActivity($, router, turn) {
503
517
  async function flushRun($, run) {
504
518
  try {
505
519
  const store = readStore(await $.store.get(STORE_KEY));
506
- const ended = { predicted: null, observed: null, runEnded: run, lateral: null, wouldDiffer: null };
520
+ const ended = { predicted: null, observed: null, runEnded: run, lateral: null, wouldDiffer: null, wouldUsd: null };
507
521
  await $.store.set(STORE_KEY, recordStore(store, ended));
508
522
  } catch {}
509
523
  }
@@ -666,6 +680,7 @@ function paneActions($, router, view) {
666
680
  .catch(() => false);
667
681
  await updateView($, router, {
668
682
  activityStats: null,
683
+ ...(cleared ? { activityStore: emptyStore() } : {}),
669
684
  notice: cleared ? 'Activity stats reset.' : 'Session activity stats reset. Saved stats could not be cleared.',
670
685
  });
671
686
  },
@@ -758,6 +773,7 @@ export function register(on, options) {
758
773
  routeDraft: null,
759
774
  lastWrite: null,
760
775
  bandDetail: (await $.store.get(BAND_DETAIL).catch(() => false)) === true,
776
+ ...(await storeView($)),
761
777
  });
762
778
  } catch (error) {
763
779
  await updateView($, router, {
@@ -923,6 +939,8 @@ export function register(on, options) {
923
939
  ...initialView(router.view?.nativeModel ?? ''),
924
940
  mode: 'auto',
925
941
  credentials: router.view?.credentials ?? null,
942
+ // Counts kept across sessions: the next session starts from them.
943
+ activityStore: router.view?.activityStore ?? null,
926
944
  health: healthOf(clientOf(router, router.config.classifier), router.config.classifier),
927
945
  configPath: router.config.nativePath,
928
946
  phase: router.view?.phase === 'unavailable' ? 'unavailable' : 'ready',
@@ -7,7 +7,7 @@ import { ACTIVITIES, ACTIVITY_VALUES } from './config.mjs';
7
7
  export const OBSERVED = ['code', 'docs', 'ops', 'read', 'talk'];
8
8
  export const STORE_KEY = 'activity:stats:v1';
9
9
 
10
- // The observed buckets each predicted activity allows. A weak oracle: it tells ops from code from explore, and cannot
10
+ // The observed buckets each predicted activity allows. A weak oracle: it tells code from ops and explore, and cannot
11
11
  // separate plan from review. That is enough to catch a cheap route on a coding turn.
12
12
  export const ALLOWED = {
13
13
  code: ['code'],
@@ -15,7 +15,8 @@ export const ALLOWED = {
15
15
  explore: ['read', 'talk'],
16
16
  plan: ['talk', 'read', 'docs'],
17
17
  review: ['read', 'talk'],
18
- ops: ['ops'],
18
+ // git status or a test listing is still running commands: a read-only Bash turn is ops too.
19
+ ops: ['ops', 'read'],
19
20
  docs: ['docs'],
20
21
  };
21
22
 
@@ -29,18 +30,34 @@ const CONFUSION_ROWS = [...ACTIVITY_VALUES, 'none'];
29
30
 
30
31
  const count = (n) => (Number.isFinite(n) && n > 0 ? Math.floor(n) : 0);
31
32
 
33
+ // The shadow readout: turns in shadow, those `on` would have routed differently, and of those the ones with a
34
+ // next-request estimate (`estimated`) and the sum of their [min, max] ranges in USD at configured list prices.
35
+ const emptyShadow = () => ({ differs: 0, turns: 0, estimated: 0, minUsd: 0, maxUsd: 0 });
36
+ const isRange = (r) => Number.isFinite(r?.minUsd) && Number.isFinite(r?.maxUsd);
37
+
38
+ function addShadow(shadow, wouldDiffer, range) {
39
+ const priced = wouldDiffer && isRange(range);
40
+ return {
41
+ differs: shadow.differs + (wouldDiffer ? 1 : 0),
42
+ turns: shadow.turns + 1,
43
+ estimated: shadow.estimated + (priced ? 1 : 0),
44
+ minUsd: shadow.minUsd + (priced ? range.minUsd : 0),
45
+ maxUsd: shadow.maxUsd + (priced ? range.maxUsd : 0),
46
+ };
47
+ }
48
+
32
49
  export function emptySession() {
33
50
  return {
34
51
  byActivity: {},
35
52
  switches: { tier: 0, activity: 0 },
36
53
  agreement: { matched: 0, total: 0 },
37
- shadow: { differs: 0, turns: 0 },
54
+ shadow: emptyShadow(),
38
55
  };
39
56
  }
40
57
 
41
58
  // One finished turn. `activity` is the turn's label (null: none), `answer` the classifier's accepted activity, which
42
- // agreement compares with the tools (null: not counted), `shadow` marks a turn in shadow mode and `wouldDiffer` says
43
- // whether 'on' would have routed it differently (null: no would-route).
59
+ // agreement compares with the tools (null: not counted), `shadow` marks a turn in shadow mode, `wouldDiffer` says
60
+ // whether 'on' would have routed it differently (null: no would-route) and `wouldUsd` prices that difference.
44
61
  export function recordTurn(
45
62
  session,
46
63
  {
@@ -53,6 +70,7 @@ export function recordTurn(
53
70
  tierSwitches,
54
71
  activitySwitches,
55
72
  wouldDiffer,
73
+ wouldUsd,
56
74
  shadow,
57
75
  },
58
76
  ) {
@@ -78,15 +96,15 @@ export function recordTurn(
78
96
  total: session.agreement.total + 1,
79
97
  }
80
98
  : session.agreement,
81
- shadow: {
82
- differs: session.shadow.differs + (shadow && wouldDiffer === true ? 1 : 0),
83
- turns: session.shadow.turns + (shadow ? 1 : 0),
84
- },
99
+ // A view saved by 1.6.0 has no estimate fields.
100
+ shadow: shadow
101
+ ? addShadow({ ...emptyShadow(), ...session.shadow }, wouldDiffer === true, wouldUsd)
102
+ : { ...emptyShadow(), ...session.shadow },
85
103
  };
86
104
  }
87
105
 
88
106
  export function emptyStore() {
89
- return { version: 1, confusion: {}, runs: {}, lateral: { taken: 0, refused: 0 }, shadow: { differs: 0, turns: 0 } };
107
+ return { version: 1, confusion: {}, runs: {}, lateral: { taken: 0, refused: 0 }, shadow: emptyShadow() };
90
108
  }
91
109
 
92
110
  const isCount = (n) => Number.isSafeInteger(n) && n >= 0;
@@ -96,7 +114,20 @@ const countsOf = (v, keys) => onlyKeys(v, keys) && Object.values(v).every(isCoun
96
114
  const exactCountsOf = (v, keys) =>
97
115
  isRecord(v) && Object.keys(v).length === keys.length && keys.every((k) => isCount(v[k]));
98
116
 
99
- // The stored value if it has the exact shape, else an empty store: a hand-edited or older file must not break stats.
117
+ const SHADOW_COUNTS = ['differs', 'turns'];
118
+ const SHADOW_ESTIMATE = ['estimated', 'minUsd', 'maxUsd'];
119
+ // 1.6.0 kept the shadow counts only; 1.6.1 adds the estimate. Either shape, never a part of the estimate.
120
+ const isShadow = (v) =>
121
+ exactCountsOf(v, SHADOW_COUNTS) ||
122
+ (onlyKeys(v, [...SHADOW_COUNTS, ...SHADOW_ESTIMATE]) &&
123
+ Object.keys(v).length === 5 &&
124
+ SHADOW_COUNTS.every((k) => isCount(v[k])) &&
125
+ isCount(v.estimated) &&
126
+ Number.isFinite(v.minUsd) &&
127
+ Number.isFinite(v.maxUsd));
128
+
129
+ // The stored value if it has a known shape, filled to the current one, else an empty store: a hand-edited or
130
+ // unknown file must not break stats.
100
131
  export function readStore(value) {
101
132
  const ok =
102
133
  onlyKeys(value, ['version', 'confusion', 'runs', 'lateral', 'shadow']) &&
@@ -108,14 +139,14 @@ export function readStore(value) {
108
139
  (lengths) => Array.isArray(lengths) && lengths.length === RUN_BUCKETS && lengths.every(isCount),
109
140
  ) &&
110
141
  exactCountsOf(value.lateral, ['taken', 'refused']) &&
111
- exactCountsOf(value.shadow, ['differs', 'turns']);
112
- return ok ? value : emptyStore();
142
+ isShadow(value.shadow);
143
+ return ok ? { ...value, shadow: { ...emptyShadow(), ...value.shadow } } : emptyStore();
113
144
  }
114
145
 
115
146
  // One finished turn. `predicted` is the classifier's answer (null: none), `runEnded` a run that closed this turn,
116
- // `lateral` the outcome of a lateral switch decision and `wouldDiffer` the shadow readout (null: not in shadow).
117
- // Unknown labels are dropped, so the store keeps a fixed set of keys.
118
- export function recordStore(store, { predicted, observed, runEnded, lateral, wouldDiffer }) {
147
+ // `lateral` the outcome of a lateral switch decision, `wouldDiffer` the shadow readout (null: not in shadow) and
148
+ // `wouldUsd` its estimate. Unknown labels are dropped, so the store keeps a fixed set of keys.
149
+ export function recordStore(store, { predicted, observed, runEnded, lateral, wouldDiffer, wouldUsd }) {
119
150
  const row = predicted ?? 'none';
120
151
  const next = {
121
152
  version: 1,
@@ -132,10 +163,7 @@ export function recordStore(store, { predicted, observed, runEnded, lateral, wou
132
163
  next.runs[runEnded.activity] = lengths;
133
164
  }
134
165
  if (lateral === 'taken' || lateral === 'refused') next.lateral[lateral] += 1;
135
- if (typeof wouldDiffer === 'boolean') {
136
- next.shadow.turns += 1;
137
- if (wouldDiffer) next.shadow.differs += 1;
138
- }
166
+ if (typeof wouldDiffer === 'boolean') next.shadow = addShadow(next.shadow, wouldDiffer, wouldUsd);
139
167
  return next;
140
168
  }
141
169
 
@@ -166,6 +194,7 @@ export function turnActivity(config, advice, previous, decision) {
166
194
  : 'tier'
167
195
  : null;
168
196
  const answer = mode === 'off' ? null : acceptActivity(config, advice);
197
+ const wouldDiffer = mode === 'shadow' ? Boolean(would && differs(would, decision)) : null;
169
198
  return {
170
199
  mode,
171
200
  answer,
@@ -174,6 +203,43 @@ export function turnActivity(config, advice, previous, decision) {
174
203
  switched,
175
204
  lateral: LATERAL[(mode === 'shadow' ? would : mode === 'on' ? decision : null)?.reason] ?? null,
176
205
  // A shadow turn always counts, so the session and store denominators agree.
177
- wouldDiffer: mode === 'shadow' ? Boolean(would && differs(would, decision)) : null,
206
+ wouldDiffer,
207
+ wouldUsd: wouldDiffer ? (would.difference ?? null) : null,
208
+ };
209
+ }
210
+
211
+ // The activities plan §11 checks agreement on before activity routing turns on by default.
212
+ const FOCUS = ['code', 'ops', 'explore'];
213
+
214
+ // The counts kept across sessions as the Usage tab and /router show them. Agreement covers the rows that name an
215
+ // activity: the store holds the raw answer, and `uncertain` or no answer never agrees; `focus` is the same over the
216
+ // FOCUS rows only. The two largest disagreements come first; ties keep the matrix order, so the readout is stable.
217
+ export function storeSummary(store) {
218
+ let labelled = 0;
219
+ let matched = 0;
220
+ let total = 0;
221
+ const focus = { matched: 0, total: 0 };
222
+ const misses = [];
223
+ for (const predicted of CONFUSION_ROWS)
224
+ for (const observed of OBSERVED) {
225
+ const n = store.confusion[predicted]?.[observed] ?? 0;
226
+ labelled += n;
227
+ if (!n || !ALLOWED[predicted]) continue;
228
+ total += n;
229
+ const agreed = agrees(predicted, observed);
230
+ if (agreed) matched += n;
231
+ else misses.push({ predicted, observed, n });
232
+ if (FOCUS.includes(predicted)) {
233
+ focus.total += n;
234
+ if (agreed) focus.matched += n;
235
+ }
236
+ }
237
+ return {
238
+ labelled,
239
+ agreement: { matched, total },
240
+ focus,
241
+ disagreements: misses.sort((a, b) => b.n - a.n).slice(0, 2),
242
+ lateral: store.lateral,
243
+ shadow: store.shadow,
178
244
  };
179
245
  }
package/lib/cost.mjs CHANGED
@@ -63,6 +63,18 @@ export function switchingTaxUsd(config, candidate, incumbent, facts, now) {
63
63
  );
64
64
  }
65
65
 
66
+ // The next request on `candidate` against `incumbent` as a range: the input bounds crossed, plus the output difference
67
+ // at the last reply's length when both routes have an output price, as shadowEconomics compares.
68
+ export function nextTurnRange(config, candidate, incumbent, facts, now) {
69
+ const tokens = nextContextTokens(facts);
70
+ const proposed = inputBounds(config, candidate, tokens, facts, now);
71
+ const current = inputBounds(config, incumbent, tokens, facts, now);
72
+ const [to, from] = [routeSpec(config, candidate), routeSpec(config, incumbent)];
73
+ const outputTokens = facts.lastRequest?.outputTokens ?? 0;
74
+ const output = to.output > 0 && from.output > 0 ? ((to.output - from.output) * outputTokens) / 1e6 : 0;
75
+ return { minUsd: proposed.min - current.max + output, maxUsd: proposed.max - current.min + output };
76
+ }
77
+
66
78
  export function shadowEconomics(config, candidate, incumbent, facts, now) {
67
79
  const proposed = routeSpec(config, candidate);
68
80
  const current = routeSpec(config, incumbent);
package/lib/display.mjs CHANGED
@@ -1,4 +1,5 @@
1
1
  import { acceptActivity } from './activity.mjs';
2
+ import { readStore, storeSummary } from './activity-stats.mjs';
2
3
  import {
3
4
  activeClassifier,
4
5
  CLASSIFIER_OPTIONS,
@@ -212,6 +213,16 @@ export function activityDetailLines(config, view) {
212
213
  ];
213
214
  }
214
215
 
216
+ // The tool agreement kept across sessions, for `/router` without a surface. The view holds the store as last read.
217
+ export function storeAgreementLine(view) {
218
+ const { agreement, focus } = storeSummary(readStore(view.activityStore));
219
+ const share = ({ matched, total }) => `${matched} of ${total}`;
220
+ if (!agreement.total) return 'Agreement across sessions: no labelled turns yet';
221
+ const all = `${share(agreement)} turns (${percent(agreement.matched / agreement.total)})`;
222
+ const code = focus.total ? ` · code/ops/explore ${share(focus)} (${percent(focus.matched / focus.total)})` : '';
223
+ return `Agreement across sessions: ${all}${code}`;
224
+ }
225
+
215
226
  export const GATEWAY_SETTINGS = 'v0.8 gateway settings remain · see /router';
216
227
  export const GATEWAY_CLEANUP = [
217
228
  'Remove from settings.json, then restart: model jev-router[1m], its modelPicker row,',
package/lib/panel.mjs CHANGED
@@ -1,4 +1,5 @@
1
1
  import { compareRoutes } from './activity.mjs';
2
+ import { readStore, storeSummary } from './activity-stats.mjs';
2
3
  import {
3
4
  ACTIVITIES,
4
5
  ACTIVITY_MODES,
@@ -700,9 +701,44 @@ function statusBar(ui, config, view, actions) {
700
701
  return out;
701
702
  }
702
703
 
703
- // Session counts by activity, the switches by cause, the tool agreement and the shadow readout. Counts only: what the
704
- // would-route turns cost is not recorded, so no dollar figure is shown. Reset is always offered: the counts kept
705
- // across sessions may exist before this session has any.
704
+ // `on` would route N of M shadow turns differently, with the summed next-request range of the differing turns that had
705
+ // one, at configured list prices; `for K` says when only some had one.
706
+ function shadowLine(shadow) {
707
+ const counted = `Shadow on would route ${shadow.differs} of ${shadow.turns} turns differently`;
708
+ if (!(shadow.estimated > 0)) return counted;
709
+ const some = shadow.estimated < shadow.differs ? ` for ${shadow.estimated}` : '';
710
+ return `${counted} · est. ${difference(shadow.minUsd)} … ${difference(shadow.maxUsd)}${some} at list prices`;
711
+ }
712
+
713
+ // The counts kept across sessions, read from the store as last loaded: what the decision to turn activity routing on
714
+ // by default reads. Agreement uses the classifier's raw answer, so it can differ from the session's.
715
+ function storeUsage(ui, store) {
716
+ const summary = storeSummary(readStore(store));
717
+ const out = [ui.head('ACROSS SESSIONS · since the last reset')];
718
+ if (summary.labelled === 0) return [...out, ui.dim(' no turns recorded yet')];
719
+ out.push(ui.text(`Labelled ${summary.labelled} turns`));
720
+ const { matched, total } = summary.agreement;
721
+ if (total > 0)
722
+ out.push(ui.text(`Agreement classifier vs tools: ${matched} of ${total} turns (${percent(matched / total)})`));
723
+ const { focus } = summary;
724
+ if (focus.total > 0)
725
+ out.push(
726
+ ui.text(
727
+ `Code/ops/explore classifier vs tools: ${focus.matched} of ${focus.total} turns (${percent(focus.matched / focus.total)})`,
728
+ ),
729
+ );
730
+ if (summary.disagreements.length)
731
+ out.push(
732
+ ui.text(`Mismatch ${summary.disagreements.map((d) => `${d.predicted} → ${d.observed} ${d.n}`).join(' · ')}`),
733
+ );
734
+ const { taken, refused } = summary.lateral;
735
+ if (taken + refused > 0) out.push(ui.text(`Lateral ${taken} taken · ${refused} refused`));
736
+ if (summary.shadow.turns > 0) out.push(ui.text(shadowLine(summary.shadow)));
737
+ return out;
738
+ }
739
+
740
+ // Session counts by activity, the switches by cause, the tool agreement and the shadow readout, then the counts kept
741
+ // across sessions. Reset is always offered: the counts kept across sessions may exist before this session has any.
706
742
  function activityUsage(ui, config, view, actions) {
707
743
  const stats = view.activityStats ?? null;
708
744
  const mode = config.activityRouting;
@@ -711,10 +747,12 @@ function activityUsage(ui, config, view, actions) {
711
747
  label: 'Reset activity stats',
712
748
  onPress: actions.resetActivityStats,
713
749
  });
750
+ const across = [ui.text(' '), ...storeUsage(ui, view.activityStore)];
714
751
  if (mode === 'off' && !stats)
715
752
  return [
716
753
  ui.head('ACTIVITY · routing off'),
717
754
  ui.dim(' Not asked. Set activity routing to shadow or on in the Routing tab.'),
755
+ ...(storeSummary(readStore(view.activityStore)).labelled > 0 ? across : []),
718
756
  reset,
719
757
  ];
720
758
  const out = [ui.line(ui.head('ACTIVITY · this session'), ui.dim(` routing ${mode}`))];
@@ -742,11 +780,9 @@ function activityUsage(ui, config, view, actions) {
742
780
  const { matched, total } = stats.agreement;
743
781
  if (total > 0)
744
782
  out.push(ui.text(`Agreement classifier vs tools: ${matched} of ${total} turns (${percent(matched / total)})`));
745
- if (mode === 'shadow' || stats.shadow.turns > 0)
746
- out.push(ui.text(`Shadow on would route ${stats.shadow.differs} of ${stats.shadow.turns} turns differently`));
783
+ if (mode === 'shadow' || stats.shadow.turns > 0) out.push(ui.text(shadowLine(stats.shadow)));
747
784
  }
748
- out.push(reset);
749
- return out;
785
+ return [...out, ...across, reset];
750
786
  }
751
787
 
752
788
  function usageTab(ui, config, view, usage, actions) {
package/lib/route.mjs CHANGED
@@ -129,7 +129,24 @@ export function chooseRoute(config, loop, input) {
129
129
  },
130
130
  );
131
131
  const { activity, tier, model, effort, reason } = run;
132
- wouldRoute = { activity, tier, model, effort, reason };
132
+ // What the would-route's next request costs against the applied one's; a route priced against itself would
133
+ // still give a spread, so only a different request is priced.
134
+ const priced =
135
+ input.contextKnown &&
136
+ loop.lastRequest &&
137
+ run.tier &&
138
+ applied.tier &&
139
+ (run.model !== applied.model || run.effort !== applied.effort);
140
+ const difference = priced
141
+ ? costs.nextTurnRange(
142
+ economicConfig(config, loop),
143
+ resolveRoute(config, run.tier, run.activity),
144
+ resolveRoute(config, applied.tier, applied.activity ?? null),
145
+ facts,
146
+ input.now,
147
+ )
148
+ : null;
149
+ wouldRoute = { activity, tier, model, effort, reason, difference };
133
150
  const stays = pin || !run.tier;
134
151
  would = {
135
152
  lastRoute: stays ? from.lastRoute : run.tier,
@@ -198,20 +215,12 @@ function routeTurn(config, loop, { facts, advice, pin, nativeModel, contextKnown
198
215
  let comparison = null;
199
216
  const candidate = pin ?? advice?.choice;
200
217
  if (contextKnown && loop.lastRequest && TIERS.includes(candidate) && candidate !== loop.lastRoute) {
201
- const tokens = costs.nextContextTokens(facts);
202
218
  const [to, from] = [resolveRoute(config, candidate, activity), routeOf(last)];
203
- const proposed = costs.inputBounds(projected, to, tokens, facts, now);
204
- const incumbentBounds = costs.inputBounds(projected, from, tokens, facts, now);
205
219
  const forecast = costs.shadowEconomics(projected, to, from, facts, now);
206
- // Without both output prices, compare input only, as shadowEconomics does.
207
- const output = forecast
208
- ? ((routeSpec(projected, to).output - routeSpec(projected, from).output) * loop.lastRequest.outputTokens) / 1e6
209
- : 0;
210
220
  comparison = {
211
221
  candidate,
212
222
  incumbent: loop.lastRoute,
213
- minUsd: proposed.min - incumbentBounds.max + output,
214
- maxUsd: proposed.max - incumbentBounds.min + output,
223
+ ...costs.nextTurnRange(projected, to, from, facts, now),
215
224
  paybackTurns: forecast?.paybackTurns ?? null,
216
225
  outputTokens: loop.lastRequest.outputTokens,
217
226
  };
package/lib/view.mjs CHANGED
@@ -38,6 +38,7 @@ export function initialView(model) {
38
38
  tiers: [],
39
39
  activities: [],
40
40
  activityStats: null,
41
+ activityStore: null,
41
42
  configPath: null,
42
43
  tuning: null,
43
44
  tuningBase: null,
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@alexeiled/claude-router",
3
- "version": "1.6.0",
3
+ "version": "1.6.1",
4
4
  "description": "Native Claude Code Mod: model and effort routing for the main conversation, advised by Jev, Clef, Clef Flash, OpenAI or Ollama.",
5
5
  "license": "MIT",
6
6
  "author": "Alexei Ledenev",
package/types/index.d.ts CHANGED
@@ -18,13 +18,26 @@ declare module 'claude-code' {
18
18
  probabilities: Record<RouterActivityAnswer, number>;
19
19
  } | null;
20
20
  }
21
- // What `on` would do, computed in `shadow`. `model` is a model id, as `decision.model`.
21
+ // A next-request cost difference in USD at configured list prices; negative is cheaper.
22
+ interface RouterUsdRange {
23
+ minUsd: number;
24
+ maxUsd: number;
25
+ }
26
+ // What `on` would do, computed in `shadow`. `model` is a model id, as `decision.model`. `difference` prices its next
27
+ // request against the applied route's; null when the routes match or there is no measured request yet.
22
28
  interface RouterWouldRoute {
23
29
  activity: RouterActivity | null;
24
30
  tier: RouterTier | null;
25
31
  model: string;
26
32
  effort: string | number | null;
27
33
  reason: string;
34
+ difference: RouterUsdRange | null;
35
+ }
36
+ // Shadow turns, those `on` would route differently, and of those the ones with a `difference`, summed.
37
+ interface RouterActivityShadow extends RouterUsdRange {
38
+ differs: number;
39
+ turns: number;
40
+ estimated: number;
28
41
  }
29
42
  interface RouterActivityCounts {
30
43
  turns: number;
@@ -37,7 +50,7 @@ declare module 'claude-code' {
37
50
  byActivity: Partial<Record<RouterActivity | 'none', RouterActivityCounts>>;
38
51
  switches: { tier: number; activity: number };
39
52
  agreement: { matched: number; total: number };
40
- shadow: { differs: number; turns: number };
53
+ shadow: RouterActivityShadow;
41
54
  }
42
55
  // Cross-session counts under store key 'activity:stats:v1'. `runs` buckets run lengths 1, 2, 3, 4, 5+.
43
56
  interface RouterActivityStore {
@@ -45,7 +58,8 @@ declare module 'claude-code' {
45
58
  confusion: Partial<Record<RouterActivityAnswer | 'none', Partial<Record<RouterObserved, number>>>>;
46
59
  runs: Partial<Record<RouterActivity, [number, number, number, number, number]>>;
47
60
  lateral: { taken: number; refused: number };
48
- shadow: { differs: number; turns: number };
61
+ // A 1.6.0 store has `differs` and `turns` only; readStore fills the rest with zeros.
62
+ shadow: RouterActivityShadow;
49
63
  }
50
64
  interface RouterComparison {
51
65
  candidate: RouterTier;
@@ -120,6 +134,8 @@ declare module 'claude-code' {
120
134
  // Aligned with `history` and `tiers`.
121
135
  activities?: (RouterActivity | null)[];
122
136
  activityStats?: RouterActivitySession | null;
137
+ // The counts kept across sessions as last read or written.
138
+ activityStore?: RouterActivityStore | null;
123
139
  configPath?: string | null;
124
140
  tuning?: Partial<RouterTuning> | null;
125
141
  tuningBase?: RouterTuning | null;