@alexeiled/claude-router 1.5.1 → 1.6.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,8 +1,8 @@
1
1
  {
2
2
  "name": "router",
3
3
  "displayName": "Router",
4
- "version": "1.5.1",
5
- "description": "Model and effort routing for the main conversation, advised by a prompt classifier (Jev, Clef, Clef Flash, OpenAI or local Ollama). Preserves subagent model choices.",
4
+ "version": "1.6.0",
5
+ "description": "Model and effort routing for the main conversation by task difficulty and activity, advised by a prompt classifier (Jev, Clef, Clef Flash, OpenAI or local Ollama). Preserves subagent model choices.",
6
6
  "author": {
7
7
  "name": "Alexei Ledenev",
8
8
  "url": "https://github.com/alexei-led"
package/CHANGELOG.md CHANGED
@@ -1,5 +1,31 @@
1
1
  # Changelog
2
2
 
3
+ ## 1.6.0 (2026-10-10)
4
+
5
+ The classifier now also names the turn's **activity**: what the turn produces (`code`, `debug`, `explore`, `plan`, `review`, `ops`, `docs`, or `uncertain`). The route can then depend on tier and activity, so a `low` coding turn can run on Sonnet 5.5 while a `low` git turn stays on Haiku 5.5. Routing does not change until you opt in.
6
+
7
+ ### Added
8
+
9
+ - Activity routing, `activityRouting` in `router.json`: `off`, `shadow` (the default), or `on`. `shadow` asks for the activity, shows it, records stats, and works out what `on` would run; the tier route still runs. `on` applies the activity overrides. Switch with `/router activities on|shadow|off` or the mode selector on the Routing tab.
10
+ - Default activity overrides on top of the unchanged tier routes: at `low`, `code`, `debug`, `plan`, `review` and `docs` run on Sonnet 5.5 at `medium`; at `medium`, `ops`, `explore` and `docs` run on Sonnet 5.5 at `medium`. Every other cell uses its tier's route. Override any cell, per field, under `activities.<activity>.<tier>`.
11
+ - `policy.activityMass` (0.6): the probability an activity needs to apply.
12
+ - Switching between routes of one tier compares routes, not labels: a move to a stronger route needs the classifier's support, a move to a cheaper one must pay back its cache write, and both pass the cash gate. A continuation may move the activity only to a stronger route. Escalation keeps the activity and skips tiers whose route is no stronger.
13
+ - The band shows the activity between the tier and the model, `(shadow)` while in shadow, and the reasons `↗ code`, `↘ ops`, `… ops: not worth a switch`. The toast names the activity of an activity move.
14
+ - Pane: **Now** has **Activity** and **Route** lines and a letter row under **Replies**; **Routing** has the **ACTIVITIES** matrix and an **OVERRIDES** editor that joins the routing draft; **Usage** has **ACTIVITY · this session**, switches by tier and by activity, tool **Agreement**, the **Shadow** line, and **Reset activity stats**.
15
+ - Tool agreement: each finished turn is labelled from its tool calls (edits, Bash, reads, none), locally and without storing any text, and compared with the classifier's activity. Counts are kept per session and across sessions.
16
+ - `scripts/probe-activity.mjs` and a 70-prompt synthetic probe set. Clef and Clef Flash each answered 69 of 70 against the live API (p50 444 and 470 ms). Jev, OpenAI and Ollama have no probe result yet.
17
+
18
+ ### Changed
19
+
20
+ - Before the first measured reply of a history (session start, `/clear`, compaction, rewind), an upgrade or downgrade needs one classifier answer that clears its bar instead of a streak of votes. A session started on the wrong tier now moves on turn 1.
21
+ - Jev, Clef and Clef Flash get the activity as one more question in the same request, and OpenAI as a second question. Ollama asks it in a second request inside the same deadline, only when at least 300 ms remain. With `off`, the requests are the same as in 1.5. A missing or malformed activity answer never costs the tier answer and never counts as a classifier failure.
22
+
23
+ ### Upgrade notes
24
+
25
+ - Nothing routes differently after the upgrade: `shadow` runs the tier routes, as 1.5 did.
26
+ - With `shadow` or `on`, every classified turn sends the activity question too. Set `"activityRouting": "off"` to send exactly what 1.5 sent.
27
+ - 1.5 rejects `activities`, `activityRouting` and `policy.activityMass`. Remove them from `router.json` before downgrading.
28
+
3
29
  ## 1.5.1 (2026-10-09)
4
30
 
5
31
  ### Changed
package/README.md CHANGED
@@ -9,21 +9,21 @@ The Mod changes only the model and effort in Claude Code's turn hook. Claude Cod
9
9
 
10
10
  ## Supported classifiers
11
11
 
12
- The active classifier is asked once per logical turn. Pick one on the pane's **Classifier** tab or in `router.json`; only the active one receives prompt text.
12
+ The active classifier is asked once per logical turn, for the tier and for the turn's activity. Pick one on the pane's **Classifier** tab or in `router.json`; only the active one receives prompt text.
13
13
 
14
- | Classifier | Service | Credential | Prompt text goes to |
15
- | ---------- | ------------------------------------------ | ------------------------------------------- | ---------------------------- |
16
- | Jev | [typesafe.ai](https://typesafe.ai) | Jev API key | typesafe.ai |
17
- | Clef | Cloudflare Workers AI, 27B | Cloudflare API token and account ID | Cloudflare |
18
- | Clef Flash | Cloudflare Workers AI, 9B | Cloudflare API token and account ID | Cloudflare |
19
- | OpenAI | OpenAI, `gpt-6-luna` (public beta) | OpenAI API key | OpenAI (`api.openai.com`) |
20
- | Ollama | A local [Ollama](https://ollama.com) model | None | Stays on this machine |
14
+ | Classifier | Service | Credential | Prompt text goes to | How it is asked for the activity |
15
+ | ---------- | ------------------------------------------ | ----------------------------------- | ------------------------- | --------------------------------------------------------------------- |
16
+ | Jev | [typesafe.ai](https://typesafe.ai) | Jev API key | typesafe.ai | A third `choice` question in the same request |
17
+ | Clef | Cloudflare Workers AI, 27B | Cloudflare API token and account ID | Cloudflare | The same |
18
+ | Clef Flash | Cloudflare Workers AI, 9B | Cloudflare API token and account ID | Cloudflare | The same |
19
+ | OpenAI | OpenAI, `gpt-6-luna` (public beta) | OpenAI API key | OpenAI (`api.openai.com`) | A second `choice` in `questions`, in the same request |
20
+ | Ollama | A local [Ollama](https://ollama.com) model | None | Stays on this machine | A second request: letters A to H, one token, if 300 ms or more remain |
21
21
 
22
22
  Jev is the default. [Configuration](docs/configuration.md#classifiers) gives the endpoints, timeouts, and how to add another service that speaks one of the supported protocols.
23
23
 
24
- ![The Router band above the Claude Code prompt in five states](docs/router-band.svg)
24
+ ![The Router band above the Claude Code prompt in seven states](docs/router-band.svg)
25
25
 
26
- The band above the prompt shows the tier, model, and reason for each turn. `/router` opens a pane to pin a tier, edit the model and effort of each tier, and tune the policy. See the [user guide](docs/user-guide.md#read-the-status-band).
26
+ The band above the prompt shows the tier, activity, model, and reason for each turn. `/router` opens a pane to pin a tier, edit the model and effort of each tier, and tune the policy. See the [user guide](docs/user-guide.md#read-the-status-band).
27
27
 
28
28
  The router needs Claude Code 2.1.289 or newer. Current savings are not measured. The panel shows Claude's reported usage and configured-price scenarios, not a savings total. See the [evaluation](docs/evaluation.md).
29
29
 
@@ -61,6 +61,8 @@ The active classifier labels a new logical turn once. Tool continuations keep th
61
61
 
62
62
  These are routing defaults. They do not claim equal model quality. [Configuration](docs/configuration.md#built-in-defaults) gives the published results behind them and the `router.json` lines that restore the 1.3 Sonnet ladder.
63
63
 
64
+ Activity routing adds a second label: what the turn produces (`code`, `debug`, `explore`, `plan`, `review`, `ops`, or `docs`). A route is then picked by tier and activity, so a `low` coding turn can run on Sonnet 5.5 while a `low` git turn stays on Haiku 5.5. It ships in `shadow` mode: the band and pane show the activity and what it would route, and the tier routes still run. Opt in with `/router activities on`. The label is optional: on `uncertain` or a weak label the tier route is used, when the activity answer fails the running activity stays and the tier answer still applies, and when the classifier fails the running route stays. See [Route by activity](docs/user-guide.md#route-by-activity), and [Evaluation](docs/evaluation.md#activity-routing) for how it is checked. Its default overrides rest on published vendor results, not measured savings.
65
+
64
66
  ## Install
65
67
 
66
68
  You need Claude Code 2.1.289 or newer and credentials for one [supported classifier](#supported-classifiers).
@@ -1,6 +1,6 @@
1
1
  # Architecture
2
2
 
3
- Claude Code owns the model request. The router Mod asks the active classifier (Jev by default; Cloudflare's Clef or Clef Flash; OpenAI; or a local Ollama model) for tier advice, applies local routing policy, and changes only the next main-conversation step's model and effort. Every subagent step passes through with Claude's model selection unchanged.
3
+ Claude Code owns the model request. The router Mod asks the active classifier (Jev by default; Cloudflare's Clef or Clef Flash; OpenAI; or a local Ollama model) for tier advice and, unless `activityRouting` is `off`, the turn's activity, applies local routing policy, and changes only the next main-conversation step's model and effort. Every subagent step passes through with Claude's model selection unchanged.
4
4
 
5
5
  ## Request flow
6
6
 
@@ -16,7 +16,7 @@ sequenceDiagram
16
16
  Code->>Mod: turn.start and turn.step
17
17
  alt Main conversation, routing on
18
18
  Mod->>Cls: Current prompt and bounded dialogue
19
- Cls-->>Mod: Tier advice
19
+ Cls-->>Mod: Tier and activity advice
20
20
  Note over Mod: Apply votes, context fit, cache costs and failure rules
21
21
  Mod-->>Code: Forward model and effort
22
22
  else Routing off, subagent, or unavailable
@@ -34,24 +34,37 @@ The active classifier receives only the current prompt and up to six recent user
34
34
 
35
35
  Each classifier names its wire protocol, `api`, and `classifier-apis.mjs` holds one adapter per protocol. `system-one` (Jev, Clef, Clef Flash) sends the typed `state` and `questions` request and reads `answers.route`; Cloudflare's REST API wraps that answer in `{ result, success, errors }`, which the parser unwraps. `openai-decisions` (OpenAI) sends the state as `input` text and reads a `choice` answer plus a `predicate` for the continuation. `ollama` asks for one letter with thinking off and reads the log-probability of each letter from the top 20 candidates. A letter's probability is its share of the letters found there; a letter missing from the list counts as zero. The Ollama adapter asks no continuation question, so `continuation` is null there. Every adapter returns the same advice shape, so the routing policy is unchanged.
36
36
 
37
+ The advice also carries `activity`: `{ choice, probabilities }` over the seven activities and `uncertain`, or null. Each protocol asks for it differently:
38
+
39
+ | Protocol | How the activity is asked |
40
+ | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------ |
41
+ | `system-one` | A third question, `activity`, of type `choice` in the same request. |
42
+ | `openai-decisions` | A second `choice` in `questions`, named `activity`. |
43
+ | `ollama` | A second request after the route answer, with the same system message and state prefix so Ollama can reuse its cache. Letters A to H, one token. |
44
+
45
+ The activity is optional at every layer. An adapter reads a missing, malformed, or unknown activity answer as `activity: null` and keeps the strict parsing of the route answer. With `activityRouting: off` no adapter builds the activity question, so the request bodies are the ones 1.5 sent; a fixture test checks this. The criteria and instructions live in `classifier-contract.mjs` beside the tier ones, and name each activity by what the turn produces. The classifier criteria keep showing the base route of each tier, not the activity overrides: the tier question is about difficulty.
46
+
37
47
  Claude Code builds each request for the model the Mod selects, so no request rewriting is needed: Haiku 4.5 got its own thinking mode and a 32K output cap. On Claude Code 2.1.289, a billed chain of pinned turns (Opus, Haiku 4.5, Sonnet at `xhigh`, Sonnet, Haiku 4.5) used a tool on every turn over the previous models' history, and no request was rejected. Sonnet and Opus accepted histories above 589K input tokens. These checks cover tested requests. They do not prove every Claude Code feature combination.
38
48
 
39
49
  ## Responsibilities
40
50
 
41
- | Component | Responsibility |
42
- | ----------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
43
- | [Native hooks](../hooks/native-router.mjs) | Turn identity, controls, lifecycle, per-step rewrites, response observation, pane actions, and every host call. |
44
- | [Classifier contract](../lib/classifier-contract.mjs) | Route criteria and instructions shared by every protocol, credentials and endpoint filling, and retry rules. |
45
- | [Classifier APIs](../lib/classifier-apis.mjs) | One adapter per wire protocol: request payload, answer validation, and the shared advice shape. |
46
- | [Classifier client](../lib/classifier-client.mjs) | Classifier HTTP, deadline, in-flight admission, and circuit breaker. |
47
- | [Router controller](../lib/route.mjs) | Context fit, model identity, engine fallback, cache observations, and one-turn pins. |
48
- | [Policy](../lib/policy.mjs) | Tier votes, escalation, and switching gates. |
49
- | [Costs](../lib/cost.mjs) | Cache uncertainty bounds and switching estimates. |
50
- | [Router view](../lib/view.mjs) | The view's initial shape and the usage readings each reply adds to it. |
51
- | [Config file](../lib/config-file.mjs) | Pane writes to `router.json`: the validated rewrite and what Undo restores. |
52
- | [Panel](../lib/panel.mjs) | Tabbed pane: route status, tier pins, the route editor, tuning, and usage. |
53
- | [Router band](../lib/band.mjs) | Status band segments fitted to the band's width, and route-change toasts. |
54
- | Claude Code | API requests, tools, authentication, stream output, and usage ledger. |
51
+ | Component | Responsibility |
52
+ | ----------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
53
+ | [Native hooks](../hooks/native-router.mjs) | Turn identity, controls, lifecycle, per-step rewrites, response observation, pane actions, and every host call. |
54
+ | [Classifier contract](../lib/classifier-contract.mjs) | Route criteria and instructions shared by every protocol, credentials and endpoint filling, and retry rules. |
55
+ | [Classifier APIs](../lib/classifier-apis.mjs) | One adapter per wire protocol: request payload, answer validation, and the shared advice shape. |
56
+ | [Classifier client](../lib/classifier-client.mjs) | Classifier HTTP, deadline, in-flight admission, and circuit breaker. |
57
+ | [Router controller](../lib/route.mjs) | Context fit, model identity, engine fallback, cache observations, one-turn pins, and the tier-and-activity cell. |
58
+ | [Activity rules](../lib/activity.mjs) | Which activity applies, how two routes compare, and when a route change inside a tier is worth it. |
59
+ | [Activity stats](../lib/activity-stats.mjs) | Session and stored activity counts, the tool-agreement table, and the per-turn record. |
60
+ | [Facts](../lib/facts.mjs) | The turn's facts for policy, and the bucket of a finished turn from its tool calls. |
61
+ | [Policy](../lib/policy.mjs) | Tier votes, escalation, and switching gates, for routes as well as tiers. |
62
+ | [Costs](../lib/cost.mjs) | Cache uncertainty bounds and switching estimates. |
63
+ | [Router view](../lib/view.mjs) | The view's initial shape and the usage readings each reply adds to it. |
64
+ | [Config file](../lib/config-file.mjs) | Pane writes to `router.json`: the validated rewrite and what Undo restores. |
65
+ | [Panel](../lib/panel.mjs) | Tabbed pane: route status, tier pins, the route editor, tuning, and usage. |
66
+ | [Router band](../lib/band.mjs) | Status band segments fitted to the band's width, and route-change toasts. |
67
+ | Claude Code | API requests, tools, authentication, stream output, and usage ledger. |
55
68
 
56
69
  Every function that takes the engine interface `$` lives in the hooks module: the engine does not follow `$` across an import, so the modules in `lib/` stay pure.
57
70
 
@@ -61,21 +74,50 @@ Node.js is not part of routing. It is used for development, the optional configu
61
74
 
62
75
  One logical turn gets one classification. Tool continuations reuse the decision and add no votes. Before each step, the controller checks whether the selected model still fits after new tool output.
63
76
 
77
+ An upgrade or downgrade needs a streak of supporting votes (`upgradeVotes`, `downgradeVotes`), because staying keeps a warm cache. Before the first measured reply of a history (session start, clear, rewind, committed compaction), no cache from it exists yet, so one vote is enough. The mass bars, the cash gate, escalation, holds and continuations apply as usual.
78
+
64
79
  A move to a smaller context window needs a main-response measurement from the current history and Claude's local context estimate. The router uses the larger available reading and reserves 20% of the model window. If it cannot establish fit, it keeps the current model with `context-unknown`. It does not infer token count from character count.
65
80
 
66
81
  An explicit context-window failure marks that model ineligible until the next history reset. An engine fallback suspends rewriting for the rest of the turn. The router does not retry an Anthropic request.
67
82
 
83
+ ## Activity routing
84
+
85
+ The router decides on a *cell*: a tier and an activity. The route is `{ ...routes[tier], ...activities[activity][tier] }`; the (model, effort) pair, not the label, is what Claude Code receives and what owns a cache. Two cells that resolve to the same route never switch.
86
+
87
+ `chooseRoute` runs one decision per logical turn:
88
+
89
+ 1. **Pin.** A pinned tier wins, with no activity: the pin's tier route.
90
+ 2. **Activity.** Only in `on`; `off` and `shadow` use none for the route they apply, and ignore the activity a turn in `on` ran with before the mode changed. With no advice, or advice without an activity answer, the incumbent's activity stays. On a new prompt the classifier scores as a continuation of the task (advice `continuation` at or above `continuationMass`; Ollama gives none), the advice's activity applies only if its route at the incumbent tier is stronger than the running route, so a task can move from `explore` up to `code` but a weaker label never takes over. Tool continuations inside a turn do not reclassify: they keep the turn's route and activity. Otherwise the activity with probability at least `activityMass` applies; below that, or for `uncertain`, there is none and the tier's base route is used.
91
+ 3. **Tier.** The tier policy runs as before, with every candidate tier priced at its route for the activity and the incumbent at the route it runs. Repeated tool errors move to the lowest tier above whose route is stronger than the running one, or sends a request that cannot be ordered against it. With no activity that is one tier up, as in 1.5. If no tier above helps, nothing escalates and the failure signature is kept.
92
+ 4. **Lateral move.** If the tier stays and the route would change, `lateralMove` decides. Routes order by configured output price, then effort; a session effort counts as the effort Claude Code sends. Two routes that send the same request do not switch. A pair that cannot be ordered (an unpriced model, a numeric effort, two models at one price) counts as a move up, so the base route is always reachable. A stronger route needs the activity's probability to clear the upgrade bar for its switching tax (a base route with no activity counts as probability 1). A cheaper route needs its downgrade tax to be paid back within `downgradeHorizonTurns`, except before the first measured reply of a history, when there is no cache to protect. The cash gate applies to both. The result is `activity-up`, `activity-down`, `activity-pending` (stay on the running route and its activity), `cash-gate`, or `hold`. While an escalation hold lasts, only a stronger route may be taken. One vote is enough: an activity is a fact about the turn, not a noisy estimate. If the tier changes, the tier gates have already priced the new route, and no lateral rule applies.
93
+ 5. **Fit.** Context fit and model availability resolve through `route(tier, activity)`. A fallback tries the last cell, the native model's cell, then each tier with the activity.
94
+
95
+ `activityRouting: shadow` runs the decision twice. The applied one uses no activity, so it is the 1.5 decision. The second runs as `on` and is kept only as `decision.wouldRoute`, shown on the pane and counted in the stats. It never changes a route. The second run keeps its own cell and policy state in `loop.would`, so each would-route starts from the previous would-route, as `on` would, not from the route actually running. Its limit: the would-route never runs, so its cache is never observed; the switching estimates price it as any cache of unknown state. A tool continuation keeps the turn's route and activity.
96
+
97
+ At session start the incumbent is the session's model. `cellForModel` looks for it in the base routes, baseline first, and only with `activityRouting: on` in the activity overrides too, so a session that starts on an override route keeps its cell until the classifier says otherwise. The start mode uses the same lookup: with `on`, a start model that only an override uses (Sonnet 5.5 on the defaults) starts with routing on.
98
+
68
99
  ## State and controls
69
100
 
70
101
  `$.state` stores route decisions, response observations, and UI state for the active session. Version-checked writes reject stale turn results. A private in-memory view merges state writes during one dispatch because state reads are frozen for that dispatch.
71
102
 
72
103
  Clear, branch, rewind, and committed compaction clear cache evidence and votes. A turn in progress keeps its route, pin, and any engine fallback. `/clear` starts the new session with routing on. A fresh session on a model that a configured tier routes to starts with routing on. A fresh session on any other model starts with routing off, and the band names the reason. Session start sets the mode; the band and pane only read it, even when they draw first. Resume restores the selected session's saved mode. Hot reload retains valid `$.state`. Running `/model` turns routing off, including an explicit choice of the baseline. `/router auto` resumes routing. `/router off` also turns routing off. Pins apply to the next turn and its tool continuations. The prior route resumes afterward.
73
104
 
105
+ ### Observed activity and stats
106
+
107
+ When a turn completes, the Mod labels it from the tool calls the assistant made after the prompt: `code` for an edit to a non-doc path, `docs` for edits only to doc paths, `ops` for a Bash command that is not read-only, `read` for other tool use, and `talk` for none. `facts.mjs` reads tool names and file paths; it keeps no text. [Evaluation](evaluation.md#tool-agreement) explains how this is compared with the classifier's answer.
108
+
109
+ Two stores hold counts:
110
+
111
+ - **Session**: `activityStats` in the view, reset when the session ends. It has turns, requests and tokens per activity, switches by tier and by activity, tool agreement, and the shadow readout. The label counted per activity is the one the turn used: the applied activity in `on`, the answer that cleared `activityMass` in `shadow`. Tool agreement always compares the answer that cleared `activityMass`, so it measures the classifier in both modes. The view also keeps an `activities` array aligned with the last 30 replies.
112
+ - **Across sessions**: one `$.store` key, `activity:stats:v1`. It holds a confusion matrix of the classifier's raw answer (including `uncertain` and `none`) against the observed bucket, run lengths per activity in five buckets, lateral switches taken and refused, and shadow turns and differences. The key set is fixed, so the size is bounded. A stored value with any other shape is read as empty. The open run is written when the session ends. **Reset activity stats** clears both stores.
113
+
114
+ Nothing is recorded with `activityRouting: off`. Stats run inside a guard: a failed transcript read or store access loses that turn's counts and never fails the turn. A turn from a session that has since changed is dropped.
115
+
74
116
  Subagent events pass directly to Claude Code. They do not create classifier requests or change main-conversation metrics. This is the selected scope because Mods do not expose whether a subagent model was explicit or inherited.
75
117
 
76
118
  ## Classifier deadline and failure handling
77
119
 
78
- Each advice attempt has a total deadline, including any transient retry: 1,500 ms for Jev, 3,000 ms for Clef, Clef Flash and OpenAI, and 5,000 ms for Ollama by default. Ollama's first call after its model unloads can exceed that while the model loads. A timeout does not retry. `Retry-After` is honored only when it fits the remaining budget. Three launched failures pause the classifier for 60 seconds. Missing-key, missing-account, busy, and paused skips do not count as failures. Each classifier has its own failure count, pause, and in-flight slot. A turn that started under one classifier finishes with it, even if the pane switches meanwhile. Switching the classifier, or starting a session with another one, starts the new one from a clean count.
120
+ Each advice attempt has a total deadline, including any transient retry: 1,500 ms for Jev, 3,000 ms for Clef, Clef Flash and OpenAI, and 5,000 ms for Ollama by default. Ollama's first call after its model unloads can exceed that while the model loads. A timeout does not retry. `Retry-After` is honored only when it fits the remaining budget. Three launched failures pause the classifier for 60 seconds. Missing-key, missing-account, busy, and paused skips do not count as failures. The Ollama activity request is a second call inside the same deadline and the same in-flight slot. It is sent only when the route answer parsed and at least 300 ms remain, gets one attempt and no retry, and a timeout or error there returns the route advice with `activity: null`. It counts as no failure and never opens a pause. Each classifier has its own failure count, pause, and in-flight slot. A turn that started under one classifier finishes with it, even if the pane switches meanwhile. Switching the classifier, or starting a session with another one, starts the new one from a clean count.
79
121
 
80
122
  The pinned host API does not expose a per-request abort option for `$.http.fetch`. The Mod returns to the incumbent at its own deadline and blocks another advice request while that wire call remains pending. Claude Code's observed host timeout is about 30 seconds. Recorded reload and unload checks closed pending sockets after 24.7 and 28.5 seconds. Replacement closed one after 31.6 seconds. These runs do not show immediate per-call cancellation. A plugin network-policy refusal degrades routing and is never bypassed with a subprocess.
81
123
 
@@ -1,6 +1,6 @@
1
1
  # Configuration
2
2
 
3
- The router works with its built-in defaults. Configure the key of the active classifier to enable advice. Change `router.json` only when you need to change the classifier, routes, model prices, cache assumptions, or policy.
3
+ The router works with its built-in defaults. Configure the key of the active classifier to enable advice. Change `router.json` only when you need to change the classifier, routes, activity routing, model prices, cache assumptions, or policy.
4
4
 
5
5
  ## Key and profile settings
6
6
 
@@ -59,9 +59,60 @@ Default model settings:
59
59
 
60
60
  Prices are configured list-price inputs for estimates. They are not a subscription bill or a claim of savings. Haiku 5.5 prices are for prompts up to 100,000 tokens; above that Anthropic charges more, so estimates run low there. A model with `"efforts": []` receives no effort field. A route in `router.json` without an `effort` takes the effort of that tier's default route. To keep the session effort, clamped to what the model supports, set `"effort": null`, for example `"routes": { "high": { "model": "opus", "effort": null } }`. A 1.3 file whose `low` route had no `effort` kept the session effort; in 1.4 it runs at `high` until you add `"effort": null`.
61
61
 
62
+ ## Activity routing
63
+
64
+ The classifier also names the activity of a turn: what the turn produces. `activityRouting` decides what the router does with it.
65
+
66
+ | `activityRouting` | Behavior |
67
+ | ----------------- | ------------------------------------------------------------------------------------------------------------------ |
68
+ | `off` | Asks nothing new. The classifier requests are the ones 1.5 sent, and routes are exactly the tier routes. |
69
+ | `shadow` | The default. Asks, shows the activity, records stats, and computes what `on` would run. The tier routes still run. |
70
+ | `on` | Applies the `activities` overrides below. |
71
+
72
+ The seven activities are `code`, `debug`, `explore`, `plan`, `review`, `ops`, and `docs`. The classifier may also answer `uncertain`. In `on`, an activity applies only when its probability is at least `policy.activityMass`. Below that, or on `uncertain`, the turn goes back to its tier's route, if the move passes the cache checks. When the classifier gives no usable activity answer, the running activity stays and the tier answer still applies. When it gives no answer at all, the running route stays. In `off` and `shadow`, the tier route runs.
73
+
74
+ `activities` maps an activity to a tier to a route override. An override may set `model`, `effort`, or both. A field it leaves out comes from the tier's route:
75
+
76
+ ```
77
+ route(tier, activity) = { ...routes[tier], ...activities[activity][tier] }
78
+ ```
79
+
80
+ The built-in overrides ship with 1.6 and run only with `activityRouting: "on"`. Cells marked `·` have no override:
81
+
82
+ | Activity | `micro` | `low` | `medium` | `high` |
83
+ | --------------------------------- | ------- | --------------- | --------------- | ------ |
84
+ | `code`, `debug`, `plan`, `review` | `·` | Sonnet · medium | `·` | `·` |
85
+ | `docs` | `·` | Sonnet · medium | Sonnet · medium | `·` |
86
+ | `ops`, `explore` | `·` | `·` | Sonnet · medium | `·` |
87
+
88
+ At `low`, `ops` and `explore` keep Haiku at `high`: they are the cheap cases the `low` route is for. Coding is where Haiku 5.5 trails most (Terminal-Bench 4.0: Haiku 39.2%, Sonnet 70.6%). On knowledge and terminal tasks Sonnet 5.5 matched Opus 5.5 (GDPval-AA 1844 against 1846; Terminal-Bench 70.6% against 66.4%). These are vendor benchmarks at max or xhigh effort, not measurements of this router, and they are not a claim of equal quality. Every override uses Sonnet at `medium`, the one pair the four base routes do not use, so activity routing adds one cache.
89
+
90
+ A user file merges over these overrides one field at a time, as `routes` does. This sets only the effort of the built-in `code` override at `low`, so the cell runs Sonnet at `high`, and adds a new override at `high`:
91
+
92
+ ```json
93
+ {
94
+ "activityRouting": "on",
95
+ "activities": {
96
+ "code": { "low": { "effort": "high" }, "high": { "effort": "max" } }
97
+ }
98
+ }
99
+ ```
100
+
101
+ - `effort: null` keeps the session effort, as in `routes`.
102
+ - A cell cannot be `null`. To drop a built-in override, set it to the tier's route, for example `"ops": { "medium": { "model": "opus", "effort": "medium" } }`. The pane writes this for you when you remove an override.
103
+ - An override applies to whatever the tier's route is. If you move `routes.low` to Opus, the built-in `code` override at `low` still runs Sonnet. Set it to match, or remove it.
104
+ - Unknown activities and tiers, unknown model aliases, and invalid efforts make the file invalid, like any other invalid setting. The pane refuses to save one and names it, such as `activities.code.low.model is not in models`.
105
+ - Each distinct (model, effort) pair is its own cache. The Routing tab counts them.
106
+
107
+ `policy.activityMass` (default `0.6`) is the lowest probability at which an activity applies. It has not been tuned on any classifier. Probabilities differ between classifiers, so recalibrate it from the [probe set](evaluation.md#activity-probe-set) if you change classifier.
108
+
109
+ Switch the mode with `/router activities off|shadow|on`, or with the selector at the top of the Routing tab's OVERRIDES section. The command writes `router.json` at once; the selector joins the Routing draft and writes on **Save**. Choosing `shadow`, the default, removes the key from the file.
110
+
111
+ **Rollback.** Router 1.5 rejects unknown keys and makes routing unavailable. Before you install 1.5 again, remove `activities`, `activityRouting`, and `policy.activityMass` from `router.json`. The pane writes them when you set the mode to `off` or `on`, or edit or remove an override.
112
+
62
113
  ## Classifiers
63
114
 
64
- `classifier` names the active entry of `classifiers`. Each entry names its wire protocol in `api`: `system-one` for Jev, Clef and Clef Flash, `openai-decisions` for OpenAI, and `ollama` for a local Ollama server. Every protocol is converted to the same advice, so the routing policy does not change with the classifier. Cloudflare wraps the System One answer in `{ result, success, errors }`; Router reads both forms.
115
+ `classifier` names the active entry of `classifiers`. Each entry names its wire protocol in `api`: `system-one` for Jev, Clef and Clef Flash, `openai-decisions` for OpenAI, and `ollama` for a local Ollama server. Every protocol is converted to the same advice, so the routing policy does not change with the classifier. Unless `activityRouting` is `off`, each one is also asked for the turn's [activity](#activity-routing); the answer is optional, and a missing or malformed one keeps the activity already running. Cloudflare wraps the System One answer in `{ result, success, errors }`; Router reads both forms.
65
116
 
66
117
  | `classifier` | Endpoint | Request `model` | Key option | `timeoutMs` |
67
118
  | ------------ | -------------------------------------------------------------------------------------------------------- | --------------- | ---------------------- | ----------: |
@@ -96,9 +147,9 @@ You can add an entry for another service that speaks one of the three protocols.
96
147
  }
97
148
  ```
98
149
 
99
- The active classifier receives the current prompt and recent dialogue, as described in the [architecture](architecture.md#request-flow). Give a Cloudflare token Workers AI permissions only. The routing policy thresholds were tuned against Jev; another classifier's probabilities can place the same prompt differently.
150
+ The active classifier receives the current prompt and recent dialogue, as described in the [architecture](architecture.md#request-flow). Give a Cloudflare token Workers AI permissions only. The routing policy thresholds were tuned against Jev, and `activityMass` against none; another classifier's probabilities can place the same prompt differently.
100
151
 
101
- To check a classifier outside Claude Code, run `node scripts/probe-classifier.mjs [classifier]` from a checkout. It reads `router.json` and the upper-case variables `TYPESAFE_API_KEY`, `CLOUDFLARE_API_TOKEN`, `CLOUDFLARE_ACCOUNT_ID`, and `OPENAI_API_KEY` from the environment or `./.env`; it does not read plugin options or the `CLAUDE_PLUGIN_OPTION_*` names. It sends the router's real request, and prints the status, latency, and parsed answer. It never prints the key or the full endpoint.
152
+ To check a classifier outside Claude Code, run `node scripts/probe-classifier.mjs [classifier]` from a checkout. It reads `router.json` and the upper-case variables `TYPESAFE_API_KEY`, `CLOUDFLARE_API_TOKEN`, `CLOUDFLARE_ACCOUNT_ID`, and `OPENAI_API_KEY` from the environment or `./.env`; it does not read plugin options or the `CLAUDE_PLUGIN_OPTION_*` names. It sends the router's real request, and prints the status, latency, and parsed answer. It never prints the key or the full endpoint. `node scripts/probe-activity.mjs [classifier]` does the same for the activity question with the [probe set](evaluation.md#activity-probe-set).
102
153
 
103
154
  ### OpenAI
104
155
 
@@ -106,7 +157,7 @@ To check a classifier outside Claude Code, run `node scripts/probe-classifier.mj
106
157
 
107
158
  ### Ollama
108
159
 
109
- `ollama` needs no key and keeps the text on this machine. Run `ollama serve`, pull the model named in `model`, and select the classifier. `model` is any tag you have pulled, such as `qwen3.5:9b`; change it under `classifiers.ollama.model`. The endpoint must stay on `localhost`, `127.0.0.1`, or `[::1]`, so a server on another machine on your network is not reachable through this entry. Each request turns thinking off. If Ollama rejects that setting for a model, the request fails and the router keeps the baseline. Ollama unloads an idle model after its keep-alive period, and the first request after that can exceed `timeoutMs` while the model loads; the router then keeps the baseline for that turn.
160
+ `ollama` needs no key and keeps the text on this machine. Run `ollama serve`, pull the model named in `model`, and select the classifier. `model` is any tag you have pulled, such as `qwen3.5:9b`; change it under `classifiers.ollama.model`. The endpoint must stay on `localhost`, `127.0.0.1`, or `[::1]`, so a server on another machine on your network is not reachable through this entry. Each request turns thinking off. The activity is a second request, sent after the route answer parses and only when at least 300 ms of the deadline remain; if it fails, the turn keeps the running activity and the classifier is not counted as failed. If Ollama rejects that setting for a model, the request fails and the router keeps the baseline. Ollama unloads an idle model after its keep-alive period, and the first request after that can exceed `timeoutMs` while the model loads; the router then keeps the baseline for that turn.
110
161
 
111
162
  ## Optional `router.json`
112
163
 
@@ -121,36 +172,39 @@ Start with the setting you need and leave the rest at defaults. For example, req
121
172
 
122
173
  The Mod validates the whole file. Unknown keys and invalid values make routing unavailable with a general configuration error. Retired gateway keys and the 1.1 `jev` section trigger a migration instruction. The migration command reports an invalid setting path and does not print its value.
123
174
 
124
- | Section | Supported fields | Defaults |
125
- | ------------------ | ------------------------------------------------------------------------------------ | -------------------------------------------------------- |
126
- | `baselineTier` | `micro`, `low`, `medium`, `high` | `low` |
127
- | `routes.<tier>` | `model`, optional `effort` (a level, or `null` for the session effort) | As in the route table above |
128
- | `models.<alias>` | `id`, `input`, optional `output`, `cacheRead`, `contextWindow`, `billing`, `efforts` | Three defaults above |
129
- | `cache` | `writeMultiplier`, `ttlMs`, `warmMarginMs` | `5m: 1.25`, `1h: 2`; `300000` / `3600000` ms; `30000` ms |
130
- | `policy` | Fields below | Values below |
131
- | `classifier` | An id in `classifiers` | `jev` |
132
- | `classifiers.<id>` | `label`, `api`, `endpoint`, `model`, `keyOption`, `timeoutMs` | [Classifiers](#classifiers) above |
133
- | `context` | `recentTurns`, `maxTextChars` | `6`, `1200` |
175
+ | Section | Supported fields | Defaults |
176
+ | ------------------------------ | ------------------------------------------------------------------------------------ | -------------------------------------------------------- |
177
+ | `baselineTier` | `micro`, `low`, `medium`, `high` | `low` |
178
+ | `routes.<tier>` | `model`, optional `effort` (a level, or `null` for the session effort) | As in the route table above |
179
+ | `activityRouting` | `off`, `shadow`, `on` | `shadow` |
180
+ | `activities.<activity>.<tier>` | optional `model` and `effort`, as in `routes` | The overrides in [Activity routing](#activity-routing) |
181
+ | `models.<alias>` | `id`, `input`, optional `output`, `cacheRead`, `contextWindow`, `billing`, `efforts` | Three defaults above |
182
+ | `cache` | `writeMultiplier`, `ttlMs`, `warmMarginMs` | `5m: 1.25`, `1h: 2`; `300000` / `3600000` ms; `30000` ms |
183
+ | `policy` | Fields below | Values below |
184
+ | `classifier` | An id in `classifiers` | `jev` |
185
+ | `classifiers.<id>` | `label`, `api`, `endpoint`, `model`, `keyOption`, `timeoutMs` | [Classifiers](#classifiers) above |
186
+ | `context` | `recentTurns`, `maxTextChars` | `6`, `1200` |
134
187
 
135
188
  Policy defaults:
136
189
 
137
- | Field | Default | Meaning |
138
- | ----------------------- | ------: | -------------------------------------------------------------------------------------------------- |
139
- | `upgradeVotes` | `2` | Consecutive supporting votes before an upgrade. |
140
- | `upgradeBase` | `0.75` | Minimum probability mass required for an upgrade. |
141
- | `upgradeSlope` | `0.15` | Maximum increase to the upgrade bar from estimated switching cost. |
142
- | `upgradePivotUsd` | `0.5` | Cost scale used by the upgrade bar. Must be positive. |
143
- | `jumpConfidence` | `0.95` | Support for a two-tier jump without waiting for votes. |
144
- | `downgradeVotes` | `2` | Consecutive supporting votes before a downgrade. |
145
- | `downgradeMass` | `0.9` | Minimum probability mass required for a downgrade. |
146
- | `downgradeSlope` | `0.08` | Maximum increase to the downgrade bar from estimated switching cost. |
147
- | `downgradePivotUsd` | `0.5` | Cost scale used by the downgrade bar. Must be positive. |
148
- | `downgradeHorizonTurns` | `5` | Later turns included in a downgrade payback estimate. |
149
- | `continuationMass` | `0.7` | Advice probability that keeps the current route for a continuation. |
150
- | `escalationHoldTurns` | `2` | Turns held after a repeated tool error escalates the route. |
151
- | `cashCapUsd` | `2` | Maximum estimated cold cache write for a `credits` model. It does not cap output or session spend. |
152
-
153
- The pane writes a subset of these settings. The **Routing** tab sets `routes`, `baselineTier`, `policy.downgradeVotes`, `policy.downgradeHorizonTurns`, and `policy.cashCapUsd` in one save: it writes only values that differ from the defaults and removes the rest. The **Classifier** tab sets `classifier` and the active classifier's `timeoutMs` at once. **Undo** reverts the settings the last pane write changed. Each save validates the whole file, keeps other keys, and applies from the next turn. A failed check names the setting and leaves the file unchanged. The pane refuses to write through a symlink. Model aliases and the other settings require a direct edit.
190
+ | Field | Default | Meaning |
191
+ | ----------------------- | ------: | --------------------------------------------------------------------------------------------------------- |
192
+ | `upgradeVotes` | `2` | Consecutive supporting votes before an upgrade; one before a history's first measured reply. |
193
+ | `upgradeBase` | `0.75` | Minimum probability mass required for an upgrade. |
194
+ | `upgradeSlope` | `0.15` | Maximum increase to the upgrade bar from estimated switching cost. |
195
+ | `upgradePivotUsd` | `0.5` | Cost scale used by the upgrade bar. Must be positive. |
196
+ | `jumpConfidence` | `0.95` | Support for a two-tier jump without waiting for votes. |
197
+ | `downgradeVotes` | `2` | Consecutive supporting votes before a downgrade; one before a history's first measured reply. |
198
+ | `downgradeMass` | `0.9` | Minimum probability mass required for a downgrade. |
199
+ | `downgradeSlope` | `0.08` | Maximum increase to the downgrade bar from estimated switching cost. |
200
+ | `downgradePivotUsd` | `0.5` | Cost scale used by the downgrade bar. Must be positive. |
201
+ | `downgradeHorizonTurns` | `5` | Later turns included in a downgrade payback estimate. |
202
+ | `continuationMass` | `0.7` | Advice probability that keeps the current route for a continuation. |
203
+ | `escalationHoldTurns` | `2` | Turns held after a repeated tool error escalates the route. |
204
+ | `cashCapUsd` | `2` | Maximum estimated cold cache write for a `credits` model. It does not cap output or session spend. |
205
+ | `activityMass` | `0.6` | Minimum probability for the classifier's activity to apply. Not tuned on any classifier. Not in the pane. |
206
+
207
+ The pane writes a subset of these settings. The **Routing** tab sets `routes`, `baselineTier`, `activities`, `activityRouting`, `policy.downgradeVotes`, `policy.downgradeHorizonTurns`, and `policy.cashCapUsd` in one save: it writes only values that differ from the defaults and removes the rest. `/router activities <mode>` writes `activityRouting` at once, with **Undo**. The **Classifier** tab sets `classifier` and the active classifier's `timeoutMs` at once. **Undo** reverts the settings the last pane write changed. Each save validates the whole file, keeps other keys, and applies from the next turn. A failed check names the setting and leaves the file unchanged. The pane refuses to write through a symlink. Model aliases and the other settings require a direct edit.
154
208
 
155
209
  Native cache freshness is unknown after 270 seconds or after a history reset. Claude usage does not report the cache TTL. The policy evaluates price bounds for five-minute and one-hour writes, but the one-hour case is not observed fact. See the [architecture](architecture.md#cache-and-cost) for the estimate rules.
156
210
 
@@ -2,7 +2,7 @@
2
2
 
3
3
  The native Mod has no measured savings result. The pane reports Claude's own usage and configured-price scenarios. It does not maintain a counterfactual bill or a cumulative savings total.
4
4
 
5
- This page separates an older gateway experiment from a shadow replay of native policy. Neither result measures answer quality or proves lower real spend.
5
+ This page separates an older gateway experiment from a shadow replay of native policy. Neither result measures answer quality or proves lower real spend. [Activity routing](#activity-routing) adds two checks of the activity label, with no savings result either.
6
6
 
7
7
  ## Historical gateway experiment
8
8
 
@@ -52,6 +52,57 @@ node --test experiments/mod-router/scripts/trace-eval.test.mjs
52
52
 
53
53
  The script writes aggregate counts, reason histograms, route transitions, and shadow estimates. It excludes prompts, session identifiers, agent names, error text, headers, keys, and filesystem paths from the result. It reads the local profile log and writes the JSON result file.
54
54
 
55
+ ## Activity routing
56
+
57
+ Activity routing has no measured result. The replay above predates it and covers tiers only. The pane counts turns, requests, switches, and the turns `on` would route differently, and it shows no dollar figure for them. Two checks look at the classifier's activity label.
58
+
59
+ ### Tool agreement
60
+
61
+ Each finished turn is labelled from the tools the assistant used, locally and at no cost. Only the bucket and counts are kept; no text or path is stored.
62
+
63
+ | Observed bucket | Rule, first match |
64
+ | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
65
+ | `code` | An edit tool (`Edit`, `Write`, `MultiEdit`, `NotebookEdit`) on a path that is not a doc path |
66
+ | `docs` | Edits only to paths ending `.md`, `.mdx`, `.rst`, or `.txt`, or under a `docs/` directory |
67
+ | `ops` | A Bash command that is not read-only |
68
+ | `read` | Other tool use: reads, searches, web fetches, and Bash commands that only read, such as `ls`, `grep`, `cat`, or `git status` and `git diff` |
69
+ | `talk` | No tool calls |
70
+
71
+ A Bash command counts as read-only only when every command in the pipeline or list is on a short list of readers. Redirects, command substitution, and `find` with `-exec` or `-delete` are not read-only, and neither is any command the list does not name.
72
+
73
+ The classifier's activity agrees with the turn when the bucket is one it allows:
74
+
75
+ | Activity | Allowed buckets |
76
+ | --------- | ---------------------- |
77
+ | `code` | `code` |
78
+ | `debug` | `code`, `ops`, `read` |
79
+ | `explore` | `read`, `talk` |
80
+ | `plan` | `talk`, `read`, `docs` |
81
+ | `review` | `read`, `talk` |
82
+ | `ops` | `ops` |
83
+ | `docs` | `docs` |
84
+
85
+ Agreement is a weak check. It tells `ops` from `code` from `explore` well. It cannot tell `plan` from `review`, and it says nothing about whether the route was good enough. It does catch the costly mistake, a cheap route on a coding turn. The Usage tab shows the session's count as `Agreement`, over the turns where the classifier gave an activity at or above `activityMass`, in `on` too when the route kept another activity. The counts kept across sessions hold the classifier's raw answer, `uncertain` and no answer included, against the bucket. They are not shown in the pane.
86
+
87
+ ### Activity probe set
88
+
89
+ `test/fixtures/activity-probe.json` holds 70 synthetic prompts, ten for each activity, weighted to the boundary cases: "fix the deploy script" against "deploy the fix", "why does the test fail" against "make the test pass", short replies after a dialogue, and a review followed by a fix. It holds no real user text.
90
+
91
+ ```sh
92
+ node scripts/probe-activity.mjs [classifier]
93
+ ```
94
+
95
+ The script sends each prompt to one classifier as the router would ask it, with activity routing forced to `shadow`. It reads credentials as `probe-classifier.mjs` does and needs a classifier you can reach. The run is billed on a paid classifier, and it is run by hand, not in CI. It writes accuracy, an expected-by-answered confusion matrix, p50 and p95 latency against the classifier's deadline, and error counts to `experiments/mod-router/results/activity-probe-<classifier>.json`. For Ollama the latency covers both requests. It never prints a key.
96
+
97
+ Two probe results are checked in, both run on 2026-10-10 against the live Cloudflare API with the synthetic set above:
98
+
99
+ | Classifier | Correct | p50 latency | p95 latency | Result file |
100
+ | ---------- | -------: | ----------: | ----------: | ---------------------------------------------------------------------------------------------------- |
101
+ | Clef | 69 of 70 | 444 ms | 883 ms | [`activity-probe-clef.json`](../experiments/mod-router/results/activity-probe-clef.json) |
102
+ | Clef Flash | 69 of 70 | 470 ms | 874 ms | [`activity-probe-clef-flash.json`](../experiments/mod-router/results/activity-probe-clef-flash.json) |
103
+
104
+ They measure the label on synthetic prompts, not routing quality or savings. Jev, OpenAI and Ollama have no probe result yet. The pane shows no probe accuracy. `policy.activityMass` was not tuned on any classifier; run the probe before you trust it with a new one.
105
+
55
106
  ## Historical method and limits
56
107
 
57
108
  - Costs use configured USD list prices per million tokens. They are not subscription cash charges.