@alexeiled/claude-router 1.22.0 → 1.22.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/plugin.json +1 -1
- package/CHANGELOG.md +11 -0
- package/README.md +7 -7
- package/docs/activity-routing.md +28 -11
- package/docs/architecture.md +28 -16
- package/docs/configuration.md +3 -3
- package/docs/evaluation.md +20 -5
- package/docs/user-guide.md +11 -11
- package/hooks/native-router.mjs +109 -8
- package/lib/activity-stats.mjs +5 -3
- package/lib/activity.mjs +29 -3
- package/lib/band.mjs +7 -2
- package/lib/classifier-apis.mjs +79 -24
- package/lib/classifier-client.mjs +16 -5
- package/lib/classifier-contract.mjs +47 -11
- package/lib/config.mjs +1 -1
- package/lib/decisions.mjs +16 -1
- package/lib/display.mjs +10 -6
- package/lib/facts.mjs +32 -1
- package/lib/panel.mjs +5 -3
- package/lib/route.mjs +20 -6
- package/lib/view.mjs +11 -1
- package/package.json +1 -1
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "router",
|
|
3
3
|
"displayName": "Router",
|
|
4
|
-
"version": "1.22.
|
|
4
|
+
"version": "1.22.1",
|
|
5
5
|
"description": "Native model and effort routing within your provider. Anthropic, DeepSeek, z.ai, MiniMax, MiMo and Kimi defaults, custom provider profiles, classifier advice and one model per subagent.",
|
|
6
6
|
"author": {
|
|
7
7
|
"name": "Alexei Ledenev",
|
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,16 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 1.22.1 (2026-11-13)
|
|
4
|
+
|
|
5
|
+
### Changed
|
|
6
|
+
|
|
7
|
+
- Separate the task goal from the current phase. Implementation stays `code` through verification, commit and reporting; a supported command-only phase can still use `ops` after the normal routing gates. Mixed tests and fixes are not execution-only.
|
|
8
|
+
- Keep a bounded goal in session memory only on explicit new-task/continuation evidence. History-prefix and request-generation fences reject replaced context and stale reviews. Unassociated notifications cannot justify downgrades.
|
|
9
|
+
- Show goal, phase and effective activity separately. Start a separate goal-contract counter series; older turn-label counters remain stored until **Delete history**. Tool compatibility is explicitly not accuracy.
|
|
10
|
+
- Use Jev `jev-1.14.0` by default. Explicit model overrides remain unchanged; Ollama remains goal-only without phase or continuity.
|
|
11
|
+
|
|
12
|
+
No configuration migration is required. Restart the plugin session after updating. Synthetic checks do not establish better real-task quality, calibration or savings.
|
|
13
|
+
|
|
3
14
|
## 1.22.0 (2026-11-11)
|
|
4
15
|
|
|
5
16
|
### Added
|
package/README.md
CHANGED
|
@@ -35,18 +35,18 @@ Execution provider selection is separate from classifier selection. Observed usa
|
|
|
35
35
|
|
|
36
36
|
## Supported classifiers
|
|
37
37
|
|
|
38
|
-
The active classifier is asked at the start of each logical turn, for the tier
|
|
38
|
+
The active classifier is asked at the start of each logical turn, for the tier, task goal and current phase, and again in the background every 4 requests while the turn runs (`policy.reviewEveryRequests`). Pick one on the pane's **Classifier** tab or in the selected router file; only the active one receives prompt text. With subagent routing `on` (the default), it is also asked once per routed subagent, with the task's description and prompt; set `"subagentRouting": "off"` to keep that text local.
|
|
39
39
|
|
|
40
40
|
| Classifier | Service | Credential | Prompt text goes to | How it is asked for the activity |
|
|
41
41
|
| ---------- | ------------------------------------------ | ----------------------------------- | ------------------------- | --------------------------------------------------------------------- |
|
|
42
|
-
| Jev | [typesafe.ai](https://typesafe.ai) | Jev API key | typesafe.ai |
|
|
42
|
+
| Jev | [typesafe.ai](https://typesafe.ai) | Jev API key | typesafe.ai | Goal and phase `choice` questions in the same request |
|
|
43
43
|
| Clef | Cloudflare Workers AI, 27B | Cloudflare API token and account ID | Cloudflare | The same |
|
|
44
44
|
| Clef Flash | Cloudflare Workers AI, 9B | Cloudflare API token and account ID | Cloudflare | The same |
|
|
45
45
|
| Clef Omni | Cloudflare Workers AI, 30B MoE | Cloudflare API token and account ID | Cloudflare | The same |
|
|
46
|
-
| OpenAI | OpenAI, `gpt-6-luna`
|
|
47
|
-
| Ollama | A local [Ollama](https://ollama.com) model | None | Stays on this machine |
|
|
46
|
+
| OpenAI | OpenAI, `gpt-6-luna` | OpenAI API key | OpenAI (`api.openai.com`) | Goal and phase choices in `questions`, in the same request |
|
|
47
|
+
| Ollama | A local [Ollama](https://ollama.com) model | None | Stays on this machine | Goal only in a second request; no phase or continuation answer |
|
|
48
48
|
|
|
49
|
-
Jev is the default. [Configuration](docs/configuration.md#classifiers) gives the endpoints, timeouts, and how to add another service that speaks one of the supported protocols.
|
|
49
|
+
Jev 1.14 is the default; a configured model override stays in force. [Configuration](docs/configuration.md#classifiers) gives the endpoints, timeouts, and how to add another service that speaks one of the supported protocols.
|
|
50
50
|
|
|
51
51
|

|
|
52
52
|
|
|
@@ -81,7 +81,7 @@ sequenceDiagram
|
|
|
81
81
|
Mod-->>Dev: Status band and router pane
|
|
82
82
|
```
|
|
83
83
|
|
|
84
|
-
The active classifier labels each new logical turn, and the router checks the route again while the task runs: a turn labelled `ops` that edits code moves up to the `code` route at once, and a background classifier review every 4 requests can move the tier or the activity either way. A cheaper route on the same task waits for a dwell and a checkpoint its work can show: a passed test, build or lint run, or for commands, docs and read-only work the success the router can see. A
|
|
84
|
+
The active classifier labels each new logical turn, and the router checks the route again while the task runs: a turn labelled `ops` that edits code moves up to the `code` route at once, and a background classifier review every 4 requests can move the tier or the activity either way. A cheaper route on the same task waits for a dwell and a checkpoint its work can show: a passed test, build or lint run, or for commands, docs and read-only work the success the router can see. A notification is treated as progress, but cannot justify a downgrade because it may belong to an older task. `/router decisions` lists why each turn ran where it did. See [Changes during a task](docs/user-guide.md#changes-during-a-task). A local policy applies vote, context, failure, and cache-cost rules before the Mod changes the next step. Claude Code owns request construction, model credentials, tools, streaming, and the API cost ledger.
|
|
85
85
|
|
|
86
86
|
With the Anthropic profile, the base routes are:
|
|
87
87
|
|
|
@@ -94,7 +94,7 @@ With the Anthropic profile, the base routes are:
|
|
|
94
94
|
|
|
95
95
|
These are routing defaults. They do not claim equal model quality. [Configuration](docs/configuration.md#built-in-defaults) gives the published results behind them and the `router.json` lines that restore the 1.3 Sonnet ladder.
|
|
96
96
|
|
|
97
|
-
Activity routing
|
|
97
|
+
Activity routing classifies the task's goal (`code`, `debug`, `explore`, `plan`, `review`, `ops`, or `docs`) separately from the current phase. Bounded verification or execution can use the `ops` route without relabeling the goal. A route is then picked by tier and effective activity. It is on by default since 1.7. A `·` cell uses the tier's own route, the base route:
|
|
98
98
|
|
|
99
99
|
The Anthropic profile uses these activity defaults. Other profiles have [their own defaults](docs/providers.md#activity-overrides).
|
|
100
100
|
|
package/docs/activity-routing.md
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
# Activity routing
|
|
2
2
|
|
|
3
|
-
The router
|
|
4
|
-
|
|
5
|
-
|
|
3
|
+
The router uses **tier** for difficulty, **goal** for the task's final deliverable, and **phase** for the current step.
|
|
4
|
+
The route uses the goal's activity unless a supported verification-only or execution-only phase qualifies for `ops`.
|
|
5
|
+
Activity routing is `on` by default. This page shows the default choices, why each one is there, and how to
|
|
6
6
|
change them.
|
|
7
7
|
|
|
8
8
|
The numbers below come from Anthropic's system cards and from a replay of one developer's sessions at list prices.
|
|
@@ -10,19 +10,36 @@ They do not measure this router's answer quality or your bill.
|
|
|
10
10
|
|
|
11
11
|
## What it does
|
|
12
12
|
|
|
13
|
-
For each new turn
|
|
14
|
-
|
|
13
|
+
For each new turn, Jev, Clef and OpenAI answer tier, continuation, goal and phase questions together. The existing
|
|
14
|
+
`activity` field and route matrix name the goal; the phase can select the `ops` cell for bounded command-only work.
|
|
15
|
+
Ollama asks tier and goal in two calls and does not provide a phase, so it uses the goal route. A route is a model and an effort.
|
|
15
16
|
|
|
16
|
-
|
|
|
17
|
+
| Goal | The task produces |
|
|
17
18
|
| --------- | --------------------------------------------------------------- |
|
|
18
|
-
| `code` | Changed code or config, including tests
|
|
19
|
+
| `code` | Changed code or config, including tests and their verification |
|
|
19
20
|
| `debug` | A found cause: failing tests, stack traces, "why does X..." |
|
|
20
21
|
| `explore` | An answer: explain code, find where something happens, research |
|
|
21
22
|
| `plan` | A decision or plan: design, task breakdown, trade-offs |
|
|
22
23
|
| `review` | Findings about existing code: a PR, a diff, an audit |
|
|
23
|
-
| `ops` |
|
|
24
|
+
| `ops` | Specified operations as the task's final deliverable |
|
|
24
25
|
| `docs` | Prose for people: README, docs, comments, commit messages |
|
|
25
26
|
|
|
27
|
+
The phases are `produce`, `reason`, `verify`, `execute`, `report` and `uncertain`. A request to implement
|
|
28
|
+
an endpoint and run tests has goal `code`; “run the tests” can have phase `verify` while the goal stays `code`.
|
|
29
|
+
Tests **and fixes** are not verification-only. A terminal command by itself does not make the task an `ops` goal.
|
|
30
|
+
|
|
31
|
+
When both the goal and a `verify` or `execute` phase meet `policy.activityMass`, the router considers the
|
|
32
|
+
`ops` cell. It uses the lower of their supports for the existing activity-switch gates; it does not multiply them
|
|
33
|
+
or claim they are independently calibrated. A missing, malformed, uncertain or weak phase keeps the goal route.
|
|
34
|
+
The usual cache, dwell, escalation, model availability, context and credits checks still apply. A background phase answer expires when newer messages have arrived since its snapshot; stale bounded-command evidence cannot justify a cheaper step. A successful
|
|
35
|
+
classifier decision displays goal, phase and the route activity separately. Pins ignore activity routing.
|
|
36
|
+
|
|
37
|
+
The router keeps a bounded goal in memory for this session. It sends it as untrusted classifier context; a positively
|
|
38
|
+
identified continuation keeps it, a new task replaces it, and an ambiguous continuation drops it. A progress
|
|
39
|
+
notification uses the active goal but does not replace it or authorize a downgrade, because its originating task is not reliably known. The goal clears on session end, compaction or a detected
|
|
40
|
+
history rewind. A late notification may concern an older task; it is not proof of the current goal.
|
|
41
|
+
Goal memory is created only on positive new-task evidence; an unanchored continuation does not become a new goal. Ollama has no continuity answer and does not retain the anchor. No goal text is written to the activity or decision records. Classifier providers receive the goal and recent text.
|
|
42
|
+
|
|
26
43
|
Most cells use the tier's own route, the **base route**. A few cells have an **override**. When the activity is
|
|
27
44
|
unclear, or the classifier gives it less than 60% probability, the router moves back to the base route if the move
|
|
28
45
|
passes the cache check below. When the classifier gives no activity answer at all, the running activity stays. A pinned
|
|
@@ -32,7 +49,7 @@ The router compares routes, not labels. If the new activity resolves to the rout
|
|
|
32
49
|
A move to a cheaper route must pay back its cache write first. A move to a stronger route needs the classifier's
|
|
33
50
|
support. The [user guide](user-guide.md#route-by-activity) has the full rules.
|
|
34
51
|
|
|
35
|
-
The
|
|
52
|
+
The goal and phase come from the prompt, so they can be wrong. A prompt such as "run the worker brief" reads as `ops` and the
|
|
36
53
|
turn may then spend 80 requests writing code. The router therefore also watches the turn's own tool calls. Once the
|
|
37
54
|
turn edits a file, it moves to the `code` route of its tier (or `docs` for doc files) if that route is stronger. This
|
|
38
55
|
move only goes up. The cache cost does not hold it back; only the cash cap on a credits-billed model can. A later
|
|
@@ -199,7 +216,7 @@ Two other directions, with what they trade:
|
|
|
199
216
|
|
|
200
217
|
**Activity threshold.** An activity applies only when the classifier gives it at least this probability. Below it, a
|
|
201
218
|
new task goes back to the base route if that move passes the usual checks; the running task follows
|
|
202
|
-
[Changes during a task](user-guide.md#changes-during-a-task). The default is 60% (`policy.activityMass: 0.6`). Change it on the Policy tab under **Activities**,
|
|
219
|
+
[Changes during a task](user-guide.md#changes-during-a-task). The default is 60% (`policy.activityMass: 0.6`); the same threshold applies independently to the phase. Change it on the Policy tab under **Activities**,
|
|
203
220
|
or in `router.json`. There is no data to tune it on yet: the probe results record whether the
|
|
204
221
|
label was right, not the classifier's probabilities.
|
|
205
222
|
|
|
@@ -248,7 +265,7 @@ Limits:
|
|
|
248
265
|
|
|
249
266
|
- The answer quality of routed sessions.
|
|
250
267
|
- How well the classifier labels real turns. On 70 synthetic prompts, Jev and OpenAI labelled all 70 right; see
|
|
251
|
-
[Evaluation](evaluation.md#activity-probe-set). The **
|
|
268
|
+
[Evaluation](evaluation.md#activity-probe-set). The **Tool compatibility** lines, on the
|
|
252
269
|
Classifier tab for this session and across sessions, check your own turns against the tools they used.
|
|
253
270
|
- Whether an effort change on the same model always keeps the cache. Claude Code's prompt-caching docs say it does on
|
|
254
271
|
the 5.5 models and Fable 5.1 with an API key or a subscription, and the router prices it so. On 2026-11-11 we counted
|
package/docs/architecture.md
CHANGED
|
@@ -30,19 +30,19 @@ sequenceDiagram
|
|
|
30
30
|
|
|
31
31
|
The Mod does not construct an Anthropic request or proxy its response. Claude Code owns authentication, request fields, streaming, tools, model fallback, and its native usage ledger. No local gateway process or port is part of the runtime. The Ollama classifier calls an Ollama server that you run; the Mod starts none.
|
|
32
32
|
|
|
33
|
-
The active classifier receives
|
|
33
|
+
The active classifier receives the current prompt, up to six recent user/assistant text messages, and optional bounded active-goal and current-step text. Each text is capped at 1,200 characters. System messages, tool inputs, and tool results are not included. Only the active classifier is asked: Jev sends that text to typesafe.ai, Clef, Clef Flash and Clef Omni to Cloudflare Workers AI, OpenAI to api.openai.com. Ollama runs on this machine, so the text stays here. The Jev, Cloudflare, and OpenAI keys are sensitive plugin options; the Cloudflare account ID is a plain one that fills the endpoint. Ollama takes no key. The Mod does not receive Claude's Anthropic credentials.
|
|
34
34
|
|
|
35
|
-
Each classifier names its wire protocol, `api`, and `classifier-apis.mjs` holds one adapter per protocol. `system-one` (Jev, Clef, Clef Flash, Clef Omni) sends the typed `state` and `questions` request and reads `answers.route`; Cloudflare's REST API wraps that answer in `{ result, success, errors }`, which the parser unwraps. `openai-decisions` (OpenAI) sends the state as `input` text and reads a `choice` answer plus a `predicate` for the continuation. `ollama` asks for one letter with thinking off and reads the log-probability of each letter from the top 20 candidates. A letter's probability is its share of the letters found there; a letter missing from the list counts as zero. The Ollama adapter asks no continuation question, so `continuation` is null there. Every adapter returns the
|
|
35
|
+
Each classifier names its wire protocol, `api`, and `classifier-apis.mjs` holds one adapter per protocol. `system-one` (Jev, Clef, Clef Flash, Clef Omni) sends the typed `state` and `questions` request and reads `answers.route`; Cloudflare's REST API wraps that answer in `{ result, success, errors }`, which the parser unwraps. `openai-decisions` (OpenAI) sends the state as `input` text and reads a `choice` answer plus a `predicate` for the continuation. `ollama` asks for one letter with thinking off and reads the log-probability of each letter from the top 20 candidates. A letter's probability is its share of the letters found there; a letter missing from the list counts as zero. The Ollama adapter asks no continuation question, so `continuation` is null there. Every adapter returns the common advice shape. Ollama has neither phase nor continuity; it uses the goal's route and does not retain a cross-turn goal anchor.
|
|
36
36
|
|
|
37
|
-
|
|
37
|
+
Advice carries `activity` (the task goal) and optional `phase`, each with a choice and probabilities. The effective routing activity can differ from the goal for bounded command phases. Protocols ask them independently:
|
|
38
38
|
|
|
39
39
|
| Protocol | How the activity is asked |
|
|
40
40
|
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
41
|
-
| `system-one` |
|
|
42
|
-
| `openai-decisions` |
|
|
41
|
+
| `system-one` | Independent `activity` (goal) and `phase` choice questions in the same request. |
|
|
42
|
+
| `openai-decisions` | Independent goal and phase choices in `questions`. |
|
|
43
43
|
| `ollama` | A second request after the route answer, with the same system message and state prefix so Ollama can reuse its cache. Letters A to H, one token. |
|
|
44
44
|
|
|
45
|
-
The activity is optional at every layer. An adapter reads a missing, malformed, or unknown activity answer as `activity: null` and keeps the strict parsing of the route answer. With `activityRouting: off` no adapter builds the activity question, so the request bodies are the ones 1.5 sent; a fixture test checks this. The criteria and instructions live in `classifier-contract.mjs` beside the tier ones, and
|
|
45
|
+
The activity is optional at every layer. An adapter reads a missing, malformed, or unknown activity answer as `activity: null` and keeps the strict parsing of the route answer. With `activityRouting: off` no adapter builds the activity question, so the request bodies are the ones 1.5 sent; a fixture test checks this. The criteria and instructions live in `classifier-contract.mjs` beside the tier ones, and describe the final task deliverable separately from the current step. Unknown phase answers never invent a command-only route. The classifier criteria keep showing the base route of each tier, not the activity overrides: the tier question is about difficulty.
|
|
46
46
|
|
|
47
47
|
Claude Code builds each request for the model the Mod selects, so no request rewriting is needed: Haiku 4.5 got its own thinking mode and a 32K output cap. On Claude Code 2.1.289, a billed chain of pinned turns (Opus, Haiku 4.5, Sonnet at `xhigh`, Sonnet, Haiku 4.5) used a tool on every turn over the previous models' history, and no request was rejected. Sonnet and Opus accepted histories above 589K input tokens. These checks cover tested requests. They do not prove every Claude Code feature combination.
|
|
48
48
|
|
|
@@ -91,9 +91,9 @@ The router decides on a _cell_: a tier and an activity. The route is `{ ...route
|
|
|
91
91
|
`chooseRoute` decides at a turn's first step, and again on each review answered inside the turn:
|
|
92
92
|
|
|
93
93
|
1. **Pin.** A pinned tier wins, with no activity: the pin's tier route.
|
|
94
|
-
2. **Activity.** Only in `on`; `off` uses none, and ignores the activity a turn in `on` ran with before the mode changed. With no advice, or advice without an activity answer, the incumbent's activity stays. On the same task (`sameTaskOf`;
|
|
94
|
+
2. **Activity.** Only in `on`; derive the route activity from the goal and a supported phase, with bounded verify/execute using `ops`. Missing phase keeps goal routing. Progress origins cannot justify downgrades. `off` uses none, and ignores the activity a turn in `on` ran with before the mode changed. With no advice, or advice without an activity answer, the incumbent's activity stays. On the same task (`sameTaskOf`; reviews and progress origins count as same-task even without a continuation answer, including Ollama) a label whose route at the incumbent tier is stronger applies at once, and a weaker one only with at least `downgradeMass` probability once `facts.downAllowed` holds; an unclear label keeps the running activity. Otherwise the activity with probability at least `activityMass` applies; below that, or for `uncertain`, there is none and the tier's base route is used.
|
|
95
95
|
3. **Tier.** The tier policy runs as before, with every candidate tier priced at its route for the activity and the incumbent at the route it runs. Repeated tool errors move to the lowest tier above whose route is stronger than the running one, or sends a request that cannot be ordered against it. With no activity that is one tier up, as in 1.5. If no tier above helps, nothing escalates and the failure signature is kept.
|
|
96
|
-
4. **Lateral move.** If the tier stays and the route would change, `lateralMove` decides. Routes order by explicit model strength when both models declare it, otherwise by output price, then effort; a session effort counts as the effort Claude Code sends. Two routes that send the same request do not switch. A pair that cannot be ordered (an unpriced model, a numeric effort, two models at one price) counts as a move up, so the base route is always reachable. A stronger route needs the activity's probability to clear the upgrade bar for its switching tax (a base route with no activity counts as probability 1). A cheaper route needs its downgrade tax to be paid back within `downgradeHorizonTurns`, except before the first measured reply of a history, when there is no cache to protect. The cash gate applies to both. The result is `activity-up`, `activity-down`, `activity-pending` (stay on the running route and its activity), `cash-gate`, or `hold`. While an escalation hold lasts, only a stronger route may be taken. One vote is enough
|
|
96
|
+
4. **Lateral move.** If the tier stays and the route would change, `lateralMove` decides. Routes order by explicit model strength when both models declare it, otherwise by output price, then effort; a session effort counts as the effort Claude Code sends. Two routes that send the same request do not switch. A pair that cannot be ordered (an unpriced model, a numeric effort, two models at one price) counts as a move up, so the base route is always reachable. A stronger route needs the activity's probability to clear the upgrade bar for its switching tax (a base route with no activity counts as probability 1). A cheaper route needs its downgrade tax to be paid back within `downgradeHorizonTurns`, except before the first measured reply of a history, when there is no cache to protect. The cash gate applies to both. The result is `activity-up`, `activity-down`, `activity-pending` (stay on the running route and its activity), `cash-gate`, or `hold`. While an escalation hold lasts, only a stronger route may be taken. One vote is enough for a proposed activity cell, but the labels remain model estimates, not observed facts. If the tier changes, the tier gates have already priced the new route, and no lateral rule applies.
|
|
97
97
|
5. **Fit.** Context fit and model availability resolve through `route(tier, activity)`. A fallback tries the last cell, the native model's cell, then each tier with the activity. The last and the native cell count only when they carry no activity or the turn's; otherwise, or when nothing fits, the native model runs at the session effort.
|
|
98
98
|
|
|
99
99
|
**Observed move.** On each later step of the turn, `observedMove` reads the turn's tool calls since its prompt, or since the transcript snapshot of the last review applied inside the turn (`loop.observedFrom`), since the review already judged the earlier ones, (`observedActivity`: `code` after an edit to a non-doc file, `docs` after edits to doc files only, else `ops`, `read` or `talk`). When that activity's route at the decision's tier is stronger than the running one, the remaining steps take it, with reason `activity-up` and `lastActivity` set, so the next turn starts there. The tier is fitted to the context first, so the gates price the route that will run. It goes through `lateralMove` with probability 1: the tools are the evidence, so the switching tax never holds it back (the upgrade bar tops out near 0.9), and only the cash gate can. The turn's reply label (`turnActivities.label`) follows the move; the classifier's answer stays as recorded. It never moves down, and skips pinned turns and `off`. `canMoveUp` skips the transcript read when no activity's route at the tier is stronger, so a step on the strongest route costs nothing. A move to a smaller window waits for a measured context, as in `routeTurn`. Context fit and model availability then apply as for any continuation.
|
|
@@ -124,14 +124,26 @@ Clear, branch, rewind, and committed compaction clear cache evidence and votes.
|
|
|
124
124
|
|
|
125
125
|
### Observed activity and stats
|
|
126
126
|
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
- **Session**: `activityStats` in the view
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
127
|
+
Completed turns have tool buckets (`code`, `docs`, `shell`, `ops`, `check`, `read`, `talk`).
|
|
128
|
+
They describe tools, not the semantic goal. Shell-written code, delegated work and unknown tools limit this oracle.
|
|
129
|
+
[Evaluation](evaluation.md#tool-agreement) describes the many-to-many compatibility check.
|
|
130
|
+
|
|
131
|
+
- **Session**: `activityStats` in the view counts turns, requests, tokens, priced replies/routes, switches and
|
|
132
|
+
tool compatibility. Its activity breakdown is the effective route activity, not necessarily the task goal.
|
|
133
|
+
The `activityContract` marker starts a fresh session counter series when the label contract changes.
|
|
134
|
+
- **Across sessions**: `activity:goals:v1:<session id>` stores the raw goal-vs-tool confusion counts,
|
|
135
|
+
bounded run-length buckets, lateral moves and `since`. Records under the old
|
|
136
|
+
`activity:stats:v1` namespace remain stored but are not merged into this new series. They are removed only
|
|
137
|
+
by **Delete history**. Goal and old turn-label percentages are not one accuracy series.
|
|
138
|
+
- **Metrics**: `activity:metrics:v1:<session id>` retains the existing latency histograms, cheaper moves and
|
|
139
|
+
following escalations, and unpriced replies by cause. These are operational readings, not semantic accuracy.
|
|
140
|
+
|
|
141
|
+
Each key has one session writer. The view's `activityStore` and `activityMetrics` are read at session start
|
|
142
|
+
and when the pane opens, then updated after a turn writes its record. The shared reset marker prevents an
|
|
143
|
+
older held record from undoing another session's reset, subject to the existing millisecond/clock limits.
|
|
144
|
+
**Delete history** clears old and new counters, metrics, usage and subagent records. No prompt text is stored.
|
|
145
|
+
A failed transcript/store read loses that turn's counts and never fails generation. Activity routing off records
|
|
146
|
+
no activity counters.
|
|
135
147
|
|
|
136
148
|
### Routing vs no routing
|
|
137
149
|
|
package/docs/configuration.md
CHANGED
|
@@ -153,7 +153,7 @@ Prices are configured list-price inputs for estimates. They are not a subscripti
|
|
|
153
153
|
|
|
154
154
|
## Activity routing
|
|
155
155
|
|
|
156
|
-
The classifier
|
|
156
|
+
The classifier names the task goal separately from its current phase. The existing `activities` matrix still selects routes; a supported command-only phase can select `ops` without changing the goal. `activityRouting` decides what the router does with it. [Activity routing](activity-routing.md) explains the defaults for end users.
|
|
157
157
|
|
|
158
158
|
| `activityRouting` | Behavior |
|
|
159
159
|
| ----------------- | ------------------------------------------------------------------------------------------ |
|
|
@@ -221,7 +221,7 @@ Switch the mode with `/router activities off|on`, or with the **Activity routing
|
|
|
221
221
|
|
|
222
222
|
| `classifier` | Endpoint | Request `model` | Key option | `timeoutMs` |
|
|
223
223
|
| ------------ | -------------------------------------------------------------------------------------------------------- | --------------- | ---------------------- | ----------: |
|
|
224
|
-
| `jev` | `https://api.typesafe.ai/v1/systemone` | `jev-1.
|
|
224
|
+
| `jev` | `https://api.typesafe.ai/v1/systemone` | `jev-1.14.0` | `typesafe_api_key` | `1500` |
|
|
225
225
|
| `clef` | `https://api.cloudflare.com/client/v4/accounts/{cloudflare_account_id}/ai/run/@cf/cloudflare/clef` | `clef` | `cloudflare_api_token` | `3000` |
|
|
226
226
|
| `clef-flash` | `https://api.cloudflare.com/client/v4/accounts/{cloudflare_account_id}/ai/run/@cf/cloudflare/clef-flash` | `clef-flash` | `cloudflare_api_token` | `3000` |
|
|
227
227
|
| `clef-omni` | `https://api.cloudflare.com/client/v4/accounts/{cloudflare_account_id}/ai/run/@cf/cloudflare/clef-omni` | `clef-omni` | `cloudflare_api_token` | `3000` |
|
|
@@ -245,7 +245,7 @@ You can add an entry for another service that speaks one of the three protocols.
|
|
|
245
245
|
"jev-proxy": {
|
|
246
246
|
"label": "Jev",
|
|
247
247
|
"endpoint": "https://jev-proxy.example.internal/v1/systemone",
|
|
248
|
-
"model": "jev-1.
|
|
248
|
+
"model": "jev-1.14.0",
|
|
249
249
|
"keyOption": "typesafe_api_key",
|
|
250
250
|
"timeoutMs": 1500
|
|
251
251
|
}
|
package/docs/evaluation.md
CHANGED
|
@@ -92,9 +92,9 @@ The script writes aggregate counts, reason histograms, route transitions, and sh
|
|
|
92
92
|
|
|
93
93
|
## Activity routing
|
|
94
94
|
|
|
95
|
-
|
|
95
|
+
Goal/phase routing has no measured answer-quality or savings result. The activity contract changed from a turn's result to a task goal, so older goal/phase and tool-compatibility counts cannot be compared as one accuracy series. The results below are historical turn-label evidence, not evaluation of the new contract. The replay above predates it and covers tiers only. It is `on` by default since 1.7, with defaults chosen from Anthropic's published benchmarks and a list-price replay of one developer's sessions; [Activity routing](activity-routing.md#what-the-estimates-mean) gives that replay and its limits. The pane counts turns, requests, switches, and the turns `on` would route differently, with a configured list-price estimate of those turns' next requests, not a measured saving. Two checks look at the classifier's activity label.
|
|
96
96
|
|
|
97
|
-
The figures plan §11 tracks (tool agreement of at least 80% on `code` against `ops` and `explore`, Jev p95 latency of at most 450 ms, at most 1 tool-error escalation in 10 cheaper activity moves, the number of activity switches) are read from the Classifier tab's **ACROSS SESSIONS** block, which shows the counts kept since the last reset: **Code/ops/explore** for the tool agreement on turns the classifier answered `code`, `ops` or `explore`, **
|
|
97
|
+
The figures plan §11 tracks (tool agreement of at least 80% on `code` against `ops` and `explore`, Jev p95 latency of at most 450 ms, at most 1 tool-error escalation in 10 cheaper activity moves, the number of activity switches) are read from the Classifier tab's **ACROSS SESSIONS** block, which shows the counts kept since the last reset: **Code/ops/explore** for the tool agreement on turns the classifier answered `code`, `ops` or `explore`, **Tool compatibility** and **Mismatch** for all of them, **Activity moves** for moves inside a tier taken and refused (a refusal by a hold or the credits cap included), **Escalations** for the tool-error escalations that followed a cheaper activity move in `on` while the turns stayed on its route in `on` (turning routing off, a turn with routing off, or a turn in `off` ends the watch, a pin in between does not, and an escalation the credits cap held counts), **Unpriced** for the routed replies the usage record left out, by cause: no usage in the reply, a model without a configured price, an unknown cache lifetime, or a failed comparison. Before 1.16 those replies were dropped without a count, so usage covered only the priced part of the routed traffic, and a session whose replies were all unpriced left no usage record at all. The active classifier's p95 wait is on its **Deadline** line on the same tab, above that block. Latency is kept in fixed buckets, so the p95 reads as a bound (`≤ 350 ms`); it is the wait for the classifier's answer, transient retries included, without the routing after it, so it compares with the probe's p95 below, and is not recorded with activity routing `off`. `/router` without a UI surface prints both agreements. The underlying counts are the per-session `activity:goals:v1:<session id>` and `activity:metrics:v1:<session id>` records described in [Architecture](architecture.md).
|
|
98
98
|
|
|
99
99
|
### Tool agreement
|
|
100
100
|
|
|
@@ -126,7 +126,22 @@ The classifier's activity agrees with the turn when the bucket is one it allows:
|
|
|
126
126
|
|
|
127
127
|
Before 1.16 a shell write, a check run and any other command were all `ops`. A `code` turn that edited through the shell therefore read as a miss, and so did a review that ran the tests. The finer buckets keep that apart. Because a shell write cannot be told from routine work, it agrees with every activity that may produce it, so it cannot catch a cheap route on shell-written code. Agreement from before 1.16 and after it is not comparable, and a store holding the new buckets reads as empty in an older version. Routing reads `shell` and `check` as `ops`, as before.
|
|
128
128
|
|
|
129
|
-
|
|
129
|
+
Tool compatibility is a weak check, not classification accuracy or a quality score. A task goal can remain `code` while a step only runs tests, and the tool bucket cannot establish the goal. Shell edits, delegated work and unknown tools create additional blind spots. It cannot reliably identify every cheap route on a coding task. The Classifier tab shows the session's count as `Tool compatibility` under **THIS SESSION**, over the turns where the classifier gave an activity at or above `activityMass`, in `on` too when the route kept another activity. The counts kept across sessions hold the classifier's raw answer, `uncertain` and no answer included, against the bucket. The pane's **ACROSS SESSIONS** block counts agreement over the answers that name an activity; `uncertain` and no answer appear only in **Labelled**.
|
|
130
|
+
|
|
131
|
+
### Goal/phase development checks
|
|
132
|
+
|
|
133
|
+
An earlier 24-case synthetic development pilot separated schema changes from goal-anchor context and compared Jev
|
|
134
|
+
1.13 with 1.14. A work-account Sonnet label check agreed with the 24 supplied goal/phase references, without seeing
|
|
135
|
+
those labels. This is model agreement under a contract, not human ground truth.
|
|
136
|
+
|
|
137
|
+
A later eight-case check through the production Jev 1.14 adapter returned eight usable goal/phase answers and matched
|
|
138
|
+
seven author reference pairs. The missing-brief case returned goal `uncertain`, phase `reason`, where the reference
|
|
139
|
+
phase was `uncertain`. This development result is retained, not relabeled as a success. Subsequent parser, rubric and
|
|
140
|
+
lifecycle changes mean it is not a benchmark of the final release commit.
|
|
141
|
+
|
|
142
|
+
Run `node --env-file=/path/to/private.env test/goal-phase-live.mjs --live` for a billed synthetic check.
|
|
143
|
+
It is not run in CI and sends no private conversation text. Offline regression tests cover the actual request,
|
|
144
|
+
parser, effective route and hook lifecycle. No paired real-task completion, calibration or savings study was run.
|
|
130
145
|
|
|
131
146
|
### Activity probe set
|
|
132
147
|
|
|
@@ -138,7 +153,7 @@ node scripts/probe-activity.mjs [classifier]
|
|
|
138
153
|
|
|
139
154
|
The script sends each prompt to one classifier as the router would ask it, with activity routing forced to `on`. It reads credentials as `probe-classifier.mjs` does and needs a classifier you can reach. The run is billed on a paid classifier, and it is run by hand, not in CI. It writes accuracy, an expected-by-answered confusion matrix, p50 and p95 latency against the classifier's deadline, and error counts to `experiments/mod-router/results/activity-probe-<classifier>.json`. For Ollama the latency covers both requests. It never prints a key.
|
|
140
155
|
|
|
141
|
-
Five probe results are checked in, run on 2026-10-09 and 2026-10-10 (UTC) with the synthetic set above: Jev against the live typesafe.ai API, OpenAI against the live Decisions API, Clef and Clef Flash against the live Cloudflare API, Ollama with `qwen3.5:9b` on an Apple M2 Pro laptop:
|
|
156
|
+
Five historical turn-activity probe results are checked in, run on 2026-10-09 and 2026-10-10 (UTC) with the synthetic set above: Jev against the live typesafe.ai API, OpenAI against the live Decisions API, Clef and Clef Flash against the live Cloudflare API, Ollama with `qwen3.5:9b` on an Apple M2 Pro laptop:
|
|
142
157
|
|
|
143
158
|
| Classifier | Correct | p50 latency | p95 latency | Result file |
|
|
144
159
|
| ---------- | -------: | ----------: | ----------: | ---------------------------------------------------------------------------------------------------- |
|
|
@@ -148,7 +163,7 @@ Five probe results are checked in, run on 2026-10-09 and 2026-10-10 (UTC) with t
|
|
|
148
163
|
| Clef Flash | 69 of 70 | 470 ms | 874 ms | [`activity-probe-clef-flash.json`](../experiments/mod-router/results/activity-probe-clef-flash.json) |
|
|
149
164
|
| Ollama | 66 of 70 | 2986 ms | 3483 ms | [`activity-probe-ollama.json`](../experiments/mod-router/results/activity-probe-ollama.json) |
|
|
150
165
|
|
|
151
|
-
Ollama's latency covers both requests and stays inside its 5000 ms deadline; on slower hardware the activity step is skipped when less than 300 ms remain. They
|
|
166
|
+
Ollama's latency covers both requests and stays inside its 5000 ms deadline; on slower hardware the activity step is skipped when less than 300 ms remain. They measured the old turn-activity contract on synthetic prompts, not the new goal/phase contract, routing quality or savings. The activity question costs little time: 30 alternating pairs of the same prompts, with and without it, gave Jev p50 259 ms both ways and p95 300 ms without, 306 ms with; OpenAI p50 163 ms and 162 ms, p95 296 ms and 349 ms. OpenAI's first request of a run took about 2.3 s of its 3000 ms deadline. The Classifier tab shows each result under its classifier, such as `activity probe 70/70 · p95 299 ms · Oct 10` (UTC date), when the classifier runs the probed model. The script regenerates `lib/probe-results.mjs` from the result files after each run; `npm run probe:results` does it alone, and a test fails when the two disagree. A run without a single activity answer (an activity or `uncertain`) or without a p95 latency, such as one with a revoked key or a model whose answers all lack the activity, exits with an error and writes neither file. `policy.activityMass` was not tuned on any classifier; run the probe before you trust it with a new one. Clef Omni has no probe result yet, so its Classifier tab row shows none.
|
|
152
167
|
|
|
153
168
|
## Routing vs no routing
|
|
154
169
|
|
package/docs/user-guide.md
CHANGED
|
@@ -227,9 +227,9 @@ The router keeps no bill. Its dollars are estimates at configured list prices an
|
|
|
227
227
|
|
|
228
228
|
## Route by activity
|
|
229
229
|
|
|
230
|
-
A tier says how hard the work is. The
|
|
230
|
+
A tier says how hard the work is. The goal says what the active task must deliver, while phase describes this step. The existing activity names describe goals:
|
|
231
231
|
|
|
232
|
-
|
|
|
232
|
+
| Goal | The task produces |
|
|
233
233
|
| --------- | --------------------------------------------------------------- |
|
|
234
234
|
| `code` | Changed code or config, including tests |
|
|
235
235
|
| `debug` | A found cause: failing tests, stack traces, "why does X..." |
|
|
@@ -239,7 +239,7 @@ A tier says how hard the work is. The activity says what kind of work it is. The
|
|
|
239
239
|
| `ops` | Executed commands: git, builds, test runs, CI, deploys |
|
|
240
240
|
| `docs` | Prose for people: README, docs, comments, commit messages |
|
|
241
241
|
|
|
242
|
-
A mixed turn such as "fix it and commit"
|
|
242
|
+
A mixed turn such as "fix it and commit" keeps goal `code`. Verification and delivery steps can retain that goal while changing phase. A positively supported verification-only or execution-only phase can use the `ops` cell; mixed checks and fixes cannot. The router then picks the route by tier and activity. With the default overrides, a `low` coding turn runs on Sonnet 5.5 at `high`, a `low` git turn stays on Haiku 5.5 at `high`, and a `medium` git turn moves from Opus 5.5 to Haiku 5.5 at `high`. [Activity routing](activity-routing.md) explains the default matrix, why each cell is there, and how to change it.
|
|
243
243
|
|
|
244
244
|
The mode decides what happens with the label:
|
|
245
245
|
|
|
@@ -255,13 +255,13 @@ When an override runs, **NEXT TURN** on the Now tab adds a line with the tier's
|
|
|
255
255
|
How the router moves inside a tier:
|
|
256
256
|
|
|
257
257
|
- It compares routes, not labels. If the new activity resolves to the route already running, nothing switches.
|
|
258
|
-
- A move to a stronger route needs the classifier's support, as an upgrade between tiers does. A move to a cheaper one must pay back its cache write within the payback horizon. One vote is enough for either, because the
|
|
259
|
-
-
|
|
258
|
+
- A move to a stronger route needs the classifier's support, as an upgrade between tiers does. A move to a cheaper one must pay back its cache write within the payback horizon. One vote is enough for either, because this proposes a cell for the current step, not a new task goal.
|
|
259
|
+
- A goal applies only when the classifier gives it at least 60% probability (`policy.activityMass`). Phase must independently meet that same bar before a verification/execution step can propose `ops`; its support is capped at the lower of goal and phase support, not a multiplied probability. Below that, or on `uncertain`, a new task goes back to its base route if the move passes the same checks. On the running task the activity can move either way while the turn runs; see [Changes during a task](#changes-during-a-task).
|
|
260
260
|
- A pinned turn uses the pin's base route and shows no activity.
|
|
261
261
|
- When the classifier gives no answer at all (a timeout, a failure, a pause, or a missing key), the running route stays. When it answers the tier but gives no usable activity, only the running activity stays: the tier answer still applies, so the turn can change tier and keeps its activity there. In `off` the base route always runs.
|
|
262
262
|
- A fresh session starts with routing on when the session model is a tier's route. With activity routing `on`, a model that only an activity override uses also counts: Sonnet 5.5 on the defaults. In `off` it does not.
|
|
263
263
|
|
|
264
|
-
The band
|
|
264
|
+
The band names the accepted goal and phase when available, and distinguishes the effective route activity: `code / verify → ops`. These are classifier estimates, not observed proof of the work done. The two-row band and the **Replies** strip show the activities of recent replies.
|
|
265
265
|
|
|
266
266
|
### Changes during a task
|
|
267
267
|
|
|
@@ -269,7 +269,7 @@ The classifier labels a turn from its prompt, and the work can change after that
|
|
|
269
269
|
|
|
270
270
|
- **On every later step of a turn** it reads the turn's tool calls. A turn labelled `ops` or `explore` that edits a code file moves at once to the tier's `code` route (`docs` when it edits only Markdown or `docs/` files). The tools are the evidence, so this needs no classifier support, and the cache cost does not hold it back; only the cash cap on a credits-billed model can. This move only goes up. The toast and the band show it as `↗ coding`.
|
|
271
271
|
- **Every 4 requests inside a turn** (`policy.reviewEveryRequests`) it asks the classifier again, in the background, with the turn's prompt and the recent dialogue. A step never waits for the answer: it applies at the first step after it arrives. Each review sends the classifier the same kind of text a new turn does. `0` turns reviews off.
|
|
272
|
-
- **A new prompt that continues the task** is decided the same way as a review. Claude Code stamps each prompt with where it came from. A task notification,
|
|
272
|
+
- **A new prompt that continues the task** is decided the same way as a review. Claude Code stamps each prompt with where it came from. A task notification, observer report or automatic continuation is treated as progress, not as a new goal. It is asked about the known goal with the report as dialogue. Such a report may belong to an older task, so it cannot justify a downgrade. A message from you or another agent, or "go on", counts as the running task when the classifier scores it as a continuation. After a resume the router does not know your last request yet, so a notification is judged as it reads.
|
|
273
273
|
|
|
274
274
|
On the running task:
|
|
275
275
|
|
|
@@ -282,7 +282,7 @@ On the running task:
|
|
|
282
282
|
|
|
283
283
|
A failed check blocks every one of these, and so does a check whose result is not the check's own (chained, redirected, piped, after `;` or `||`, or in the background). While no checkpoint is met, the route stays: for a tier the band shows `… down after a checkpoint`; a held activity keeps the running one and shows `= fits`. A cheaper activity also needs 90% probability (`policy.downgradeMass`; 65% in `cost`).
|
|
284
284
|
- A cheaper tier lands where the answer as a whole points: when the classifier says `micro` 63% and `low` 30%, the turn moves to `low`, which 93% of the answer supports, instead of staying until `micro` alone reaches 90%.
|
|
285
|
-
- A pinned turn does not move. A
|
|
285
|
+
- A pinned turn does not move. A prompt not scored as a continuation is routed as a fresh turn. The bounded in-memory goal anchor is created or replaced only on positive new-task evidence, retained only on positive continuation, and dropped on ambiguity, history replacement or session end. A continuation without an anchor cannot seed a short reply as a new goal. Ollama provides neither phase nor continuity, so it does not retain this anchor; recent dialogue can still help its goal answer.
|
|
286
286
|
|
|
287
287
|
These thresholds are starting values. They are not tuned on data yet.
|
|
288
288
|
|
|
@@ -293,11 +293,11 @@ On Claude Code 2.1.322 an effort that the router changed inside a turn was not s
|
|
|
293
293
|
- **Now**: the first line of **NEXT TURN** names the classifier's activity with its share and the next answer, such as `code 84% debug 8%`. An `uncertain` answer is shown dim: it applies no activity. In `on`, a line under it names the cell when the route is an override, such as `code at low runs on Sonnet 5.5 · high · tier route Haiku 5.5 · high`, and **why** reads `↑ code` or `↓ ops` when the activity moved the route; a refused move, a hold, or a tier change keeps its reason.
|
|
294
294
|
- **Routes**: see [Edit routes and policy](#edit-routes-and-policy). Each activity is a row of the grid, and a cell shows the model the activity runs at that tier.
|
|
295
295
|
- **Usage**: **BY** with **Activity** lists the period's replies, cost and difference against no routing per activity, and `mostly on`, the route most of its replies used, such as `Sonnet 5.5 · high`; opening a row lists its routes. Replies whose turn had no applied activity count as `no activity`. The **cache rebuilds** row counts the route switches by tier and by activity together. See [Compare routing with no routing](#compare-routing-with-no-routing).
|
|
296
|
-
- **Classifier**: **THIS SESSION** gives **
|
|
296
|
+
- **Classifier**: **THIS SESSION** gives **Tool compatibility (not accuracy)**, how often the tools fit the classifier's goal (see [Evaluation](evaluation.md#tool-agreement)).
|
|
297
297
|
|
|
298
|
-
On the **Classifier** tab, **ACROSS SESSIONS · since the last reset** shows the counts kept across sessions, once there are any: the turns labelled from their tools, **
|
|
298
|
+
On the **Classifier** tab, **ACROSS SESSIONS · since the last reset** shows the counts kept across sessions, once there are any: the turns labelled from their tools, **Tool compatibility** over them, **Code/ops/explore**, the same agreement over the turns the classifier answered `code`, `ops` or `explore` (omitted when there are none), the two most frequent mismatches as `predicted → observed count` (`code → read 3`), **Activity moves**, the moves to another route inside a tier, taken and refused for any reason, **Escalations**, such as `after a cheaper activity move: 1 in 12 moves`, the tool-error escalations that followed a move to a cheaper route in `on` while the turns stayed on it in `on` (turning routing off, a turn with routing off, or a turn in `off` ends the watch, a pin in between does not, and an escalation the credits cap held counts). Each of these lines appears only when it has data. The active classifier's p95 wait across sessions shows on its **Deadline** line instead (not recorded with activity routing `off`). This agreement compares the classifier's raw answer, so it can differ from the session's. The pane reads these counts when the session starts and when it opens, and adds this session's turns as they finish; another session's new latency and escalation readings show the next time the pane opens. **Delete history** on the Usage tab deletes these counts with the usage and subagent records, for this session and across sessions.
|
|
299
299
|
|
|
300
|
-
`/router` without a UI surface
|
|
300
|
+
`/router` without a UI surface distinguishes `Goal: code (95%)`, `Phase: verify` and `Route activity: ops`, followed by the resolved route. It labels tool compatibility explicitly as not accuracy. Goal-contract counters start a separate series; pre-change turn-label counters remain stored but are excluded until **Delete history** removes them. It also prints the session in the band's words, such as `Session: $6.40 billed · −$1.84 (−22%) vs Opus 5.5 $8.24 · 64 replies`, after a `Plan: 5h 48% · 7d 43%` line on a Claude plan.
|
|
301
301
|
|
|
302
302
|
## Change routing
|
|
303
303
|
|