pi-auto-route 1.1.0 → 1.3.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +21 -0
- package/README.md +61 -45
- package/auto-route.ts +190 -48
- package/package.json +4 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,26 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 1.3.1
|
|
4
|
+
|
|
5
|
+
- npm version badge and package link in the README.
|
|
6
|
+
- Releases now publish to npm through trusted publishing (GitHub Actions OIDC): pushing a `v*` tag publishes the matching package.json version, no npm token on disk.
|
|
7
|
+
|
|
8
|
+
## 1.3.0
|
|
9
|
+
|
|
10
|
+
- Bilingual heuristic: every signal regex gained English alternations; the French patterns stay. Paths, code blocks, message length and the terminal-paste penalty were already language-neutral.
|
|
11
|
+
- Hybrid classifier mode: when a classifier model is available — auto-detected across all providers (TypeSafe Jev, OpenRouter decision models, Cloudflare Clef, GitHub Copilot decision models, OpenCode `jev-1.13-free`) — the router asks it one multiple-choice question (light / standard / complex / heavy) and maps the answer straight to the four tiers with their thinking levels. Only the user message is sent, truncated to 16k chars; never the system prompt, history or tool results. Free classifiers are preferred, then the cheaper ones; one classification per new turn, never on retries or continuations. The status line now shows the source: `local` or the classifier id (e.g. `jev-1.13-free`).
|
|
12
|
+
- The zero-requirements install keeps working: confidence below 0.50, a 4-second timeout, any classifier error, or no classifier at all silently falls back to the local bilingual heuristic — no key, no config, no network.
|
|
13
|
+
|
|
14
|
+
## 1.2.0
|
|
15
|
+
|
|
16
|
+
- Economy and quality fixed together, measured on 926 real turns:
|
|
17
|
+
- turns routed to an expensive tier: 104 → 85;
|
|
18
|
+
- heaviest-quartile turns sent to the cheapest tier: 82/232 → 53/232;
|
|
19
|
+
- Spearman correlation with real difficulty: +0.288 → +0.339.
|
|
20
|
+
- The `ACTION` verb list now recognises the spoken French imperatives (`fais`, `faire`, `regarde`, `prend`, `donne`, `met`, `lance`, `écris`, `essaie`, `montre`). They were as frequent as the formal verbs in real prompts but matched nothing, so "fais le portage français" — 49 tool calls, 82k output tokens — scored 0.10 and landed in `eco`.
|
|
21
|
+
- Upper thresholds raised from 0.50/0.70 to 0.60/0.75: a path plus one verb is a routine request and no longer reaches `strong` ("corrige le bug dans src/foo.ts" used to).
|
|
22
|
+
- Tests: 20 cases (the threshold table, the spoken imperatives, and a regression guard on the path-plus-verb over-provisioning).
|
|
23
|
+
|
|
3
24
|
## 1.1.0
|
|
4
25
|
|
|
5
26
|
- Provider-agnostic: the failover now recognizes provider-wide limits from any provider (Anthropic rate limits, Google `RESOURCE_EXHAUSTED`, OpenAI `insufficient_quota`, OpenCode usage limits), not only OpenCode-specific errors.
|
package/README.md
CHANGED
|
@@ -1,85 +1,101 @@
|
|
|
1
1
|
# auto-route
|
|
2
2
|
|
|
3
|
-
A virtual model for [pi](https://github.com/earendil-works/pi-coding-agent) that
|
|
3
|
+
A virtual model for [pi](https://github.com/earendil-works/pi-coding-agent) that sends every request to the right physical model based on how hard the task looks, to cut token spend without losing quality.
|
|
4
4
|
|
|
5
|
-

|
|
5
|
+
[](https://www.npmjs.com/package/pi-auto-route) 
|
|
6
6
|
|
|
7
|
-
**pi-specific.** This extension uses pi's extension API (`pi.registerVirtualModel`, `ModelRegistry`)
|
|
7
|
+
**pi-specific.** This extension uses pi's extension API (`pi.registerVirtualModel`, `ModelRegistry`); it does not port as-is to Claude Code, Codex or other harnesses. See [Portability](#portability).
|
|
8
8
|
|
|
9
|
-
##
|
|
9
|
+
## Install
|
|
10
10
|
|
|
11
|
-
|
|
11
|
+
```bash
|
|
12
|
+
pi install npm:pi-auto-route
|
|
13
|
+
```
|
|
12
14
|
|
|
13
|
-
|
|
14
|
-
- heaviest-quartile messages sent to `eco`: **195/220 → 72/220** (the real win: less under-routing);
|
|
15
|
-
- Spearman score vs. tool calls: +0.16 → **+0.28**;
|
|
16
|
-
- overall quartile agreement: **30 % against 25 % for chance** — the score is an *asymmetric umbrella* (it shields heavy messages from the `eco` tier and at worst bumps simple ones one step), not a precise classifier. Do not overestimate fine-grained prediction.
|
|
17
|
-
2. **Tier choice** inside YOUR real catalog (discovered at runtime via `getAvailable()`, nothing hardcoded):
|
|
18
|
-
- score ≥ 0.7 — heavy work, several signals → most capable tier, thinking `xhigh`
|
|
19
|
-
- score ≥ 0.5 → strong tier, thinking `high`
|
|
20
|
-
- score ≥ 0.3 → standard tier, thinking `medium`
|
|
21
|
-
- score < 0.3 — short follow-ups, quick questions → lightest tier, thinking `low`
|
|
22
|
-
3. **One switch per turn**: the starting model plans and makes the first `edit`/`write`; the rest of the turn continues on the lightest model. The prompt cache breaks only once per turn.
|
|
23
|
-
4. **Compaction and invisible summaries** always go to the lightest model — the token leak nobody sees.
|
|
24
|
-
5. **Automatic failover instead of a crash**: a timeout or overload (429/5xx) moves to the next model of the ladder; a **429 "provider quota"** error (which pi treats as non-retryable) is rewritten into a retryable one, and the router then avoids the **whole provider** — the turn continues instead of dying. Permanent errors (400/404/410) ban the model through the denylist. See `failover.test.ts`.
|
|
15
|
+
Then in pi: `/reload`, and pick **auto → auto** in the model selector (`/model`). It shows up as "Auto (routing + thinking + failover)".
|
|
25
16
|
|
|
26
|
-
|
|
17
|
+
- update: `pi update --extensions`
|
|
18
|
+
- remove: `pi remove npm:pi-auto-route`
|
|
19
|
+
- without npm: `pi install git:github.com/rzph/pi-auto-route`, or copy `auto-route.ts` into `~/.pi/agent/extensions/` by hand
|
|
27
20
|
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
- **`contributor`** models (OpenCode endpoints that train on your requests);
|
|
32
|
-
- retired or deprecated generations: `gemini-2.5-*` (Google 404), `mimo-v2.5` (OpenCode 410);
|
|
33
|
-
- specialized non-code models (`computer-use`, `robotics`, `deep-research`);
|
|
34
|
-
- anything listed in pi's agent dir as `auto-route-denylist.json` (`$PI_CODING_AGENT_DIR` if set, else `~/.pi/agent/`) — banned automatically and **persistently** as soon as it returns a permanent error (400/404/410, "trains on request data", unsupported region…). Delete an entry from the file to restore a model.
|
|
21
|
+
Selecting `auto` starts at thinking `low`. That level is a **ceiling**, not a target: raise it with `shift+tab` when you want the router to think harder.
|
|
22
|
+
|
|
23
|
+
## Usage
|
|
35
24
|
|
|
36
|
-
|
|
25
|
+
Once selected, you do nothing else. On every turn the router picks a model on its own and shows one discreet status line: `⚡ provider/model · thinking · local` — the last tag is the source of the difficulty call, `local` or the classifier id (e.g. `jev-1.13-free`).
|
|
26
|
+
|
|
27
|
+
Three things worth knowing up front:
|
|
28
|
+
|
|
29
|
+
- **Language.** The local heuristic reads both French and English: every signal regex carries French and English alternations. When a classifier model is available on any of your providers, routing uses it instead — classifiers are language-agnostic. The regex path stays as the zero-dependency fallback.
|
|
30
|
+
- **One switch per turn.** The first model plans and makes the first `edit`/`write`; the rest of the turn runs on the cheapest tier. The prompt cache therefore breaks only once per turn, which is why the trade-off favours turns with several tool calls.
|
|
31
|
+
- **You stay in control.** `/model` to any real model takes over immediately; switch back to `auto` on the next turn to resume routing. To force one model for a single turn, switch away and back.
|
|
37
32
|
|
|
38
33
|
## Requirements
|
|
39
34
|
|
|
40
35
|
- pi with at least one provider that has valid credentials. Nothing is hardcoded to a specific provider: the candidate catalog, the tiers and the failover are all built from your own model registry at runtime.
|
|
41
|
-
- No API key of its own
|
|
36
|
+
- No API key of its own: without a classifier, the difficulty heuristic runs entirely locally, with no network calls.
|
|
37
|
+
- A classifier model from any provider is optional and auto-detected; without one the local heuristic is used, so no extra credential is ever required.
|
|
42
38
|
- Only to run the test suite: Node.js 22.6 or newer (`node --test *.test.ts`).
|
|
43
39
|
|
|
44
|
-
##
|
|
45
|
-
|
|
46
|
-
```bash
|
|
47
|
-
pi install npm:pi-auto-route
|
|
48
|
-
```
|
|
40
|
+
## How it works
|
|
49
41
|
|
|
50
|
-
|
|
42
|
+
On each turn, in order:
|
|
51
43
|
|
|
52
|
-
|
|
44
|
+
1. **Score the message.** The router tries a classifier first. Any classifier model in your registry is auto-detected across all providers (TypeSafe Jev, OpenRouter decision models, Cloudflare Clef, GitHub Copilot decision models, OpenCode `jev-1.13-free`…); free ones are preferred, then the cheaper ones. The router asks it one multiple-choice question on your last message — light, standard, complex or heavy — sending only that message, truncated to 16k characters: never the system prompt, the history or tool results. One classification runs per new turn, never on retries or continuations. Confidence below 0.50, a 4-second timeout, any classifier error, or no classifier at all falls back silently to the local heuristic. That heuristic is now bilingual: regex patterns read your last message in French and English and produce a score from 0 to 1. Signals of real work: a path or repo, action verbs, explicit intent, a conceptual question, code blocks, length. A terminal paste (three or more `$`/`sudo` lines) is penalised — a wall of shell output is not a hard task.
|
|
45
|
+
2. **Pick a tier.** Your model catalog is sorted by output cost, used as a proxy for capacity (ties broken by the larger context window), and split into four tiers:
|
|
46
|
+
- score ≥ 0.75 — heavy work, several independent signals → most capable tier, thinking `xhigh`
|
|
47
|
+
- score ≥ 0.6 → strong tier, thinking `high`
|
|
48
|
+
- score ≥ 0.3 → standard tier, thinking `medium`
|
|
49
|
+
- score < 0.3 — short follow-ups, quick questions → lightest tier, thinking `low`
|
|
53
50
|
|
|
51
|
+
The two upper thresholds are deliberately high: a path plus one verb is a routine request, not a hard one, and should not buy an expensive model. The thresholds apply to the local score only; a classifier answer (light / standard / complex / heavy) maps straight onto the same four tiers with their thinking levels.
|
|
52
|
+
3. **Hand off after the first edit.** The chosen model plans and makes the first `edit`/`write`; the turn continues on the lightest tier (see Usage).
|
|
53
|
+
4. **Compaction and invisible summaries** always go to the lightest model — the token leak nobody sees.
|
|
54
|
+
5. **Failover instead of a crash.** A timeout or overload (429/5xx) moves to the next model of the ladder. A **429 provider-quota** error — which pi treats as non-retryable, so the turn would die — is rewritten into a retryable one, and the router then avoids the **whole provider**: the turn continues on another provider instead. Permanent errors (400/404/410, and messages like "trains on request data" or "unsupported region") ban the model persistently. See `failover.test.ts`.
|
|
54
55
|
|
|
55
|
-
|
|
56
|
-
# for the session:
|
|
57
|
-
pi -e /path/to/auto-route.ts
|
|
56
|
+
### Excluded from the candidates
|
|
58
57
|
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
58
|
+
- **free** models (tight quotas, constant timeouts);
|
|
59
|
+
- **non-reasoning** models;
|
|
60
|
+
- any model whose provider has no valid credentials;
|
|
61
|
+
- **`contributor`** models (OpenCode endpoints that train on your requests);
|
|
62
|
+
- retired or deprecated generations: `gemini-2.5-*` (Google 404), `mimo-v2.5` (OpenCode 410);
|
|
63
|
+
- specialized non-code models (`computer-use`, `robotics`, `deep-research`);
|
|
64
|
+
- anything in pi's agent dir as `auto-route-denylist.json` (`$PI_CODING_AGENT_DIR` if set, else `~/.pi/agent/`) — models banned automatically and **persistently** as soon as they return a permanent error. Delete an entry from the file to restore a model.
|
|
62
65
|
|
|
63
|
-
|
|
66
|
+
A model is excluded as free only when its pricing says 0; missing pricing means unknown and stays a candidate, ranked at a mid cost. An empty tier falls back to the next one; if everything fails, you get an explicit error.
|
|
64
67
|
|
|
65
68
|
## Customize
|
|
66
69
|
|
|
67
|
-
Everything is set at the top of
|
|
70
|
+
Everything is set at the top of `auto-route.ts`:
|
|
68
71
|
|
|
69
72
|
- `difficultyScore()`: the heuristic (signals plus length thresholds) — each signal is one commented regex; recalibrate and run `node --test *.test.ts` after any change (`heuristics.test.ts` for the score, `failover.test.ts` for the 429 handling and the default thinking);
|
|
70
73
|
- `tierAndThinking()`: the score → tier + thinking mapping;
|
|
71
74
|
- the filter in `candidateModels()` if you want free or non-reasoning models back.
|
|
72
75
|
|
|
73
|
-
|
|
76
|
+
## Calibration details
|
|
77
|
+
|
|
78
|
+
The heuristic is a hand-written rule, not a trained model; the numbers below only measure it, and they measure the local fallback path. They come from 926 real turns taken from local session logs (see `heuristics.test.ts`), scored against a real-difficulty proxy (the combined rank of a turn's tool calls and output tokens).
|
|
79
|
+
|
|
80
|
+
Current rule against the previous one:
|
|
81
|
+
|
|
82
|
+
- heaviest-quartile turns sent to the `eco` tier: **82/232 → 53/232** — fewer heavy turns left on the cheapest model;
|
|
83
|
+
- turns routed to an expensive tier: **104 → 85**;
|
|
84
|
+
- light turns that still reached an expensive tier: **52 → 44**;
|
|
85
|
+
- Spearman correlation between score and real difficulty: **+0.288 → +0.339**.
|
|
86
|
+
|
|
87
|
+
Both directions improved at once because two separate faults were fixed together: the verb list had missed the spoken imperatives (`fais`, `regarde`, `prend`, `donne`, `met`, `lance`), which sent turns like "fais le portage français" — 49 tool calls, 82k output tokens — to `eco`; and the upper thresholds let a path plus one verb reach `strong`. The first version of the rule scored +0.16 with 195/220 of the heaviest turns in `eco`.
|
|
88
|
+
|
|
89
|
+
Read the correlation honestly: the score is an **asymmetric umbrella** — it shields heavy messages from the `eco` tier and at worst bumps a simple one step up. It is not a precise classifier of difficulty, and fine-grained prediction should not be expected from it. Some heavy turns open with a single word ("tous") that no text heuristic can recognise from the message alone; those stay in `eco`. When a classifier model is available, the classifier path is the more precise one and needs no calibration.
|
|
74
90
|
|
|
75
91
|
## Portability
|
|
76
92
|
|
|
77
93
|
- **pi**: supported (this extension).
|
|
78
|
-
- **Claude Code**: not portable; close ideas: `jev-router` (gargpratyush), "model-router" skills
|
|
94
|
+
- **Claude Code**: not portable; close ideas: `jev-router` (gargpratyush), "model-router" skills.
|
|
79
95
|
- **Codex**: not portable; `jev-codex-router` (0xNatoshi): model + thinking per turn.
|
|
80
96
|
- **Hermes**: not portable; `hermes-jev-skills` (kerpopule).
|
|
81
97
|
|
|
82
|
-
The idea (plan strong, execute cheap, one switch per turn,
|
|
98
|
+
The idea (plan strong, execute cheap, one switch per turn, compaction on a light model) ports everywhere; this implementation is tied to pi. Want a port for another harness? Open an issue.
|
|
83
99
|
|
|
84
100
|
## Key concepts
|
|
85
101
|
|
package/auto-route.ts
CHANGED
|
@@ -6,7 +6,13 @@
|
|
|
6
6
|
* free models, training models (`contributor`) and retired generations
|
|
7
7
|
* (`gemini-2.5-*`);
|
|
8
8
|
* - capacity tiers built at runtime (output cost as a proxy);
|
|
9
|
-
* -
|
|
9
|
+
* - difficulty of each NEW turn: ONE classification when a classifier model is
|
|
10
|
+
* available (auto-detected from any provider with working credentials, free
|
|
11
|
+
* ones first); otherwise — or on any failure (no classifier, low
|
|
12
|
+
* confidence, timeout, error) — a local bilingual (FR + EN) heuristic scores
|
|
13
|
+
* the message, as a silent fallback;
|
|
14
|
+
* - difficulty → tier + thinking: heuristic thresholds and classifier answers
|
|
15
|
+
* map onto the same four tiers;
|
|
10
16
|
* - the thinking level chosen in pi (`shift+tab`) acts as a CEILING;
|
|
11
17
|
* - failover on retry (429/5xx) plus a `message_end` listener that bans
|
|
12
18
|
* PERMANENTLY models returning 400/404 (pi does not retry non-429 4xx);
|
|
@@ -15,7 +21,7 @@
|
|
|
15
21
|
* runs instead of letting the turn die;
|
|
16
22
|
* - selecting `auto` starts at thinking `low` by default (ceiling, raise it
|
|
17
23
|
* later with shift+tab);
|
|
18
|
-
* - ONE discreet status line (`⚡ model · thinking`);
|
|
24
|
+
* - ONE discreet status line (`⚡ model · thinking · source`);
|
|
19
25
|
* - one model switch per turn (prompt cache preserved);
|
|
20
26
|
* - compaction/summaries → the lightest model.
|
|
21
27
|
*
|
|
@@ -72,6 +78,8 @@ interface RouteState {
|
|
|
72
78
|
thinking: Thinking;
|
|
73
79
|
/** provider/id entries avoided for THIS turn (transient failures). */
|
|
74
80
|
avoid: string[];
|
|
81
|
+
/** Where the tier decision came from: "local" or the classifier's key. */
|
|
82
|
+
source?: string;
|
|
75
83
|
}
|
|
76
84
|
|
|
77
85
|
function modelKey(m: { provider: string; id: string }): string {
|
|
@@ -85,7 +93,8 @@ function sameState(a: RouteState | undefined, b: RouteState): boolean {
|
|
|
85
93
|
a.providerId === b.providerId &&
|
|
86
94
|
a.thinking === b.thinking &&
|
|
87
95
|
a.avoid.length === b.avoid.length &&
|
|
88
|
-
a.avoid.every((v, i) => v === b.avoid[i])
|
|
96
|
+
a.avoid.every((v, i) => v === b.avoid[i]) &&
|
|
97
|
+
a.source === b.source
|
|
89
98
|
);
|
|
90
99
|
}
|
|
91
100
|
|
|
@@ -119,9 +128,9 @@ function banPermanently(provider: string, modelId: string): void {
|
|
|
119
128
|
}
|
|
120
129
|
}
|
|
121
130
|
|
|
122
|
-
/** Permanent failure (model retired, 404/410, refuses training, unsupported region…). */
|
|
123
|
-
function isPermanentFailure(message: string): boolean {
|
|
124
|
-
return /no longer available|not found|\b404\b|\b410\b|ModelDeprecated|deprecated|discontinued|trains on request data|hosted outside Europe|requires Global regions|does not exist|no API implementation|unsupported/i.test(
|
|
131
|
+
/** Permanent failure (model retired, 404/410, refuses training, unsupported region, disabled model access…). */
|
|
132
|
+
export function isPermanentFailure(message: string): boolean {
|
|
133
|
+
return /no longer available|not found|\b404\b|\b410\b|ModelDeprecated|deprecated|discontinued|trains on request data|hosted outside Europe|requires Global regions|does not exist|no API implementation|unsupported|permission_error|model access is disabled|access to this model|do not have access/i.test(
|
|
125
134
|
message,
|
|
126
135
|
);
|
|
127
136
|
}
|
|
@@ -131,17 +140,28 @@ function isPermanentFailure(message: string): boolean {
|
|
|
131
140
|
* to decide provider-wide avoidance — an account-level limit hits every model
|
|
132
141
|
* of that provider, so failing over within it would fail again immediately.
|
|
133
142
|
*/
|
|
134
|
-
const PROVIDER_LIMIT_RE =
|
|
143
|
+
export const PROVIDER_LIMIT_RE =
|
|
135
144
|
/gousagelimiterror|freeusagelimiterror|usage limit|available balance|credit balance|insufficient_quota|out of budget|out of credit|quota|billing|subscription_sharing|resource_exhausted|rate.?limit|too many requests|\b429\b/i;
|
|
136
145
|
|
|
137
146
|
/**
|
|
138
147
|
* The subset pi classifies as NON-retryable (its retry would never start, so
|
|
139
148
|
* the turn dies). Only those get rewritten into a retryable 429 by
|
|
140
149
|
* `toFailoverRetryMessage`; already-retryable errors (transient rate limits,
|
|
141
|
-
* 5xx…) keep their original text untouched.
|
|
150
|
+
* 5xx…) keep their original text untouched. Two families: provider-wide
|
|
151
|
+
* quota/billing limits, and model access disabled on the account (a 403 the
|
|
152
|
+
* provider never lifts for this key — same account, other models, still
|
|
153
|
+
* usable, so the model alone is avoided and the turn continues elsewhere).
|
|
142
154
|
*/
|
|
143
155
|
const NON_RETRYABLE_QUOTA_RE =
|
|
144
|
-
/gousagelimiterror|freeusagelimiterror|monthly usage limit reached|available balance|credit balance|insufficient_quota|out of budget|out of credit|quota exceeded|exceeded your current quota|billing|subscription_sharing|resource_exhausted/i;
|
|
156
|
+
/gousagelimiterror|freeusagelimiterror|monthly usage limit reached|available balance|insufficient balance|credit balance|insufficient_quota|out of budget|out of credit|quota exceeded|exceeded your current quota|billing|subscription_sharing|resource_exhausted|permission_error|model access is disabled|access to this model|do not have access/i;
|
|
157
|
+
|
|
158
|
+
/**
|
|
159
|
+
* Model-access family (subset of NON_RETRYABLE_QUOTA_RE): the account cannot
|
|
160
|
+
* use THIS model. Unlike a quota, other models of the same provider may still
|
|
161
|
+
* work, so the retry avoids the model alone, not the whole provider.
|
|
162
|
+
*/
|
|
163
|
+
const MODEL_ACCESS_RE =
|
|
164
|
+
/permission_error|model access is disabled|access to this model|do not have access|not enabled on this account/i;
|
|
145
165
|
|
|
146
166
|
/**
|
|
147
167
|
* Turns a provider-quota error into a "429" pi is willing to retry. Pi drops
|
|
@@ -162,9 +182,12 @@ export function toFailoverRetryMessage(msg: AssistantMessage): AssistantMessage
|
|
|
162
182
|
) {
|
|
163
183
|
return undefined;
|
|
164
184
|
}
|
|
185
|
+
const modelAccess = MODEL_ACCESS_RE.test(msg.errorMessage);
|
|
165
186
|
return {
|
|
166
187
|
...msg,
|
|
167
|
-
errorMessage:
|
|
188
|
+
errorMessage: modelAccess
|
|
189
|
+
? `auto-route: 429 model access — ${msg.provider}/${msg.model} not enabled on this account; retrying on another model`
|
|
190
|
+
: `auto-route: 429 provider quota — ${msg.provider}/${msg.model} at limit; retrying on another model`,
|
|
168
191
|
};
|
|
169
192
|
}
|
|
170
193
|
|
|
@@ -219,17 +242,20 @@ function setStatusLine(
|
|
|
219
242
|
ctx: ExtensionContext,
|
|
220
243
|
m: Model<never>,
|
|
221
244
|
thinking: Thinking,
|
|
245
|
+
source?: string,
|
|
222
246
|
): void {
|
|
247
|
+
const shown = source ?? "local";
|
|
223
248
|
try {
|
|
224
249
|
const dim = ctx.ui.theme?.fg?.bind(ctx.ui.theme) ?? ((_: string, s: string) => s);
|
|
225
|
-
ctx.ui.setStatus("auto", dim("dim", `⚡ ${modelKey(m)} · ${thinking}`));
|
|
250
|
+
ctx.ui.setStatus("auto", dim("dim", `⚡ ${modelKey(m)} · ${thinking} · ${shown}`));
|
|
226
251
|
} catch {
|
|
227
|
-
ctx.ui.setStatus("auto", `${modelKey(m)} · ${thinking}`);
|
|
252
|
+
ctx.ui.setStatus("auto", `${modelKey(m)} · ${thinking} · ${shown}`);
|
|
228
253
|
}
|
|
229
254
|
pi.events.emit("auto-route:routed", {
|
|
230
255
|
provider: m.provider,
|
|
231
256
|
modelId: m.id,
|
|
232
257
|
thinking,
|
|
258
|
+
source: shown,
|
|
233
259
|
});
|
|
234
260
|
}
|
|
235
261
|
|
|
@@ -238,10 +264,11 @@ function toRoute(
|
|
|
238
264
|
ctx: ExtensionContext,
|
|
239
265
|
m: Model<never>,
|
|
240
266
|
thinking: Thinking,
|
|
267
|
+
source: string | undefined,
|
|
241
268
|
prevState?: RouteState,
|
|
242
269
|
nextState?: RouteState,
|
|
243
270
|
): ModelRoute<RouteState> {
|
|
244
|
-
setStatusLine(pi, ctx, m, thinking);
|
|
271
|
+
setStatusLine(pi, ctx, m, thinking, source);
|
|
245
272
|
// Only return a new state object when it actually changed (avoids writing a
|
|
246
273
|
// session-tree entry on every continuation request).
|
|
247
274
|
const stateToStore =
|
|
@@ -269,19 +296,29 @@ function toRoute(
|
|
|
269
296
|
/** Reference to a path, file or repo → work on an existing codebase. */
|
|
270
297
|
const PATHY =
|
|
271
298
|
/(?:\/(?:home|data|tmp|etc|usr|opt|var|srv)\/|~\/|\b[\w.+-]+\/[\w.-]+\.[A-Za-z0-9]{1,6}\b|\b[\w.+-]+\/[\w-]*-[\w-]+\b)/;
|
|
272
|
-
/**
|
|
299
|
+
/**
|
|
300
|
+
* Construction / modification / analysis verbs → the agent is about to do work.
|
|
301
|
+
* The colloquial imperatives (fais, regarde, prend, donne, met, lance, écris,
|
|
302
|
+
* essaie, montre) are as frequent as the formal ones in real prompts: leaving
|
|
303
|
+
* them out scored "fais le portage français" at 0.10 and routed a turn of 49
|
|
304
|
+
* tool calls and 82k output tokens to the eco tier. English stems a French
|
|
305
|
+
* pattern already covers (impl-, develop-, gener-, analy-, test, install: the
|
|
306
|
+
* accent classes hold the plain letters too) are not repeated.
|
|
307
|
+
*/
|
|
273
308
|
const ACTION =
|
|
274
|
-
/\b(reprend\w*|construi\w*|impl[ée]ment\w*|cré[ée]r?|cree\w*|g[ée]n[èe]r\w*|d[ée]velop\w*|migre\w*|refais|refaire|transforme\w*|modifi\w*|ajout\w*|supprim\w*|corrige\w*|corrig\w*|install\w*|configure\w*|[ée]crase\w*|r[ée]pare\w*|optimise\w*|nettoie\w*|remplace\w*|relance\w*|trouve\w*|cherch\w*|analy\w*|examin\w*|v[ée]rifi\w*|test\w*)\b/i;
|
|
275
|
-
/** Explicit intent from the requester. */
|
|
276
|
-
const WANT =
|
|
309
|
+
/\b(reprend\w*|construi\w*|impl[ée]ment\w*|cré[ée]r?|cree\w*|g[ée]n[èe]r\w*|d[ée]velop\w*|migre\w*|refais|refaire|transforme\w*|modifi\w*|ajout\w*|supprim\w*|corrige\w*|corrig\w*|install\w*|configure\w*|[ée]crase\w*|r[ée]pare\w*|optimise\w*|nettoie\w*|remplace\w*|relance\w*|trouve\w*|cherch\w*|analy\w*|examin\w*|v[ée]rifi\w*|test\w*|fais\w*|faire|regarde\w*|prends?\b|donnes?\b|mets?\b|lance\w*|[ée]cris\b|essaie\w*|montre\w*|make|do(es)?\b|build|creat\w*|migrat\w*|redo|transform|modif\w*|add\w*|remov\w*|delet\w*|fix\w*|repair|configur\w*|overwrit\w*|optimi[sz]\w*|clean|replac\w*|relaunch|restart|find\w*|search\w*|exam\w*|check\w*|look\w*|take|put|launch\w*|run\b|writes?\b|try\b|show\w*|open\w*|read\w*|updat\w*|refactor\w*)\b/i;
|
|
310
|
+
/** Explicit intent from the requester, in French or English. */
|
|
311
|
+
const WANT =
|
|
312
|
+
/\b(je veux|fais en sorte|fais[- ]moi|il faut|on pourrait|peux[- ]tu|besoin de|i want|make sure|i need|can you|could you|we could|need to|please\b)\b/i;
|
|
277
313
|
/** Conceptual question → asks for a solid explanation. */
|
|
278
314
|
const QUESTION =
|
|
279
|
-
/\b(pourquoi|explique\w*|c'est quoi|c est quoi|comment (?:ça|ca) marche|diff[ée]rence entre|quelle est|quel est|est[- ]ce que)\b/i;
|
|
280
|
-
/** Concept keywords, counted (up to 3). */
|
|
315
|
+
/\b(pourquoi|explique\w*|c'est quoi|c est quoi|comment (?:ça|ca) marche|diff[ée]rence entre|quelle est|quel est|est[- ]ce que|why\b|explain\w*|what (?:is|are|does)|what's|how (?:does|do|to)|difference between|which (?:is|one)|is there)\b/i;
|
|
316
|
+
/** Concept keywords, counted (up to 3); French and English stems. */
|
|
281
317
|
const CONCEPTS =
|
|
282
|
-
/\b(architecture|conception|refactor|d[ée]bogu|debug|
|
|
283
|
-
/** Explicit breakage / regression report. */
|
|
284
|
-
const REGRESSION =
|
|
318
|
+
/\b(architecture|conception|refactor|d[ée]bogu|debug|optimi[sz]|s[ée]curit|security|timing|algorithme|algorithm|compar|pourquoi|analyse|syst[èe]me|system|protocole|protocol|concept|tradeoff|modul\w*|scal\w*)\b/gi;
|
|
319
|
+
/** Explicit breakage / regression report, in French or English. */
|
|
320
|
+
const REGRESSION =
|
|
321
|
+
/\b(pas le m[êe]me|r[ée]gression|cass[ée]|fuit|bloqu[ée]|corriger|not the same|broken|broke\b|leak\w*|blocked|stuck|crash\w*|fails?\b|error\w*|wrong)\b/i;
|
|
285
322
|
/** Shell prompt line (terminal paste: form, not difficulty). */
|
|
286
323
|
const SHELLLINE = /^\s*(?:\$|sudo\s|#\s)/gm;
|
|
287
324
|
|
|
@@ -303,11 +340,14 @@ export function difficultyScore(text: string): number {
|
|
|
303
340
|
}
|
|
304
341
|
|
|
305
342
|
export function tierAndThinking(score: number): { tier: TierName; thinking: Thinking } {
|
|
306
|
-
// Thresholds
|
|
307
|
-
//
|
|
308
|
-
//
|
|
309
|
-
|
|
310
|
-
|
|
343
|
+
// Thresholds raised from 0.50/0.70 to 0.60/0.75 on 926 real turns, after the
|
|
344
|
+
// ACTION regex picked up the colloquial imperatives: with both changes a
|
|
345
|
+
// path + verb pair no longer justifies an expensive tier ("corrige le bug
|
|
346
|
+
// dans src/foo.ts" used to reach strong). Measured against the real-difficulty
|
|
347
|
+
// proxy: expensive routings 104 → 85, heaviest-quartile turns sent to eco
|
|
348
|
+
// 82 → 53, light turns in an expensive tier 52 → 44, Spearman +0.288 → +0.339.
|
|
349
|
+
if (score >= 0.75) return { tier: "frontier", thinking: "xhigh" };
|
|
350
|
+
if (score >= 0.6) return { tier: "strong", thinking: "high" };
|
|
311
351
|
if (score >= 0.3) return { tier: "standard", thinking: "medium" };
|
|
312
352
|
return { tier: "eco", thinking: "low" };
|
|
313
353
|
}
|
|
@@ -316,6 +356,87 @@ function capThinking(auto: Thinking, userSelected: Thinking): Thinking {
|
|
|
316
356
|
return THINKING_ORDER[Math.min(THINKING_ORDER.indexOf(auto), THINKING_ORDER.indexOf(userSelected))];
|
|
317
357
|
}
|
|
318
358
|
|
|
359
|
+
// Hybrid classifier (optional; auto-detected, silent fallback)
|
|
360
|
+
|
|
361
|
+
/** Tier decision handed back by a classifier: target tier plus provenance. */
|
|
362
|
+
export interface TierDecision {
|
|
363
|
+
tier: TierName;
|
|
364
|
+
thinking: Thinking;
|
|
365
|
+
via: string;
|
|
366
|
+
}
|
|
367
|
+
|
|
368
|
+
/** Classifier answer → tier, mirroring `tierAndThinking`'s thresholds. */
|
|
369
|
+
const CHOICE_TO_TIER: Record<string, { tier: TierName; thinking: Thinking }> = {
|
|
370
|
+
light: { tier: "eco", thinking: "low" },
|
|
371
|
+
standard: { tier: "standard", thinking: "medium" },
|
|
372
|
+
complex: { tier: "strong", thinking: "high" },
|
|
373
|
+
heavy: { tier: "frontier", thinking: "xhigh" },
|
|
374
|
+
};
|
|
375
|
+
|
|
376
|
+
/**
|
|
377
|
+
* Asks an available classifier model to sort a turn into a tier: the catalog is
|
|
378
|
+
* auto-detected across every provider with working credentials, free
|
|
379
|
+
* classifiers first, then the cheaper ones. Only the user prompt is sent,
|
|
380
|
+
* truncated to 16k characters — never the system prompt, the history or tool
|
|
381
|
+
* results. The question is asked in English: classifiers are multilingual.
|
|
382
|
+
* Never throws: no classifier, a 4-second timeout, an abort, an unexpected
|
|
383
|
+
* answer or one below 0.50 confidence all return undefined, so the caller
|
|
384
|
+
* silently falls back to the local heuristic.
|
|
385
|
+
*/
|
|
386
|
+
export async function classifyTier(
|
|
387
|
+
ctx: ExtensionContext,
|
|
388
|
+
text: string,
|
|
389
|
+
signal?: AbortSignal,
|
|
390
|
+
): Promise<TierDecision | undefined> {
|
|
391
|
+
try {
|
|
392
|
+
const models = await ctx.modelRegistry.getAvailableOfType("classifier");
|
|
393
|
+
if (!models || models.length === 0) return undefined;
|
|
394
|
+
// Free classifiers first, then ascending input cost; unknown pricing
|
|
395
|
+
// ranks last. Array sort is stable, so the catalog order survives.
|
|
396
|
+
const model = [...models].sort((a, b) => {
|
|
397
|
+
const freeA = a.cost?.input === 0;
|
|
398
|
+
const freeB = b.cost?.input === 0;
|
|
399
|
+
if (freeA !== freeB) return freeA ? -1 : 1;
|
|
400
|
+
return (
|
|
401
|
+
(a.cost?.input ?? Number.POSITIVE_INFINITY) -
|
|
402
|
+
(b.cost?.input ?? Number.POSITIVE_INFINITY)
|
|
403
|
+
);
|
|
404
|
+
})[0];
|
|
405
|
+
if (!model) return undefined;
|
|
406
|
+
const timeoutSignal = AbortSignal.timeout(4_000);
|
|
407
|
+
const result = await ctx.modelRegistry.classify(
|
|
408
|
+
model,
|
|
409
|
+
{
|
|
410
|
+
state: { prompt: text.slice(0, 16_000) },
|
|
411
|
+
questions: {
|
|
412
|
+
difficulty: {
|
|
413
|
+
type: "choice",
|
|
414
|
+
instructions:
|
|
415
|
+
"How demanding is the software engineering work requested in prompt?",
|
|
416
|
+
criteria: {
|
|
417
|
+
light: "A quick question, a short follow-up, or a trivial change",
|
|
418
|
+
standard: "An ordinary feature, fix, review, or explanation",
|
|
419
|
+
complex: "Subtle design, cross-cutting changes, or hard debugging",
|
|
420
|
+
heavy: "Architecture-level rework or a long multi-step implementation",
|
|
421
|
+
},
|
|
422
|
+
},
|
|
423
|
+
},
|
|
424
|
+
},
|
|
425
|
+
{ signal: signal ? AbortSignal.any([signal, timeoutSignal]) : timeoutSignal },
|
|
426
|
+
);
|
|
427
|
+
if (result.stopReason !== "stop") return undefined;
|
|
428
|
+
const answer = result.answers?.difficulty;
|
|
429
|
+
if (answer?.type !== "choice") return undefined;
|
|
430
|
+
const decision = CHOICE_TO_TIER[answer.choice];
|
|
431
|
+
if (!decision) return undefined;
|
|
432
|
+
const probability = answer.probabilities[answer.choice] ?? answer.confidence ?? 0;
|
|
433
|
+
if (probability < 0.5) return undefined;
|
|
434
|
+
return { ...decision, via: modelKey(model) };
|
|
435
|
+
} catch {
|
|
436
|
+
return undefined;
|
|
437
|
+
}
|
|
438
|
+
}
|
|
439
|
+
|
|
319
440
|
// Utilities
|
|
320
441
|
|
|
321
442
|
function lastUserText(messages: readonly Message[]): string {
|
|
@@ -340,26 +461,40 @@ function editedThisTurn(messages: readonly Message[]): boolean {
|
|
|
340
461
|
|
|
341
462
|
/**
|
|
342
463
|
* Shared entry point for a new turn and for the safety fallback of a
|
|
343
|
-
* continuation (current model banned/missing):
|
|
344
|
-
*
|
|
464
|
+
* continuation (current model banned/missing): decide the tier of the current
|
|
465
|
+
* message — ONE classifier call when a classifier is available, the local
|
|
466
|
+
* bilingual heuristic otherwise. `avoid` is empty on a new turn, preserved on
|
|
467
|
+
* a safety re-route.
|
|
345
468
|
*/
|
|
346
|
-
function rerouteCurrentTurn(
|
|
469
|
+
async function rerouteCurrentTurn(
|
|
347
470
|
pi: ExtensionAPI,
|
|
348
471
|
ctx: ExtensionContext,
|
|
349
|
-
request:
|
|
472
|
+
request: ModelRouteRequest<RouteState>,
|
|
350
473
|
avoid: string[],
|
|
351
474
|
userCap: Thinking,
|
|
352
475
|
prevState?: RouteState,
|
|
353
|
-
): ModelRoute<RouteState
|
|
354
|
-
const
|
|
355
|
-
|
|
356
|
-
const
|
|
476
|
+
): Promise<ModelRoute<RouteState>> {
|
|
477
|
+
const text = lastUserText(request.messages);
|
|
478
|
+
// Classifier first; the local bilingual heuristic is the silent fallback.
|
|
479
|
+
const decision = await classifyTier(ctx, text, request.signal);
|
|
480
|
+
let tier: TierName;
|
|
481
|
+
let thinking: Thinking;
|
|
482
|
+
let source: string;
|
|
483
|
+
if (decision) {
|
|
484
|
+
({ tier, thinking } = decision);
|
|
485
|
+
source = decision.via;
|
|
486
|
+
} else {
|
|
487
|
+
({ tier, thinking } = tierAndThinking(difficultyScore(text)));
|
|
488
|
+
source = "local";
|
|
489
|
+
}
|
|
490
|
+
const th = capThinking(thinking, userCap);
|
|
357
491
|
const m = buildTiers(ctx, avoid)[tier];
|
|
358
|
-
return toRoute(pi, ctx, m, th, prevState, {
|
|
492
|
+
return toRoute(pi, ctx, m, th, source, prevState, {
|
|
359
493
|
modelId: m.id,
|
|
360
494
|
providerId: m.provider,
|
|
361
495
|
thinking: th,
|
|
362
496
|
avoid,
|
|
497
|
+
source,
|
|
363
498
|
});
|
|
364
499
|
}
|
|
365
500
|
|
|
@@ -425,11 +560,15 @@ export default function (pi: ExtensionAPI) {
|
|
|
425
560
|
const avoid = [
|
|
426
561
|
...new Set([
|
|
427
562
|
...baseState.avoid,
|
|
428
|
-
//
|
|
429
|
-
//
|
|
430
|
-
|
|
431
|
-
|
|
432
|
-
|
|
563
|
+
// Model access disabled on the account: a per-model problem — other
|
|
564
|
+
// models of the same provider may still work, so avoid the model
|
|
565
|
+
// alone. Provider quota: avoid the whole provider, otherwise the
|
|
566
|
+
// failover lands on another of its models and fails again at once.
|
|
567
|
+
MODEL_ACCESS_RE.test(request.failed.message?.errorMessage ?? "")
|
|
568
|
+
? failedKey
|
|
569
|
+
: PROVIDER_LIMIT_RE.test(request.failed.message?.errorMessage ?? "")
|
|
570
|
+
? request.failed.model.provider
|
|
571
|
+
: failedKey,
|
|
433
572
|
]),
|
|
434
573
|
];
|
|
435
574
|
const tiers = buildTiers(ctx, avoid);
|
|
@@ -441,7 +580,7 @@ export default function (pi: ExtensionAPI) {
|
|
|
441
580
|
.find((m) => m && !avoid.includes(modelKey(m)) && !avoid.includes(m.provider)) ??
|
|
442
581
|
tiers.frontier;
|
|
443
582
|
const th = capThinking(baseState.thinking, userCap);
|
|
444
|
-
return toRoute(pi, ctx, chosen, th, prevState, {
|
|
583
|
+
return toRoute(pi, ctx, chosen, th, baseState.source, prevState, {
|
|
445
584
|
...baseState,
|
|
446
585
|
modelId: chosen.id,
|
|
447
586
|
providerId: chosen.provider,
|
|
@@ -459,7 +598,7 @@ export default function (pi: ExtensionAPI) {
|
|
|
459
598
|
if (editedThisTurn(request.messages)) {
|
|
460
599
|
const eco = buildTiers(ctx, avoid).eco;
|
|
461
600
|
const th = capThinking("medium", userCap);
|
|
462
|
-
return toRoute(pi, ctx, eco, th, prevState, {
|
|
601
|
+
return toRoute(pi, ctx, eco, th, prevState.source, prevState, {
|
|
463
602
|
...prevState,
|
|
464
603
|
modelId: eco.id,
|
|
465
604
|
providerId: eco.provider,
|
|
@@ -482,16 +621,19 @@ export default function (pi: ExtensionAPI) {
|
|
|
482
621
|
| undefined;
|
|
483
622
|
if (m) {
|
|
484
623
|
const th = capThinking(prevState.thinking, userCap);
|
|
485
|
-
return toRoute(pi, ctx, m, th, prevState,
|
|
624
|
+
return toRoute(pi, ctx, m, th, prevState.source, prevState, {
|
|
625
|
+
...prevState,
|
|
626
|
+
thinking: th,
|
|
627
|
+
});
|
|
486
628
|
}
|
|
487
629
|
}
|
|
488
630
|
// Safety fallback: current model banned/missing → re-route the SAME
|
|
489
|
-
// turn (
|
|
490
|
-
return rerouteCurrentTurn(pi, ctx, request, prevState.avoid, userCap, prevState);
|
|
631
|
+
// turn (classifier or local score on the current message, `avoid` kept).
|
|
632
|
+
return await rerouteCurrentTurn(pi, ctx, request, prevState.avoid, userCap, prevState);
|
|
491
633
|
}
|
|
492
634
|
|
|
493
|
-
// NEW TURN (user message)
|
|
494
|
-
return rerouteCurrentTurn(pi, ctx, request, [], userCap, prevState);
|
|
635
|
+
// NEW TURN (user message): one classifier call, else the local heuristic.
|
|
636
|
+
return await rerouteCurrentTurn(pi, ctx, request, [], userCap, prevState);
|
|
495
637
|
},
|
|
496
638
|
});
|
|
497
639
|
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-auto-route",
|
|
3
|
-
"version": "1.1
|
|
3
|
+
"version": "1.3.1",
|
|
4
4
|
"description": "A pi virtual model that routes each turn to the right physical model by task difficulty and adjusts the thinking level.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"pi-package",
|
|
@@ -23,6 +23,9 @@
|
|
|
23
23
|
"CHANGELOG.md",
|
|
24
24
|
"LICENSE"
|
|
25
25
|
],
|
|
26
|
+
"scripts": {
|
|
27
|
+
"test": "node --test *.test.ts"
|
|
28
|
+
},
|
|
26
29
|
"pi": {
|
|
27
30
|
"extensions": [
|
|
28
31
|
"./auto-route.ts"
|