pi-auto-route 1.1.0 → 1.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,21 @@
1
1
  # Changelog
2
2
 
3
+ ## 1.3.0
4
+
5
+ - Bilingual heuristic: every signal regex gained English alternations; the French patterns stay. Paths, code blocks, message length and the terminal-paste penalty were already language-neutral.
6
+ - Hybrid classifier mode: when a classifier model is available — auto-detected across all providers (TypeSafe Jev, OpenRouter decision models, Cloudflare Clef, GitHub Copilot decision models, OpenCode `jev-1.13-free`) — the router asks it one multiple-choice question (light / standard / complex / heavy) and maps the answer straight to the four tiers with their thinking levels. Only the user message is sent, truncated to 16k chars; never the system prompt, history or tool results. Free classifiers are preferred, then the cheaper ones; one classification per new turn, never on retries or continuations. The status line now shows the source: `local` or the classifier id (e.g. `jev-1.13-free`).
7
+ - The zero-requirements install keeps working: confidence below 0.50, a 4-second timeout, any classifier error, or no classifier at all silently falls back to the local bilingual heuristic — no key, no config, no network.
8
+
9
+ ## 1.2.0
10
+
11
+ - Economy and quality fixed together, measured on 926 real turns:
12
+ - turns routed to an expensive tier: 104 → 85;
13
+ - heaviest-quartile turns sent to the cheapest tier: 82/232 → 53/232;
14
+ - Spearman correlation with real difficulty: +0.288 → +0.339.
15
+ - The `ACTION` verb list now recognises the spoken French imperatives (`fais`, `faire`, `regarde`, `prend`, `donne`, `met`, `lance`, `écris`, `essaie`, `montre`). They were as frequent as the formal verbs in real prompts but matched nothing, so "fais le portage français" — 49 tool calls, 82k output tokens — scored 0.10 and landed in `eco`.
16
+ - Upper thresholds raised from 0.50/0.70 to 0.60/0.75: a path plus one verb is a routine request and no longer reaches `strong` ("corrige le bug dans src/foo.ts" used to).
17
+ - Tests: 20 cases (the threshold table, the spoken imperatives, and a regression guard on the path-plus-verb over-provisioning).
18
+
3
19
  ## 1.1.0
4
20
 
5
21
  - Provider-agnostic: the failover now recognizes provider-wide limits from any provider (Anthropic rate limits, Google `RESOURCE_EXHAUSTED`, OpenAI `insufficient_quota`, OpenCode usage limits), not only OpenCode-specific errors.
package/README.md CHANGED
@@ -1,85 +1,101 @@
1
1
  # auto-route
2
2
 
3
- A virtual model for [pi](https://github.com/earendil-works/pi-coding-agent) that routes every turn to the right physical model by task difficulty and adjusts the thinking level, to cut token spend without losing quality.
3
+ A virtual model for [pi](https://github.com/earendil-works/pi-coding-agent) that sends every request to the right physical model based on how hard the task looks, to cut token spend without losing quality.
4
4
 
5
5
  ![harness](https://img.shields.io/badge/harness-pi--coding--agent-blue)
6
6
 
7
- **pi-specific.** This extension uses pi's extension API (`pi.registerVirtualModel`, `ModelRegistry`). It does not port as-is to Claude Code, Codex or other harnesses; see [Portability](#portability) for close equivalents.
7
+ **pi-specific.** This extension uses pi's extension API (`pi.registerVirtualModel`, `ModelRegistry`); it does not port as-is to Claude Code, Codex or other harnesses. See [Portability](#portability).
8
8
 
9
- ## How it works
9
+ ## Install
10
10
 
11
- Pick **auto → auto** in the model selector (`/model`). Then, on every turn:
11
+ ```bash
12
+ pi install npm:pi-auto-route
13
+ ```
12
14
 
13
- 1. **Difficulty estimate** (local heuristic, no external API): a cited path or repo, action verbs ("resume", "modify", "find"…), explicit intent ("I want"), a conceptual question, keyword matches (debugging, architecture…), length, code, a terminal-paste penalty → a continuous score from 0 to 1. Calibrated on 880 real turns (see `heuristics.test.ts`); read the numbers honestly:
14
- - heaviest-quartile messages sent to `eco`: **195/220 → 72/220** (the real win: less under-routing);
15
- - Spearman score vs. tool calls: +0.16 → **+0.28**;
16
- - overall quartile agreement: **30 % against 25 % for chance** — the score is an *asymmetric umbrella* (it shields heavy messages from the `eco` tier and at worst bumps simple ones one step), not a precise classifier. Do not overestimate fine-grained prediction.
17
- 2. **Tier choice** inside YOUR real catalog (discovered at runtime via `getAvailable()`, nothing hardcoded):
18
- - score ≥ 0.7 — heavy work, several signals → most capable tier, thinking `xhigh`
19
- - score ≥ 0.5 → strong tier, thinking `high`
20
- - score ≥ 0.3 → standard tier, thinking `medium`
21
- - score < 0.3 — short follow-ups, quick questions → lightest tier, thinking `low`
22
- 3. **One switch per turn**: the starting model plans and makes the first `edit`/`write`; the rest of the turn continues on the lightest model. The prompt cache breaks only once per turn.
23
- 4. **Compaction and invisible summaries** always go to the lightest model — the token leak nobody sees.
24
- 5. **Automatic failover instead of a crash**: a timeout or overload (429/5xx) moves to the next model of the ladder; a **429 "provider quota"** error (which pi treats as non-retryable) is rewritten into a retryable one, and the router then avoids the **whole provider** — the turn continues instead of dying. Permanent errors (400/404/410) ban the model through the denylist. See `failover.test.ts`.
15
+ Then in pi: `/reload`, and pick **auto → auto** in the model selector (`/model`). It shows up as "Auto (routing + thinking + failover)".
25
16
 
26
- ### Excluded from the candidates
17
+ - update: `pi update --extensions`
18
+ - remove: `pi remove npm:pi-auto-route`
19
+ - without npm: `pi install git:github.com/rzph/pi-auto-route`, or copy `auto-route.ts` into `~/.pi/agent/extensions/` by hand
27
20
 
28
- - **free** models (tight quotas, constant timeouts);
29
- - **non-reasoning** models;
30
- - any model whose provider has no valid credentials;
31
- - **`contributor`** models (OpenCode endpoints that train on your requests);
32
- - retired or deprecated generations: `gemini-2.5-*` (Google 404), `mimo-v2.5` (OpenCode 410);
33
- - specialized non-code models (`computer-use`, `robotics`, `deep-research`);
34
- - anything listed in pi's agent dir as `auto-route-denylist.json` (`$PI_CODING_AGENT_DIR` if set, else `~/.pi/agent/`) — banned automatically and **persistently** as soon as it returns a permanent error (400/404/410, "trains on request data", unsupported region…). Delete an entry from the file to restore a model.
21
+ Selecting `auto` starts at thinking `low`. That level is a **ceiling**, not a target: raise it with `shift+tab` when you want the router to think harder.
22
+
23
+ ## Usage
35
24
 
36
- Capacity ranking uses **output cost** as a proxy (ties broken by the largest contextWindow). A model is excluded as free only when its pricing says 0; missing pricing means unknown and stays a candidate, ranked at a mid cost. An empty tier falls back to the next one; if everything fails, you get an explicit error.
25
+ Once selected, you do nothing else. On every turn the router picks a model on its own and shows one discreet status line: `⚡ provider/model · thinking · local` — the last tag is the source of the difficulty call, `local` or the classifier id (e.g. `jev-1.13-free`).
26
+
27
+ Three things worth knowing up front:
28
+
29
+ - **Language.** The local heuristic reads both French and English: every signal regex carries French and English alternations. When a classifier model is available on any of your providers, routing uses it instead — classifiers are language-agnostic. The regex path stays as the zero-dependency fallback.
30
+ - **One switch per turn.** The first model plans and makes the first `edit`/`write`; the rest of the turn runs on the cheapest tier. The prompt cache therefore breaks only once per turn, which is why the trade-off favours turns with several tool calls.
31
+ - **You stay in control.** `/model` to any real model takes over immediately; switch back to `auto` on the next turn to resume routing. To force one model for a single turn, switch away and back.
37
32
 
38
33
  ## Requirements
39
34
 
40
35
  - pi with at least one provider that has valid credentials. Nothing is hardcoded to a specific provider: the candidate catalog, the tiers and the failover are all built from your own model registry at runtime.
41
- - No API key of its own and no network calls: the difficulty heuristic runs entirely locally.
36
+ - No API key of its own: without a classifier, the difficulty heuristic runs entirely locally, with no network calls.
37
+ - A classifier model from any provider is optional and auto-detected; without one the local heuristic is used, so no extra credential is ever required.
42
38
  - Only to run the test suite: Node.js 22.6 or newer (`node --test *.test.ts`).
43
39
 
44
- ## Install
45
-
46
- ```bash
47
- pi install npm:pi-auto-route
48
- ```
40
+ ## How it works
49
41
 
50
- Then `/reload` and pick **auto → auto** in the model selector (`/model`). Updates come with `pi update --extensions`; removal with `pi remove npm:pi-auto-route`.
42
+ On each turn, in order:
51
43
 
52
- Without npm, a local checkout works too: `pi install git:github.com/rzph/pi-auto-route`, or copy `auto-route.ts` into `~/.pi/agent/extensions/` by hand.
44
+ 1. **Score the message.** The router tries a classifier first. Any classifier model in your registry is auto-detected across all providers (TypeSafe Jev, OpenRouter decision models, Cloudflare Clef, GitHub Copilot decision models, OpenCode `jev-1.13-free`…); free ones are preferred, then the cheaper ones. The router asks it one multiple-choice question on your last message — light, standard, complex or heavy — sending only that message, truncated to 16k characters: never the system prompt, the history or tool results. One classification runs per new turn, never on retries or continuations. Confidence below 0.50, a 4-second timeout, any classifier error, or no classifier at all falls back silently to the local heuristic. That heuristic is now bilingual: regex patterns read your last message in French and English and produce a score from 0 to 1. Signals of real work: a path or repo, action verbs, explicit intent, a conceptual question, code blocks, length. A terminal paste (three or more `$`/`sudo` lines) is penalised — a wall of shell output is not a hard task.
45
+ 2. **Pick a tier.** Your model catalog is sorted by output cost, used as a proxy for capacity (ties broken by the larger context window), and split into four tiers:
46
+ - score ≥ 0.75 — heavy work, several independent signals → most capable tier, thinking `xhigh`
47
+ - score ≥ 0.6 → strong tier, thinking `high`
48
+ - score ≥ 0.3 → standard tier, thinking `medium`
49
+ - score < 0.3 — short follow-ups, quick questions → lightest tier, thinking `low`
53
50
 
51
+ The two upper thresholds are deliberately high: a path plus one verb is a routine request, not a hard one, and should not buy an expensive model. The thresholds apply to the local score only; a classifier answer (light / standard / complex / heavy) maps straight onto the same four tiers with their thinking levels.
52
+ 3. **Hand off after the first edit.** The chosen model plans and makes the first `edit`/`write`; the turn continues on the lightest tier (see Usage).
53
+ 4. **Compaction and invisible summaries** always go to the lightest model — the token leak nobody sees.
54
+ 5. **Failover instead of a crash.** A timeout or overload (429/5xx) moves to the next model of the ladder. A **429 provider-quota** error — which pi treats as non-retryable, so the turn would die — is rewritten into a retryable one, and the router then avoids the **whole provider**: the turn continues on another provider instead. Permanent errors (400/404/410, and messages like "trains on request data" or "unsupported region") ban the model persistently. See `failover.test.ts`.
54
55
 
55
- ```bash
56
- # for the session:
57
- pi -e /path/to/auto-route.ts
56
+ ### Excluded from the candidates
58
57
 
59
- # or permanent (loaded at every pi start):
60
- cp /path/to/auto-route.ts ~/.pi/agent/extensions/auto-route.ts
61
- ```
58
+ - **free** models (tight quotas, constant timeouts);
59
+ - **non-reasoning** models;
60
+ - any model whose provider has no valid credentials;
61
+ - **`contributor`** models (OpenCode endpoints that train on your requests);
62
+ - retired or deprecated generations: `gemini-2.5-*` (Google 404), `mimo-v2.5` (OpenCode 410);
63
+ - specialized non-code models (`computer-use`, `robotics`, `deep-research`);
64
+ - anything in pi's agent dir as `auto-route-denylist.json` (`$PI_CODING_AGENT_DIR` if set, else `~/.pi/agent/`) — models banned automatically and **persistently** as soon as they return a permanent error. Delete an entry from the file to restore a model.
62
65
 
63
- Then `/model` → **auto → auto** (shown as "Auto (routing + thinking + failover)"). Selecting it starts at thinking **`low`** — that level is a ceiling: raise it with `shift+tab` when you want the router to think harder.
66
+ A model is excluded as free only when its pricing says 0; missing pricing means unknown and stays a candidate, ranked at a mid cost. An empty tier falls back to the next one; if everything fails, you get an explicit error.
64
67
 
65
68
  ## Customize
66
69
 
67
- Everything is set at the top of the file:
70
+ Everything is set at the top of `auto-route.ts`:
68
71
 
69
72
  - `difficultyScore()`: the heuristic (signals plus length thresholds) — each signal is one commented regex; recalibrate and run `node --test *.test.ts` after any change (`heuristics.test.ts` for the score, `failover.test.ts` for the 429 handling and the default thinking);
70
73
  - `tierAndThinking()`: the score → tier + thinking mapping;
71
74
  - the filter in `candidateModels()` if you want free or non-reasoning models back.
72
75
 
73
- To force one specific model for a single turn: switch to a real model with `/model`, then back to `auto` on the next turn.
76
+ ## Calibration details
77
+
78
+ The heuristic is a hand-written rule, not a trained model; the numbers below only measure it, and they measure the local fallback path. They come from 926 real turns taken from local session logs (see `heuristics.test.ts`), scored against a real-difficulty proxy (the combined rank of a turn's tool calls and output tokens).
79
+
80
+ Current rule against the previous one:
81
+
82
+ - heaviest-quartile turns sent to the `eco` tier: **82/232 → 53/232** — fewer heavy turns left on the cheapest model;
83
+ - turns routed to an expensive tier: **104 → 85**;
84
+ - light turns that still reached an expensive tier: **52 → 44**;
85
+ - Spearman correlation between score and real difficulty: **+0.288 → +0.339**.
86
+
87
+ Both directions improved at once because two separate faults were fixed together: the verb list had missed the spoken imperatives (`fais`, `regarde`, `prend`, `donne`, `met`, `lance`), which sent turns like "fais le portage français" — 49 tool calls, 82k output tokens — to `eco`; and the upper thresholds let a path plus one verb reach `strong`. The first version of the rule scored +0.16 with 195/220 of the heaviest turns in `eco`.
88
+
89
+ Read the correlation honestly: the score is an **asymmetric umbrella** — it shields heavy messages from the `eco` tier and at worst bumps a simple one step up. It is not a precise classifier of difficulty, and fine-grained prediction should not be expected from it. Some heavy turns open with a single word ("tous") that no text heuristic can recognise from the message alone; those stay in `eco`. When a classifier model is available, the classifier path is the more precise one and needs no calibration.
74
90
 
75
91
  ## Portability
76
92
 
77
93
  - **pi**: supported (this extension).
78
- - **Claude Code**: not portable; close ideas: `jev-router` (gargpratyush), "model-router" skills (Claude Code internals).
94
+ - **Claude Code**: not portable; close ideas: `jev-router` (gargpratyush), "model-router" skills.
79
95
  - **Codex**: not portable; `jev-codex-router` (0xNatoshi): model + thinking per turn.
80
96
  - **Hermes**: not portable; `hermes-jev-skills` (kerpopule).
81
97
 
82
- The idea (plan strong, execute cheap, one switch per turn, degenerate compaction on a light model) ports everywhere; this implementation is tied to pi. Want a port for another harness? Open an issue.
98
+ The idea (plan strong, execute cheap, one switch per turn, compaction on a light model) ports everywhere; this implementation is tied to pi. Want a port for another harness? Open an issue.
83
99
 
84
100
  ## Key concepts
85
101
 
package/auto-route.ts CHANGED
@@ -6,7 +6,13 @@
6
6
  * free models, training models (`contributor`) and retired generations
7
7
  * (`gemini-2.5-*`);
8
8
  * - capacity tiers built at runtime (output cost as a proxy);
9
- * - locally estimated difficulty → tier + thinking;
9
+ * - difficulty of each NEW turn: ONE classification when a classifier model is
10
+ * available (auto-detected from any provider with working credentials, free
11
+ * ones first); otherwise — or on any failure (no classifier, low
12
+ * confidence, timeout, error) — a local bilingual (FR + EN) heuristic scores
13
+ * the message, as a silent fallback;
14
+ * - difficulty → tier + thinking: heuristic thresholds and classifier answers
15
+ * map onto the same four tiers;
10
16
  * - the thinking level chosen in pi (`shift+tab`) acts as a CEILING;
11
17
  * - failover on retry (429/5xx) plus a `message_end` listener that bans
12
18
  * PERMANENTLY models returning 400/404 (pi does not retry non-429 4xx);
@@ -15,7 +21,7 @@
15
21
  * runs instead of letting the turn die;
16
22
  * - selecting `auto` starts at thinking `low` by default (ceiling, raise it
17
23
  * later with shift+tab);
18
- * - ONE discreet status line (`⚡ model · thinking`);
24
+ * - ONE discreet status line (`⚡ model · thinking · source`);
19
25
  * - one model switch per turn (prompt cache preserved);
20
26
  * - compaction/summaries → the lightest model.
21
27
  *
@@ -72,6 +78,8 @@ interface RouteState {
72
78
  thinking: Thinking;
73
79
  /** provider/id entries avoided for THIS turn (transient failures). */
74
80
  avoid: string[];
81
+ /** Where the tier decision came from: "local" or the classifier's key. */
82
+ source?: string;
75
83
  }
76
84
 
77
85
  function modelKey(m: { provider: string; id: string }): string {
@@ -85,7 +93,8 @@ function sameState(a: RouteState | undefined, b: RouteState): boolean {
85
93
  a.providerId === b.providerId &&
86
94
  a.thinking === b.thinking &&
87
95
  a.avoid.length === b.avoid.length &&
88
- a.avoid.every((v, i) => v === b.avoid[i])
96
+ a.avoid.every((v, i) => v === b.avoid[i]) &&
97
+ a.source === b.source
89
98
  );
90
99
  }
91
100
 
@@ -119,9 +128,9 @@ function banPermanently(provider: string, modelId: string): void {
119
128
  }
120
129
  }
121
130
 
122
- /** Permanent failure (model retired, 404/410, refuses training, unsupported region…). */
123
- function isPermanentFailure(message: string): boolean {
124
- return /no longer available|not found|\b404\b|\b410\b|ModelDeprecated|deprecated|discontinued|trains on request data|hosted outside Europe|requires Global regions|does not exist|no API implementation|unsupported/i.test(
131
+ /** Permanent failure (model retired, 404/410, refuses training, unsupported region, disabled model access…). */
132
+ export function isPermanentFailure(message: string): boolean {
133
+ return /no longer available|not found|\b404\b|\b410\b|ModelDeprecated|deprecated|discontinued|trains on request data|hosted outside Europe|requires Global regions|does not exist|no API implementation|unsupported|permission_error|model access is disabled|access to this model|do not have access/i.test(
125
134
  message,
126
135
  );
127
136
  }
@@ -131,17 +140,28 @@ function isPermanentFailure(message: string): boolean {
131
140
  * to decide provider-wide avoidance — an account-level limit hits every model
132
141
  * of that provider, so failing over within it would fail again immediately.
133
142
  */
134
- const PROVIDER_LIMIT_RE =
143
+ export const PROVIDER_LIMIT_RE =
135
144
  /gousagelimiterror|freeusagelimiterror|usage limit|available balance|credit balance|insufficient_quota|out of budget|out of credit|quota|billing|subscription_sharing|resource_exhausted|rate.?limit|too many requests|\b429\b/i;
136
145
 
137
146
  /**
138
147
  * The subset pi classifies as NON-retryable (its retry would never start, so
139
148
  * the turn dies). Only those get rewritten into a retryable 429 by
140
149
  * `toFailoverRetryMessage`; already-retryable errors (transient rate limits,
141
- * 5xx…) keep their original text untouched.
150
+ * 5xx…) keep their original text untouched. Two families: provider-wide
151
+ * quota/billing limits, and model access disabled on the account (a 403 the
152
+ * provider never lifts for this key — same account, other models, still
153
+ * usable, so the model alone is avoided and the turn continues elsewhere).
142
154
  */
143
155
  const NON_RETRYABLE_QUOTA_RE =
144
- /gousagelimiterror|freeusagelimiterror|monthly usage limit reached|available balance|credit balance|insufficient_quota|out of budget|out of credit|quota exceeded|exceeded your current quota|billing|subscription_sharing|resource_exhausted/i;
156
+ /gousagelimiterror|freeusagelimiterror|monthly usage limit reached|available balance|insufficient balance|credit balance|insufficient_quota|out of budget|out of credit|quota exceeded|exceeded your current quota|billing|subscription_sharing|resource_exhausted|permission_error|model access is disabled|access to this model|do not have access/i;
157
+
158
+ /**
159
+ * Model-access family (subset of NON_RETRYABLE_QUOTA_RE): the account cannot
160
+ * use THIS model. Unlike a quota, other models of the same provider may still
161
+ * work, so the retry avoids the model alone, not the whole provider.
162
+ */
163
+ const MODEL_ACCESS_RE =
164
+ /permission_error|model access is disabled|access to this model|do not have access|not enabled on this account/i;
145
165
 
146
166
  /**
147
167
  * Turns a provider-quota error into a "429" pi is willing to retry. Pi drops
@@ -162,9 +182,12 @@ export function toFailoverRetryMessage(msg: AssistantMessage): AssistantMessage
162
182
  ) {
163
183
  return undefined;
164
184
  }
185
+ const modelAccess = MODEL_ACCESS_RE.test(msg.errorMessage);
165
186
  return {
166
187
  ...msg,
167
- errorMessage: `auto-route: 429 provider quota — ${msg.provider}/${msg.model} at limit; retrying on another model`,
188
+ errorMessage: modelAccess
189
+ ? `auto-route: 429 model access — ${msg.provider}/${msg.model} not enabled on this account; retrying on another model`
190
+ : `auto-route: 429 provider quota — ${msg.provider}/${msg.model} at limit; retrying on another model`,
168
191
  };
169
192
  }
170
193
 
@@ -219,17 +242,20 @@ function setStatusLine(
219
242
  ctx: ExtensionContext,
220
243
  m: Model<never>,
221
244
  thinking: Thinking,
245
+ source?: string,
222
246
  ): void {
247
+ const shown = source ?? "local";
223
248
  try {
224
249
  const dim = ctx.ui.theme?.fg?.bind(ctx.ui.theme) ?? ((_: string, s: string) => s);
225
- ctx.ui.setStatus("auto", dim("dim", `⚡ ${modelKey(m)} · ${thinking}`));
250
+ ctx.ui.setStatus("auto", dim("dim", `⚡ ${modelKey(m)} · ${thinking} · ${shown}`));
226
251
  } catch {
227
- ctx.ui.setStatus("auto", `${modelKey(m)} · ${thinking}`);
252
+ ctx.ui.setStatus("auto", `${modelKey(m)} · ${thinking} · ${shown}`);
228
253
  }
229
254
  pi.events.emit("auto-route:routed", {
230
255
  provider: m.provider,
231
256
  modelId: m.id,
232
257
  thinking,
258
+ source: shown,
233
259
  });
234
260
  }
235
261
 
@@ -238,10 +264,11 @@ function toRoute(
238
264
  ctx: ExtensionContext,
239
265
  m: Model<never>,
240
266
  thinking: Thinking,
267
+ source: string | undefined,
241
268
  prevState?: RouteState,
242
269
  nextState?: RouteState,
243
270
  ): ModelRoute<RouteState> {
244
- setStatusLine(pi, ctx, m, thinking);
271
+ setStatusLine(pi, ctx, m, thinking, source);
245
272
  // Only return a new state object when it actually changed (avoids writing a
246
273
  // session-tree entry on every continuation request).
247
274
  const stateToStore =
@@ -269,19 +296,29 @@ function toRoute(
269
296
  /** Reference to a path, file or repo → work on an existing codebase. */
270
297
  const PATHY =
271
298
  /(?:\/(?:home|data|tmp|etc|usr|opt|var|srv)\/|~\/|\b[\w.+-]+\/[\w.-]+\.[A-Za-z0-9]{1,6}\b|\b[\w.+-]+\/[\w-]*-[\w-]+\b)/;
272
- /** Construction / modification / analysis verbs → the agent is about to do work. */
299
+ /**
300
+ * Construction / modification / analysis verbs → the agent is about to do work.
301
+ * The colloquial imperatives (fais, regarde, prend, donne, met, lance, écris,
302
+ * essaie, montre) are as frequent as the formal ones in real prompts: leaving
303
+ * them out scored "fais le portage français" at 0.10 and routed a turn of 49
304
+ * tool calls and 82k output tokens to the eco tier. English stems a French
305
+ * pattern already covers (impl-, develop-, gener-, analy-, test, install: the
306
+ * accent classes hold the plain letters too) are not repeated.
307
+ */
273
308
  const ACTION =
274
- /\b(reprend\w*|construi\w*|impl[ée]ment\w*|cré[ée]r?|cree\w*|g[ée]n[èe]r\w*|d[ée]velop\w*|migre\w*|refais|refaire|transforme\w*|modifi\w*|ajout\w*|supprim\w*|corrige\w*|corrig\w*|install\w*|configure\w*|[ée]crase\w*|r[ée]pare\w*|optimise\w*|nettoie\w*|remplace\w*|relance\w*|trouve\w*|cherch\w*|analy\w*|examin\w*|v[ée]rifi\w*|test\w*)\b/i;
275
- /** Explicit intent from the requester. */
276
- const WANT = /\b(je veux|fais en sorte|fais[- ]moi|il faut|on pourrait|peux[- ]tu|besoin de)\b/i;
309
+ /\b(reprend\w*|construi\w*|impl[ée]ment\w*|cré[ée]r?|cree\w*|g[ée]n[èe]r\w*|d[ée]velop\w*|migre\w*|refais|refaire|transforme\w*|modifi\w*|ajout\w*|supprim\w*|corrige\w*|corrig\w*|install\w*|configure\w*|[ée]crase\w*|r[ée]pare\w*|optimise\w*|nettoie\w*|remplace\w*|relance\w*|trouve\w*|cherch\w*|analy\w*|examin\w*|v[ée]rifi\w*|test\w*|fais\w*|faire|regarde\w*|prends?\b|donnes?\b|mets?\b|lance\w*|[ée]cris\b|essaie\w*|montre\w*|make|do(es)?\b|build|creat\w*|migrat\w*|redo|transform|modif\w*|add\w*|remov\w*|delet\w*|fix\w*|repair|configur\w*|overwrit\w*|optimi[sz]\w*|clean|replac\w*|relaunch|restart|find\w*|search\w*|exam\w*|check\w*|look\w*|take|put|launch\w*|run\b|writes?\b|try\b|show\w*|open\w*|read\w*|updat\w*|refactor\w*)\b/i;
310
+ /** Explicit intent from the requester, in French or English. */
311
+ const WANT =
312
+ /\b(je veux|fais en sorte|fais[- ]moi|il faut|on pourrait|peux[- ]tu|besoin de|i want|make sure|i need|can you|could you|we could|need to|please\b)\b/i;
277
313
  /** Conceptual question → asks for a solid explanation. */
278
314
  const QUESTION =
279
- /\b(pourquoi|explique\w*|c'est quoi|c est quoi|comment (?:ça|ca) marche|diff[ée]rence entre|quelle est|quel est|est[- ]ce que)\b/i;
280
- /** Concept keywords, counted (up to 3). */
315
+ /\b(pourquoi|explique\w*|c'est quoi|c est quoi|comment (?:ça|ca) marche|diff[ée]rence entre|quelle est|quel est|est[- ]ce que|why\b|explain\w*|what (?:is|are|does)|what's|how (?:does|do|to)|difference between|which (?:is|one)|is there)\b/i;
316
+ /** Concept keywords, counted (up to 3); French and English stems. */
281
317
  const CONCEPTS =
282
- /\b(architecture|conception|refactor|d[ée]bogu|debug|optimis|s[ée]curit|timing|algorithme|compar|pourquoi|analyse|syst[èe]me|protocole)\b/gi;
283
- /** Explicit breakage / regression report. */
284
- const REGRESSION = /\b(pas le m[êe]me|r[ée]gression|cass[ée]|fuit|bloqu[ée]|corriger)\b/i;
318
+ /\b(architecture|conception|refactor|d[ée]bogu|debug|optimi[sz]|s[ée]curit|security|timing|algorithme|algorithm|compar|pourquoi|analyse|syst[èe]me|system|protocole|protocol|concept|tradeoff|modul\w*|scal\w*)\b/gi;
319
+ /** Explicit breakage / regression report, in French or English. */
320
+ const REGRESSION =
321
+ /\b(pas le m[êe]me|r[ée]gression|cass[ée]|fuit|bloqu[ée]|corriger|not the same|broken|broke\b|leak\w*|blocked|stuck|crash\w*|fails?\b|error\w*|wrong)\b/i;
285
322
  /** Shell prompt line (terminal paste: form, not difficulty). */
286
323
  const SHELLLINE = /^\s*(?:\$|sudo\s|#\s)/gm;
287
324
 
@@ -303,11 +340,14 @@ export function difficultyScore(text: string): number {
303
340
  }
304
341
 
305
342
  export function tierAndThinking(score: number): { tier: TierName; thinking: Thinking } {
306
- // Thresholds calibrated on the 880 real turns (score → real-difficulty
307
- // quartile table: 0.10-0.20 ≈ quartile 1, 0.30-0.45 ≈ quartile 2,
308
- // ≥0.55 ≈ quartile 3).
309
- if (score >= 0.7) return { tier: "frontier", thinking: "xhigh" };
310
- if (score >= 0.5) return { tier: "strong", thinking: "high" };
343
+ // Thresholds raised from 0.50/0.70 to 0.60/0.75 on 926 real turns, after the
344
+ // ACTION regex picked up the colloquial imperatives: with both changes a
345
+ // path + verb pair no longer justifies an expensive tier ("corrige le bug
346
+ // dans src/foo.ts" used to reach strong). Measured against the real-difficulty
347
+ // proxy: expensive routings 104 → 85, heaviest-quartile turns sent to eco
348
+ // 82 → 53, light turns in an expensive tier 52 → 44, Spearman +0.288 → +0.339.
349
+ if (score >= 0.75) return { tier: "frontier", thinking: "xhigh" };
350
+ if (score >= 0.6) return { tier: "strong", thinking: "high" };
311
351
  if (score >= 0.3) return { tier: "standard", thinking: "medium" };
312
352
  return { tier: "eco", thinking: "low" };
313
353
  }
@@ -316,6 +356,87 @@ function capThinking(auto: Thinking, userSelected: Thinking): Thinking {
316
356
  return THINKING_ORDER[Math.min(THINKING_ORDER.indexOf(auto), THINKING_ORDER.indexOf(userSelected))];
317
357
  }
318
358
 
359
+ // Hybrid classifier (optional; auto-detected, silent fallback)
360
+
361
+ /** Tier decision handed back by a classifier: target tier plus provenance. */
362
+ export interface TierDecision {
363
+ tier: TierName;
364
+ thinking: Thinking;
365
+ via: string;
366
+ }
367
+
368
+ /** Classifier answer → tier, mirroring `tierAndThinking`'s thresholds. */
369
+ const CHOICE_TO_TIER: Record<string, { tier: TierName; thinking: Thinking }> = {
370
+ light: { tier: "eco", thinking: "low" },
371
+ standard: { tier: "standard", thinking: "medium" },
372
+ complex: { tier: "strong", thinking: "high" },
373
+ heavy: { tier: "frontier", thinking: "xhigh" },
374
+ };
375
+
376
+ /**
377
+ * Asks an available classifier model to sort a turn into a tier: the catalog is
378
+ * auto-detected across every provider with working credentials, free
379
+ * classifiers first, then the cheaper ones. Only the user prompt is sent,
380
+ * truncated to 16k characters — never the system prompt, the history or tool
381
+ * results. The question is asked in English: classifiers are multilingual.
382
+ * Never throws: no classifier, a 4-second timeout, an abort, an unexpected
383
+ * answer or one below 0.50 confidence all return undefined, so the caller
384
+ * silently falls back to the local heuristic.
385
+ */
386
+ export async function classifyTier(
387
+ ctx: ExtensionContext,
388
+ text: string,
389
+ signal?: AbortSignal,
390
+ ): Promise<TierDecision | undefined> {
391
+ try {
392
+ const models = await ctx.modelRegistry.getAvailableOfType("classifier");
393
+ if (!models || models.length === 0) return undefined;
394
+ // Free classifiers first, then ascending input cost; unknown pricing
395
+ // ranks last. Array sort is stable, so the catalog order survives.
396
+ const model = [...models].sort((a, b) => {
397
+ const freeA = a.cost?.input === 0;
398
+ const freeB = b.cost?.input === 0;
399
+ if (freeA !== freeB) return freeA ? -1 : 1;
400
+ return (
401
+ (a.cost?.input ?? Number.POSITIVE_INFINITY) -
402
+ (b.cost?.input ?? Number.POSITIVE_INFINITY)
403
+ );
404
+ })[0];
405
+ if (!model) return undefined;
406
+ const timeoutSignal = AbortSignal.timeout(4_000);
407
+ const result = await ctx.modelRegistry.classify(
408
+ model,
409
+ {
410
+ state: { prompt: text.slice(0, 16_000) },
411
+ questions: {
412
+ difficulty: {
413
+ type: "choice",
414
+ instructions:
415
+ "How demanding is the software engineering work requested in prompt?",
416
+ criteria: {
417
+ light: "A quick question, a short follow-up, or a trivial change",
418
+ standard: "An ordinary feature, fix, review, or explanation",
419
+ complex: "Subtle design, cross-cutting changes, or hard debugging",
420
+ heavy: "Architecture-level rework or a long multi-step implementation",
421
+ },
422
+ },
423
+ },
424
+ },
425
+ { signal: signal ? AbortSignal.any([signal, timeoutSignal]) : timeoutSignal },
426
+ );
427
+ if (result.stopReason !== "stop") return undefined;
428
+ const answer = result.answers?.difficulty;
429
+ if (answer?.type !== "choice") return undefined;
430
+ const decision = CHOICE_TO_TIER[answer.choice];
431
+ if (!decision) return undefined;
432
+ const probability = answer.probabilities[answer.choice] ?? answer.confidence ?? 0;
433
+ if (probability < 0.5) return undefined;
434
+ return { ...decision, via: modelKey(model) };
435
+ } catch {
436
+ return undefined;
437
+ }
438
+ }
439
+
319
440
  // Utilities
320
441
 
321
442
  function lastUserText(messages: readonly Message[]): string {
@@ -340,26 +461,40 @@ function editedThisTurn(messages: readonly Message[]): boolean {
340
461
 
341
462
  /**
342
463
  * Shared entry point for a new turn and for the safety fallback of a
343
- * continuation (current model banned/missing): score the current message →
344
- * tier. `avoid` is empty on a new turn, preserved on a safety re-route.
464
+ * continuation (current model banned/missing): decide the tier of the current
465
+ * message — ONE classifier call when a classifier is available, the local
466
+ * bilingual heuristic otherwise. `avoid` is empty on a new turn, preserved on
467
+ * a safety re-route.
345
468
  */
346
- function rerouteCurrentTurn(
469
+ async function rerouteCurrentTurn(
347
470
  pi: ExtensionAPI,
348
471
  ctx: ExtensionContext,
349
- request: Req,
472
+ request: ModelRouteRequest<RouteState>,
350
473
  avoid: string[],
351
474
  userCap: Thinking,
352
475
  prevState?: RouteState,
353
- ): ModelRoute<RouteState> {
354
- const score = difficultyScore(lastUserText(request.messages));
355
- const { tier, thinking: autoTh } = tierAndThinking(score);
356
- const th = capThinking(autoTh, userCap);
476
+ ): Promise<ModelRoute<RouteState>> {
477
+ const text = lastUserText(request.messages);
478
+ // Classifier first; the local bilingual heuristic is the silent fallback.
479
+ const decision = await classifyTier(ctx, text, request.signal);
480
+ let tier: TierName;
481
+ let thinking: Thinking;
482
+ let source: string;
483
+ if (decision) {
484
+ ({ tier, thinking } = decision);
485
+ source = decision.via;
486
+ } else {
487
+ ({ tier, thinking } = tierAndThinking(difficultyScore(text)));
488
+ source = "local";
489
+ }
490
+ const th = capThinking(thinking, userCap);
357
491
  const m = buildTiers(ctx, avoid)[tier];
358
- return toRoute(pi, ctx, m, th, prevState, {
492
+ return toRoute(pi, ctx, m, th, source, prevState, {
359
493
  modelId: m.id,
360
494
  providerId: m.provider,
361
495
  thinking: th,
362
496
  avoid,
497
+ source,
363
498
  });
364
499
  }
365
500
 
@@ -425,11 +560,15 @@ export default function (pi: ExtensionAPI) {
425
560
  const avoid = [
426
561
  ...new Set([
427
562
  ...baseState.avoid,
428
- // Provider quota: avoid the whole provider, otherwise the failover
429
- // lands on another of its models and fails again immediately.
430
- PROVIDER_LIMIT_RE.test(request.failed.message?.errorMessage ?? "")
431
- ? request.failed.model.provider
432
- : failedKey,
563
+ // Model access disabled on the account: a per-model problem — other
564
+ // models of the same provider may still work, so avoid the model
565
+ // alone. Provider quota: avoid the whole provider, otherwise the
566
+ // failover lands on another of its models and fails again at once.
567
+ MODEL_ACCESS_RE.test(request.failed.message?.errorMessage ?? "")
568
+ ? failedKey
569
+ : PROVIDER_LIMIT_RE.test(request.failed.message?.errorMessage ?? "")
570
+ ? request.failed.model.provider
571
+ : failedKey,
433
572
  ]),
434
573
  ];
435
574
  const tiers = buildTiers(ctx, avoid);
@@ -441,7 +580,7 @@ export default function (pi: ExtensionAPI) {
441
580
  .find((m) => m && !avoid.includes(modelKey(m)) && !avoid.includes(m.provider)) ??
442
581
  tiers.frontier;
443
582
  const th = capThinking(baseState.thinking, userCap);
444
- return toRoute(pi, ctx, chosen, th, prevState, {
583
+ return toRoute(pi, ctx, chosen, th, baseState.source, prevState, {
445
584
  ...baseState,
446
585
  modelId: chosen.id,
447
586
  providerId: chosen.provider,
@@ -459,7 +598,7 @@ export default function (pi: ExtensionAPI) {
459
598
  if (editedThisTurn(request.messages)) {
460
599
  const eco = buildTiers(ctx, avoid).eco;
461
600
  const th = capThinking("medium", userCap);
462
- return toRoute(pi, ctx, eco, th, prevState, {
601
+ return toRoute(pi, ctx, eco, th, prevState.source, prevState, {
463
602
  ...prevState,
464
603
  modelId: eco.id,
465
604
  providerId: eco.provider,
@@ -482,16 +621,19 @@ export default function (pi: ExtensionAPI) {
482
621
  | undefined;
483
622
  if (m) {
484
623
  const th = capThinking(prevState.thinking, userCap);
485
- return toRoute(pi, ctx, m, th, prevState, { ...prevState, thinking: th });
624
+ return toRoute(pi, ctx, m, th, prevState.source, prevState, {
625
+ ...prevState,
626
+ thinking: th,
627
+ });
486
628
  }
487
629
  }
488
630
  // Safety fallback: current model banned/missing → re-route the SAME
489
- // turn (difficultyScore on the current message, `avoid` kept).
490
- return rerouteCurrentTurn(pi, ctx, request, prevState.avoid, userCap, prevState);
631
+ // turn (classifier or local score on the current message, `avoid` kept).
632
+ return await rerouteCurrentTurn(pi, ctx, request, prevState.avoid, userCap, prevState);
491
633
  }
492
634
 
493
- // NEW TURN (user message)
494
- return rerouteCurrentTurn(pi, ctx, request, [], userCap, prevState);
635
+ // NEW TURN (user message): one classifier call, else the local heuristic.
636
+ return await rerouteCurrentTurn(pi, ctx, request, [], userCap, prevState);
495
637
  },
496
638
  });
497
639
  }
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "pi-auto-route",
3
- "version": "1.1.0",
3
+ "version": "1.3.0",
4
4
  "description": "A pi virtual model that routes each turn to the right physical model by task difficulty and adjusts the thinking level.",
5
5
  "keywords": [
6
6
  "pi-package",