pi-auto-route 0.0.0-stage → 1.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +18 -0
- package/LICENSE +21 -0
- package/README.md +91 -2
- package/auto-route.ts +497 -0
- package/package.json +36 -5
package/CHANGELOG.md
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## 1.1.0
|
|
4
|
+
|
|
5
|
+
- Provider-agnostic: the failover now recognizes provider-wide limits from any provider (Anthropic rate limits, Google `RESOURCE_EXHAUSTED`, OpenAI `insufficient_quota`, OpenCode usage limits), not only OpenCode-specific errors.
|
|
6
|
+
- Transient rate limits keep their original message text; only errors pi refuses to retry are rewritten into a retryable 429.
|
|
7
|
+
- Models without pricing data are no longer dropped from the catalog; they rank at a mid cost.
|
|
8
|
+
- The denylist path honors `PI_CODING_AGENT_DIR` instead of a hardcoded home directory.
|
|
9
|
+
- Packaged as a pi package: install with `pi install npm:pi-auto-route`.
|
|
10
|
+
- Tests: 18 cases covering the heuristic calibration and the failover lifecycle.
|
|
11
|
+
|
|
12
|
+
## 1.0.0
|
|
13
|
+
|
|
14
|
+
- Initial release.
|
|
15
|
+
- `auto` virtual model routing every turn by a locally calibrated difficulty heuristic (0.30 / 0.50 / 0.70 thresholds, 880 real turns).
|
|
16
|
+
- Four tiers (eco, standard, strong, frontier) with a thinking-level cap per tier.
|
|
17
|
+
- One model switch per turn: the starting model plans and makes the first `edit`/`write`.
|
|
18
|
+
- Automatic failover on timeouts and overloads, provider-wide avoidance on quota 429s, persistent denylist for permanent errors (400/404/410).
|
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 rzph
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
CHANGED
|
@@ -1,3 +1,92 @@
|
|
|
1
|
-
#
|
|
1
|
+
# auto-route
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
A virtual model for [pi](https://github.com/earendil-works/pi-coding-agent) that routes every turn to the right physical model by task difficulty and adjusts the thinking level, to cut token spend without losing quality.
|
|
4
|
+
|
|
5
|
+

|
|
6
|
+
|
|
7
|
+
**pi-specific.** This extension uses pi's extension API (`pi.registerVirtualModel`, `ModelRegistry`). It does not port as-is to Claude Code, Codex or other harnesses; see [Portability](#portability) for close equivalents.
|
|
8
|
+
|
|
9
|
+
## How it works
|
|
10
|
+
|
|
11
|
+
Pick **auto → auto** in the model selector (`/model`). Then, on every turn:
|
|
12
|
+
|
|
13
|
+
1. **Difficulty estimate** (local heuristic, no external API): a cited path or repo, action verbs ("resume", "modify", "find"…), explicit intent ("I want"), a conceptual question, keyword matches (debugging, architecture…), length, code, a terminal-paste penalty → a continuous score from 0 to 1. Calibrated on 880 real turns (see `heuristics.test.ts`); read the numbers honestly:
|
|
14
|
+
- heaviest-quartile messages sent to `eco`: **195/220 → 72/220** (the real win: less under-routing);
|
|
15
|
+
- Spearman score vs. tool calls: +0.16 → **+0.28**;
|
|
16
|
+
- overall quartile agreement: **30 % against 25 % for chance** — the score is an *asymmetric umbrella* (it shields heavy messages from the `eco` tier and at worst bumps simple ones one step), not a precise classifier. Do not overestimate fine-grained prediction.
|
|
17
|
+
2. **Tier choice** inside YOUR real catalog (discovered at runtime via `getAvailable()`, nothing hardcoded):
|
|
18
|
+
- score ≥ 0.7 — heavy work, several signals → most capable tier, thinking `xhigh`
|
|
19
|
+
- score ≥ 0.5 → strong tier, thinking `high`
|
|
20
|
+
- score ≥ 0.3 → standard tier, thinking `medium`
|
|
21
|
+
- score < 0.3 — short follow-ups, quick questions → lightest tier, thinking `low`
|
|
22
|
+
3. **One switch per turn**: the starting model plans and makes the first `edit`/`write`; the rest of the turn continues on the lightest model. The prompt cache breaks only once per turn.
|
|
23
|
+
4. **Compaction and invisible summaries** always go to the lightest model — the token leak nobody sees.
|
|
24
|
+
5. **Automatic failover instead of a crash**: a timeout or overload (429/5xx) moves to the next model of the ladder; a **429 "provider quota"** error (which pi treats as non-retryable) is rewritten into a retryable one, and the router then avoids the **whole provider** — the turn continues instead of dying. Permanent errors (400/404/410) ban the model through the denylist. See `failover.test.ts`.
|
|
25
|
+
|
|
26
|
+
### Excluded from the candidates
|
|
27
|
+
|
|
28
|
+
- **free** models (tight quotas, constant timeouts);
|
|
29
|
+
- **non-reasoning** models;
|
|
30
|
+
- any model whose provider has no valid credentials;
|
|
31
|
+
- **`contributor`** models (OpenCode endpoints that train on your requests);
|
|
32
|
+
- retired or deprecated generations: `gemini-2.5-*` (Google 404), `mimo-v2.5` (OpenCode 410);
|
|
33
|
+
- specialized non-code models (`computer-use`, `robotics`, `deep-research`);
|
|
34
|
+
- anything listed in pi's agent dir as `auto-route-denylist.json` (`$PI_CODING_AGENT_DIR` if set, else `~/.pi/agent/`) — banned automatically and **persistently** as soon as it returns a permanent error (400/404/410, "trains on request data", unsupported region…). Delete an entry from the file to restore a model.
|
|
35
|
+
|
|
36
|
+
Capacity ranking uses **output cost** as a proxy (ties broken by the largest contextWindow). A model is excluded as free only when its pricing says 0; missing pricing means unknown and stays a candidate, ranked at a mid cost. An empty tier falls back to the next one; if everything fails, you get an explicit error.
|
|
37
|
+
|
|
38
|
+
## Requirements
|
|
39
|
+
|
|
40
|
+
- pi with at least one provider that has valid credentials. Nothing is hardcoded to a specific provider: the candidate catalog, the tiers and the failover are all built from your own model registry at runtime.
|
|
41
|
+
- No API key of its own and no network calls: the difficulty heuristic runs entirely locally.
|
|
42
|
+
- Only to run the test suite: Node.js 22.6 or newer (`node --test *.test.ts`).
|
|
43
|
+
|
|
44
|
+
## Install
|
|
45
|
+
|
|
46
|
+
```bash
|
|
47
|
+
pi install npm:pi-auto-route
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
Then `/reload` and pick **auto → auto** in the model selector (`/model`). Updates come with `pi update --extensions`; removal with `pi remove npm:pi-auto-route`.
|
|
51
|
+
|
|
52
|
+
Without npm, a local checkout works too: `pi install git:github.com/rzph/pi-auto-route`, or copy `auto-route.ts` into `~/.pi/agent/extensions/` by hand.
|
|
53
|
+
|
|
54
|
+
|
|
55
|
+
```bash
|
|
56
|
+
# for the session:
|
|
57
|
+
pi -e /path/to/auto-route.ts
|
|
58
|
+
|
|
59
|
+
# or permanent (loaded at every pi start):
|
|
60
|
+
cp /path/to/auto-route.ts ~/.pi/agent/extensions/auto-route.ts
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
Then `/model` → **auto → auto** (shown as "Auto (routing + thinking + failover)"). Selecting it starts at thinking **`low`** — that level is a ceiling: raise it with `shift+tab` when you want the router to think harder.
|
|
64
|
+
|
|
65
|
+
## Customize
|
|
66
|
+
|
|
67
|
+
Everything is set at the top of the file:
|
|
68
|
+
|
|
69
|
+
- `difficultyScore()`: the heuristic (signals plus length thresholds) — each signal is one commented regex; recalibrate and run `node --test *.test.ts` after any change (`heuristics.test.ts` for the score, `failover.test.ts` for the 429 handling and the default thinking);
|
|
70
|
+
- `tierAndThinking()`: the score → tier + thinking mapping;
|
|
71
|
+
- the filter in `candidateModels()` if you want free or non-reasoning models back.
|
|
72
|
+
|
|
73
|
+
To force one specific model for a single turn: switch to a real model with `/model`, then back to `auto` on the next turn.
|
|
74
|
+
|
|
75
|
+
## Portability
|
|
76
|
+
|
|
77
|
+
- **pi**: supported (this extension).
|
|
78
|
+
- **Claude Code**: not portable; close ideas: `jev-router` (gargpratyush), "model-router" skills (Claude Code internals).
|
|
79
|
+
- **Codex**: not portable; `jev-codex-router` (0xNatoshi): model + thinking per turn.
|
|
80
|
+
- **Hermes**: not portable; `hermes-jev-skills` (kerpopule).
|
|
81
|
+
|
|
82
|
+
The idea (plan strong, execute cheap, one switch per turn, degenerate compaction on a light model) ports everywhere; this implementation is tied to pi. Want a port for another harness? Open an issue.
|
|
83
|
+
|
|
84
|
+
## Key concepts
|
|
85
|
+
|
|
86
|
+
- **Prompt cache**: the router switches *once* per turn, after planning — every cross-provider switch invalidates the cache, so the trade-off favors turns with several tool calls.
|
|
87
|
+
- **Route state**: stored on the session branch by pi; it follows the session tree and survives compaction.
|
|
88
|
+
- **Capacity proxy**: output cost ranks models in the absence of per-task benchmarks. Home-grown benchmark data (reviewer catch rate, say) is a natural extension.
|
|
89
|
+
|
|
90
|
+
## License
|
|
91
|
+
|
|
92
|
+
MIT — see [LICENSE](LICENSE).
|
package/auto-route.ts
ADDED
|
@@ -0,0 +1,497 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* auto-route — virtual model `auto/auto`: full automatic routing.
|
|
3
|
+
*
|
|
4
|
+
* /model → "auto":
|
|
5
|
+
* - candidates = paid reasoning chat models with valid credentials, excluding
|
|
6
|
+
* free models, training models (`contributor`) and retired generations
|
|
7
|
+
* (`gemini-2.5-*`);
|
|
8
|
+
* - capacity tiers built at runtime (output cost as a proxy);
|
|
9
|
+
* - locally estimated difficulty → tier + thinking;
|
|
10
|
+
* - the thinking level chosen in pi (`shift+tab`) acts as a CEILING;
|
|
11
|
+
* - failover on retry (429/5xx) plus a `message_end` listener that bans
|
|
12
|
+
* PERMANENTLY models returning 400/404 (pi does not retry non-429 4xx);
|
|
13
|
+
* 429 "provider quota" errors, which pi treats as non-retryable, are
|
|
14
|
+
* rewritten into retryable 429s so the same failover (cross-provider)
|
|
15
|
+
* runs instead of letting the turn die;
|
|
16
|
+
* - selecting `auto` starts at thinking `low` by default (ceiling, raise it
|
|
17
|
+
* later with shift+tab);
|
|
18
|
+
* - ONE discreet status line (`⚡ model · thinking`);
|
|
19
|
+
* - one model switch per turn (prompt cache preserved);
|
|
20
|
+
* - compaction/summaries → the lightest model.
|
|
21
|
+
*
|
|
22
|
+
* Usage: pi -e ~/.pi/agent/extensions/auto-route.ts then /model → auto
|
|
23
|
+
*/
|
|
24
|
+
|
|
25
|
+
import type { AssistantMessage, Message, Model } from "@earendil-works/pi-ai";
|
|
26
|
+
import type {
|
|
27
|
+
ExtensionAPI,
|
|
28
|
+
ExtensionContext,
|
|
29
|
+
ModelRoute,
|
|
30
|
+
ModelRouteRequest,
|
|
31
|
+
} from "@earendil-works/pi-coding-agent";
|
|
32
|
+
|
|
33
|
+
import { mkdirSync, readFileSync, writeFileSync } from "node:fs";
|
|
34
|
+
import { homedir } from "node:os";
|
|
35
|
+
import { dirname, join } from "node:path";
|
|
36
|
+
|
|
37
|
+
const EDIT_TOOLS = new Set(["edit", "write"]);
|
|
38
|
+
|
|
39
|
+
/** Output cost assumed for models without pricing data: mid-tier, neither free nor frontier. */
|
|
40
|
+
const UNKNOWN_OUTPUT_COST = 1;
|
|
41
|
+
|
|
42
|
+
/**
|
|
43
|
+
* ID/name patterns excluded up front:
|
|
44
|
+
* - `free`: tight quotas / constant timeouts;
|
|
45
|
+
* - `contributor`: OpenCode endpoints that train on your requests;
|
|
46
|
+
* - `gemini-2.5-`: retired by Google for new accounts (404);
|
|
47
|
+
* - `mimo-v2.5`: deprecated on OpenCode Go (410 ModelDeprecated);
|
|
48
|
+
* - `computer-use` / `robotics` / `deep-research`: specialized non-code models.
|
|
49
|
+
*/
|
|
50
|
+
const EXCLUDED_MODEL_RE =
|
|
51
|
+
/free|contributor|gemini-2\.5-|mimo-v2\.5\b|computer-use|robotics|deep-research/i;
|
|
52
|
+
|
|
53
|
+
/**
|
|
54
|
+
* pi's agent config dir: honors PI_CODING_AGENT_DIR (its own env var, tilde
|
|
55
|
+
* expanded), else falls back to ~/.pi/agent.
|
|
56
|
+
*/
|
|
57
|
+
function agentDir(): string {
|
|
58
|
+
const envDir = process.env.PI_CODING_AGENT_DIR?.trim();
|
|
59
|
+
if (envDir) return envDir.startsWith("~") ? join(homedir(), envDir.slice(1)) : envDir;
|
|
60
|
+
return join(homedir(), ".pi", "agent");
|
|
61
|
+
}
|
|
62
|
+
/** File of permanently banned models (400/404/410 discovered on the fly). */
|
|
63
|
+
const DENYLIST_FILE = join(agentDir(), "auto-route-denylist.json");
|
|
64
|
+
|
|
65
|
+
type Thinking = "off" | "low" | "medium" | "high" | "xhigh";
|
|
66
|
+
type TierName = "frontier" | "strong" | "standard" | "eco";
|
|
67
|
+
const THINKING_ORDER: Thinking[] = ["off", "low", "medium", "high", "xhigh"];
|
|
68
|
+
|
|
69
|
+
interface RouteState {
|
|
70
|
+
modelId: string;
|
|
71
|
+
providerId: string;
|
|
72
|
+
thinking: Thinking;
|
|
73
|
+
/** provider/id entries avoided for THIS turn (transient failures). */
|
|
74
|
+
avoid: string[];
|
|
75
|
+
}
|
|
76
|
+
|
|
77
|
+
function modelKey(m: { provider: string; id: string }): string {
|
|
78
|
+
return `${m.provider}/${m.id}`;
|
|
79
|
+
}
|
|
80
|
+
|
|
81
|
+
function sameState(a: RouteState | undefined, b: RouteState): boolean {
|
|
82
|
+
if (!a) return false;
|
|
83
|
+
return (
|
|
84
|
+
a.modelId === b.modelId &&
|
|
85
|
+
a.providerId === b.providerId &&
|
|
86
|
+
a.thinking === b.thinking &&
|
|
87
|
+
a.avoid.length === b.avoid.length &&
|
|
88
|
+
a.avoid.every((v, i) => v === b.avoid[i])
|
|
89
|
+
);
|
|
90
|
+
}
|
|
91
|
+
|
|
92
|
+
// Persistent denylist
|
|
93
|
+
|
|
94
|
+
function loadDenylist(): Set<string> {
|
|
95
|
+
try {
|
|
96
|
+
const raw = JSON.parse(readFileSync(DENYLIST_FILE, "utf8")) as string[];
|
|
97
|
+
return new Set(Array.isArray(raw) ? raw : []);
|
|
98
|
+
} catch {
|
|
99
|
+
return new Set();
|
|
100
|
+
}
|
|
101
|
+
}
|
|
102
|
+
|
|
103
|
+
function saveDenylist(set: Set<string>): void {
|
|
104
|
+
try {
|
|
105
|
+
mkdirSync(dirname(DENYLIST_FILE), { recursive: true });
|
|
106
|
+
writeFileSync(DENYLIST_FILE, JSON.stringify([...set], null, 1));
|
|
107
|
+
} catch {
|
|
108
|
+
// best effort
|
|
109
|
+
}
|
|
110
|
+
}
|
|
111
|
+
|
|
112
|
+
const permanentDenylist = loadDenylist();
|
|
113
|
+
|
|
114
|
+
function banPermanently(provider: string, modelId: string): void {
|
|
115
|
+
const k = modelKey({ provider, id: modelId });
|
|
116
|
+
if (!permanentDenylist.has(k)) {
|
|
117
|
+
permanentDenylist.add(k);
|
|
118
|
+
saveDenylist(permanentDenylist);
|
|
119
|
+
}
|
|
120
|
+
}
|
|
121
|
+
|
|
122
|
+
/** Permanent failure (model retired, 404/410, refuses training, unsupported region…). */
|
|
123
|
+
function isPermanentFailure(message: string): boolean {
|
|
124
|
+
return /no longer available|not found|\b404\b|\b410\b|ModelDeprecated|deprecated|discontinued|trains on request data|hosted outside Europe|requires Global regions|does not exist|no API implementation|unsupported/i.test(
|
|
125
|
+
message,
|
|
126
|
+
);
|
|
127
|
+
}
|
|
128
|
+
|
|
129
|
+
/**
|
|
130
|
+
* Any provider-wide limit (quota, rate limit, billing, bare 429): used on retry
|
|
131
|
+
* to decide provider-wide avoidance — an account-level limit hits every model
|
|
132
|
+
* of that provider, so failing over within it would fail again immediately.
|
|
133
|
+
*/
|
|
134
|
+
const PROVIDER_LIMIT_RE =
|
|
135
|
+
/gousagelimiterror|freeusagelimiterror|usage limit|available balance|credit balance|insufficient_quota|out of budget|out of credit|quota|billing|subscription_sharing|resource_exhausted|rate.?limit|too many requests|\b429\b/i;
|
|
136
|
+
|
|
137
|
+
/**
|
|
138
|
+
* The subset pi classifies as NON-retryable (its retry would never start, so
|
|
139
|
+
* the turn dies). Only those get rewritten into a retryable 429 by
|
|
140
|
+
* `toFailoverRetryMessage`; already-retryable errors (transient rate limits,
|
|
141
|
+
* 5xx…) keep their original text untouched.
|
|
142
|
+
*/
|
|
143
|
+
const NON_RETRYABLE_QUOTA_RE =
|
|
144
|
+
/gousagelimiterror|freeusagelimiterror|monthly usage limit reached|available balance|credit balance|insufficient_quota|out of budget|out of credit|quota exceeded|exceeded your current quota|billing|subscription_sharing|resource_exhausted/i;
|
|
145
|
+
|
|
146
|
+
/**
|
|
147
|
+
* Turns a provider-quota error into a "429" pi is willing to retry. Pi drops
|
|
148
|
+
* these errors (retrying the same provider is pointless); under auto-route the
|
|
149
|
+
* failover changes provider, so a retry becomes useful: pi then calls
|
|
150
|
+
* `route(reason: "retry")` and the router switches. The rewritten text carries
|
|
151
|
+
* pi's retryable "429" pattern and avoids its non-retryable ones (billing,
|
|
152
|
+
* quota exceeded…); it also reads as a clear marker in the transcript.
|
|
153
|
+
* Returns undefined when there is nothing to rewrite.
|
|
154
|
+
*/
|
|
155
|
+
export function toFailoverRetryMessage(msg: AssistantMessage): AssistantMessage | undefined {
|
|
156
|
+
if (
|
|
157
|
+
msg.stopReason !== "error" ||
|
|
158
|
+
!msg.errorMessage ||
|
|
159
|
+
!msg.provider ||
|
|
160
|
+
!msg.model ||
|
|
161
|
+
!NON_RETRYABLE_QUOTA_RE.test(msg.errorMessage)
|
|
162
|
+
) {
|
|
163
|
+
return undefined;
|
|
164
|
+
}
|
|
165
|
+
return {
|
|
166
|
+
...msg,
|
|
167
|
+
errorMessage: `auto-route: 429 provider quota — ${msg.provider}/${msg.model} at limit; retrying on another model`,
|
|
168
|
+
};
|
|
169
|
+
}
|
|
170
|
+
|
|
171
|
+
// Dynamic catalog
|
|
172
|
+
|
|
173
|
+
function candidateModels(ctx: ExtensionContext): Model<never>[] {
|
|
174
|
+
return (ctx.modelRegistry.getAvailable() as Model<never>[]).filter((m) => {
|
|
175
|
+
if (permanentDenylist.has(modelKey(m))) return false;
|
|
176
|
+
if (EXCLUDED_MODEL_RE.test(m.id) || EXCLUDED_MODEL_RE.test(m.name)) return false;
|
|
177
|
+
// A model is free only when its pricing says so; missing pricing means
|
|
178
|
+
// unknown, not free, so it stays a candidate.
|
|
179
|
+
const free = m.cost !== undefined && m.cost.input === 0 && m.cost.output === 0;
|
|
180
|
+
return m.reasoning !== false && !free;
|
|
181
|
+
});
|
|
182
|
+
}
|
|
183
|
+
|
|
184
|
+
function buildTiers(ctx: ExtensionContext, avoid: string[] = []) {
|
|
185
|
+
const allCandidates = candidateModels(ctx);
|
|
186
|
+
if (allCandidates.length === 0) {
|
|
187
|
+
throw new Error("auto-route: no candidate model available (check your API keys)");
|
|
188
|
+
}
|
|
189
|
+
const avoidSet = new Set(avoid);
|
|
190
|
+
// `avoid` holds either `provider/id` keys (one model) or a bare `provider`
|
|
191
|
+
// (whole provider, quota exhausted: all of its models would fail alike).
|
|
192
|
+
const filtered = allCandidates.filter(
|
|
193
|
+
(m) => !avoidSet.has(modelKey(m)) && !avoidSet.has(m.provider),
|
|
194
|
+
);
|
|
195
|
+
const pool = filtered.length > 0 ? filtered : allCandidates;
|
|
196
|
+
const sorted = [...pool].sort((a, b) => {
|
|
197
|
+
// Unknown pricing ranks as mid cost so pricing-less models land in the
|
|
198
|
+
// middle tiers instead of being dumped into eco or promoted to frontier.
|
|
199
|
+
const ca = a.cost?.output ?? UNKNOWN_OUTPUT_COST;
|
|
200
|
+
const cb = b.cost?.output ?? UNKNOWN_OUTPUT_COST;
|
|
201
|
+
if (Math.abs(ca - cb) < 0.01) return (b.contextWindow ?? 0) - (a.contextWindow ?? 0);
|
|
202
|
+
return cb - ca;
|
|
203
|
+
});
|
|
204
|
+
const at = (frac: number): Model<never> =>
|
|
205
|
+
sorted[Math.min(sorted.length - 1, Math.floor(frac * sorted.length))];
|
|
206
|
+
return {
|
|
207
|
+
frontier: at(0),
|
|
208
|
+
strong: at(0.33),
|
|
209
|
+
standard: at(0.66),
|
|
210
|
+
eco: sorted[sorted.length - 1],
|
|
211
|
+
background: sorted[sorted.length - 1],
|
|
212
|
+
} as const;
|
|
213
|
+
}
|
|
214
|
+
|
|
215
|
+
// Status (one discreet line)
|
|
216
|
+
|
|
217
|
+
function setStatusLine(
|
|
218
|
+
pi: ExtensionAPI,
|
|
219
|
+
ctx: ExtensionContext,
|
|
220
|
+
m: Model<never>,
|
|
221
|
+
thinking: Thinking,
|
|
222
|
+
): void {
|
|
223
|
+
try {
|
|
224
|
+
const dim = ctx.ui.theme?.fg?.bind(ctx.ui.theme) ?? ((_: string, s: string) => s);
|
|
225
|
+
ctx.ui.setStatus("auto", dim("dim", `⚡ ${modelKey(m)} · ${thinking}`));
|
|
226
|
+
} catch {
|
|
227
|
+
ctx.ui.setStatus("auto", `${modelKey(m)} · ${thinking}`);
|
|
228
|
+
}
|
|
229
|
+
pi.events.emit("auto-route:routed", {
|
|
230
|
+
provider: m.provider,
|
|
231
|
+
modelId: m.id,
|
|
232
|
+
thinking,
|
|
233
|
+
});
|
|
234
|
+
}
|
|
235
|
+
|
|
236
|
+
function toRoute(
|
|
237
|
+
pi: ExtensionAPI,
|
|
238
|
+
ctx: ExtensionContext,
|
|
239
|
+
m: Model<never>,
|
|
240
|
+
thinking: Thinking,
|
|
241
|
+
prevState?: RouteState,
|
|
242
|
+
nextState?: RouteState,
|
|
243
|
+
): ModelRoute<RouteState> {
|
|
244
|
+
setStatusLine(pi, ctx, m, thinking);
|
|
245
|
+
// Only return a new state object when it actually changed (avoids writing a
|
|
246
|
+
// session-tree entry on every continuation request).
|
|
247
|
+
const stateToStore =
|
|
248
|
+
nextState && !sameState(prevState, nextState) ? nextState : undefined;
|
|
249
|
+
return {
|
|
250
|
+
model: m,
|
|
251
|
+
thinkingLevel: thinking,
|
|
252
|
+
state: stateToStore,
|
|
253
|
+
};
|
|
254
|
+
}
|
|
255
|
+
|
|
256
|
+
// Difficulty heuristic
|
|
257
|
+
//
|
|
258
|
+
// Calibrated on 2026-11-21 over 880 real turns extracted from
|
|
259
|
+
// ~/.pi/agent/sessions/*.jsonl. Before/after measurements (analysis script,
|
|
260
|
+
// real-difficulty proxy = combined rank of the turn's tool calls and output
|
|
261
|
+
// tokens):
|
|
262
|
+
// - Spearman score vs. tool calls: v1 +0.16 → v2 +0.28;
|
|
263
|
+
// - heaviest-quartile messages sent to the eco tier: v1 195/220 → v2 72/220;
|
|
264
|
+
// - distribution: v1 = 92 % of messages in {0.10, 0.15} → v2 spread over 8+
|
|
265
|
+
// values.
|
|
266
|
+
// Every signal is independent and interpretable; no training, no data leaves
|
|
267
|
+
// the machine: the rule stays hand-written, the numbers only verify it.
|
|
268
|
+
|
|
269
|
+
/** Reference to a path, file or repo → work on an existing codebase. */
|
|
270
|
+
const PATHY =
|
|
271
|
+
/(?:\/(?:home|data|tmp|etc|usr|opt|var|srv)\/|~\/|\b[\w.+-]+\/[\w.-]+\.[A-Za-z0-9]{1,6}\b|\b[\w.+-]+\/[\w-]*-[\w-]+\b)/;
|
|
272
|
+
/** Construction / modification / analysis verbs → the agent is about to do work. */
|
|
273
|
+
const ACTION =
|
|
274
|
+
/\b(reprend\w*|construi\w*|impl[ée]ment\w*|cré[ée]r?|cree\w*|g[ée]n[èe]r\w*|d[ée]velop\w*|migre\w*|refais|refaire|transforme\w*|modifi\w*|ajout\w*|supprim\w*|corrige\w*|corrig\w*|install\w*|configure\w*|[ée]crase\w*|r[ée]pare\w*|optimise\w*|nettoie\w*|remplace\w*|relance\w*|trouve\w*|cherch\w*|analy\w*|examin\w*|v[ée]rifi\w*|test\w*)\b/i;
|
|
275
|
+
/** Explicit intent from the requester. */
|
|
276
|
+
const WANT = /\b(je veux|fais en sorte|fais[- ]moi|il faut|on pourrait|peux[- ]tu|besoin de)\b/i;
|
|
277
|
+
/** Conceptual question → asks for a solid explanation. */
|
|
278
|
+
const QUESTION =
|
|
279
|
+
/\b(pourquoi|explique\w*|c'est quoi|c est quoi|comment (?:ça|ca) marche|diff[ée]rence entre|quelle est|quel est|est[- ]ce que)\b/i;
|
|
280
|
+
/** Concept keywords, counted (up to 3). */
|
|
281
|
+
const CONCEPTS =
|
|
282
|
+
/\b(architecture|conception|refactor|d[ée]bogu|debug|optimis|s[ée]curit|timing|algorithme|compar|pourquoi|analyse|syst[èe]me|protocole)\b/gi;
|
|
283
|
+
/** Explicit breakage / regression report. */
|
|
284
|
+
const REGRESSION = /\b(pas le m[êe]me|r[ée]gression|cass[ée]|fuit|bloqu[ée]|corriger)\b/i;
|
|
285
|
+
/** Shell prompt line (terminal paste: form, not difficulty). */
|
|
286
|
+
const SHELLLINE = /^\s*(?:\$|sudo\s|#\s)/gm;
|
|
287
|
+
|
|
288
|
+
export function difficultyScore(text: string): number {
|
|
289
|
+
const t = text.trim();
|
|
290
|
+
let sig = Math.min(0.3, ((t.match(CONCEPTS) ?? []).length) * 0.1);
|
|
291
|
+
if (PATHY.test(t)) sig += 0.25;
|
|
292
|
+
if (ACTION.test(t)) sig += 0.2;
|
|
293
|
+
if (WANT.test(t)) sig += 0.1;
|
|
294
|
+
if (QUESTION.test(t)) sig += 0.1;
|
|
295
|
+
if (t.includes("```")) sig += 0.1;
|
|
296
|
+
if (t.length > 800) sig += 0.1;
|
|
297
|
+
if (t.length > 2000) sig += 0.1;
|
|
298
|
+
if (REGRESSION.test(t)) sig += 0.1;
|
|
299
|
+
// Terminal paste (≥3 shell prompt lines): form, not difficulty.
|
|
300
|
+
if ((t.match(SHELLLINE) ?? []).length >= 3) sig -= 0.15;
|
|
301
|
+
// Base 0.10: short follow-up and neutral message → lightest tier.
|
|
302
|
+
return Math.round(Math.min(1, 0.1 + Math.max(0, sig)) * 100) / 100;
|
|
303
|
+
}
|
|
304
|
+
|
|
305
|
+
export function tierAndThinking(score: number): { tier: TierName; thinking: Thinking } {
|
|
306
|
+
// Thresholds calibrated on the 880 real turns (score → real-difficulty
|
|
307
|
+
// quartile table: 0.10-0.20 ≈ quartile 1, 0.30-0.45 ≈ quartile 2,
|
|
308
|
+
// ≥0.55 ≈ quartile 3).
|
|
309
|
+
if (score >= 0.7) return { tier: "frontier", thinking: "xhigh" };
|
|
310
|
+
if (score >= 0.5) return { tier: "strong", thinking: "high" };
|
|
311
|
+
if (score >= 0.3) return { tier: "standard", thinking: "medium" };
|
|
312
|
+
return { tier: "eco", thinking: "low" };
|
|
313
|
+
}
|
|
314
|
+
|
|
315
|
+
function capThinking(auto: Thinking, userSelected: Thinking): Thinking {
|
|
316
|
+
return THINKING_ORDER[Math.min(THINKING_ORDER.indexOf(auto), THINKING_ORDER.indexOf(userSelected))];
|
|
317
|
+
}
|
|
318
|
+
|
|
319
|
+
// Utilities
|
|
320
|
+
|
|
321
|
+
function lastUserText(messages: readonly Message[]): string {
|
|
322
|
+
const content = messages.filter((m) => m.role === "user").at(-1)?.content ?? "";
|
|
323
|
+
if (typeof content === "string") return content;
|
|
324
|
+
return content
|
|
325
|
+
.flatMap((b) => (b.type === "text" ? [b.text] : []))
|
|
326
|
+
.join("\n");
|
|
327
|
+
}
|
|
328
|
+
|
|
329
|
+
function editedThisTurn(messages: readonly Message[]): boolean {
|
|
330
|
+
const lastUser = messages.findLastIndex((m) => m.role === "user");
|
|
331
|
+
return messages
|
|
332
|
+
.slice(lastUser + 1)
|
|
333
|
+
.some(
|
|
334
|
+
(m) =>
|
|
335
|
+
m.role === "toolResult" &&
|
|
336
|
+
EDIT_TOOLS.has((m as { toolName?: string }).toolName ?? "") &&
|
|
337
|
+
!(m as { isError?: boolean }).isError,
|
|
338
|
+
);
|
|
339
|
+
}
|
|
340
|
+
|
|
341
|
+
/**
|
|
342
|
+
* Shared entry point for a new turn and for the safety fallback of a
|
|
343
|
+
* continuation (current model banned/missing): score the current message →
|
|
344
|
+
* tier. `avoid` is empty on a new turn, preserved on a safety re-route.
|
|
345
|
+
*/
|
|
346
|
+
function rerouteCurrentTurn(
|
|
347
|
+
pi: ExtensionAPI,
|
|
348
|
+
ctx: ExtensionContext,
|
|
349
|
+
request: Req,
|
|
350
|
+
avoid: string[],
|
|
351
|
+
userCap: Thinking,
|
|
352
|
+
prevState?: RouteState,
|
|
353
|
+
): ModelRoute<RouteState> {
|
|
354
|
+
const score = difficultyScore(lastUserText(request.messages));
|
|
355
|
+
const { tier, thinking: autoTh } = tierAndThinking(score);
|
|
356
|
+
const th = capThinking(autoTh, userCap);
|
|
357
|
+
const m = buildTiers(ctx, avoid)[tier];
|
|
358
|
+
return toRoute(pi, ctx, m, th, prevState, {
|
|
359
|
+
modelId: m.id,
|
|
360
|
+
providerId: m.provider,
|
|
361
|
+
thinking: th,
|
|
362
|
+
avoid,
|
|
363
|
+
});
|
|
364
|
+
}
|
|
365
|
+
|
|
366
|
+
export default function (pi: ExtensionAPI) {
|
|
367
|
+
// Pi does NOT trigger `route(reason: "retry")` on 400/404 errors nor on
|
|
368
|
+
// provider-quota 429s (non-retryable classes). We listen to `message_end` to:
|
|
369
|
+
// 1. permanently ban models with permanent failures (400/404/410…);
|
|
370
|
+
// 2. rewrite quota 429s into a retryable error, so pi starts its retry →
|
|
371
|
+
// route(reason: "retry") → cross-provider failover instead of a dead turn.
|
|
372
|
+
pi.on("message_end", (event, ctx) => {
|
|
373
|
+
const msg = event.message as AssistantMessage;
|
|
374
|
+
if (msg?.role !== "assistant" || msg.stopReason !== "error" || !msg.errorMessage) return;
|
|
375
|
+
if (isPermanentFailure(msg.errorMessage) && msg.provider && msg.model) {
|
|
376
|
+
banPermanently(msg.provider, msg.model);
|
|
377
|
+
}
|
|
378
|
+
if (ctx.model?.provider === "auto" && ctx.model?.id === "auto") {
|
|
379
|
+
const replacement = toFailoverRetryMessage(msg);
|
|
380
|
+
if (replacement) return { message: replacement };
|
|
381
|
+
}
|
|
382
|
+
});
|
|
383
|
+
|
|
384
|
+
// Selecting `auto` starts at thinking `low` by default (ceiling, raise it
|
|
385
|
+
// with shift+tab). `model_select` only fires on a user selection: pi never
|
|
386
|
+
// emits it at startup or when restoring a session.
|
|
387
|
+
pi.on("model_select", (event) => {
|
|
388
|
+
if (event.model.provider === "auto" && event.model.id === "auto") {
|
|
389
|
+
pi.setThinkingLevel("low");
|
|
390
|
+
}
|
|
391
|
+
});
|
|
392
|
+
|
|
393
|
+
pi.registerVirtualModel<RouteState>({
|
|
394
|
+
provider: "auto",
|
|
395
|
+
id: "auto",
|
|
396
|
+
name: "Auto (routing + thinking + failover)",
|
|
397
|
+
thinkingLevels: ["off", "low", "medium", "high", "xhigh"],
|
|
398
|
+
contextWindow: 200_000,
|
|
399
|
+
maxTokens: 64_000,
|
|
400
|
+
async route(request, ctx) {
|
|
401
|
+
const userCap = (request.thinkingLevel ?? "xhigh") as Thinking;
|
|
402
|
+
|
|
403
|
+
// Compaction/summaries outside the agent loop: no session state.
|
|
404
|
+
if (request.reason === "direct") {
|
|
405
|
+
return {
|
|
406
|
+
model: buildTiers(ctx).background,
|
|
407
|
+
thinkingLevel: "low",
|
|
408
|
+
};
|
|
409
|
+
}
|
|
410
|
+
|
|
411
|
+
const prevState = request.state;
|
|
412
|
+
|
|
413
|
+
// RETRY / failover (429 / 5xx)
|
|
414
|
+
if (request.reason === "retry" && request.failed) {
|
|
415
|
+
const failedKey = modelKey(request.failed.model);
|
|
416
|
+
if (isPermanentFailure(request.failed.message?.errorMessage ?? "")) {
|
|
417
|
+
banPermanently(request.failed.model.provider, request.failed.model.id);
|
|
418
|
+
}
|
|
419
|
+
const baseState: RouteState = prevState ?? {
|
|
420
|
+
modelId: request.failed.model.id,
|
|
421
|
+
providerId: request.failed.model.provider,
|
|
422
|
+
thinking: (request.failed.thinkingLevel ?? "medium") as Thinking,
|
|
423
|
+
avoid: [],
|
|
424
|
+
};
|
|
425
|
+
const avoid = [
|
|
426
|
+
...new Set([
|
|
427
|
+
...baseState.avoid,
|
|
428
|
+
// Provider quota: avoid the whole provider, otherwise the failover
|
|
429
|
+
// lands on another of its models and fails again immediately.
|
|
430
|
+
PROVIDER_LIMIT_RE.test(request.failed.message?.errorMessage ?? "")
|
|
431
|
+
? request.failed.model.provider
|
|
432
|
+
: failedKey,
|
|
433
|
+
]),
|
|
434
|
+
];
|
|
435
|
+
const tiers = buildTiers(ctx, avoid);
|
|
436
|
+
const ladder: TierName[] =
|
|
437
|
+
avoid.length >= 3 ? ["standard", "eco"] : ["strong", "standard", "eco"];
|
|
438
|
+
const chosen =
|
|
439
|
+
ladder
|
|
440
|
+
.map((t) => tiers[t])
|
|
441
|
+
.find((m) => m && !avoid.includes(modelKey(m)) && !avoid.includes(m.provider)) ??
|
|
442
|
+
tiers.frontier;
|
|
443
|
+
const th = capThinking(baseState.thinking, userCap);
|
|
444
|
+
return toRoute(pi, ctx, chosen, th, prevState, {
|
|
445
|
+
...baseState,
|
|
446
|
+
modelId: chosen.id,
|
|
447
|
+
providerId: chosen.provider,
|
|
448
|
+
thinking: th,
|
|
449
|
+
avoid,
|
|
450
|
+
});
|
|
451
|
+
}
|
|
452
|
+
|
|
453
|
+
// CONTINUATION (same turn, after tool calls)
|
|
454
|
+
if (request.reason === "continuation" && prevState) {
|
|
455
|
+
const avoid = prevState.avoid ?? [];
|
|
456
|
+
|
|
457
|
+
// First successful edit/write of the turn → switch to the eco tier for
|
|
458
|
+
// the rest of the turn (editedThisTurn stays true: toolResults accumulate).
|
|
459
|
+
if (editedThisTurn(request.messages)) {
|
|
460
|
+
const eco = buildTiers(ctx, avoid).eco;
|
|
461
|
+
const th = capThinking("medium", userCap);
|
|
462
|
+
return toRoute(pi, ctx, eco, th, prevState, {
|
|
463
|
+
...prevState,
|
|
464
|
+
modelId: eco.id,
|
|
465
|
+
providerId: eco.provider,
|
|
466
|
+
thinking: th,
|
|
467
|
+
avoid,
|
|
468
|
+
});
|
|
469
|
+
}
|
|
470
|
+
|
|
471
|
+
// No edit yet → keep the current model to preserve the prompt cache
|
|
472
|
+
// and the `avoid` list; if it was banned meanwhile, fall through to
|
|
473
|
+
// the safety fallback below.
|
|
474
|
+
const currentKey = modelKey({ provider: prevState.providerId, id: prevState.modelId });
|
|
475
|
+
if (
|
|
476
|
+
!permanentDenylist.has(currentKey) &&
|
|
477
|
+
!avoid.includes(currentKey) &&
|
|
478
|
+
!avoid.includes(prevState.providerId)
|
|
479
|
+
) {
|
|
480
|
+
const m = ctx.modelRegistry.find(prevState.providerId, prevState.modelId) as
|
|
481
|
+
| Model<never>
|
|
482
|
+
| undefined;
|
|
483
|
+
if (m) {
|
|
484
|
+
const th = capThinking(prevState.thinking, userCap);
|
|
485
|
+
return toRoute(pi, ctx, m, th, prevState, { ...prevState, thinking: th });
|
|
486
|
+
}
|
|
487
|
+
}
|
|
488
|
+
// Safety fallback: current model banned/missing → re-route the SAME
|
|
489
|
+
// turn (difficultyScore on the current message, `avoid` kept).
|
|
490
|
+
return rerouteCurrentTurn(pi, ctx, request, prevState.avoid, userCap, prevState);
|
|
491
|
+
}
|
|
492
|
+
|
|
493
|
+
// NEW TURN (user message)
|
|
494
|
+
return rerouteCurrentTurn(pi, ctx, request, [], userCap, prevState);
|
|
495
|
+
},
|
|
496
|
+
});
|
|
497
|
+
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,37 @@
|
|
|
1
1
|
{
|
|
2
|
-
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
2
|
+
"name": "pi-auto-route",
|
|
3
|
+
"version": "1.1.0",
|
|
4
|
+
"description": "A pi virtual model that routes each turn to the right physical model by task difficulty and adjusts the thinking level.",
|
|
5
|
+
"keywords": [
|
|
6
|
+
"pi-package",
|
|
7
|
+
"pi",
|
|
8
|
+
"virtual-model"
|
|
9
|
+
],
|
|
10
|
+
"license": "MIT",
|
|
11
|
+
"author": "rzph",
|
|
12
|
+
"repository": {
|
|
13
|
+
"type": "git",
|
|
14
|
+
"url": "git+https://github.com/rzph/pi-auto-route.git"
|
|
15
|
+
},
|
|
16
|
+
"homepage": "https://github.com/rzph/pi-auto-route#readme",
|
|
17
|
+
"bugs": {
|
|
18
|
+
"url": "https://github.com/rzph/pi-auto-route/issues"
|
|
19
|
+
},
|
|
20
|
+
"files": [
|
|
21
|
+
"auto-route.ts",
|
|
22
|
+
"README.md",
|
|
23
|
+
"CHANGELOG.md",
|
|
24
|
+
"LICENSE"
|
|
25
|
+
],
|
|
26
|
+
"pi": {
|
|
27
|
+
"extensions": [
|
|
28
|
+
"./auto-route.ts"
|
|
29
|
+
]
|
|
30
|
+
},
|
|
31
|
+
"peerDependencies": {
|
|
32
|
+
"@earendil-works/pi-ai": "*",
|
|
33
|
+
"@earendil-works/pi-coding-agent": "*",
|
|
34
|
+
"@earendil-works/pi-tui": "*",
|
|
35
|
+
"typebox": "*"
|
|
36
|
+
}
|
|
37
|
+
}
|