pi-provider-litellm 4.5.0 → 4.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +14 -3
- package/package.json +1 -1
- package/src/backend-identity.ts +1 -1
- package/src/budget.ts +70 -1
- package/src/discover.ts +23 -5
- package/src/index.ts +38 -9
package/README.md
CHANGED
|
@@ -345,6 +345,14 @@ LiteLLM ("litellm") budget
|
|
|
345
345
|
|
|
346
346
|
Automatic polls need Pi's interactive UI and do not run under `LITELLM_OFFLINE=1`, `LITELLM_DISCOVERY_TIMEOUT_MS=0`, or `PI_OFFLINE`; header updates still apply. `/litellm-budget` is stopped by the first two only. The credential and custom headers are kept in memory only as a keyed digest, and proxy responses are reduced to the numbers shown and never logged.
|
|
347
347
|
|
|
348
|
+
When LiteLLM refuses a request because a budget is used up, the error says which budget, its spend and limit, and when it resets, in local time, instead of the proxy's reply:
|
|
349
|
+
|
|
350
|
+
```
|
|
351
|
+
Error: LiteLLM ("litellm"): out of budget. The team budget is used up ($1,000.50 of $1,000) and resets in 2d (2026-11-01 01:00).
|
|
352
|
+
```
|
|
353
|
+
|
|
354
|
+
LiteLLM 1.104.0 and later answer such a request with 422, and earlier releases with 429 or 400; the extension recognises the refusal by its text, whatever the status. Pi does not retry it. It used to retry LiteLLM's own text whenever that held a number like `500`. The figures and the reset come from the budget status, polled again for each refusal under the conditions automatic polls need. Without them, with `budget.enabled` set to `false`, or when the level cannot be read, the error says only which budget is used up. Budgets the footer does not show (end user, tag, project, model access group, and the proxy's own budget), budget windows (`budget_limits`), and per-model budgets keep LiteLLM's text after `out of budget.`, because they reset on their own schedules.
|
|
355
|
+
|
|
348
356
|
## Model transport
|
|
349
357
|
|
|
350
358
|
A `/model/info` route group uses native Anthropic `/v1/messages` only when every deployment explicitly reports Chat mode, identifies a Claude backend through an Anthropic-, Bedrock-, or Vertex-capable adapter, and every declared backend identity resolves the same Anthropic compatibility policy. A `claude-<name>-<major>-<minor>` backend id resolves through its `claude-<name>-<major>` entry when the exact id is absent; an unknown generation stays on Chat Completions. When `supported_endpoints` is a list, it must include `/v1/messages`; an explicit endpoint list without it keeps the group on Chat Completions. A `null` value, which LiteLLM v1.100 and later report for models absent from their model map, counts as absent. An unresolved routing or base-model identity denies native Messages rather than being discarded from the unanimity check. Mixed-generation Claude groups can use Messages when their serializer policies match, with reasoning levels restricted to their common supported set. Groups with incompatible policies, mixed families, or unknown backend evidence remain on Chat Completions; unanimous explicit Responses mode takes precedence. Routes discovered through `/health` never select native Messages: Messages candidates are downgraded to Chat Completions with reasoning controls closed, while correlated Responses detail remains on Responses. Evidence-free `/v1/models` and wholly health-only `/health` groups take transport and presentation metadata from the bounded Pi catalog lookup: `openai-responses` when the catalog says so, otherwise Chat Completions. They expose no speculative reasoning selector and never select native Messages. Catalog identity such as Amazon Bedrock remains separate from the selected wire protocol.
|
|
@@ -456,6 +464,8 @@ The extension registers at startup and updates the registration before each agen
|
|
|
456
464
|
|
|
457
465
|
Before registering, the extension sends the proxy one MCP `initialize` with the credential and headers Pi will send, and ends the session it opened. LiteLLM 1.102.0 and later refuse it with 403 when no MCP servers are granted to the key, its team, or its organization, or none of them is allowed from your IP address. The extension then registers the server disabled: Pi does not connect to it, warn about it at startup, or turn on `codemode` for it, and `/mcp` lists it as disabled. Once servers are granted, enable it in `/mcp` for the session, or restart Pi, which checks again, as does a login. When a key helper (a `!command` key) or Google ADC is configured, the check is skipped and the server is registered enabled.
|
|
458
466
|
|
|
467
|
+
A proxy that checks budgets on `initialize` refuses it with its budget error once a budget is used up, and every MCP call then fails until the budget resets. The extension registers the server disabled in that case too, and says so once: `LiteLLM MCP ("litellm"): out of budget, so its MCP server is off for this session. Turn it on in /mcp once the budget resets.` After the reset, enable it in `/mcp` or restart Pi.
|
|
468
|
+
|
|
459
469
|
Pi reads the provider's credential when it sends each request, not when the server was registered. If another Pi process stores a login for a different proxy, this session's open connection can send that credential to the previous proxy until the next turn moves the server.
|
|
460
470
|
|
|
461
471
|
Earlier versions discovered tools through `/mcp-rest/tools/list` and registered them as `mcp_<server>_<tool>_<hash>`. Those names and the `<server>/<tool>` form of `toolExposure` keys no longer apply, and the `~/.pi/agent/litellm-mcp-pauses/` directory is no longer used.
|
|
@@ -545,7 +555,7 @@ Each distinct availability diagnostic is written once per session on stderr. `LI
|
|
|
545
555
|
|
|
546
556
|
Discovery selects native Messages for proven Claude groups under the [Model transport](#model-transport) rules, Responses for OpenAI-family backends, and Chat Completions for other OpenAI-compatible backends. Azure deployments stay on Chat Completions unless an explicit Responses mode or a `supported_endpoints` list containing `/v1/responses` authorizes Responses; `api_version` alone does not prove support. Azure AI (`azure_ai/` prefix or `custom_llm_provider: azure_ai`) deployments additionally depend on the proxy version. On LiteLLM before v1.103.0, or when the version is unknown, they ignore a `supported_endpoints` list that contains `/v1/responses` and need an explicit `mode: "responses"`: those releases have no native Responses config for that provider, so they bridge `/v1/responses` through Chat Completions, and the bridge fails streaming with `list index out of range` on Azure's empty-choices chunks (BerriAI/litellm#34455) while the endpoint list, copied from the model cost map, still advertises `/v1/responses`. From v1.103.0 the list is honored again. Discovery reads the version from the `x-litellm-version` header of one extra request, `GET /v1/responses/resp_version_probe`, sent only when such a deployment is published; LiteLLM answers it with an error and calls no model. A pre-release counts as older than the release it names, and a proxy that omits the header keeps these deployments on Chat Completions. Azure is detected from the adapter field, an `azure/` or `azure_ai/` model prefix, or the reported provider; other adapters follow the backend family because LiteLLM bridges `/v1/responses` to Chat Completions when the provider has no native Responses config (`litellm/responses/main.py`, `_bridges_to_chat_completions`). The family comes from the model id, not from LiteLLM's `openai/`, `openai_like/` or `custom_openai/` adapter prefix, so `openai/qwen3` stays on Chat Completions while `openai/gpt-5.5` uses Responses; a deployment opts into Responses with `mode: "responses"` or a `supported_endpoints` list containing `/v1/responses`. A route whose model prefix and `custom_llm_provider` name different providers is treated as unidentified and stays on Chat Completions, so it also does not get the local 128-tool preflight. When `/model/info` supplies `supported_endpoints`, that list takes precedence, and `mode: "responses"` remains an explicit Responses signal. An evidence-free fallback entry—a bare `/v1/models` id or a `/health` entry with no detail row—takes its protocol from the Pi catalog entry for its id; both evidence-free fallback paths also inherit presentation metadata from the bounded catalog lookup. An unknown id stays on Chat Completions. The route name alone authorizes nothing. A concrete id expanded from wildcard `/model/info` routes inherits the deployment evidence of the wildcard route LiteLLM would select for it, with the requested id substituted into the row's `model` the way LiteLLM serves it; the id is omitted when that selected route is not a published chat-style route. Chat Completions and Responses use `<root>/v1`; native Messages uses the proxy root directly. All three protocols pass through the provider's host guard. When an evidence-free cached `(no metadata)` entry can be enriched from the Pi catalog, it is restored on Responses only for a catalog Responses model and otherwise on Chat Completions, never Anthropic Messages. A model supplied only through `models.json` whose `api` this provider does not implement takes Pi's global API fallback rather than this provider's host guard; the credential-root pin (see [Model host enforcement](#model-host-enforcement)) still keeps the credential on the active proxy.
|
|
547
557
|
|
|
548
|
-
`cacheControlFormat: "anthropic"` applies only to Chat Completions, where Pi adds Anthropic `cache_control` markers. The Responses transport has a different compatibility type and uses native `prompt_cache_key` (plus supported retention fields), so Responses models must not receive `cacheControlFormat`. Freshly discovered OpenAI-family models forced onto Chat Completions are rejected locally when a request contains more than 128 tools; route them through Responses or reduce enabled extensions. Moonshot request-side reasoning suppression follows LiteLLM's routing provider, not `model_info.base_model`; response-side `<think>` normalization follows discovered Kimi-family evidence. Opaque aliases retain display normalization without sending Moonshot-only parameters to Azure- or Bedrock-hosted Kimi models. Strict tool-message repair requires every deployment to identify Kimi; Gemini effort normalization likewise requires consistent backend-family evidence. Claude-backed routes on native Messages or Chat Completions declare Pi's `promptCache` lifetimes (`short: 300`, `long: 3600` seconds, the values Pi's built-in Anthropic models use), so Pi's `cacheWarming` can keep their caches warm. Responses routes, routes identified only by their name, and other backend families declare none, so Pi does not warm them. A `modelOverrides` entry for the route in the global `models.json` can change the values, and `cacheWarming: "off"` disables warming.
|
|
558
|
+
`cacheControlFormat: "anthropic"` applies only to Chat Completions, where Pi adds Anthropic `cache_control` markers. The Responses transport has a different compatibility type and uses native `prompt_cache_key` (plus supported retention fields), so Responses models must not receive `cacheControlFormat`. Freshly discovered OpenAI-family models forced onto Chat Completions are rejected locally when a request contains more than 128 tools; route them through Responses or reduce enabled extensions. Moonshot request-side reasoning suppression follows LiteLLM's routing provider, not `model_info.base_model`; response-side `<think>` normalization follows discovered Kimi-family evidence. Opaque aliases retain display normalization without sending Moonshot-only parameters to Azure- or Bedrock-hosted Kimi models. Strict tool-message repair requires every deployment to identify Kimi; Gemini effort normalization likewise requires consistent backend-family evidence. Claude-backed routes on native Messages or Chat Completions declare Pi's `promptCache` lifetimes (`short: 300`, `long: 3600` seconds, the values Pi's built-in Anthropic models use), so Pi's `cacheWarming` can keep their caches warm. Responses routes whose every deployment is GPT-5.6 or later and routes through LiteLLM's `openai`, `azure` or `azure_ai` provider declare `short: 1800`, the 30-minute default minimum lifetime OpenAI and Azure document for those models, and no `long` value. Claude routes on Responses, Chat routes to OpenAI models, earlier GPT models, GPT models served through a relay such as OpenRouter or GitHub Copilot, routes identified only by their name, and other backend families declare none, so Pi does not warm them. A `modelOverrides` entry for the route in the global `models.json` can change the values, and `cacheWarming: "off"` disables warming.
|
|
549
559
|
|
|
550
560
|
Discovery policy is versioned in Pi's persisted model catalog. After upgrading, an online refresh replaces legacy entries. Offline legacy entries remain usable without a diagnostic; the extension reports the discovery-version mismatch once only when an attempted online refresh fails. To force rediscovery or roll back across this policy change, delete the `litellm` entry from `~/.pi/agent/models-store.json` (or delete the file) and refresh once online.
|
|
551
561
|
|
|
@@ -608,14 +618,15 @@ The development probe runs against minimized snapshots with `npm run probe:proxy
|
|
|
608
618
|
| `LiteLLM discovery: ... route group(s) mix chat-style and explicitly incompatible deployment modes` | A public route can select both a chat-style deployment and an incompatible deployment such as an embedding model. Give the embedding deployment its own `model_name`. |
|
|
609
619
|
| `LiteLLM (...): the request for "<route>" was answered by a fallback` | A successful HTTP response reports a LiteLLM router fallback (`x-litellm-attempted-fallbacks`). `/model/info` does not expose fallbacks, so protocol and model handling follow the requested route's own deployments. If the primary and fallback groups support Chat Completions, set `model_info.supported_endpoints: ["/v1/chat/completions"]` on the primary to avoid Responses-to-Chat bridging. This warning does not change routing or fix upstream bridging defects. It names only the requested route; LiteLLM reports the group and model that answered in the `x-litellm-model-group` and `x-litellm-model-name` response headers. Warned once per provider and route until the extension reloads; shown in the UI, or stderr without a UI. |
|
|
610
620
|
| A model is marked ` (incomplete metadata)` | `/model/info` or `/health` did not provide enough authoritative metadata. Explicit fields remain usable, but unknown cost fields are shown as zero and route-name cache enrichment stays disabled. |
|
|
611
|
-
| Cache miss notices after idle, no cache warming | The model declares no `promptCache`: only Claude-backed Messages and Chat routes get one. Run `/litellm-refresh` after updating so stored models are rediscovered. |
|
|
621
|
+
| Cache miss notices after idle, no cache warming | The model declares no `promptCache`: only Claude-backed Messages and Chat routes and GPT-5.6+ Responses routes on OpenAI or Azure get one. Run `/litellm-refresh` after updating so stored models are rediscovered. |
|
|
612
622
|
| `401 Token expired` | Set `LITELLM_API_KEY_HELPER`. |
|
|
613
623
|
| No models with gcloud auth | Verify `gcloud auth application-default login` has been run or set `GOOGLE_APPLICATION_CREDENTIALS` to an `authorized_user` ADC file |
|
|
614
624
|
| Enterprise SSO waits for token insertion | The proxy returned 404/405 for `/sso/cli/start`, so Pi used the legacy flow — upgrade LiteLLM or paste the UI token |
|
|
615
625
|
| Enterprise CLI SSO start/poll fails | Check the proxy logs and verify `/sso/cli/start` and `/sso/cli/poll/{login_id}` are reachable; only 404/405 falls back to legacy login |
|
|
616
626
|
| Enterprise SSO login shows "virtual key generation failed" | The LiteLLM instance may lack a database (`/key/generate` requires one), your user account may lack key-generation permission, or the request timed out; the JWT is used directly as a fallback |
|
|
617
627
|
| Enterprise SSO token prompt fails with "SSO token is required" | The token field was left empty — paste the token copied from the LiteLLM UI |
|
|
618
|
-
|
|
|
628
|
+
| `LiteLLM (...): out of budget` | A LiteLLM budget is used up. The error names it and, when the budget status can read it, its reset time; `/litellm-budget` shows every level. Ask the proxy admin to raise the limit, or wait for the reset — see [Budget status](#budget-status) |
|
|
629
|
+
| MCP tools not showing | Run `/mcp` and check the provider's server: its state shows a connection or sign-in error. `disabled` means the proxy refused the key at startup, usually because no MCP servers are granted to it, its team, or its organization, or because a budget is used up, which is reported at startup; after a grant or the budget's reset, enable it in `/mcp` or restart Pi. Verify the proxy serves `/mcp` and that the key has MCP access |
|
|
619
630
|
| MCP tools listed but the model cannot call them | `codemode` and `deferred` tools are reached through Pi's `codemode` and `tool_search` tools. Check that neither is disabled; `--no-extensions` disables them and Pi's MCP client alike. Setting `litellm.mcp.exposure` to `direct` declares the tools on every request instead |
|
|
620
631
|
| Skills not affecting prompts | Verify the proxy exposes `/claude-code/marketplace.json` or `/v1/skills` and returns enabled skills |
|
|
621
632
|
|
package/package.json
CHANGED
package/src/backend-identity.ts
CHANGED
package/src/budget.ts
CHANGED
|
@@ -311,6 +311,65 @@ export function formatBudgetDetails(
|
|
|
311
311
|
return lines.join("\n");
|
|
312
312
|
}
|
|
313
313
|
|
|
314
|
+
// LiteLLM refuses a used-up budget with `type: "budget_exceeded"` (status 422 since 1.104, 429 or 400 before), and
|
|
315
|
+
// /mcp with the bare message, whose wording varies by level and check.
|
|
316
|
+
const BUDGET_EXCEEDED = /budget_exceeded|Budget has been exceeded!|ExceededBudget:|exceeded budget for model=/;
|
|
317
|
+
// The refusals of a level's own budget. Budget windows and per-model budgets reset on schedules of their own.
|
|
318
|
+
const EXCEEDED_LEVEL =
|
|
319
|
+
/Budget has been exceeded! (Key|Team|TeamMember|User|Organization)=\S*( in Team=)?|ExceededBudget: (User)=.+? over budget\./;
|
|
320
|
+
const EXCEEDED_LEVELS: Record<string, BudgetLevelName> = {
|
|
321
|
+
Key: "key",
|
|
322
|
+
Team: "team",
|
|
323
|
+
TeamMember: "member",
|
|
324
|
+
User: "user",
|
|
325
|
+
Organization: "org",
|
|
326
|
+
};
|
|
327
|
+
const LEVEL_LABELS: Record<BudgetLevelName, string> = {
|
|
328
|
+
key: "key",
|
|
329
|
+
user: "user",
|
|
330
|
+
team: "team",
|
|
331
|
+
member: "team member",
|
|
332
|
+
org: "organization",
|
|
333
|
+
};
|
|
334
|
+
|
|
335
|
+
export const isBudgetExceeded = (text: string): boolean => BUDGET_EXCEEDED.test(text);
|
|
336
|
+
|
|
337
|
+
// Local time in one fixed format, which the Go port prints the same way.
|
|
338
|
+
function resetClock(resetAt: number): string {
|
|
339
|
+
const at = new Date(resetAt);
|
|
340
|
+
const pad = (n: number) => String(n).padStart(2, "0");
|
|
341
|
+
return `${at.getFullYear()}-${pad(at.getMonth() + 1)}-${pad(at.getDate())} ${pad(at.getHours())}:${pad(at.getMinutes())}`;
|
|
342
|
+
}
|
|
343
|
+
|
|
344
|
+
// Starts with "out of budget", which Pi and PiG never retry. A refusal whose level is not tracked keeps the proxy's
|
|
345
|
+
// text, as that names the budget.
|
|
346
|
+
export function formatBudgetExceeded(errorText: string, levels: BudgetLevels, now: number): string {
|
|
347
|
+
const match = EXCEEDED_LEVEL.exec(errorText);
|
|
348
|
+
if (!match) return `out of budget. ${errorText}`;
|
|
349
|
+
const name = match[2] ? "member" : EXCEEDED_LEVELS[match[1] ?? match[3]!]!;
|
|
350
|
+
const l = levels[name];
|
|
351
|
+
const amounts = l ? ` (${money(l.spend, 2)} of ${money(l.maxBudget, Number.isInteger(l.maxBudget) ? 0 : 2)})` : "";
|
|
352
|
+
const reset = formatRelativeReset(l?.resetAt, now);
|
|
353
|
+
const resets = reset ? ` and resets in ${reset} (${resetClock(l!.resetAt!)})` : "";
|
|
354
|
+
return `out of budget. The ${LEVEL_LABELS[name]} budget is used up${amounts}${resets}.`;
|
|
355
|
+
}
|
|
356
|
+
|
|
357
|
+
export function setupLiteLLMBudgetErrors(
|
|
358
|
+
pi: ExtensionAPI,
|
|
359
|
+
providerNames: ReadonlySet<string>,
|
|
360
|
+
levelsAfterError?: (ctx: ExtensionContext, name: string) => Promise<BudgetLevels>,
|
|
361
|
+
): void {
|
|
362
|
+
pi.on("message_end", async (event, ctx) => {
|
|
363
|
+
const message = event.message;
|
|
364
|
+
if (message.role !== "assistant" || message.stopReason !== "error" || !providerNames.has(message.provider)) return;
|
|
365
|
+
const text = message.errorMessage ?? "";
|
|
366
|
+
if (!isBudgetExceeded(text)) return;
|
|
367
|
+
const levels = (await levelsAfterError?.(ctx, message.provider).catch(() => undefined)) ?? {};
|
|
368
|
+
const explained = formatBudgetExceeded(text, levels, Date.now());
|
|
369
|
+
return { message: { ...message, errorMessage: `LiteLLM (${JSON.stringify(message.provider)}): ${explained}` } };
|
|
370
|
+
});
|
|
371
|
+
}
|
|
372
|
+
|
|
314
373
|
export function budgetDisplaySetting(budgetSettings: unknown): { display: BudgetDisplay; warning?: string } {
|
|
315
374
|
const display = obj(budgetSettings)?.display;
|
|
316
375
|
if (display === undefined || display === "all" || display === "tightest") return { display: display ?? "all" };
|
|
@@ -344,7 +403,10 @@ interface ProviderState {
|
|
|
344
403
|
inFlight?: Promise<PollOutcome>;
|
|
345
404
|
}
|
|
346
405
|
|
|
347
|
-
export function setupLiteLLMBudget(
|
|
406
|
+
export function setupLiteLLMBudget(
|
|
407
|
+
pi: ExtensionAPI,
|
|
408
|
+
options: BudgetOptions,
|
|
409
|
+
): (ctx: ExtensionContext, name: string) => Promise<BudgetLevels> {
|
|
348
410
|
const displayNames = new Map(options.providers.map(({ name, displayName }) => [name, displayName]));
|
|
349
411
|
const states = new Map<string, ProviderState>();
|
|
350
412
|
// Keyed like the MCP registration identity in index.ts, because custom headers can carry gateway credentials.
|
|
@@ -558,4 +620,11 @@ export function setupLiteLLMBudget(pi: ExtensionAPI, options: BudgetOptions): vo
|
|
|
558
620
|
}
|
|
559
621
|
},
|
|
560
622
|
});
|
|
623
|
+
|
|
624
|
+
// The levels a budget refusal is explained with. Polled for each refusal, joining a poll in flight, because only a
|
|
625
|
+
// poll notices that the credential changed since the last one.
|
|
626
|
+
return async (ctx, name) => {
|
|
627
|
+
if (ctx?.hasUI && !gated()) await poll(ctx, name);
|
|
628
|
+
return stateFor(name).levels;
|
|
629
|
+
};
|
|
561
630
|
}
|
package/src/discover.ts
CHANGED
|
@@ -55,6 +55,11 @@ const DEFAULT_TIMEOUT_MS = 5000;
|
|
|
55
55
|
// shortcut: `long` assumes the host offers the 1-hour TTL (Bedrock only does for Claude 4.5+), gate it on
|
|
56
56
|
// deployment routing if LiteLLM ever reports per-route TTL support.
|
|
57
57
|
const CLAUDE_PROMPT_CACHE = { short: 300, long: 3600 } as const;
|
|
58
|
+
// OpenAI's and Azure's documented 30-minute default minimum for GPT-5.6 and later. No `long`: the
|
|
59
|
+
// 12-hour lifetime needs `prompt_cache_options.ttl`, which Pi's long retention does not request.
|
|
60
|
+
const OPENAI_PROMPT_CACHE = { short: 1800 } as const;
|
|
61
|
+
// Relays (OpenRouter, Copilot, resellers) document no lifetime of their own and may bill or route differently.
|
|
62
|
+
const OPENAI_PROMPT_CACHE_ROUTES: ReadonlySet<string> = new Set(["openai", "azure", "azure_ai"]);
|
|
58
63
|
const HEALTH_DETAIL_CONCURRENCY = 8;
|
|
59
64
|
const reportedConflictingFamilyRoutes = new Set<string>();
|
|
60
65
|
const reportedWithheldRepairRoutes = new Set<string>();
|
|
@@ -132,6 +137,12 @@ export function isGpt55OrNewerModel(modelId: string): boolean {
|
|
|
132
137
|
return generation !== undefined && (generation[0] > 5 || (generation[0] === 5 && generation[1] >= 5));
|
|
133
138
|
}
|
|
134
139
|
|
|
140
|
+
// GPT-5.6 and later keep a cached prefix for at least 30 minutes by default.
|
|
141
|
+
function isGpt56OrNewerModel(modelId: string): boolean {
|
|
142
|
+
const generation = gptGeneration(modelId);
|
|
143
|
+
return generation !== undefined && (generation[0] > 5 || (generation[0] === 5 && generation[1] >= 6));
|
|
144
|
+
}
|
|
145
|
+
|
|
135
146
|
// GPT-6 also rejects an omitted effort there: it falls back to its default effort.
|
|
136
147
|
export function isGpt6OrNewerModel(modelId: string): boolean {
|
|
137
148
|
const generation = gptGeneration(modelId);
|
|
@@ -726,7 +737,8 @@ function mapFromModelInfoGroup(
|
|
|
726
737
|
const protocols = entries.map((entry) => modelProtocol(reduced.id, entry, options.proxyVersion));
|
|
727
738
|
const protocol = protocols.find((candidate) => candidate.api === "openai-completions") ?? protocols[0]!;
|
|
728
739
|
const api = reduced.api === "anthropic-messages" ? reduced.api : protocol.api;
|
|
729
|
-
const
|
|
740
|
+
const identities = entries.map((entry) => resolveBackendIdentity({ ...entry, model_name: undefined }));
|
|
741
|
+
const families = new Set(identities.map((identity) => identity?.family));
|
|
730
742
|
const [family] = families;
|
|
731
743
|
const hasBackendEvidence = entries.some(hasReadableBackendEvidence);
|
|
732
744
|
const reasoningPolicy =
|
|
@@ -788,10 +800,7 @@ function mapFromModelInfoGroup(
|
|
|
788
800
|
);
|
|
789
801
|
// One GPT-5.5+ deployment is enough: a tool request routed to it fails with
|
|
790
802
|
// reasoning attached, while dropping reasoning elsewhere only loses effort.
|
|
791
|
-
const chatBackendIds =
|
|
792
|
-
api === "openai-completions"
|
|
793
|
-
? entries.flatMap((entry) => resolveBackendIdentity({ ...entry, model_name: undefined })?.modelId ?? [])
|
|
794
|
-
: [];
|
|
803
|
+
const chatBackendIds = api === "openai-completions" ? identities.flatMap((identity) => identity?.modelId ?? []) : [];
|
|
795
804
|
const modelPolicy: LiteLLMModelPolicy | undefined = chatBackendIds.some(isGpt55OrNewerModel)
|
|
796
805
|
? {
|
|
797
806
|
normalizeStrictToolMessages: false,
|
|
@@ -821,6 +830,15 @@ function mapFromModelInfoGroup(
|
|
|
821
830
|
...(families.size === 1 && family === "claude" && relaysAnthropicCacheMarkers
|
|
822
831
|
? { promptCache: { ...CLAUDE_PROMPT_CACHE } }
|
|
823
832
|
: {}),
|
|
833
|
+
...(api === "openai-responses" &&
|
|
834
|
+
identities.every(
|
|
835
|
+
(identity, index) =>
|
|
836
|
+
identity?.family === "openai" &&
|
|
837
|
+
isGpt56OrNewerModel(identity.modelId) &&
|
|
838
|
+
routesOnlyThrough(entries[index]!, OPENAI_PROMPT_CACHE_ROUTES),
|
|
839
|
+
)
|
|
840
|
+
? { promptCache: { ...OPENAI_PROMPT_CACHE } }
|
|
841
|
+
: {}),
|
|
824
842
|
...(modelPolicy ? { litellmPolicy: modelPolicy } : {}),
|
|
825
843
|
// closeSerializerPolicy derives compat from `api`, a correlation its return type cannot carry.
|
|
826
844
|
} as DiscoveredModel;
|
package/src/index.ts
CHANGED
|
@@ -18,7 +18,7 @@ import type { ExtensionAPI, ExtensionContext, McpServerConfig } from "@earendil-
|
|
|
18
18
|
// A namespace import, so a Pi without `VERSION` reaches the version check instead of failing to link.
|
|
19
19
|
import * as piCodingAgent from "@earendil-works/pi-coding-agent";
|
|
20
20
|
import { getAgentDir, readStoredCredential } from "@earendil-works/pi-coding-agent";
|
|
21
|
-
import { budgetDisplaySetting, setupLiteLLMBudget } from "./budget.js";
|
|
21
|
+
import { budgetDisplaySetting, isBudgetExceeded, setupLiteLLMBudget, setupLiteLLMBudgetErrors } from "./budget.js";
|
|
22
22
|
import type { LiteLLMClassifierModel } from "./classifiers.js";
|
|
23
23
|
import { setupLiteLLMCostTracking } from "./cost.js";
|
|
24
24
|
import {
|
|
@@ -723,6 +723,19 @@ function piLiteral(value: string): string {
|
|
|
723
723
|
return escaped.startsWith("!") ? `$${escaped}` : escaped;
|
|
724
724
|
}
|
|
725
725
|
|
|
726
|
+
// The first 64 KiB of a body, as much as the MCP access check reads of a refusal.
|
|
727
|
+
async function readBodyPrefix(response: Response, limit = 64 * 1024): Promise<string> {
|
|
728
|
+
if (!response.body) return "";
|
|
729
|
+
const decoder = new TextDecoder();
|
|
730
|
+
let text = "";
|
|
731
|
+
for await (const chunk of response.body) {
|
|
732
|
+
text += decoder.decode(chunk, { stream: true });
|
|
733
|
+
// Leaving the loop cancels the rest of the stream.
|
|
734
|
+
if (text.length >= limit) break;
|
|
735
|
+
}
|
|
736
|
+
return text;
|
|
737
|
+
}
|
|
738
|
+
|
|
726
739
|
/**
|
|
727
740
|
* LiteLLM's /model/info is the metadata authority, so models.dev is an opt-in escape hatch.
|
|
728
741
|
* Without a cache path and with refresh suppressed, discovery reads neither the models.dev
|
|
@@ -2501,8 +2514,12 @@ export default async function (pi: ExtensionAPI): Promise<void> {
|
|
|
2501
2514
|
// LiteLLM 1.102.0 and later answer `initialize` with 403 when the key has no MCP servers granted,
|
|
2502
2515
|
// and Pi warns at every start about a server that fails to connect. Asking first lets such a
|
|
2503
2516
|
// server be registered disabled: Pi neither connects to it nor warns, and /mcp still lists it, so
|
|
2504
|
-
// the user can enable it once servers are granted.
|
|
2505
|
-
|
|
2517
|
+
// the user can enable it once servers are granted. A used-up budget fails every MCP request until
|
|
2518
|
+
// it resets, so it is refused the same way, with a notice.
|
|
2519
|
+
async function mcpAccessRefused(
|
|
2520
|
+
definition: ProviderDefinition,
|
|
2521
|
+
url: string,
|
|
2522
|
+
): Promise<"no-servers" | "out-of-budget" | undefined> {
|
|
2506
2523
|
try {
|
|
2507
2524
|
const stored = readStoredCredential(definition.name, join(getAgentDir(), "auth.json"));
|
|
2508
2525
|
// The check must send the credential Pi will send. Like seeding, it runs no key helper and
|
|
@@ -2515,11 +2532,11 @@ export default async function (pi: ExtensionAPI): Promise<void> {
|
|
|
2515
2532
|
definition.apiKeyConfig?.startsWith("!") ||
|
|
2516
2533
|
useGcloudToken
|
|
2517
2534
|
) {
|
|
2518
|
-
return
|
|
2535
|
+
return undefined;
|
|
2519
2536
|
}
|
|
2520
2537
|
const auth = await authForCredential(definition, stored, false);
|
|
2521
2538
|
// The credential goes only to its own root.
|
|
2522
|
-
if (`${auth.baseUrl}/mcp` !== url) return
|
|
2539
|
+
if (`${auth.baseUrl}/mcp` !== url) return undefined;
|
|
2523
2540
|
// The credential and headers Pi's client sends, so the proxy answers as it will answer Pi.
|
|
2524
2541
|
const headers = { ...auth.headers, Authorization: `Bearer ${auth.apiKey}` };
|
|
2525
2542
|
const response = await fetch(url, {
|
|
@@ -2538,7 +2555,9 @@ export default async function (pi: ExtensionAPI): Promise<void> {
|
|
|
2538
2555
|
redirect: "manual",
|
|
2539
2556
|
signal: AbortSignal.timeout(getSeedTimeoutMs()),
|
|
2540
2557
|
});
|
|
2541
|
-
|
|
2558
|
+
// Only a refusal is read, and only its start: a granted session can answer with an open event stream.
|
|
2559
|
+
const refusal = response.ok ? undefined : await readBodyPrefix(response).catch(() => "");
|
|
2560
|
+
if (refusal === undefined) await response.body?.cancel().catch(() => undefined);
|
|
2542
2561
|
// Pi opens its own session; end the one this check opened.
|
|
2543
2562
|
const session = response.headers.get("mcp-session-id");
|
|
2544
2563
|
if (session) {
|
|
@@ -2551,9 +2570,11 @@ export default async function (pi: ExtensionAPI): Promise<void> {
|
|
|
2551
2570
|
.then((closed) => closed.body?.cancel())
|
|
2552
2571
|
.catch(() => undefined);
|
|
2553
2572
|
}
|
|
2554
|
-
|
|
2573
|
+
// The budget refusal's status is configurable on the proxy, so its text is checked first.
|
|
2574
|
+
if (refusal !== undefined && isBudgetExceeded(refusal)) return "out-of-budget";
|
|
2575
|
+
return response.status === 403 ? "no-servers" : undefined;
|
|
2555
2576
|
} catch {
|
|
2556
|
-
return
|
|
2577
|
+
return undefined;
|
|
2557
2578
|
}
|
|
2558
2579
|
}
|
|
2559
2580
|
|
|
@@ -2574,6 +2595,12 @@ export default async function (pi: ExtensionAPI): Promise<void> {
|
|
|
2574
2595
|
try {
|
|
2575
2596
|
pi.registerMcpServer(definition.name, refused ? { ...config, enabled: false } : config);
|
|
2576
2597
|
attempt.registered = true;
|
|
2598
|
+
if (refused === "out-of-budget") {
|
|
2599
|
+
notifyMcp(
|
|
2600
|
+
`LiteLLM MCP (${JSON.stringify(definition.name)}): out of budget, so its MCP server is off for this ` +
|
|
2601
|
+
"session. Turn it on in /mcp once the budget resets.",
|
|
2602
|
+
);
|
|
2603
|
+
}
|
|
2577
2604
|
} catch (error) {
|
|
2578
2605
|
// The claim stays, so the refusal is reported once rather than on every turn.
|
|
2579
2606
|
notifyMcp(
|
|
@@ -2878,10 +2905,11 @@ export default async function (pi: ExtensionAPI): Promise<void> {
|
|
|
2878
2905
|
|
|
2879
2906
|
setupLiteLLMCostTracking(pi, [...providerNames]);
|
|
2880
2907
|
|
|
2908
|
+
let budgetLevels: ReturnType<typeof setupLiteLLMBudget> | undefined;
|
|
2881
2909
|
if (budgetEnabled) {
|
|
2882
2910
|
const { display, warning } = budgetDisplaySetting(settings?.budget);
|
|
2883
2911
|
const definitionNamed = (name: string) => definitions.find((definition) => definition.name === name)!;
|
|
2884
|
-
setupLiteLLMBudget(pi, {
|
|
2912
|
+
budgetLevels = setupLiteLLMBudget(pi, {
|
|
2885
2913
|
providers: definitions.map(({ name, displayName }) => ({ name, displayName })),
|
|
2886
2914
|
display,
|
|
2887
2915
|
displayWarning: warning,
|
|
@@ -2892,6 +2920,7 @@ export default async function (pi: ExtensionAPI): Promise<void> {
|
|
|
2892
2920
|
missingCredentials: (name) => missingCredentials(definitionNamed(name)),
|
|
2893
2921
|
});
|
|
2894
2922
|
}
|
|
2923
|
+
setupLiteLLMBudgetErrors(pi, providerNames, budgetLevels);
|
|
2895
2924
|
|
|
2896
2925
|
if (skillsEnabled) {
|
|
2897
2926
|
for (const tool of createSkillToolDefinitions(resolveDefaultRuntimeAuth)) {
|