pi-provider-litellm 4.4.2 → 4.6.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -345,6 +345,14 @@ LiteLLM ("litellm") budget
345
345
 
346
346
  Automatic polls need Pi's interactive UI and do not run under `LITELLM_OFFLINE=1`, `LITELLM_DISCOVERY_TIMEOUT_MS=0`, or `PI_OFFLINE`; header updates still apply. `/litellm-budget` is stopped by the first two only. The credential and custom headers are kept in memory only as a keyed digest, and proxy responses are reduced to the numbers shown and never logged.
347
347
 
348
+ When LiteLLM refuses a request because a budget is used up, the error says which budget, its spend and limit, and when it resets, in local time, instead of the proxy's reply:
349
+
350
+ ```
351
+ Error: LiteLLM ("litellm"): out of budget. The team budget is used up ($1,000.50 of $1,000) and resets in 2d (2026-11-01 01:00).
352
+ ```
353
+
354
+ LiteLLM 1.104.0 and later answer such a request with 422, and earlier releases with 429 or 400; the extension recognises the refusal by its text, whatever the status. Pi does not retry it. It used to retry LiteLLM's own text whenever that held a number like `500`. The figures and the reset come from the budget status, polled again for each refusal under the conditions automatic polls need. Without them, with `budget.enabled` set to `false`, or when the level cannot be read, the error says only which budget is used up. Budgets the footer does not show (end user, tag, project, model access group, and the proxy's own budget), budget windows (`budget_limits`), and per-model budgets keep LiteLLM's text after `out of budget.`, because they reset on their own schedules.
355
+
348
356
  ## Model transport
349
357
 
350
358
  A `/model/info` route group uses native Anthropic `/v1/messages` only when every deployment explicitly reports Chat mode, identifies a Claude backend through an Anthropic-, Bedrock-, or Vertex-capable adapter, and every declared backend identity resolves the same Anthropic compatibility policy. A `claude-<name>-<major>-<minor>` backend id resolves through its `claude-<name>-<major>` entry when the exact id is absent; an unknown generation stays on Chat Completions. When `supported_endpoints` is a list, it must include `/v1/messages`; an explicit endpoint list without it keeps the group on Chat Completions. A `null` value, which LiteLLM v1.100 and later report for models absent from their model map, counts as absent. An unresolved routing or base-model identity denies native Messages rather than being discarded from the unanimity check. Mixed-generation Claude groups can use Messages when their serializer policies match, with reasoning levels restricted to their common supported set. Groups with incompatible policies, mixed families, or unknown backend evidence remain on Chat Completions; unanimous explicit Responses mode takes precedence. Routes discovered through `/health` never select native Messages: Messages candidates are downgraded to Chat Completions with reasoning controls closed, while correlated Responses detail remains on Responses. Evidence-free `/v1/models` and wholly health-only `/health` groups take transport and presentation metadata from the bounded Pi catalog lookup: `openai-responses` when the catalog says so, otherwise Chat Completions. They expose no speculative reasoning selector and never select native Messages. Catalog identity such as Amazon Bedrock remains separate from the selected wire protocol.
@@ -375,6 +383,38 @@ and groups that mix image and other modes, or wildcard routes, are withheld. The
375
383
  need `/v1/images/edits`, which this provider does not call, and backends that answer with image URLs instead of
376
384
  base64 data are reported as errors rather than fetched.
377
385
 
386
+ ## Classification
387
+
388
+ Decision routes are published as Pi classifier models (for example `litellm/decision-1`, backed by
389
+ `azure_ai/Microsoft-Decision-1`, or a `typesafe/jev-latest` route). A route qualifies when every deployment behind it
390
+ reports `mode: "evaluation"` or `"decisions"` in `/model/info`, or lists `/v1/systemone` or `/v1/decisions` in
391
+ `supported_endpoints`; a declared endpoint list without them withholds the route. Like Pi's other classifiers they do
392
+ not appear in `/model`; scripts reach them through Pi's `codemode` tool:
393
+
394
+ ```js
395
+ const judge = await models.getModelOfType("classifier", "litellm", "decision-1");
396
+ const result = await models.classify(judge, {
397
+ state: { text: "Please send me the report by Friday." },
398
+ questions: {
399
+ polite: {
400
+ type: "bool",
401
+ instructions: "Is the message polite?",
402
+ criteria: { true: "The message is polite.", false: "The message is rude." },
403
+ },
404
+ },
405
+ });
406
+ if (result.stopReason !== "stop") return result.errorMessage;
407
+ return result.answers.polite.probability;
408
+ ```
409
+
410
+ Question types are `bool`, `choice` and `score`. Requests go to the proxy's `/v1/systemone` (LiteLLM v1.104.2 and
411
+ later) with the provider's own credential and headers, and are budgeted and logged by the proxy like chat requests. A
412
+ `bool` question is sent as System One's `noul`. Usage cost comes from LiteLLM's `x-litellm-response-cost` header when
413
+ present; otherwise it is estimated from `input_cost_per_token` and `output_cost_per_token`. Only routes found through
414
+ `/model/info` qualify, and groups that mix decision and other deployments, or wildcard routes, are withheld. The state
415
+ is text only: images in the context return an error without a request. A chat route that also lists `/v1/decisions`,
416
+ such as OpenAI's GPT-6 Luna, is published as both a chat model and a classifier.
417
+
378
418
  ## Optional environment variables
379
419
 
380
420
  | Variable | Default | Effect |
@@ -393,6 +433,7 @@ base64 data are reported as errors rather than fetched.
393
433
  | `LITELLM_CLI_JWT_EXPIRATION_HOURS` | `24` | CLI SSO token lifetime fallback for older proxies whose poll response omits `expires_in`; mirror a non-default proxy setting locally |
394
434
  | `LITELLM_VERBOSE_DISCOVERY` | unset | If `1`, enable progress messages during model discovery (login, refresh, startup), including startup skip reasons and defaulted `/model/info` context windows. Progress messages are off by default |
395
435
  | `LITELLM_DEFAULT_CONTEXT_WINDOW` | `128000` | Context window assumed when neither LiteLLM's `/model/info` nor the catalog reports `max_input_tokens`. Set a positive integer for proxies whose custom aliases carry no metadata; an unset or unusable value keeps 128K |
436
+ | `LITELLM_TRUST_PROXY_LIMITS` | unset | If `1`, a deployment's explicit `model_info.max_input_tokens` / `max_output_tokens` win over Pi's catalog instead of taking the smaller of the two; the catalog still fills missing values and the group minimum still applies. A `models.json` `modelOverrides` entry still wins first |
396
437
  | `LITELLM_MODELS_DEV` | unset | If `1`, enrich discovered metadata (limits, prices, effort lists) from models.dev, cached for 28 days in `litellm-models-dev.json`. Off by default because LiteLLM's `/model/info` is authoritative; use it when LiteLLM's model map lacks a model's metadata |
397
438
 
398
439
  Only use a trusted `LITELLM_API_KEY_HELPER` or `!command`. Prefer an absolute executable path, keep secrets out of command arguments, and print only the token to stdout without logging it to stderr.
@@ -423,6 +464,8 @@ The extension registers at startup and updates the registration before each agen
423
464
 
424
465
  Before registering, the extension sends the proxy one MCP `initialize` with the credential and headers Pi will send, and ends the session it opened. LiteLLM 1.102.0 and later refuse it with 403 when no MCP servers are granted to the key, its team, or its organization, or none of them is allowed from your IP address. The extension then registers the server disabled: Pi does not connect to it, warn about it at startup, or turn on `codemode` for it, and `/mcp` lists it as disabled. Once servers are granted, enable it in `/mcp` for the session, or restart Pi, which checks again, as does a login. When a key helper (a `!command` key) or Google ADC is configured, the check is skipped and the server is registered enabled.
425
466
 
467
+ A proxy that checks budgets on `initialize` refuses it with its budget error once a budget is used up, and every MCP call then fails until the budget resets. The extension registers the server disabled in that case too, and says so once: `LiteLLM MCP ("litellm"): out of budget, so its MCP server is off for this session. Turn it on in /mcp once the budget resets.` After the reset, enable it in `/mcp` or restart Pi.
468
+
426
469
  Pi reads the provider's credential when it sends each request, not when the server was registered. If another Pi process stores a login for a different proxy, this session's open connection can send that credential to the previous proxy until the next turn moves the server.
427
470
 
428
471
  Earlier versions discovered tools through `/mcp-rest/tools/list` and registered them as `mcp_<server>_<tool>_<hash>`. Those names and the `<server>/<tool>` form of `toolExposure` keys no longer apply, and the `~/.pi/agent/litellm-mcp-pauses/` directory is no longer used.
@@ -512,7 +555,7 @@ Each distinct availability diagnostic is written once per session on stderr. `LI
512
555
 
513
556
  Discovery selects native Messages for proven Claude groups under the [Model transport](#model-transport) rules, Responses for OpenAI-family backends, and Chat Completions for other OpenAI-compatible backends. Azure deployments stay on Chat Completions unless an explicit Responses mode or a `supported_endpoints` list containing `/v1/responses` authorizes Responses; `api_version` alone does not prove support. Azure AI (`azure_ai/` prefix or `custom_llm_provider: azure_ai`) deployments additionally depend on the proxy version. On LiteLLM before v1.103.0, or when the version is unknown, they ignore a `supported_endpoints` list that contains `/v1/responses` and need an explicit `mode: "responses"`: those releases have no native Responses config for that provider, so they bridge `/v1/responses` through Chat Completions, and the bridge fails streaming with `list index out of range` on Azure's empty-choices chunks (BerriAI/litellm#34455) while the endpoint list, copied from the model cost map, still advertises `/v1/responses`. From v1.103.0 the list is honored again. Discovery reads the version from the `x-litellm-version` header of one extra request, `GET /v1/responses/resp_version_probe`, sent only when such a deployment is published; LiteLLM answers it with an error and calls no model. A pre-release counts as older than the release it names, and a proxy that omits the header keeps these deployments on Chat Completions. Azure is detected from the adapter field, an `azure/` or `azure_ai/` model prefix, or the reported provider; other adapters follow the backend family because LiteLLM bridges `/v1/responses` to Chat Completions when the provider has no native Responses config (`litellm/responses/main.py`, `_bridges_to_chat_completions`). The family comes from the model id, not from LiteLLM's `openai/`, `openai_like/` or `custom_openai/` adapter prefix, so `openai/qwen3` stays on Chat Completions while `openai/gpt-5.5` uses Responses; a deployment opts into Responses with `mode: "responses"` or a `supported_endpoints` list containing `/v1/responses`. A route whose model prefix and `custom_llm_provider` name different providers is treated as unidentified and stays on Chat Completions, so it also does not get the local 128-tool preflight. When `/model/info` supplies `supported_endpoints`, that list takes precedence, and `mode: "responses"` remains an explicit Responses signal. An evidence-free fallback entry—a bare `/v1/models` id or a `/health` entry with no detail row—takes its protocol from the Pi catalog entry for its id; both evidence-free fallback paths also inherit presentation metadata from the bounded catalog lookup. An unknown id stays on Chat Completions. The route name alone authorizes nothing. A concrete id expanded from wildcard `/model/info` routes inherits the deployment evidence of the wildcard route LiteLLM would select for it, with the requested id substituted into the row's `model` the way LiteLLM serves it; the id is omitted when that selected route is not a published chat-style route. Chat Completions and Responses use `<root>/v1`; native Messages uses the proxy root directly. All three protocols pass through the provider's host guard. When an evidence-free cached `(no metadata)` entry can be enriched from the Pi catalog, it is restored on Responses only for a catalog Responses model and otherwise on Chat Completions, never Anthropic Messages. A model supplied only through `models.json` whose `api` this provider does not implement takes Pi's global API fallback rather than this provider's host guard; the credential-root pin (see [Model host enforcement](#model-host-enforcement)) still keeps the credential on the active proxy.
514
557
 
515
- `cacheControlFormat: "anthropic"` applies only to Chat Completions, where Pi adds Anthropic `cache_control` markers. The Responses transport has a different compatibility type and uses native `prompt_cache_key` (plus supported retention fields), so Responses models must not receive `cacheControlFormat`. Freshly discovered OpenAI-family models forced onto Chat Completions are rejected locally when a request contains more than 128 tools; route them through Responses or reduce enabled extensions. Moonshot request-side reasoning suppression follows LiteLLM's routing provider, not `model_info.base_model`; response-side `<think>` normalization follows discovered Kimi-family evidence. Opaque aliases retain display normalization without sending Moonshot-only parameters to Azure- or Bedrock-hosted Kimi models. Strict tool-message repair requires every deployment to identify Kimi; Gemini effort normalization likewise requires consistent backend-family evidence.
558
+ `cacheControlFormat: "anthropic"` applies only to Chat Completions, where Pi adds Anthropic `cache_control` markers. The Responses transport has a different compatibility type and uses native `prompt_cache_key` (plus supported retention fields), so Responses models must not receive `cacheControlFormat`. Freshly discovered OpenAI-family models forced onto Chat Completions are rejected locally when a request contains more than 128 tools; route them through Responses or reduce enabled extensions. Moonshot request-side reasoning suppression follows LiteLLM's routing provider, not `model_info.base_model`; response-side `<think>` normalization follows discovered Kimi-family evidence. Opaque aliases retain display normalization without sending Moonshot-only parameters to Azure- or Bedrock-hosted Kimi models. Strict tool-message repair requires every deployment to identify Kimi; Gemini effort normalization likewise requires consistent backend-family evidence. Claude-backed routes on native Messages or Chat Completions declare Pi's `promptCache` lifetimes (`short: 300`, `long: 3600` seconds, the values Pi's built-in Anthropic models use), so Pi's `cacheWarming` can keep their caches warm. Responses routes whose every deployment is GPT-5.6 or later and routes through LiteLLM's `openai`, `azure` or `azure_ai` provider declare `short: 1800`, the 30-minute default minimum lifetime OpenAI and Azure document for those models, and no `long` value. Claude routes on Responses, Chat routes to OpenAI models, earlier GPT models, GPT models served through a relay such as OpenRouter or GitHub Copilot, routes identified only by their name, and other backend families declare none, so Pi does not warm them. A `modelOverrides` entry for the route in the global `models.json` can change the values, and `cacheWarming: "off"` disables warming.
516
559
 
517
560
  Discovery policy is versioned in Pi's persisted model catalog. After upgrading, an online refresh replaces legacy entries. Offline legacy entries remain usable without a diagnostic; the extension reports the discovery-version mismatch once only when an attempted online refresh fails. To force rediscovery or roll back across this policy change, delete the `litellm` entry from `~/.pi/agent/models-store.json` (or delete the file) and refresh once online.
518
561
 
@@ -522,7 +565,19 @@ LiteLLM may load-balance one public `model_name` across deployments with differe
522
565
 
523
566
  - Responses is selected only when every deployment supports it under the protocol rules above; a deployment that requires Chat keeps the group on Chat.
524
567
  - Vision and reasoning are advertised only when every routable deployment resolves them as supported. An explicit `supports_reasoning: false` suppresses all thinking controls, even when catalog or router effort metadata exists. Catalog thinking controls are intersected per level: a level is exposed only when every deployment maps it to the same value, while disagreement or absence becomes an explicit denial. Catalog maps are not complete effort lists, so absence has one exception: an omitted `minimal`, `low`, `medium` or `high` counts as Pi's default spelling of that level. An omitted `off`, `xhigh` or `max` still differs from an explicit mapping. Reasoning selectors additionally require an accepted wire carrier on every deployment. Public effort lists are intersected, explicit LiteLLM denials win, and `xhigh`/`max` need unanimous explicit support. Null or absent standard-effort flags have no opinion. A deployment that declares `model_info.reasoning_effort_levels` answers every level from that list, ahead of its per-level flags, as LiteLLM itself reads it.
525
- - Context and output limits use the minimum resolved value across deployments. Pi catalogue lookup prefers the deployment's provider-specific entry before falling back to the generic vendor entry; Azure and Azure AI therefore retain Azure-specific limits rather than being capped by a different OpenAI catalogue entry. When models.dev supplies a sparse vendor record, missing fields still use the provider-specific Pi catalogue.
568
+ - Context and output limits use the minimum resolved value across deployments. Per deployment, a Pi `models.json` `modelOverrides` entry for the identified backend model (Pi provider id plus model id, resolved from backend identity, never from the route name; the provider id is the one whose Pi catalog entry identifies the backend, such as `openai-codex` for a `chatgpt/` deployment, not the LiteLLM provider label) sets `contextWindow` / `maxTokens` ahead of both the router figure and the catalog; it applies only under unanimous catalog authority, and the group minimum still applies. Without an override, the smaller of the router's explicit figure and the catalog figure is used, or the router's figure first under `LITELLM_TRUST_PROXY_LIMITS=1`. Overrides are read from the global `models.json` only and take effect at the next discovery (`/litellm-refresh` or cache expiry). Pi applies `modelOverrides` only to that provider's own models, so one entry covers every LiteLLM route backed by the model:
569
+
570
+ ```json
571
+ {
572
+ "providers": {
573
+ "openai": {
574
+ "modelOverrides": { "gpt-5.2": { "contextWindow": 500000 } }
575
+ }
576
+ }
577
+ }
578
+ ```
579
+
580
+ Pi catalogue lookup prefers the deployment's provider-specific entry before falling back to the generic vendor entry; Azure and Azure AI therefore retain Azure-specific limits rather than being capped by a different OpenAI catalogue entry. When models.dev supplies a sparse vendor record, missing fields still use the provider-specific Pi catalogue.
526
581
  - Each displayed base-price field uses the maximum only when every deployment resolves that field; unresolved fields remain zero and the model name is suffixed with ` (incomplete metadata)`.
527
582
  - When complete base pricing comes from the catalog under unanimous catalog authority, tiered pricing is the conservative worst-case envelope. Its usable finite non-negative thresholds are the sorted union of every deployment's thresholds, and each price field at each interval is the maximum applicable rate across all deployments, floored by each deployment's base rate. Differing tier breakpoints are therefore preserved rather than treated as incompatible. An explicit `/model/info` base-price field replaces catalog pricing for that field at every threshold; catalog tiers remain for unaffected fields and are omitted only when every base-price field is explicit.
528
583
  - Reduced route groups omit tier ladders when complete base-pricing or catalog authority is unavailable. Wildcard expansion retains every known tier from matching groups even when another match has incomplete metadata: suppressing a complete sibling's higher tier could understate the known worst-case rate. The ` (incomplete metadata)` marker remains because the resulting envelope is only as complete as the available ladders.
@@ -563,13 +618,15 @@ The development probe runs against minimized snapshots with `npm run probe:proxy
563
618
  | `LiteLLM discovery: ... route group(s) mix chat-style and explicitly incompatible deployment modes` | A public route can select both a chat-style deployment and an incompatible deployment such as an embedding model. Give the embedding deployment its own `model_name`. |
564
619
  | `LiteLLM (...): the request for "<route>" was answered by a fallback` | A successful HTTP response reports a LiteLLM router fallback (`x-litellm-attempted-fallbacks`). `/model/info` does not expose fallbacks, so protocol and model handling follow the requested route's own deployments. If the primary and fallback groups support Chat Completions, set `model_info.supported_endpoints: ["/v1/chat/completions"]` on the primary to avoid Responses-to-Chat bridging. This warning does not change routing or fix upstream bridging defects. It names only the requested route; LiteLLM reports the group and model that answered in the `x-litellm-model-group` and `x-litellm-model-name` response headers. Warned once per provider and route until the extension reloads; shown in the UI, or stderr without a UI. |
565
620
  | A model is marked ` (incomplete metadata)` | `/model/info` or `/health` did not provide enough authoritative metadata. Explicit fields remain usable, but unknown cost fields are shown as zero and route-name cache enrichment stays disabled. |
621
+ | Cache miss notices after idle, no cache warming | The model declares no `promptCache`: only Claude-backed Messages and Chat routes and GPT-5.6+ Responses routes on OpenAI or Azure get one. Run `/litellm-refresh` after updating so stored models are rediscovered. |
566
622
  | `401 Token expired` | Set `LITELLM_API_KEY_HELPER`. |
567
623
  | No models with gcloud auth | Verify `gcloud auth application-default login` has been run or set `GOOGLE_APPLICATION_CREDENTIALS` to an `authorized_user` ADC file |
568
624
  | Enterprise SSO waits for token insertion | The proxy returned 404/405 for `/sso/cli/start`, so Pi used the legacy flow — upgrade LiteLLM or paste the UI token |
569
625
  | Enterprise CLI SSO start/poll fails | Check the proxy logs and verify `/sso/cli/start` and `/sso/cli/poll/{login_id}` are reachable; only 404/405 falls back to legacy login |
570
626
  | Enterprise SSO login shows "virtual key generation failed" | The LiteLLM instance may lack a database (`/key/generate` requires one), your user account may lack key-generation permission, or the request timed out; the JWT is used directly as a fallback |
571
627
  | Enterprise SSO token prompt fails with "SSO token is required" | The token field was left empty — paste the token copied from the LiteLLM UI |
572
- | MCP tools not showing | Run `/mcp` and check the provider's server: its state shows a connection or sign-in error. `disabled` means the proxy refused the key at startup, usually because no MCP servers are granted to it, its team, or its organization; after a grant, enable it in `/mcp` or restart Pi. Verify the proxy serves `/mcp` and that the key has MCP access |
628
+ | `LiteLLM (...): out of budget` | A LiteLLM budget is used up. The error names it and, when the budget status can read it, its reset time; `/litellm-budget` shows every level. Ask the proxy admin to raise the limit, or wait for the reset — see [Budget status](#budget-status) |
629
+ | MCP tools not showing | Run `/mcp` and check the provider's server: its state shows a connection or sign-in error. `disabled` means the proxy refused the key at startup, usually because no MCP servers are granted to it, its team, or its organization, or because a budget is used up, which is reported at startup; after a grant or the budget's reset, enable it in `/mcp` or restart Pi. Verify the proxy serves `/mcp` and that the key has MCP access |
573
630
  | MCP tools listed but the model cannot call them | `codemode` and `deferred` tools are reached through Pi's `codemode` and `tool_search` tools. Check that neither is disabled; `--no-extensions` disables them and Pi's MCP client alike. Setting `litellm.mcp.exposure` to `direct` declares the tools on every request instead |
574
631
  | Skills not affecting prompts | Verify the proxy exposes `/claude-code/marketplace.json` or `/v1/skills` and returns enabled skills |
575
632
 
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "pi-provider-litellm",
3
- "version": "4.4.2",
3
+ "version": "4.6.0",
4
4
  "description": "LiteLLM proxy provider extension for Pi",
5
5
  "type": "module",
6
6
  "license": "MIT",
@@ -1,4 +1,4 @@
1
- export const LITELLM_DISCOVERY_VERSION = 7 as const;
1
+ export const LITELLM_DISCOVERY_VERSION = 10 as const;
2
2
 
3
3
  export type BackendFamily = "claude" | "deepseek" | "gemini" | "kimi" | "openai";
4
4
 
package/src/budget.ts CHANGED
@@ -311,6 +311,65 @@ export function formatBudgetDetails(
311
311
  return lines.join("\n");
312
312
  }
313
313
 
314
+ // LiteLLM refuses a used-up budget with `type: "budget_exceeded"` (status 422 since 1.104, 429 or 400 before), and
315
+ // /mcp with the bare message, whose wording varies by level and check.
316
+ const BUDGET_EXCEEDED = /budget_exceeded|Budget has been exceeded!|ExceededBudget:|exceeded budget for model=/;
317
+ // The refusals of a level's own budget. Budget windows and per-model budgets reset on schedules of their own.
318
+ const EXCEEDED_LEVEL =
319
+ /Budget has been exceeded! (Key|Team|TeamMember|User|Organization)=\S*( in Team=)?|ExceededBudget: (User)=.+? over budget\./;
320
+ const EXCEEDED_LEVELS: Record<string, BudgetLevelName> = {
321
+ Key: "key",
322
+ Team: "team",
323
+ TeamMember: "member",
324
+ User: "user",
325
+ Organization: "org",
326
+ };
327
+ const LEVEL_LABELS: Record<BudgetLevelName, string> = {
328
+ key: "key",
329
+ user: "user",
330
+ team: "team",
331
+ member: "team member",
332
+ org: "organization",
333
+ };
334
+
335
+ export const isBudgetExceeded = (text: string): boolean => BUDGET_EXCEEDED.test(text);
336
+
337
+ // Local time in one fixed format, which the Go port prints the same way.
338
+ function resetClock(resetAt: number): string {
339
+ const at = new Date(resetAt);
340
+ const pad = (n: number) => String(n).padStart(2, "0");
341
+ return `${at.getFullYear()}-${pad(at.getMonth() + 1)}-${pad(at.getDate())} ${pad(at.getHours())}:${pad(at.getMinutes())}`;
342
+ }
343
+
344
+ // Starts with "out of budget", which Pi and PiG never retry. A refusal whose level is not tracked keeps the proxy's
345
+ // text, as that names the budget.
346
+ export function formatBudgetExceeded(errorText: string, levels: BudgetLevels, now: number): string {
347
+ const match = EXCEEDED_LEVEL.exec(errorText);
348
+ if (!match) return `out of budget. ${errorText}`;
349
+ const name = match[2] ? "member" : EXCEEDED_LEVELS[match[1] ?? match[3]!]!;
350
+ const l = levels[name];
351
+ const amounts = l ? ` (${money(l.spend, 2)} of ${money(l.maxBudget, Number.isInteger(l.maxBudget) ? 0 : 2)})` : "";
352
+ const reset = formatRelativeReset(l?.resetAt, now);
353
+ const resets = reset ? ` and resets in ${reset} (${resetClock(l!.resetAt!)})` : "";
354
+ return `out of budget. The ${LEVEL_LABELS[name]} budget is used up${amounts}${resets}.`;
355
+ }
356
+
357
+ export function setupLiteLLMBudgetErrors(
358
+ pi: ExtensionAPI,
359
+ providerNames: ReadonlySet<string>,
360
+ levelsAfterError?: (ctx: ExtensionContext, name: string) => Promise<BudgetLevels>,
361
+ ): void {
362
+ pi.on("message_end", async (event, ctx) => {
363
+ const message = event.message;
364
+ if (message.role !== "assistant" || message.stopReason !== "error" || !providerNames.has(message.provider)) return;
365
+ const text = message.errorMessage ?? "";
366
+ if (!isBudgetExceeded(text)) return;
367
+ const levels = (await levelsAfterError?.(ctx, message.provider).catch(() => undefined)) ?? {};
368
+ const explained = formatBudgetExceeded(text, levels, Date.now());
369
+ return { message: { ...message, errorMessage: `LiteLLM (${JSON.stringify(message.provider)}): ${explained}` } };
370
+ });
371
+ }
372
+
314
373
  export function budgetDisplaySetting(budgetSettings: unknown): { display: BudgetDisplay; warning?: string } {
315
374
  const display = obj(budgetSettings)?.display;
316
375
  if (display === undefined || display === "all" || display === "tightest") return { display: display ?? "all" };
@@ -344,7 +403,10 @@ interface ProviderState {
344
403
  inFlight?: Promise<PollOutcome>;
345
404
  }
346
405
 
347
- export function setupLiteLLMBudget(pi: ExtensionAPI, options: BudgetOptions): void {
406
+ export function setupLiteLLMBudget(
407
+ pi: ExtensionAPI,
408
+ options: BudgetOptions,
409
+ ): (ctx: ExtensionContext, name: string) => Promise<BudgetLevels> {
348
410
  const displayNames = new Map(options.providers.map(({ name, displayName }) => [name, displayName]));
349
411
  const states = new Map<string, ProviderState>();
350
412
  // Keyed like the MCP registration identity in index.ts, because custom headers can carry gateway credentials.
@@ -558,4 +620,11 @@ export function setupLiteLLMBudget(pi: ExtensionAPI, options: BudgetOptions): vo
558
620
  }
559
621
  },
560
622
  });
623
+
624
+ // The levels a budget refusal is explained with. Polled for each refusal, joining a poll in flight, because only a
625
+ // poll notices that the credential changed since the last one.
626
+ return async (ctx, name) => {
627
+ if (ctx?.hasUI && !gated()) await poll(ctx, name);
628
+ return stateFor(name).levels;
629
+ };
561
630
  }
@@ -0,0 +1,174 @@
1
+ import type { ClassifierAnswer, ClassifierModel, ClassifierResult, ProviderClassifier } from "@earendil-works/pi-ai";
2
+ import { LITELLM_DISCOVERY_VERSION } from "./backend-identity.js";
3
+ import { errorDetail, parseUsage, perMillion, requestHeaders, requestSignal, responseCost } from "./images.js";
4
+ import { defaultContextWindow, wireString } from "./model-groups.js";
5
+ import type { ModelInfoEntry } from "./types.js";
6
+
7
+ /** Decision models through the proxy's System One `/v1/systemone` endpoint. */
8
+ export const LITELLM_DECISIONS_API = "litellm-decisions" as const;
9
+ export type LiteLLMDecisionsApi = typeof LITELLM_DECISIONS_API;
10
+
11
+ export type DiscoveredClassifierModel = Omit<ClassifierModel<LiteLLMDecisionsApi>, "provider" | "baseUrl"> & {
12
+ litellmDiscoveryVersion?: typeof LITELLM_DISCOVERY_VERSION;
13
+ };
14
+
15
+ export type LiteLLMClassifierModel = ClassifierModel<LiteLLMDecisionsApi> & {
16
+ litellmDiscoveryVersion?: typeof LITELLM_DISCOVERY_VERSION;
17
+ };
18
+
19
+ const DECISION_MODES = new Set(["evaluation", "decisions"]);
20
+ const DECISION_ENDPOINTS = new Set(["/v1/systemone", "/v1/decisions"]);
21
+
22
+ /**
23
+ * LiteLLM's own rule: a declared `supported_endpoints` list decides alone, so a list without a
24
+ * decision endpoint denies even with an evaluation mode; without a list, the mode decides.
25
+ */
26
+ export function isDecisionDeployment(entry: ModelInfoEntry): boolean {
27
+ const info = entry.model_info;
28
+ const endpoints: unknown = info?.supported_endpoints;
29
+ if (Array.isArray(endpoints)) {
30
+ return endpoints.some((endpoint) => typeof endpoint === "string" && DECISION_ENDPOINTS.has(endpoint.trim()));
31
+ }
32
+ const mode = wireString(info?.mode);
33
+ return mode !== undefined && DECISION_MODES.has(mode.trim().toLowerCase());
34
+ }
35
+
36
+ /**
37
+ * Reduce one `/model/info` route group to a classifier model. Every deployment behind the route
38
+ * must be a decision deployment: LiteLLM load-balances across them, so a mixed group could answer
39
+ * from a non-decision deployment.
40
+ */
41
+ export function mapFromClassifierModelInfoGroup(
42
+ route: string,
43
+ entries: readonly ModelInfoEntry[],
44
+ ): DiscoveredClassifierModel | undefined {
45
+ // A wildcard row is not addressable, and `/v1/models` expansion carries no mode evidence.
46
+ if (route.includes("*")) return undefined;
47
+ const deployments = entries.filter((entry) => wireString(entry.model_name) === route);
48
+ if (deployments.length === 0) return undefined;
49
+ if (!deployments.every(isDecisionDeployment)) return undefined;
50
+ // Prices are estimates for Pi's usage display; the `x-litellm-response-cost` header replaces them.
51
+ const rate = (read: (entry: ModelInfoEntry) => number | undefined): number => {
52
+ const values = deployments.map(read);
53
+ return values.every((value) => value !== undefined) ? Math.max(...(values as number[])) : 0;
54
+ };
55
+ // The smallest declared window; an undeclared one counts as the default so it never raises the result.
56
+ const windows = deployments.map((entry) => entry.model_info?.max_input_tokens);
57
+ const declared = windows.filter(
58
+ (value): value is number => typeof value === "number" && Number.isFinite(value) && value > 0,
59
+ );
60
+ const contextWindow = Math.min(...declared, ...(declared.length < windows.length ? [defaultContextWindow()] : []));
61
+ return {
62
+ type: "classifier",
63
+ id: route,
64
+ name: route,
65
+ api: LITELLM_DECISIONS_API,
66
+ input: ["text"],
67
+ cost: {
68
+ input: rate((entry) => perMillion(entry.model_info?.input_cost_per_token)),
69
+ output: rate((entry) => perMillion(entry.model_info?.output_cost_per_token)),
70
+ cacheRead: 0,
71
+ cacheWrite: 0,
72
+ },
73
+ contextWindow,
74
+ litellmDiscoveryVersion: LITELLM_DISCOVERY_VERSION,
75
+ };
76
+ }
77
+
78
+ function isRecord(value: unknown): value is Record<string, unknown> {
79
+ return typeof value === "object" && value !== null && !Array.isArray(value);
80
+ }
81
+
82
+ const isFiniteNumber = (value: unknown): value is number => typeof value === "number" && Number.isFinite(value);
83
+
84
+ function parseAnswer(model: string, id: string, type: "choice" | "score" | "bool", raw: unknown): ClassifierAnswer {
85
+ const wireType = type === "bool" ? "noul" : type;
86
+ if (!isRecord(raw) || raw.type !== wireType) {
87
+ throw new Error(`LiteLLM classifier ${model} did not return a ${type} answer for ${id}`);
88
+ }
89
+ const invalid = (field: string): Error =>
90
+ new Error(`LiteLLM classifier ${model} returned an invalid ${field} for ${id}`);
91
+ if (type === "bool") {
92
+ if (!isFiniteNumber(raw.noul)) throw invalid("noul");
93
+ return { type: "bool", probability: raw.noul };
94
+ }
95
+ if (!isFiniteNumber(raw.confidence)) throw invalid("confidence");
96
+ if (type === "score") {
97
+ if (!isFiniteNumber(raw.score)) throw invalid("score");
98
+ return { type: "score", score: raw.score, confidence: raw.confidence };
99
+ }
100
+ if (typeof raw.choice !== "string") throw invalid("choice");
101
+ if (!isRecord(raw.probabilities) || !Object.values(raw.probabilities).every(isFiniteNumber)) {
102
+ throw invalid("probabilities");
103
+ }
104
+ return {
105
+ type: "choice",
106
+ choice: raw.choice,
107
+ probabilities: raw.probabilities as Record<string, number>,
108
+ confidence: raw.confidence,
109
+ };
110
+ }
111
+
112
+ /** Classify through LiteLLM's `/v1/systemone`. Never rejects. */
113
+ export const classify: ProviderClassifier["classify"] = async (model, context, options) => {
114
+ const result: ClassifierResult = {
115
+ api: model.api,
116
+ provider: model.provider,
117
+ model: model.id,
118
+ answers: {},
119
+ stopReason: "stop",
120
+ timestamp: Date.now(),
121
+ };
122
+ try {
123
+ const apiKey = options?.apiKey;
124
+ if (!apiKey) throw new Error(`No API key for provider: ${model.provider}`);
125
+ if (context.images?.length) {
126
+ throw new Error(`LiteLLM classifier ${model.id} takes text state only; images are not sent to /v1/systemone`);
127
+ }
128
+ // System One calls Pi's `bool` question `noul`; the fields are otherwise the same.
129
+ const questions = Object.fromEntries(
130
+ Object.entries(context.questions).map(([id, question]) => [
131
+ id,
132
+ question.type === "bool" ? { ...question, type: "noul" } : question,
133
+ ]),
134
+ );
135
+ let params: Record<string, unknown> = { model: model.id, state: context.state, questions };
136
+ const nextParams = await options?.onPayload?.(params, model);
137
+ if (nextParams !== undefined) params = nextParams as Record<string, unknown>;
138
+ const response = await (options?.fetch ?? fetch)(`${model.baseUrl}/systemone`, {
139
+ method: "POST",
140
+ headers: requestHeaders(apiKey, model.headers, options?.headers),
141
+ body: JSON.stringify(params),
142
+ signal: requestSignal(options),
143
+ });
144
+ const responseHeaders = Object.fromEntries(response.headers.entries());
145
+ await options?.onResponse?.({ status: response.status, headers: responseHeaders }, model);
146
+ const body = await response.text();
147
+ if (!response.ok) throw new Error(`LiteLLM decisions request failed (${response.status}): ${errorDetail(body)}`);
148
+ // The proxy billed the request whatever the body holds; token counts are added once it parses.
149
+ const cost = responseCost(responseHeaders);
150
+ const billed = parseUsage(undefined, model.cost, cost);
151
+ if (billed) result.usage = billed;
152
+ const parsed: unknown = JSON.parse(body);
153
+ if (!isRecord(parsed)) throw new Error(`LiteLLM classifier ${model.id} returned an unexpected response`);
154
+ // A malformed answer set was still billed.
155
+ const usage = parseUsage(isRecord(parsed.usage) ? parsed.usage : undefined, model.cost, cost);
156
+ if (usage) result.usage = usage;
157
+ const answers = isRecord(parsed.answers) ? parsed.answers : {};
158
+ const parsedAnswers: Record<string, ClassifierAnswer> = {};
159
+ for (const [id, question] of Object.entries(context.questions)) {
160
+ parsedAnswers[id] = parseAnswer(model.id, id, question.type, answers[id]);
161
+ }
162
+ result.answers = parsedAnswers;
163
+ return result;
164
+ } catch (error) {
165
+ result.answers = {};
166
+ result.stopReason = options?.signal?.aborted ? "aborted" : "error";
167
+ result.errorMessage = error instanceof Error ? error.message : String(error);
168
+ return result;
169
+ }
170
+ };
171
+
172
+ export function createLiteLLMClassifierApis(): Record<LiteLLMDecisionsApi, ProviderClassifier> {
173
+ return { [LITELLM_DECISIONS_API]: { classify } };
174
+ }
package/src/discover.ts CHANGED
@@ -8,6 +8,7 @@ import {
8
8
  resolveCatalogProvider,
9
9
  routesOnlyThrough,
10
10
  } from "./backend-identity.js";
11
+ import { type DiscoveredClassifierModel, mapFromClassifierModelInfoGroup } from "./classifiers.js";
11
12
  import { type DiscoveredImageModel, mapFromImageModelInfoGroup } from "./images.js";
12
13
  import {
13
14
  type CatalogResolution,
@@ -50,6 +51,15 @@ import type {
50
51
  } from "./types.js";
51
52
 
52
53
  const DEFAULT_TIMEOUT_MS = 5000;
54
+ // Anthropic's documented 5-minute and 1-hour cache lifetimes; the values Pi's built-in Anthropic catalog uses.
55
+ // shortcut: `long` assumes the host offers the 1-hour TTL (Bedrock only does for Claude 4.5+), gate it on
56
+ // deployment routing if LiteLLM ever reports per-route TTL support.
57
+ const CLAUDE_PROMPT_CACHE = { short: 300, long: 3600 } as const;
58
+ // OpenAI's and Azure's documented 30-minute default minimum for GPT-5.6 and later. No `long`: the
59
+ // 12-hour lifetime needs `prompt_cache_options.ttl`, which Pi's long retention does not request.
60
+ const OPENAI_PROMPT_CACHE = { short: 1800 } as const;
61
+ // Relays (OpenRouter, Copilot, resellers) document no lifetime of their own and may bill or route differently.
62
+ const OPENAI_PROMPT_CACHE_ROUTES: ReadonlySet<string> = new Set(["openai", "azure", "azure_ai"]);
53
63
  const HEALTH_DETAIL_CONCURRENCY = 8;
54
64
  const reportedConflictingFamilyRoutes = new Set<string>();
55
65
  const reportedWithheldRepairRoutes = new Set<string>();
@@ -127,6 +137,12 @@ export function isGpt55OrNewerModel(modelId: string): boolean {
127
137
  return generation !== undefined && (generation[0] > 5 || (generation[0] === 5 && generation[1] >= 5));
128
138
  }
129
139
 
140
+ // GPT-5.6 and later keep a cached prefix for at least 30 minutes by default.
141
+ function isGpt56OrNewerModel(modelId: string): boolean {
142
+ const generation = gptGeneration(modelId);
143
+ return generation !== undefined && (generation[0] > 5 || (generation[0] === 5 && generation[1] >= 6));
144
+ }
145
+
130
146
  // GPT-6 also rejects an omitted effort there: it falls back to its default effort.
131
147
  export function isGpt6OrNewerModel(modelId: string): boolean {
132
148
  const generation = gptGeneration(modelId);
@@ -496,6 +512,7 @@ function adaptPublicCatalogRecord(
496
512
  const piCost = piCatalog?.cost;
497
513
  return {
498
514
  provider,
515
+ ...(resolved ? { piProvider: resolved.provider } : {}),
499
516
  // A models.dev hit names the model canonically even when Pi's catalog does not know it yet,
500
517
  // so two spellings of one backend model reduce to one identity instead of a conflict.
501
518
  catalogModelId: resolved?.model.id ?? record?.modelId ?? catalogModelId,
@@ -689,10 +706,16 @@ function mapFromModelInfoGroup(
689
706
  denyLevels?: boolean;
690
707
  allowMessages?: boolean;
691
708
  proxyVersion?: ProxyVersion;
709
+ modelOverrides?: DiscoveryOptions["modelOverrides"];
692
710
  } = {},
693
711
  ): DiscoveredModel | undefined {
694
712
  const reduced = reduceModelGroup(entries, (entry) => {
695
- const catalog = resolveModelInfoCatalog(entry, publicCatalog);
713
+ const resolved = resolveModelInfoCatalog(entry, publicCatalog);
714
+ const limitOverrides =
715
+ resolved?.piProvider && resolved.catalogModelId
716
+ ? options.modelOverrides?.[resolved.piProvider]?.[resolved.catalogModelId]
717
+ : undefined;
718
+ const catalog = limitOverrides ? { ...resolved, limitOverrides } : resolved;
696
719
  const family = deploymentFamily(entry);
697
720
  const generations = [entry.litellm_params?.model, entry.model_info?.base_model]
698
721
  .map((id) => wireString(id)?.trim())
@@ -714,7 +737,8 @@ function mapFromModelInfoGroup(
714
737
  const protocols = entries.map((entry) => modelProtocol(reduced.id, entry, options.proxyVersion));
715
738
  const protocol = protocols.find((candidate) => candidate.api === "openai-completions") ?? protocols[0]!;
716
739
  const api = reduced.api === "anthropic-messages" ? reduced.api : protocol.api;
717
- const families = new Set(entries.map((entry) => resolveBackendIdentity({ ...entry, model_name: undefined })?.family));
740
+ const identities = entries.map((entry) => resolveBackendIdentity({ ...entry, model_name: undefined }));
741
+ const families = new Set(identities.map((identity) => identity?.family));
718
742
  const [family] = families;
719
743
  const hasBackendEvidence = entries.some(hasReadableBackendEvidence);
720
744
  const reasoningPolicy =
@@ -756,6 +780,10 @@ function mapFromModelInfoGroup(
756
780
  acceptsResponsesReasoningControl: reduced.acceptsResponsesReasoningControl,
757
781
  denyLevels: options.denyLevels,
758
782
  });
783
+ const relaysAnthropicCacheMarkers =
784
+ api === "anthropic-messages" ||
785
+ (api === "openai-completions" &&
786
+ (policy.compat as Model<"openai-completions">["compat"])?.cacheControlFormat === "anthropic");
759
787
  const hasConflictingFamily = reduced.deploymentFamilies.includes("conflicting");
760
788
  const unanimousMoonshot =
761
789
  reduced.deploymentFamilies.length > 0 && reduced.deploymentFamilies.every((family) => family === "kimi");
@@ -772,10 +800,7 @@ function mapFromModelInfoGroup(
772
800
  );
773
801
  // One GPT-5.5+ deployment is enough: a tool request routed to it fails with
774
802
  // reasoning attached, while dropping reasoning elsewhere only loses effort.
775
- const chatBackendIds =
776
- api === "openai-completions"
777
- ? entries.flatMap((entry) => resolveBackendIdentity({ ...entry, model_name: undefined })?.modelId ?? [])
778
- : [];
803
+ const chatBackendIds = api === "openai-completions" ? identities.flatMap((identity) => identity?.modelId ?? []) : [];
779
804
  const modelPolicy: LiteLLMModelPolicy | undefined = chatBackendIds.some(isGpt55OrNewerModel)
780
805
  ? {
781
806
  normalizeStrictToolMessages: false,
@@ -802,6 +827,18 @@ function mapFromModelInfoGroup(
802
827
  ? { litellmResponsesReasoningControl: true as const }
803
828
  : {}),
804
829
  ...(families.size === 1 && family ? { litellmBackendFamily: family } : {}),
830
+ ...(families.size === 1 && family === "claude" && relaysAnthropicCacheMarkers
831
+ ? { promptCache: { ...CLAUDE_PROMPT_CACHE } }
832
+ : {}),
833
+ ...(api === "openai-responses" &&
834
+ identities.every(
835
+ (identity, index) =>
836
+ identity?.family === "openai" &&
837
+ isGpt56OrNewerModel(identity.modelId) &&
838
+ routesOnlyThrough(entries[index]!, OPENAI_PROMPT_CACHE_ROUTES),
839
+ )
840
+ ? { promptCache: { ...OPENAI_PROMPT_CACHE } }
841
+ : {}),
805
842
  ...(modelPolicy ? { litellmPolicy: modelPolicy } : {}),
806
843
  // closeSerializerPolicy derives compat from `api`, a correlation its return type cannot carry.
807
844
  } as DiscoveredModel;
@@ -930,6 +967,7 @@ async function discoverFromHealth(
930
967
  denyLevels: group.some(({ denyLevels }) => denyLevels),
931
968
  allowMessages: false,
932
969
  proxyVersion,
970
+ modelOverrides: options.modelOverrides,
933
971
  });
934
972
  })
935
973
  .filter((model): model is DiscoveredModel => model !== undefined);
@@ -1036,10 +1074,11 @@ function applyWildcardEvidence(
1036
1074
  publishedWildcardIds: ReadonlySet<string>,
1037
1075
  publicCatalog: PublicCatalog | undefined,
1038
1076
  proxyVersion: ProxyVersion | undefined,
1077
+ modelOverrides: DiscoveryOptions["modelOverrides"],
1039
1078
  ): DiscoveredModel | undefined {
1040
1079
  const parents = selectedWildcardRows(model.id, wildcardRows, publishedWildcardIds);
1041
1080
  if (!parents) return undefined;
1042
- const selected = mapFromModelInfoGroup(parents, publicCatalog, { proxyVersion });
1081
+ const selected = mapFromModelInfoGroup(parents, publicCatalog, { proxyVersion, modelOverrides });
1043
1082
  if (!selected) return undefined;
1044
1083
  const policies = [model.litellmPolicy, selected.litellmPolicy].filter((policy) => policy !== undefined);
1045
1084
  const combinedPolicy =
@@ -1061,6 +1100,7 @@ function applyWildcardEvidence(
1061
1100
  litellmBackendFamily: _family,
1062
1101
  litellmResponsesReasoningControl: _control,
1063
1102
  litellmPolicy: _policy,
1103
+ promptCache: _promptCache,
1064
1104
  thinkingLevelMap: _levels,
1065
1105
  ...rest
1066
1106
  } = model;
@@ -1071,6 +1111,7 @@ function applyWildcardEvidence(
1071
1111
  compat: selected.compat,
1072
1112
  ...(selected.thinkingLevelMap ? { thinkingLevelMap: selected.thinkingLevelMap } : {}),
1073
1113
  ...(selected.litellmBackendFamily ? { litellmBackendFamily: selected.litellmBackendFamily } : {}),
1114
+ ...(selected.promptCache ? { promptCache: selected.promptCache } : {}),
1074
1115
  ...(combinedPolicy ? { litellmPolicy: combinedPolicy } : {}),
1075
1116
  ...(selected.litellmResponsesReasoningControl ? { litellmResponsesReasoningControl: true as const } : {}),
1076
1117
  } as DiscoveredModel;
@@ -1216,6 +1257,7 @@ export async function discoverModels(
1216
1257
  withheldRepairRoutes,
1217
1258
  defaultedContextRoutes,
1218
1259
  proxyVersion,
1260
+ modelOverrides: options.modelOverrides,
1219
1261
  }),
1220
1262
  deploymentFamilies: group.map(deploymentFamily),
1221
1263
  };
@@ -1272,14 +1314,26 @@ export async function discoverModels(
1272
1314
  // The parents were reduced before the version was known; an expansion
1273
1315
  // must inherit from parents reduced under the version it is published with.
1274
1316
  sources = wildcardRoutes.flatMap(({ route, deploymentFamilies }) => {
1275
- const model = mapFromModelInfoGroup(groups.get(route)!, publicCatalog, { proxyVersion });
1317
+ const model = mapFromModelInfoGroup(groups.get(route)!, publicCatalog, {
1318
+ proxyVersion,
1319
+ modelOverrides: options.modelOverrides,
1320
+ });
1276
1321
  return model ? [{ model, deploymentFamilies }] : [];
1277
1322
  });
1278
1323
  }
1279
1324
  const expanded = candidates
1280
1325
  .map((entry) => mapFromWildcardExpansion(entry, sources, withheldRepairRoutes))
1281
1326
  .filter((model): model is DiscoveredModel => model !== undefined)
1282
- .map((model) => applyWildcardEvidence(model, wildcardRows, publishedWildcardIds, publicCatalog, proxyVersion))
1327
+ .map((model) =>
1328
+ applyWildcardEvidence(
1329
+ model,
1330
+ wildcardRows,
1331
+ publishedWildcardIds,
1332
+ publicCatalog,
1333
+ proxyVersion,
1334
+ options.modelOverrides,
1335
+ ),
1336
+ )
1283
1337
  .filter((model): model is DiscoveredModel => model !== undefined);
1284
1338
  models = [...models, ...expanded];
1285
1339
  }
@@ -1309,10 +1363,15 @@ export async function discoverModels(
1309
1363
  const imageModels = [...groups.entries()]
1310
1364
  .map(([route, group]) => mapFromImageModelInfoGroup(route, group))
1311
1365
  .filter((model): model is DiscoveredImageModel => model !== undefined);
1366
+ // Decision routes are a separate operation in Pi; the chat reducer withholds their modes.
1367
+ const classifierModels = [...groups.entries()]
1368
+ .map(([route, group]) => mapFromClassifierModelInfoGroup(route, group))
1369
+ .filter((model): model is DiscoveredClassifierModel => model !== undefined);
1312
1370
  return {
1313
1371
  source: "model_info",
1314
1372
  models,
1315
1373
  ...(imageModels.length > 0 ? { imageModels } : {}),
1374
+ ...(classifierModels.length > 0 ? { classifierModels } : {}),
1316
1375
  ...(proxyVersion ? { proxyVersion } : {}),
1317
1376
  };
1318
1377
  }
package/src/images.ts CHANGED
@@ -1,9 +1,8 @@
1
1
  import type {
2
2
  AssistantImages,
3
- ImageApi,
4
3
  ImageContent,
5
4
  ImageModel,
6
- ImagesOptions,
5
+ ModelCost,
7
6
  ProviderImages,
8
7
  Usage,
9
8
  } from "@earendil-works/pi-ai";
@@ -33,7 +32,7 @@ export function isImageGenerationMode(mode: unknown): boolean {
33
32
  return value !== undefined && IMAGE_GENERATION_MODE_PATTERN.test(value.trim());
34
33
  }
35
34
 
36
- function perMillion(value: unknown): number | undefined {
35
+ export function perMillion(value: unknown): number | undefined {
37
36
  return typeof value === "number" && Number.isFinite(value) && value >= 0 ? value * 1_000_000 : undefined;
38
37
  }
39
38
 
@@ -119,18 +118,19 @@ function tokenCount(...values: unknown[]): number {
119
118
  return 0;
120
119
  }
121
120
 
122
- function parseUsage(
123
- response: ImagesResponse,
124
- model: ImageModel<ImageApi>,
121
+ export function parseUsage(
122
+ raw:
123
+ | { input_tokens?: unknown; output_tokens?: unknown; prompt_tokens?: unknown; completion_tokens?: unknown }
124
+ | undefined,
125
+ cost: ModelCost,
125
126
  responseCost: number | undefined,
126
127
  ): Usage | undefined {
127
- const raw = response.usage;
128
128
  if (!raw && responseCost === undefined) return undefined;
129
129
  const input = tokenCount(raw?.input_tokens, raw?.prompt_tokens);
130
130
  const output = tokenCount(raw?.output_tokens, raw?.completion_tokens);
131
131
  const estimated = {
132
- input: (model.cost.input / 1_000_000) * input,
133
- output: (model.cost.output / 1_000_000) * output,
132
+ input: (cost.input / 1_000_000) * input,
133
+ output: (cost.output / 1_000_000) * output,
134
134
  };
135
135
  // LiteLLM's own cost is authoritative: it knows per-image prices that a token rate cannot express.
136
136
  const total = responseCost ?? estimated.input + estimated.output;
@@ -148,7 +148,13 @@ function parseUsage(
148
148
  };
149
149
  }
150
150
 
151
- function requestHeaders(
151
+ /** LiteLLM's `x-litellm-response-cost` header, when it is a usable amount. */
152
+ export function responseCost(headers: Record<string, string>): number | undefined {
153
+ const cost = Number.parseFloat(headers["x-litellm-response-cost"] ?? "");
154
+ return Number.isFinite(cost) && cost >= 0 ? cost : undefined;
155
+ }
156
+
157
+ export function requestHeaders(
152
158
  apiKey: string,
153
159
  ...sources: (Record<string, string | null | undefined> | undefined)[]
154
160
  ): Record<string, string> {
@@ -167,7 +173,7 @@ function requestHeaders(
167
173
  return headers;
168
174
  }
169
175
 
170
- function errorDetail(body: string): string {
176
+ export function errorDetail(body: string): string {
171
177
  try {
172
178
  const parsed = JSON.parse(body) as { error?: { message?: unknown } | unknown };
173
179
  const message =
@@ -181,7 +187,9 @@ function errorDetail(body: string): string {
181
187
  return body.slice(0, MAX_ERROR_DETAIL);
182
188
  }
183
189
 
184
- function requestSignal(options: ImagesOptions | undefined): AbortSignal | undefined {
190
+ export function requestSignal(
191
+ options: { signal?: AbortSignal; timeoutMs?: number } | undefined,
192
+ ): AbortSignal | undefined {
185
193
  const signals = [
186
194
  options?.signal,
187
195
  options?.timeoutMs !== undefined ? AbortSignal.timeout(options.timeoutMs) : undefined,
@@ -242,8 +250,7 @@ export const generateImages: ProviderImages["generateImages"] = async (model, co
242
250
  : `LiteLLM image model ${model.id} returned no images`,
243
251
  );
244
252
  }
245
- const costHeader = Number.parseFloat(responseHeaders["x-litellm-response-cost"] ?? "");
246
- const usage = parseUsage(parsed, model, Number.isFinite(costHeader) && costHeader >= 0 ? costHeader : undefined);
253
+ const usage = parseUsage(parsed.usage, model.cost, responseCost(responseHeaders));
247
254
  if (usage) result.usage = usage;
248
255
  return result;
249
256
  } catch (error) {
package/src/index.ts CHANGED
@@ -18,7 +18,8 @@ import type { ExtensionAPI, ExtensionContext, McpServerConfig } from "@earendil-
18
18
  // A namespace import, so a Pi without `VERSION` reaches the version check instead of failing to link.
19
19
  import * as piCodingAgent from "@earendil-works/pi-coding-agent";
20
20
  import { getAgentDir, readStoredCredential } from "@earendil-works/pi-coding-agent";
21
- import { budgetDisplaySetting, setupLiteLLMBudget } from "./budget.js";
21
+ import { budgetDisplaySetting, isBudgetExceeded, setupLiteLLMBudget, setupLiteLLMBudgetErrors } from "./budget.js";
22
+ import type { LiteLLMClassifierModel } from "./classifiers.js";
22
23
  import { setupLiteLLMCostTracking } from "./cost.js";
23
24
  import {
24
25
  discoverModels,
@@ -34,6 +35,7 @@ import {
34
35
  createLiteLLMProvider,
35
36
  DEFAULT_LITELLM_BASE_URL,
36
37
  isPlaceholderHost,
38
+ toNativeClassifierModels,
37
39
  toNativeImageModels,
38
40
  toNativeModels,
39
41
  } from "./provider.js";
@@ -286,6 +288,37 @@ async function readModelsProviderSettings(): Promise<Record<string, unknown> | u
286
288
  }
287
289
  }
288
290
 
291
+ // Pi `modelOverrides` limits from the global models.json, keyed by provider id then model id.
292
+ async function readModelLimitOverrides(): Promise<NonNullable<DiscoveryOptions["modelOverrides"]>> {
293
+ // Null prototypes: a `__proto__` provider or model key in the user's file stays a plain key.
294
+ const result: Record<string, Record<string, { contextWindow?: number; maxTokens?: number }>> = Object.create(null);
295
+ try {
296
+ const raw = await readFile(join(getAgentDir(), MODELS_FILENAME), "utf8");
297
+ const parsed = JSON.parse(stripJsonComments(stripBom(raw))) as Record<string, unknown>;
298
+ if (!isPlainObject(parsed.providers)) return result;
299
+ const limit = (value: unknown) =>
300
+ typeof value === "number" && Number.isFinite(value) && value > 0 ? value : undefined;
301
+ for (const [providerId, provider] of Object.entries(parsed.providers)) {
302
+ if (!isPlainObject(provider) || !isPlainObject(provider.modelOverrides)) continue;
303
+ for (const [modelId, override] of Object.entries(provider.modelOverrides)) {
304
+ if (!isPlainObject(override)) continue;
305
+ const contextWindow = limit(override.contextWindow);
306
+ const maxTokens = limit(override.maxTokens);
307
+ if (contextWindow === undefined && maxTokens === undefined) continue;
308
+ const models = result[providerId] ?? Object.create(null);
309
+ result[providerId] = models;
310
+ models[modelId] = {
311
+ ...(contextWindow !== undefined ? { contextWindow } : {}),
312
+ ...(maxTokens !== undefined ? { maxTokens } : {}),
313
+ };
314
+ }
315
+ }
316
+ } catch {
317
+ // Missing or unparsable models.json: no overrides.
318
+ }
319
+ return result;
320
+ }
321
+
289
322
  // A provider block may live in either file; merge per field with models.json winning. Non-provider
290
323
  // settings.json keys pass through untouched, which is what keeps `skills`/`mcp`/`budget` working.
291
324
  function mergeProviderSettings(
@@ -690,6 +723,19 @@ function piLiteral(value: string): string {
690
723
  return escaped.startsWith("!") ? `$${escaped}` : escaped;
691
724
  }
692
725
 
726
+ // The first 64 KiB of a body, as much as the MCP access check reads of a refusal.
727
+ async function readBodyPrefix(response: Response, limit = 64 * 1024): Promise<string> {
728
+ if (!response.body) return "";
729
+ const decoder = new TextDecoder();
730
+ let text = "";
731
+ for await (const chunk of response.body) {
732
+ text += decoder.decode(chunk, { stream: true });
733
+ // Leaving the loop cancels the rest of the stream.
734
+ if (text.length >= limit) break;
735
+ }
736
+ return text;
737
+ }
738
+
693
739
  /**
694
740
  * LiteLLM's /model/info is the metadata authority, so models.dev is an opt-in escape hatch.
695
741
  * Without a cache path and with refresh suppressed, discovery reads neither the models.dev
@@ -2468,8 +2514,12 @@ export default async function (pi: ExtensionAPI): Promise<void> {
2468
2514
  // LiteLLM 1.102.0 and later answer `initialize` with 403 when the key has no MCP servers granted,
2469
2515
  // and Pi warns at every start about a server that fails to connect. Asking first lets such a
2470
2516
  // server be registered disabled: Pi neither connects to it nor warns, and /mcp still lists it, so
2471
- // the user can enable it once servers are granted.
2472
- async function mcpAccessRefused(definition: ProviderDefinition, url: string): Promise<boolean> {
2517
+ // the user can enable it once servers are granted. A used-up budget fails every MCP request until
2518
+ // it resets, so it is refused the same way, with a notice.
2519
+ async function mcpAccessRefused(
2520
+ definition: ProviderDefinition,
2521
+ url: string,
2522
+ ): Promise<"no-servers" | "out-of-budget" | undefined> {
2473
2523
  try {
2474
2524
  const stored = readStoredCredential(definition.name, join(getAgentDir(), "auth.json"));
2475
2525
  // The check must send the credential Pi will send. Like seeding, it runs no key helper and
@@ -2482,11 +2532,11 @@ export default async function (pi: ExtensionAPI): Promise<void> {
2482
2532
  definition.apiKeyConfig?.startsWith("!") ||
2483
2533
  useGcloudToken
2484
2534
  ) {
2485
- return false;
2535
+ return undefined;
2486
2536
  }
2487
2537
  const auth = await authForCredential(definition, stored, false);
2488
2538
  // The credential goes only to its own root.
2489
- if (`${auth.baseUrl}/mcp` !== url) return false;
2539
+ if (`${auth.baseUrl}/mcp` !== url) return undefined;
2490
2540
  // The credential and headers Pi's client sends, so the proxy answers as it will answer Pi.
2491
2541
  const headers = { ...auth.headers, Authorization: `Bearer ${auth.apiKey}` };
2492
2542
  const response = await fetch(url, {
@@ -2505,7 +2555,9 @@ export default async function (pi: ExtensionAPI): Promise<void> {
2505
2555
  redirect: "manual",
2506
2556
  signal: AbortSignal.timeout(getSeedTimeoutMs()),
2507
2557
  });
2508
- await response.body?.cancel().catch(() => undefined);
2558
+ // Only a refusal is read, and only its start: a granted session can answer with an open event stream.
2559
+ const refusal = response.ok ? undefined : await readBodyPrefix(response).catch(() => "");
2560
+ if (refusal === undefined) await response.body?.cancel().catch(() => undefined);
2509
2561
  // Pi opens its own session; end the one this check opened.
2510
2562
  const session = response.headers.get("mcp-session-id");
2511
2563
  if (session) {
@@ -2518,9 +2570,11 @@ export default async function (pi: ExtensionAPI): Promise<void> {
2518
2570
  .then((closed) => closed.body?.cancel())
2519
2571
  .catch(() => undefined);
2520
2572
  }
2521
- return response.status === 403;
2573
+ // The budget refusal's status is configurable on the proxy, so its text is checked first.
2574
+ if (refusal !== undefined && isBudgetExceeded(refusal)) return "out-of-budget";
2575
+ return response.status === 403 ? "no-servers" : undefined;
2522
2576
  } catch {
2523
- return false;
2577
+ return undefined;
2524
2578
  }
2525
2579
  }
2526
2580
 
@@ -2541,6 +2595,12 @@ export default async function (pi: ExtensionAPI): Promise<void> {
2541
2595
  try {
2542
2596
  pi.registerMcpServer(definition.name, refused ? { ...config, enabled: false } : config);
2543
2597
  attempt.registered = true;
2598
+ if (refused === "out-of-budget") {
2599
+ notifyMcp(
2600
+ `LiteLLM MCP (${JSON.stringify(definition.name)}): out of budget, so its MCP server is off for this ` +
2601
+ "session. Turn it on in /mcp once the budget resets.",
2602
+ );
2603
+ }
2544
2604
  } catch (error) {
2545
2605
  // The claim stays, so the refusal is reported once rather than on every turn.
2546
2606
  notifyMcp(
@@ -2637,6 +2697,7 @@ export default async function (pi: ExtensionAPI): Promise<void> {
2637
2697
  headers: auth.headers,
2638
2698
  allowInsecureHttp: auth.allowInsecureHttp,
2639
2699
  ...modelsDevDiscoveryOptions(),
2700
+ modelOverrides: await readModelLimitOverrides(),
2640
2701
  silent: !isVerboseDiscovery(),
2641
2702
  onProgress: isVerboseDiscovery() ? (message) => process.stderr.write(`LiteLLM: ${message}\n`) : undefined,
2642
2703
  });
@@ -2653,7 +2714,9 @@ export default async function (pi: ExtensionAPI): Promise<void> {
2653
2714
  // critical path of every start -- is replaced by reading that catalog while it is fresh (models-store.ts). Nothing is
2654
2715
  // written: Pi owns the file. The stored models are handed over rather than an empty list, so that a store Pi keeps
2655
2716
  // somewhere else cannot leave the provider with nothing.
2656
- async function seedModels(definition: ProviderDefinition): Promise<(Model<LiteLLMApi> | LiteLLMImageModel)[]> {
2717
+ async function seedModels(
2718
+ definition: ProviderDefinition,
2719
+ ): Promise<(Model<LiteLLMApi> | LiteLLMImageModel | LiteLLMClassifierModel)[]> {
2657
2720
  const disabledReason = discoveryDisabledReason() ?? (isHostOffline() ? "PI_OFFLINE" : null);
2658
2721
  if (disabledReason) {
2659
2722
  if (isVerboseDiscovery()) {
@@ -2685,6 +2748,12 @@ export default async function (pi: ExtensionAPI): Promise<void> {
2685
2748
  return [
2686
2749
  ...toNativeModels(definition.name, result.baseUrl, result.models, definition.allowInsecureHttp),
2687
2750
  ...toNativeImageModels(definition.name, result.baseUrl, result.imageModels ?? [], definition.allowInsecureHttp),
2751
+ ...toNativeClassifierModels(
2752
+ definition.name,
2753
+ result.baseUrl,
2754
+ result.classifierModels ?? [],
2755
+ definition.allowInsecureHttp,
2756
+ ),
2688
2757
  ];
2689
2758
  } catch (error) {
2690
2759
  // No credentials yet is the normal unconfigured case; anything else is worth one line.
@@ -2836,10 +2905,11 @@ export default async function (pi: ExtensionAPI): Promise<void> {
2836
2905
 
2837
2906
  setupLiteLLMCostTracking(pi, [...providerNames]);
2838
2907
 
2908
+ let budgetLevels: ReturnType<typeof setupLiteLLMBudget> | undefined;
2839
2909
  if (budgetEnabled) {
2840
2910
  const { display, warning } = budgetDisplaySetting(settings?.budget);
2841
2911
  const definitionNamed = (name: string) => definitions.find((definition) => definition.name === name)!;
2842
- setupLiteLLMBudget(pi, {
2912
+ budgetLevels = setupLiteLLMBudget(pi, {
2843
2913
  providers: definitions.map(({ name, displayName }) => ({ name, displayName })),
2844
2914
  display,
2845
2915
  displayWarning: warning,
@@ -2850,6 +2920,7 @@ export default async function (pi: ExtensionAPI): Promise<void> {
2850
2920
  missingCredentials: (name) => missingCredentials(definitionNamed(name)),
2851
2921
  });
2852
2922
  }
2923
+ setupLiteLLMBudgetErrors(pi, providerNames, budgetLevels);
2853
2924
 
2854
2925
  if (skillsEnabled) {
2855
2926
  for (const tool of createSkillToolDefinitions(resolveDefaultRuntimeAuth)) {
@@ -54,6 +54,8 @@ export interface ReasoningPolicy {
54
54
 
55
55
  export interface CatalogResolution {
56
56
  provider?: string;
57
+ // The Pi provider whose catalog entry resolved catalogModelId; set only then, so the pair is Pi's providerId/model.id.
58
+ piProvider?: string;
57
59
  catalogModelId?: string;
58
60
  semanticFamily?: FamilyEvidence;
59
61
  semanticModel?: SemanticModel;
@@ -68,6 +70,8 @@ export interface CatalogResolution {
68
70
  vision?: boolean;
69
71
  contextWindow?: number;
70
72
  maxTokens?: number;
73
+ // User `modelOverrides` for the identified catalog model; read only under unanimous catalog authority.
74
+ limitOverrides?: { contextWindow?: number; maxTokens?: number };
71
75
  cost?: Partial<DiscoveredModel["cost"]>;
72
76
  }
73
77
 
@@ -657,6 +661,8 @@ function min(values: readonly number[]): number {
657
661
  }
658
662
 
659
663
  function conservativeLimit(explicit: number | undefined, catalog: number | undefined): number | undefined {
664
+ // LITELLM_TRUST_PROXY_LIMITS=1 lets the router's own figure win over the catalog.
665
+ if (process.env.LITELLM_TRUST_PROXY_LIMITS === "1") return explicitLimit(explicit) ?? explicitLimit(catalog);
660
666
  const valid = [explicitLimit(explicit), explicitLimit(catalog)].filter(
661
667
  (value): value is number => value !== undefined,
662
668
  );
@@ -819,11 +825,15 @@ export function reduceModelGroup(
819
825
  const visionEvidence = deployments.map(
820
826
  (entry, index) => wireBoolean(entry.model_info?.supports_vision) ?? catalogAuthority[index]?.vision,
821
827
  );
822
- const contextWindowEvidence = deployments.map((entry, index) =>
823
- conservativeLimit(entry.model_info?.max_input_tokens, catalogAuthority[index]?.contextWindow),
828
+ const contextWindowEvidence = deployments.map(
829
+ (entry, index) =>
830
+ explicitLimit(catalogAuthority[index]?.limitOverrides?.contextWindow) ??
831
+ conservativeLimit(entry.model_info?.max_input_tokens, catalogAuthority[index]?.contextWindow),
824
832
  );
825
- const maxTokensEvidence = deployments.map((entry, index) =>
826
- conservativeLimit(entry.model_info?.max_output_tokens, catalogAuthority[index]?.maxTokens),
833
+ const maxTokensEvidence = deployments.map(
834
+ (entry, index) =>
835
+ explicitLimit(catalogAuthority[index]?.limitOverrides?.maxTokens) ??
836
+ conservativeLimit(entry.model_info?.max_output_tokens, catalogAuthority[index]?.maxTokens),
827
837
  );
828
838
  const reasoning = reasoningEvidence.every((value) => value ?? false);
829
839
  const explicitlyUnsupported = deployments.some(
@@ -39,10 +39,11 @@
39
39
  import { readFile } from "node:fs/promises";
40
40
  import { join } from "node:path";
41
41
  import { LITELLM_DISCOVERY_VERSION } from "./backend-identity.js";
42
+ import { LITELLM_DECISIONS_API, type LiteLLMClassifierModel } from "./classifiers.js";
42
43
  import { enrichCachedModel } from "./discover.js";
43
44
  import { LITELLM_IMAGES_API, type LiteLLMImageModel } from "./images.js";
44
45
  import { isLiteLLMApi, resolveModelBaseUrl } from "./protocols.js";
45
- import { imageBaseUrl } from "./provider.js";
46
+ import { proxyV1BaseUrl } from "./provider.js";
46
47
  import type { LiteLLMModel } from "./types.js";
47
48
 
48
49
  /**
@@ -59,17 +60,20 @@ function isRecord(value: unknown): value is Record<string, unknown> {
59
60
  /** Pi's own remote catalogs skip the network for 4 hours after `checkedAt` (`REMOTE_CATALOG_REFRESH_INTERVAL_MS`). */
60
61
  const STORE_FRESH_MS = 4 * 60 * 60 * 1000;
61
62
 
62
- function isImageModel(value: Record<string, unknown>): boolean {
63
- return value.type === "image" && value.api === LITELLM_IMAGES_API;
63
+ function isOperationModel(value: Record<string, unknown>): boolean {
64
+ return (
65
+ (value.type === "image" && value.api === LITELLM_IMAGES_API) ||
66
+ (value.type === "classifier" && value.api === LITELLM_DECISIONS_API)
67
+ );
64
68
  }
65
69
 
66
70
  function isUsableModel(value: unknown, provider: string, baseUrl: string, allowInsecureHttp?: boolean): boolean {
67
71
  if (!isRecord(value) || value.provider !== provider) return false;
68
72
  if (value.litellmDiscoveryVersion !== LITELLM_DISCOVERY_VERSION) return false;
69
73
  try {
70
- // A stored baseUrl is per protocol -- `/v1` for the OpenAI APIs and images, the bare root for Messages -- so the root
74
+ // A stored baseUrl is per protocol -- `/v1` for the OpenAI APIs, images and classifiers, the bare root for Messages -- so the root
71
75
  // goes through the same resolution rather than being compared to it.
72
- if (isImageModel(value)) return value.baseUrl === imageBaseUrl(baseUrl, allowInsecureHttp);
76
+ if (isOperationModel(value)) return value.baseUrl === proxyV1BaseUrl(baseUrl, allowInsecureHttp);
73
77
  // `api` indexes LITELLM_PROTOCOLS in resolveModelBaseUrl, so an unknown one is a TypeError, not a bad model.
74
78
  return isLiteLLMApi(value.api) && value.baseUrl === resolveModelBaseUrl(baseUrl, value.api, allowInsecureHttp);
75
79
  } catch {
@@ -96,7 +100,7 @@ export async function readStoredModels(options: {
96
100
  provider: string;
97
101
  baseUrl: string;
98
102
  allowInsecureHttp?: boolean;
99
- }): Promise<(LiteLLMModel | LiteLLMImageModel)[] | undefined> {
103
+ }): Promise<(LiteLLMModel | LiteLLMImageModel | LiteLLMClassifierModel)[] | undefined> {
100
104
  const entry = (await readStore(options.agentDir))?.[options.provider];
101
105
  if (!isRecord(entry) || !Array.isArray(entry.models) || entry.models.length === 0) return undefined;
102
106
  if (typeof entry.checkedAt !== "number" || Date.now() - entry.checkedAt >= STORE_FRESH_MS) return undefined;
@@ -106,10 +110,11 @@ export async function readStoredModels(options: {
106
110
  if (!usable) return undefined;
107
111
  try {
108
112
  // Pi's restore path (provider.ts) runs the same call; when it throws, nothing is restored and the seed is the catalog.
109
- // Image models carry no request policy, so that path restores them as stored.
110
- return entry.models.map((model) => (isImageModel(model) ? model : enrichCachedModel(model))) as (
113
+ // Image and classifier models carry no request policy, so that path restores them as stored.
114
+ return entry.models.map((model) => (isOperationModel(model) ? model : enrichCachedModel(model))) as (
111
115
  | LiteLLMModel
112
116
  | LiteLLMImageModel
117
+ | LiteLLMClassifierModel
113
118
  )[];
114
119
  } catch {
115
120
  return undefined;
package/src/provider.ts CHANGED
@@ -3,6 +3,10 @@ import {
3
3
  type Api,
4
4
  type ApiStreamOptions,
5
5
  type AssistantImages,
6
+ type ClassifierContext,
7
+ type ClassifierModel,
8
+ type ClassifierOptions,
9
+ type ClassifierResult,
6
10
  type Credential,
7
11
  createProvider,
8
12
  getCurrentTools,
@@ -17,6 +21,12 @@ import {
17
21
  type TranscriptContext,
18
22
  } from "@earendil-works/pi-ai";
19
23
  import { LITELLM_DISCOVERY_VERSION } from "./backend-identity.js";
24
+ import {
25
+ createLiteLLMClassifierApis,
26
+ type DiscoveredClassifierModel,
27
+ LITELLM_DECISIONS_API,
28
+ type LiteLLMClassifierModel,
29
+ } from "./classifiers.js";
20
30
  import { enrichCachedModel, normalizeBaseUrl, restoreCachedModelPolicy } from "./discover.js";
21
31
  import {
22
32
  createLiteLLMImageApis,
@@ -34,7 +44,7 @@ export type LiteLLMProviderOptions = {
34
44
  auth: ProviderAuth;
35
45
  /** Catalog discovered during activation. Pi's startup refresh never allows network access, so
36
46
  * without a seed the provider stays empty until the user opens /model. */
37
- models?: readonly (LiteLLMModel | LiteLLMImageModel)[];
47
+ models?: readonly (LiteLLMModel | LiteLLMImageModel | LiteLLMClassifierModel)[];
38
48
  allowInsecureHttp?: boolean;
39
49
  resolveCredentialRoot: (credential?: Credential, requestBaseUrl?: string, apiKey?: string) => string | undefined;
40
50
  discover(credential: Credential, signal?: AbortSignal): Promise<DiscoveryResult & { baseUrl?: string }>;
@@ -49,17 +59,31 @@ function isLiteLLMImageModel(model: AnyModel): model is LiteLLMImageModel {
49
59
  return model.type === "image" && model.api === LITELLM_IMAGES_API;
50
60
  }
51
61
 
52
- export function imageBaseUrl(root: string, allowInsecureHttp = false): string {
62
+ function isLiteLLMClassifierModel(model: AnyModel): model is LiteLLMClassifierModel {
63
+ return model.type === "classifier" && model.api === LITELLM_DECISIONS_API;
64
+ }
65
+
66
+ /** Where image and classifier requests go: `<root>/v1`. */
67
+ export function proxyV1BaseUrl(root: string, allowInsecureHttp = false): string {
53
68
  return `${normalizeBaseUrl(root, allowInsecureHttp)}/v1`;
54
69
  }
55
70
 
71
+ export function toNativeClassifierModels(
72
+ provider: string,
73
+ baseUrl: string,
74
+ models: readonly DiscoveredClassifierModel[],
75
+ allowInsecureHttp = false,
76
+ ): LiteLLMClassifierModel[] {
77
+ return models.map((model) => ({ ...model, provider, baseUrl: proxyV1BaseUrl(baseUrl, allowInsecureHttp) }));
78
+ }
79
+
56
80
  export function toNativeImageModels(
57
81
  provider: string,
58
82
  baseUrl: string,
59
83
  models: readonly DiscoveredImageModel[],
60
84
  allowInsecureHttp = false,
61
85
  ): LiteLLMImageModel[] {
62
- return models.map((model) => ({ ...model, provider, baseUrl: imageBaseUrl(baseUrl, allowInsecureHttp) }));
86
+ return models.map((model) => ({ ...model, provider, baseUrl: proxyV1BaseUrl(baseUrl, allowInsecureHttp) }));
63
87
  }
64
88
 
65
89
  export function toNativeModels(
@@ -145,23 +169,24 @@ function modelRootError(
145
169
  }
146
170
  }
147
171
 
148
- function imageRootError(
149
- model: ImageModel<ImageApi>,
172
+ function operationRootError(
173
+ model: { baseUrl: string },
174
+ subject: string,
150
175
  active: { root: string; canonical: string; hostname: string },
151
176
  allowInsecureHttp = false,
152
177
  ): Error | undefined {
153
178
  let stored: { root: string; canonical: string; hostname: string };
154
179
  try {
155
- stored = proxyRoot(model.baseUrl, "Cached image model", allowInsecureHttp);
180
+ stored = proxyRoot(model.baseUrl, subject, allowInsecureHttp);
156
181
  } catch (error) {
157
- return error instanceof Error ? error : refreshRequired("Cached image model has an invalid LiteLLM model URL");
182
+ return error instanceof Error ? error : refreshRequired(`${subject} has an invalid LiteLLM model URL`);
158
183
  }
159
184
  if (isPlaceholderHost(stored.hostname)) {
160
- return refreshRequired("Cached image model uses a placeholder LiteLLM model host");
185
+ return refreshRequired(`${subject} uses a placeholder LiteLLM model host`);
161
186
  }
162
187
  if (stored.canonical !== active.canonical) {
163
188
  return refreshRequired(
164
- `Cached image model has stale LiteLLM model root ${stored.root}; active credentials use ${active.root}`,
189
+ `${subject} has stale LiteLLM model root ${stored.root}; active credentials use ${active.root}`,
165
190
  );
166
191
  }
167
192
  }
@@ -244,6 +269,7 @@ export function createLiteLLMProvider(options: LiteLLMProviderOptions): Provider
244
269
  return [
245
270
  ...toNativeModels(options.id, root, result.models, options.allowInsecureHttp),
246
271
  ...toNativeImageModels(options.id, root, result.imageModels ?? [], options.allowInsecureHttp),
272
+ ...toNativeClassifierModels(options.id, root, result.classifierModels ?? [], options.allowInsecureHttp),
247
273
  ];
248
274
  },
249
275
  filterModels(models, credential) {
@@ -257,17 +283,26 @@ export function createLiteLLMProvider(options: LiteLLMProviderOptions): Provider
257
283
  // Image models follow the same rule as chat: a cached root that no longer matches the
258
284
  // active credential is hidden until a network refresh proves the new proxy's catalog.
259
285
  const images = models.filter(isLiteLLMImageModel).flatMap((model) => {
260
- const error = imageRootError(model, active, options.allowInsecureHttp);
286
+ const error = operationRootError(model, "Cached image model", active, options.allowInsecureHttp);
261
287
  if (error) {
262
288
  reportUnavailable(error.message);
263
289
  return [];
264
290
  }
265
- return [{ ...model, baseUrl: imageBaseUrl(active.root, options.allowInsecureHttp) }];
291
+ return [{ ...model, baseUrl: proxyV1BaseUrl(active.root, options.allowInsecureHttp) }];
266
292
  });
267
- return [...chat, ...images];
293
+ const classifiers = models.filter(isLiteLLMClassifierModel).flatMap((model) => {
294
+ const error = operationRootError(model, "Cached classifier model", active, options.allowInsecureHttp);
295
+ if (error) {
296
+ reportUnavailable(error.message);
297
+ return [];
298
+ }
299
+ return [{ ...model, baseUrl: proxyV1BaseUrl(active.root, options.allowInsecureHttp) }];
300
+ });
301
+ return [...chat, ...images, ...classifiers];
268
302
  },
269
303
  api: createLiteLLMProtocolApis(),
270
304
  images: createLiteLLMImageApis(),
305
+ classifiers: createLiteLLMClassifierApis(),
271
306
  });
272
307
  const refreshModels = provider.refreshModels;
273
308
  const guardedProvider: Provider<LiteLLMApi> = {
@@ -307,10 +342,10 @@ export function createLiteLLMProvider(options: LiteLLMProviderOptions): Provider
307
342
  );
308
343
  if (!root) throw refreshRequired("Active credentials do not identify a LiteLLM model host");
309
344
  const active = activeCredentialRoot(root, options.allowInsecureHttp);
310
- const error = imageRootError(model, active, options.allowInsecureHttp);
345
+ const error = operationRootError(model, "Cached image model", active, options.allowInsecureHttp);
311
346
  if (error) throw error;
312
347
  return await provider.generateImages!(
313
- { ...model, provider: options.id, baseUrl: imageBaseUrl(active.root, options.allowInsecureHttp) },
348
+ { ...model, provider: options.id, baseUrl: proxyV1BaseUrl(active.root, options.allowInsecureHttp) },
314
349
  context,
315
350
  requestOptions,
316
351
  );
@@ -328,23 +363,67 @@ export function createLiteLLMProvider(options: LiteLLMProviderOptions): Provider
328
363
  return failed;
329
364
  }
330
365
  },
366
+ classify: async (
367
+ model: ClassifierModel<string>,
368
+ context: ClassifierContext,
369
+ requestOptions?: ClassifierOptions,
370
+ ) => {
371
+ try {
372
+ const root = options.resolveCredentialRoot(
373
+ undefined,
374
+ requestOptions?.env?.LITELLM_BASE_URL,
375
+ requestOptions?.apiKey,
376
+ );
377
+ if (!root) throw refreshRequired("Active credentials do not identify a LiteLLM model host");
378
+ const active = activeCredentialRoot(root, options.allowInsecureHttp);
379
+ const error = operationRootError(model, "Cached classifier model", active, options.allowInsecureHttp);
380
+ if (error) throw error;
381
+ return await provider.classify!(
382
+ { ...model, provider: options.id, baseUrl: proxyV1BaseUrl(active.root, options.allowInsecureHttp) },
383
+ context,
384
+ requestOptions,
385
+ );
386
+ } catch (error) {
387
+ // `Provider.classify` never rejects.
388
+ const failed: ClassifierResult = {
389
+ api: model.api,
390
+ provider: model.provider,
391
+ model: model.id,
392
+ answers: {},
393
+ stopReason: requestOptions?.signal?.aborted ? "aborted" : "error",
394
+ errorMessage: error instanceof Error ? error.message : String(error),
395
+ timestamp: Date.now(),
396
+ };
397
+ return failed;
398
+ }
399
+ },
331
400
  };
332
401
  if (!refreshModels) return guardedProvider;
333
- const seedKey = (model: { id: string; baseUrl: string }) => `${model.id}\0${model.baseUrl}`;
402
+ // A route can be published as both a chat model and a classifier (GPT-6 Luna lists `/v1/decisions`
403
+ // next to its chat endpoints), so the type keeps those seeds apart.
404
+ const seedKey = (model: { type?: string; id: string; baseUrl: string }) =>
405
+ `${model.type ?? "chat"}\0${model.id}\0${model.baseUrl}`;
334
406
  const seeded = new Map((options.models ?? []).map((model) => [seedKey(model), model]));
335
407
  return {
336
408
  ...guardedProvider,
337
409
  refreshModels: async (context) => {
338
410
  // Pi 0.99 persists models of every type; the policy helpers below read chat-only fields, so
339
- // chat and image entries are restored separately. Other types are not this provider's.
411
+ // chat, image and classifier entries are restored separately. Other types are not this provider's.
340
412
  const storedModels = (context.stored?.models ?? []).filter(isChatModel);
341
- const storedImageModels = (context.stored?.models ?? []).filter(isLiteLLMImageModel).flatMap((model) => {
342
- if (model.litellmDiscoveryVersion === LITELLM_DISCOVERY_VERSION) return [model];
343
- // An image entry carries no request policy to restore; a stale one is only kept when
344
- // startup discovery proved the same route, otherwise the network refresh replaces it.
345
- const fresh = seeded.get(seedKey(model));
346
- return fresh && isLiteLLMImageModel(fresh) ? [fresh] : [];
347
- });
413
+ // Image and classifier entries carry no request policy to restore; a stale one is only kept
414
+ // when startup discovery proved the same route, otherwise the network refresh replaces it.
415
+ const restoreOperationModels = <T extends AnyModel>(is: (model: AnyModel) => model is T): T[] =>
416
+ (context.stored?.models ?? []).filter(is).flatMap((model) => {
417
+ if ((model as { litellmDiscoveryVersion?: number }).litellmDiscoveryVersion === LITELLM_DISCOVERY_VERSION) {
418
+ return [model];
419
+ }
420
+ const fresh = seeded.get(seedKey(model));
421
+ return fresh && is(fresh) ? [fresh] : [];
422
+ });
423
+ const storedOperationModels = [
424
+ ...restoreOperationModels(isLiteLLMImageModel),
425
+ ...restoreOperationModels(isLiteLLMClassifierModel),
426
+ ];
348
427
  let legacyCount = 0;
349
428
  const models = storedModels.map((model) => {
350
429
  if ((model as LiteLLMModel).litellmDiscoveryVersion !== LITELLM_DISCOVERY_VERSION) {
@@ -362,7 +441,7 @@ export function createLiteLLMProvider(options: LiteLLMProviderOptions): Provider
362
441
  ...context,
363
442
  // The current pi-ai provider has no freshness gate, but force is the documented refresh contract field.
364
443
  force: context.force || (context.allowNetwork && legacyCount > 0),
365
- stored: context.stored && { ...context.stored, models: [...models, ...storedImageModels] },
444
+ stored: context.stored && { ...context.stored, models: [...models, ...storedOperationModels] },
366
445
  });
367
446
  } catch (error) {
368
447
  if (context.allowNetwork && legacyCount > 0) {
package/src/types.ts CHANGED
@@ -1,5 +1,6 @@
1
1
  import type { Model } from "@earendil-works/pi-ai";
2
2
  import type { BackendFamily, LITELLM_DISCOVERY_VERSION } from "./backend-identity.js";
3
+ import type { DiscoveredClassifierModel } from "./classifiers.js";
3
4
  import type { DiscoveredImageModel } from "./images.js";
4
5
  import type { ProxyVersion } from "./proxy-version.js";
5
6
 
@@ -67,6 +68,11 @@ export interface DiscoveryResult {
67
68
  models: DiscoveredModel[];
68
69
  /** `mode: "image_generation"` routes from `/model/info`; absent when there are none. */
69
70
  imageModels?: DiscoveredImageModel[];
71
+ /**
72
+ * Decision routes from `/model/info` (`mode: "evaluation"`/`"decisions"`, or a decision endpoint in
73
+ * `supported_endpoints`); absent when there are none.
74
+ */
75
+ classifierModels?: DiscoveredClassifierModel[];
70
76
  source: DiscoverySource;
71
77
  // Present only when a `/model/info` discovery had to read it and the proxy
72
78
  // answered. Absent means unknown, not old: discovery asks only when it must.
@@ -82,6 +88,9 @@ export interface DiscoveryOptions {
82
88
  // path, models.dev is not consulted at all.
83
89
  modelsDev?: boolean;
84
90
  modelsDevCachePath?: string;
91
+ // Pi `models.json` `providers.<providerId>.modelOverrides.<modelId>` limits. Pi applies them to that
92
+ // provider's own models only, so discovery applies them to routes backed by the same catalog model.
93
+ modelOverrides?: Readonly<Record<string, Readonly<Record<string, { contextWindow?: number; maxTokens?: number }>>>>;
85
94
  }
86
95
 
87
96
  export interface ModelInfoEntry {