pi-provider-litellm 4.4.2 → 4.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +60 -3
- package/package.json +1 -1
- package/src/backend-identity.ts +1 -1
- package/src/budget.ts +70 -1
- package/src/classifiers.ts +174 -0
- package/src/discover.ts +68 -9
- package/src/images.ts +21 -14
- package/src/index.ts +81 -10
- package/src/model-groups.ts +14 -4
- package/src/models-store.ts +13 -8
- package/src/provider.ts +103 -24
- package/src/types.ts +9 -0
package/README.md
CHANGED
|
@@ -345,6 +345,14 @@ LiteLLM ("litellm") budget
|
|
|
345
345
|
|
|
346
346
|
Automatic polls need Pi's interactive UI and do not run under `LITELLM_OFFLINE=1`, `LITELLM_DISCOVERY_TIMEOUT_MS=0`, or `PI_OFFLINE`; header updates still apply. `/litellm-budget` is stopped by the first two only. The credential and custom headers are kept in memory only as a keyed digest, and proxy responses are reduced to the numbers shown and never logged.
|
|
347
347
|
|
|
348
|
+
When LiteLLM refuses a request because a budget is used up, the error says which budget, its spend and limit, and when it resets, in local time, instead of the proxy's reply:
|
|
349
|
+
|
|
350
|
+
```
|
|
351
|
+
Error: LiteLLM ("litellm"): out of budget. The team budget is used up ($1,000.50 of $1,000) and resets in 2d (2026-11-01 01:00).
|
|
352
|
+
```
|
|
353
|
+
|
|
354
|
+
LiteLLM 1.104.0 and later answer such a request with 422, and earlier releases with 429 or 400; the extension recognises the refusal by its text, whatever the status. Pi does not retry it. It used to retry LiteLLM's own text whenever that held a number like `500`. The figures and the reset come from the budget status, polled again for each refusal under the conditions automatic polls need. Without them, with `budget.enabled` set to `false`, or when the level cannot be read, the error says only which budget is used up. Budgets the footer does not show (end user, tag, project, model access group, and the proxy's own budget), budget windows (`budget_limits`), and per-model budgets keep LiteLLM's text after `out of budget.`, because they reset on their own schedules.
|
|
355
|
+
|
|
348
356
|
## Model transport
|
|
349
357
|
|
|
350
358
|
A `/model/info` route group uses native Anthropic `/v1/messages` only when every deployment explicitly reports Chat mode, identifies a Claude backend through an Anthropic-, Bedrock-, or Vertex-capable adapter, and every declared backend identity resolves the same Anthropic compatibility policy. A `claude-<name>-<major>-<minor>` backend id resolves through its `claude-<name>-<major>` entry when the exact id is absent; an unknown generation stays on Chat Completions. When `supported_endpoints` is a list, it must include `/v1/messages`; an explicit endpoint list without it keeps the group on Chat Completions. A `null` value, which LiteLLM v1.100 and later report for models absent from their model map, counts as absent. An unresolved routing or base-model identity denies native Messages rather than being discarded from the unanimity check. Mixed-generation Claude groups can use Messages when their serializer policies match, with reasoning levels restricted to their common supported set. Groups with incompatible policies, mixed families, or unknown backend evidence remain on Chat Completions; unanimous explicit Responses mode takes precedence. Routes discovered through `/health` never select native Messages: Messages candidates are downgraded to Chat Completions with reasoning controls closed, while correlated Responses detail remains on Responses. Evidence-free `/v1/models` and wholly health-only `/health` groups take transport and presentation metadata from the bounded Pi catalog lookup: `openai-responses` when the catalog says so, otherwise Chat Completions. They expose no speculative reasoning selector and never select native Messages. Catalog identity such as Amazon Bedrock remains separate from the selected wire protocol.
|
|
@@ -375,6 +383,38 @@ and groups that mix image and other modes, or wildcard routes, are withheld. The
|
|
|
375
383
|
need `/v1/images/edits`, which this provider does not call, and backends that answer with image URLs instead of
|
|
376
384
|
base64 data are reported as errors rather than fetched.
|
|
377
385
|
|
|
386
|
+
## Classification
|
|
387
|
+
|
|
388
|
+
Decision routes are published as Pi classifier models (for example `litellm/decision-1`, backed by
|
|
389
|
+
`azure_ai/Microsoft-Decision-1`, or a `typesafe/jev-latest` route). A route qualifies when every deployment behind it
|
|
390
|
+
reports `mode: "evaluation"` or `"decisions"` in `/model/info`, or lists `/v1/systemone` or `/v1/decisions` in
|
|
391
|
+
`supported_endpoints`; a declared endpoint list without them withholds the route. Like Pi's other classifiers they do
|
|
392
|
+
not appear in `/model`; scripts reach them through Pi's `codemode` tool:
|
|
393
|
+
|
|
394
|
+
```js
|
|
395
|
+
const judge = await models.getModelOfType("classifier", "litellm", "decision-1");
|
|
396
|
+
const result = await models.classify(judge, {
|
|
397
|
+
state: { text: "Please send me the report by Friday." },
|
|
398
|
+
questions: {
|
|
399
|
+
polite: {
|
|
400
|
+
type: "bool",
|
|
401
|
+
instructions: "Is the message polite?",
|
|
402
|
+
criteria: { true: "The message is polite.", false: "The message is rude." },
|
|
403
|
+
},
|
|
404
|
+
},
|
|
405
|
+
});
|
|
406
|
+
if (result.stopReason !== "stop") return result.errorMessage;
|
|
407
|
+
return result.answers.polite.probability;
|
|
408
|
+
```
|
|
409
|
+
|
|
410
|
+
Question types are `bool`, `choice` and `score`. Requests go to the proxy's `/v1/systemone` (LiteLLM v1.104.2 and
|
|
411
|
+
later) with the provider's own credential and headers, and are budgeted and logged by the proxy like chat requests. A
|
|
412
|
+
`bool` question is sent as System One's `noul`. Usage cost comes from LiteLLM's `x-litellm-response-cost` header when
|
|
413
|
+
present; otherwise it is estimated from `input_cost_per_token` and `output_cost_per_token`. Only routes found through
|
|
414
|
+
`/model/info` qualify, and groups that mix decision and other deployments, or wildcard routes, are withheld. The state
|
|
415
|
+
is text only: images in the context return an error without a request. A chat route that also lists `/v1/decisions`,
|
|
416
|
+
such as OpenAI's GPT-6 Luna, is published as both a chat model and a classifier.
|
|
417
|
+
|
|
378
418
|
## Optional environment variables
|
|
379
419
|
|
|
380
420
|
| Variable | Default | Effect |
|
|
@@ -393,6 +433,7 @@ base64 data are reported as errors rather than fetched.
|
|
|
393
433
|
| `LITELLM_CLI_JWT_EXPIRATION_HOURS` | `24` | CLI SSO token lifetime fallback for older proxies whose poll response omits `expires_in`; mirror a non-default proxy setting locally |
|
|
394
434
|
| `LITELLM_VERBOSE_DISCOVERY` | unset | If `1`, enable progress messages during model discovery (login, refresh, startup), including startup skip reasons and defaulted `/model/info` context windows. Progress messages are off by default |
|
|
395
435
|
| `LITELLM_DEFAULT_CONTEXT_WINDOW` | `128000` | Context window assumed when neither LiteLLM's `/model/info` nor the catalog reports `max_input_tokens`. Set a positive integer for proxies whose custom aliases carry no metadata; an unset or unusable value keeps 128K |
|
|
436
|
+
| `LITELLM_TRUST_PROXY_LIMITS` | unset | If `1`, a deployment's explicit `model_info.max_input_tokens` / `max_output_tokens` win over Pi's catalog instead of taking the smaller of the two; the catalog still fills missing values and the group minimum still applies. A `models.json` `modelOverrides` entry still wins first |
|
|
396
437
|
| `LITELLM_MODELS_DEV` | unset | If `1`, enrich discovered metadata (limits, prices, effort lists) from models.dev, cached for 28 days in `litellm-models-dev.json`. Off by default because LiteLLM's `/model/info` is authoritative; use it when LiteLLM's model map lacks a model's metadata |
|
|
397
438
|
|
|
398
439
|
Only use a trusted `LITELLM_API_KEY_HELPER` or `!command`. Prefer an absolute executable path, keep secrets out of command arguments, and print only the token to stdout without logging it to stderr.
|
|
@@ -423,6 +464,8 @@ The extension registers at startup and updates the registration before each agen
|
|
|
423
464
|
|
|
424
465
|
Before registering, the extension sends the proxy one MCP `initialize` with the credential and headers Pi will send, and ends the session it opened. LiteLLM 1.102.0 and later refuse it with 403 when no MCP servers are granted to the key, its team, or its organization, or none of them is allowed from your IP address. The extension then registers the server disabled: Pi does not connect to it, warn about it at startup, or turn on `codemode` for it, and `/mcp` lists it as disabled. Once servers are granted, enable it in `/mcp` for the session, or restart Pi, which checks again, as does a login. When a key helper (a `!command` key) or Google ADC is configured, the check is skipped and the server is registered enabled.
|
|
425
466
|
|
|
467
|
+
A proxy that checks budgets on `initialize` refuses it with its budget error once a budget is used up, and every MCP call then fails until the budget resets. The extension registers the server disabled in that case too, and says so once: `LiteLLM MCP ("litellm"): out of budget, so its MCP server is off for this session. Turn it on in /mcp once the budget resets.` After the reset, enable it in `/mcp` or restart Pi.
|
|
468
|
+
|
|
426
469
|
Pi reads the provider's credential when it sends each request, not when the server was registered. If another Pi process stores a login for a different proxy, this session's open connection can send that credential to the previous proxy until the next turn moves the server.
|
|
427
470
|
|
|
428
471
|
Earlier versions discovered tools through `/mcp-rest/tools/list` and registered them as `mcp_<server>_<tool>_<hash>`. Those names and the `<server>/<tool>` form of `toolExposure` keys no longer apply, and the `~/.pi/agent/litellm-mcp-pauses/` directory is no longer used.
|
|
@@ -512,7 +555,7 @@ Each distinct availability diagnostic is written once per session on stderr. `LI
|
|
|
512
555
|
|
|
513
556
|
Discovery selects native Messages for proven Claude groups under the [Model transport](#model-transport) rules, Responses for OpenAI-family backends, and Chat Completions for other OpenAI-compatible backends. Azure deployments stay on Chat Completions unless an explicit Responses mode or a `supported_endpoints` list containing `/v1/responses` authorizes Responses; `api_version` alone does not prove support. Azure AI (`azure_ai/` prefix or `custom_llm_provider: azure_ai`) deployments additionally depend on the proxy version. On LiteLLM before v1.103.0, or when the version is unknown, they ignore a `supported_endpoints` list that contains `/v1/responses` and need an explicit `mode: "responses"`: those releases have no native Responses config for that provider, so they bridge `/v1/responses` through Chat Completions, and the bridge fails streaming with `list index out of range` on Azure's empty-choices chunks (BerriAI/litellm#34455) while the endpoint list, copied from the model cost map, still advertises `/v1/responses`. From v1.103.0 the list is honored again. Discovery reads the version from the `x-litellm-version` header of one extra request, `GET /v1/responses/resp_version_probe`, sent only when such a deployment is published; LiteLLM answers it with an error and calls no model. A pre-release counts as older than the release it names, and a proxy that omits the header keeps these deployments on Chat Completions. Azure is detected from the adapter field, an `azure/` or `azure_ai/` model prefix, or the reported provider; other adapters follow the backend family because LiteLLM bridges `/v1/responses` to Chat Completions when the provider has no native Responses config (`litellm/responses/main.py`, `_bridges_to_chat_completions`). The family comes from the model id, not from LiteLLM's `openai/`, `openai_like/` or `custom_openai/` adapter prefix, so `openai/qwen3` stays on Chat Completions while `openai/gpt-5.5` uses Responses; a deployment opts into Responses with `mode: "responses"` or a `supported_endpoints` list containing `/v1/responses`. A route whose model prefix and `custom_llm_provider` name different providers is treated as unidentified and stays on Chat Completions, so it also does not get the local 128-tool preflight. When `/model/info` supplies `supported_endpoints`, that list takes precedence, and `mode: "responses"` remains an explicit Responses signal. An evidence-free fallback entry—a bare `/v1/models` id or a `/health` entry with no detail row—takes its protocol from the Pi catalog entry for its id; both evidence-free fallback paths also inherit presentation metadata from the bounded catalog lookup. An unknown id stays on Chat Completions. The route name alone authorizes nothing. A concrete id expanded from wildcard `/model/info` routes inherits the deployment evidence of the wildcard route LiteLLM would select for it, with the requested id substituted into the row's `model` the way LiteLLM serves it; the id is omitted when that selected route is not a published chat-style route. Chat Completions and Responses use `<root>/v1`; native Messages uses the proxy root directly. All three protocols pass through the provider's host guard. When an evidence-free cached `(no metadata)` entry can be enriched from the Pi catalog, it is restored on Responses only for a catalog Responses model and otherwise on Chat Completions, never Anthropic Messages. A model supplied only through `models.json` whose `api` this provider does not implement takes Pi's global API fallback rather than this provider's host guard; the credential-root pin (see [Model host enforcement](#model-host-enforcement)) still keeps the credential on the active proxy.
|
|
514
557
|
|
|
515
|
-
`cacheControlFormat: "anthropic"` applies only to Chat Completions, where Pi adds Anthropic `cache_control` markers. The Responses transport has a different compatibility type and uses native `prompt_cache_key` (plus supported retention fields), so Responses models must not receive `cacheControlFormat`. Freshly discovered OpenAI-family models forced onto Chat Completions are rejected locally when a request contains more than 128 tools; route them through Responses or reduce enabled extensions. Moonshot request-side reasoning suppression follows LiteLLM's routing provider, not `model_info.base_model`; response-side `<think>` normalization follows discovered Kimi-family evidence. Opaque aliases retain display normalization without sending Moonshot-only parameters to Azure- or Bedrock-hosted Kimi models. Strict tool-message repair requires every deployment to identify Kimi; Gemini effort normalization likewise requires consistent backend-family evidence.
|
|
558
|
+
`cacheControlFormat: "anthropic"` applies only to Chat Completions, where Pi adds Anthropic `cache_control` markers. The Responses transport has a different compatibility type and uses native `prompt_cache_key` (plus supported retention fields), so Responses models must not receive `cacheControlFormat`. Freshly discovered OpenAI-family models forced onto Chat Completions are rejected locally when a request contains more than 128 tools; route them through Responses or reduce enabled extensions. Moonshot request-side reasoning suppression follows LiteLLM's routing provider, not `model_info.base_model`; response-side `<think>` normalization follows discovered Kimi-family evidence. Opaque aliases retain display normalization without sending Moonshot-only parameters to Azure- or Bedrock-hosted Kimi models. Strict tool-message repair requires every deployment to identify Kimi; Gemini effort normalization likewise requires consistent backend-family evidence. Claude-backed routes on native Messages or Chat Completions declare Pi's `promptCache` lifetimes (`short: 300`, `long: 3600` seconds, the values Pi's built-in Anthropic models use), so Pi's `cacheWarming` can keep their caches warm. Responses routes whose every deployment is GPT-5.6 or later and routes through LiteLLM's `openai`, `azure` or `azure_ai` provider declare `short: 1800`, the 30-minute default minimum lifetime OpenAI and Azure document for those models, and no `long` value. Claude routes on Responses, Chat routes to OpenAI models, earlier GPT models, GPT models served through a relay such as OpenRouter or GitHub Copilot, routes identified only by their name, and other backend families declare none, so Pi does not warm them. A `modelOverrides` entry for the route in the global `models.json` can change the values, and `cacheWarming: "off"` disables warming.
|
|
516
559
|
|
|
517
560
|
Discovery policy is versioned in Pi's persisted model catalog. After upgrading, an online refresh replaces legacy entries. Offline legacy entries remain usable without a diagnostic; the extension reports the discovery-version mismatch once only when an attempted online refresh fails. To force rediscovery or roll back across this policy change, delete the `litellm` entry from `~/.pi/agent/models-store.json` (or delete the file) and refresh once online.
|
|
518
561
|
|
|
@@ -522,7 +565,19 @@ LiteLLM may load-balance one public `model_name` across deployments with differe
|
|
|
522
565
|
|
|
523
566
|
- Responses is selected only when every deployment supports it under the protocol rules above; a deployment that requires Chat keeps the group on Chat.
|
|
524
567
|
- Vision and reasoning are advertised only when every routable deployment resolves them as supported. An explicit `supports_reasoning: false` suppresses all thinking controls, even when catalog or router effort metadata exists. Catalog thinking controls are intersected per level: a level is exposed only when every deployment maps it to the same value, while disagreement or absence becomes an explicit denial. Catalog maps are not complete effort lists, so absence has one exception: an omitted `minimal`, `low`, `medium` or `high` counts as Pi's default spelling of that level. An omitted `off`, `xhigh` or `max` still differs from an explicit mapping. Reasoning selectors additionally require an accepted wire carrier on every deployment. Public effort lists are intersected, explicit LiteLLM denials win, and `xhigh`/`max` need unanimous explicit support. Null or absent standard-effort flags have no opinion. A deployment that declares `model_info.reasoning_effort_levels` answers every level from that list, ahead of its per-level flags, as LiteLLM itself reads it.
|
|
525
|
-
- Context and output limits use the minimum resolved value across deployments. Pi
|
|
568
|
+
- Context and output limits use the minimum resolved value across deployments. Per deployment, a Pi `models.json` `modelOverrides` entry for the identified backend model (Pi provider id plus model id, resolved from backend identity, never from the route name; the provider id is the one whose Pi catalog entry identifies the backend, such as `openai-codex` for a `chatgpt/` deployment, not the LiteLLM provider label) sets `contextWindow` / `maxTokens` ahead of both the router figure and the catalog; it applies only under unanimous catalog authority, and the group minimum still applies. Without an override, the smaller of the router's explicit figure and the catalog figure is used, or the router's figure first under `LITELLM_TRUST_PROXY_LIMITS=1`. Overrides are read from the global `models.json` only and take effect at the next discovery (`/litellm-refresh` or cache expiry). Pi applies `modelOverrides` only to that provider's own models, so one entry covers every LiteLLM route backed by the model:
|
|
569
|
+
|
|
570
|
+
```json
|
|
571
|
+
{
|
|
572
|
+
"providers": {
|
|
573
|
+
"openai": {
|
|
574
|
+
"modelOverrides": { "gpt-5.2": { "contextWindow": 500000 } }
|
|
575
|
+
}
|
|
576
|
+
}
|
|
577
|
+
}
|
|
578
|
+
```
|
|
579
|
+
|
|
580
|
+
Pi catalogue lookup prefers the deployment's provider-specific entry before falling back to the generic vendor entry; Azure and Azure AI therefore retain Azure-specific limits rather than being capped by a different OpenAI catalogue entry. When models.dev supplies a sparse vendor record, missing fields still use the provider-specific Pi catalogue.
|
|
526
581
|
- Each displayed base-price field uses the maximum only when every deployment resolves that field; unresolved fields remain zero and the model name is suffixed with ` (incomplete metadata)`.
|
|
527
582
|
- When complete base pricing comes from the catalog under unanimous catalog authority, tiered pricing is the conservative worst-case envelope. Its usable finite non-negative thresholds are the sorted union of every deployment's thresholds, and each price field at each interval is the maximum applicable rate across all deployments, floored by each deployment's base rate. Differing tier breakpoints are therefore preserved rather than treated as incompatible. An explicit `/model/info` base-price field replaces catalog pricing for that field at every threshold; catalog tiers remain for unaffected fields and are omitted only when every base-price field is explicit.
|
|
528
583
|
- Reduced route groups omit tier ladders when complete base-pricing or catalog authority is unavailable. Wildcard expansion retains every known tier from matching groups even when another match has incomplete metadata: suppressing a complete sibling's higher tier could understate the known worst-case rate. The ` (incomplete metadata)` marker remains because the resulting envelope is only as complete as the available ladders.
|
|
@@ -563,13 +618,15 @@ The development probe runs against minimized snapshots with `npm run probe:proxy
|
|
|
563
618
|
| `LiteLLM discovery: ... route group(s) mix chat-style and explicitly incompatible deployment modes` | A public route can select both a chat-style deployment and an incompatible deployment such as an embedding model. Give the embedding deployment its own `model_name`. |
|
|
564
619
|
| `LiteLLM (...): the request for "<route>" was answered by a fallback` | A successful HTTP response reports a LiteLLM router fallback (`x-litellm-attempted-fallbacks`). `/model/info` does not expose fallbacks, so protocol and model handling follow the requested route's own deployments. If the primary and fallback groups support Chat Completions, set `model_info.supported_endpoints: ["/v1/chat/completions"]` on the primary to avoid Responses-to-Chat bridging. This warning does not change routing or fix upstream bridging defects. It names only the requested route; LiteLLM reports the group and model that answered in the `x-litellm-model-group` and `x-litellm-model-name` response headers. Warned once per provider and route until the extension reloads; shown in the UI, or stderr without a UI. |
|
|
565
620
|
| A model is marked ` (incomplete metadata)` | `/model/info` or `/health` did not provide enough authoritative metadata. Explicit fields remain usable, but unknown cost fields are shown as zero and route-name cache enrichment stays disabled. |
|
|
621
|
+
| Cache miss notices after idle, no cache warming | The model declares no `promptCache`: only Claude-backed Messages and Chat routes and GPT-5.6+ Responses routes on OpenAI or Azure get one. Run `/litellm-refresh` after updating so stored models are rediscovered. |
|
|
566
622
|
| `401 Token expired` | Set `LITELLM_API_KEY_HELPER`. |
|
|
567
623
|
| No models with gcloud auth | Verify `gcloud auth application-default login` has been run or set `GOOGLE_APPLICATION_CREDENTIALS` to an `authorized_user` ADC file |
|
|
568
624
|
| Enterprise SSO waits for token insertion | The proxy returned 404/405 for `/sso/cli/start`, so Pi used the legacy flow — upgrade LiteLLM or paste the UI token |
|
|
569
625
|
| Enterprise CLI SSO start/poll fails | Check the proxy logs and verify `/sso/cli/start` and `/sso/cli/poll/{login_id}` are reachable; only 404/405 falls back to legacy login |
|
|
570
626
|
| Enterprise SSO login shows "virtual key generation failed" | The LiteLLM instance may lack a database (`/key/generate` requires one), your user account may lack key-generation permission, or the request timed out; the JWT is used directly as a fallback |
|
|
571
627
|
| Enterprise SSO token prompt fails with "SSO token is required" | The token field was left empty — paste the token copied from the LiteLLM UI |
|
|
572
|
-
|
|
|
628
|
+
| `LiteLLM (...): out of budget` | A LiteLLM budget is used up. The error names it and, when the budget status can read it, its reset time; `/litellm-budget` shows every level. Ask the proxy admin to raise the limit, or wait for the reset — see [Budget status](#budget-status) |
|
|
629
|
+
| MCP tools not showing | Run `/mcp` and check the provider's server: its state shows a connection or sign-in error. `disabled` means the proxy refused the key at startup, usually because no MCP servers are granted to it, its team, or its organization, or because a budget is used up, which is reported at startup; after a grant or the budget's reset, enable it in `/mcp` or restart Pi. Verify the proxy serves `/mcp` and that the key has MCP access |
|
|
573
630
|
| MCP tools listed but the model cannot call them | `codemode` and `deferred` tools are reached through Pi's `codemode` and `tool_search` tools. Check that neither is disabled; `--no-extensions` disables them and Pi's MCP client alike. Setting `litellm.mcp.exposure` to `direct` declares the tools on every request instead |
|
|
574
631
|
| Skills not affecting prompts | Verify the proxy exposes `/claude-code/marketplace.json` or `/v1/skills` and returns enabled skills |
|
|
575
632
|
|
package/package.json
CHANGED
package/src/backend-identity.ts
CHANGED
package/src/budget.ts
CHANGED
|
@@ -311,6 +311,65 @@ export function formatBudgetDetails(
|
|
|
311
311
|
return lines.join("\n");
|
|
312
312
|
}
|
|
313
313
|
|
|
314
|
+
// LiteLLM refuses a used-up budget with `type: "budget_exceeded"` (status 422 since 1.104, 429 or 400 before), and
|
|
315
|
+
// /mcp with the bare message, whose wording varies by level and check.
|
|
316
|
+
const BUDGET_EXCEEDED = /budget_exceeded|Budget has been exceeded!|ExceededBudget:|exceeded budget for model=/;
|
|
317
|
+
// The refusals of a level's own budget. Budget windows and per-model budgets reset on schedules of their own.
|
|
318
|
+
const EXCEEDED_LEVEL =
|
|
319
|
+
/Budget has been exceeded! (Key|Team|TeamMember|User|Organization)=\S*( in Team=)?|ExceededBudget: (User)=.+? over budget\./;
|
|
320
|
+
const EXCEEDED_LEVELS: Record<string, BudgetLevelName> = {
|
|
321
|
+
Key: "key",
|
|
322
|
+
Team: "team",
|
|
323
|
+
TeamMember: "member",
|
|
324
|
+
User: "user",
|
|
325
|
+
Organization: "org",
|
|
326
|
+
};
|
|
327
|
+
const LEVEL_LABELS: Record<BudgetLevelName, string> = {
|
|
328
|
+
key: "key",
|
|
329
|
+
user: "user",
|
|
330
|
+
team: "team",
|
|
331
|
+
member: "team member",
|
|
332
|
+
org: "organization",
|
|
333
|
+
};
|
|
334
|
+
|
|
335
|
+
export const isBudgetExceeded = (text: string): boolean => BUDGET_EXCEEDED.test(text);
|
|
336
|
+
|
|
337
|
+
// Local time in one fixed format, which the Go port prints the same way.
|
|
338
|
+
function resetClock(resetAt: number): string {
|
|
339
|
+
const at = new Date(resetAt);
|
|
340
|
+
const pad = (n: number) => String(n).padStart(2, "0");
|
|
341
|
+
return `${at.getFullYear()}-${pad(at.getMonth() + 1)}-${pad(at.getDate())} ${pad(at.getHours())}:${pad(at.getMinutes())}`;
|
|
342
|
+
}
|
|
343
|
+
|
|
344
|
+
// Starts with "out of budget", which Pi and PiG never retry. A refusal whose level is not tracked keeps the proxy's
|
|
345
|
+
// text, as that names the budget.
|
|
346
|
+
export function formatBudgetExceeded(errorText: string, levels: BudgetLevels, now: number): string {
|
|
347
|
+
const match = EXCEEDED_LEVEL.exec(errorText);
|
|
348
|
+
if (!match) return `out of budget. ${errorText}`;
|
|
349
|
+
const name = match[2] ? "member" : EXCEEDED_LEVELS[match[1] ?? match[3]!]!;
|
|
350
|
+
const l = levels[name];
|
|
351
|
+
const amounts = l ? ` (${money(l.spend, 2)} of ${money(l.maxBudget, Number.isInteger(l.maxBudget) ? 0 : 2)})` : "";
|
|
352
|
+
const reset = formatRelativeReset(l?.resetAt, now);
|
|
353
|
+
const resets = reset ? ` and resets in ${reset} (${resetClock(l!.resetAt!)})` : "";
|
|
354
|
+
return `out of budget. The ${LEVEL_LABELS[name]} budget is used up${amounts}${resets}.`;
|
|
355
|
+
}
|
|
356
|
+
|
|
357
|
+
export function setupLiteLLMBudgetErrors(
|
|
358
|
+
pi: ExtensionAPI,
|
|
359
|
+
providerNames: ReadonlySet<string>,
|
|
360
|
+
levelsAfterError?: (ctx: ExtensionContext, name: string) => Promise<BudgetLevels>,
|
|
361
|
+
): void {
|
|
362
|
+
pi.on("message_end", async (event, ctx) => {
|
|
363
|
+
const message = event.message;
|
|
364
|
+
if (message.role !== "assistant" || message.stopReason !== "error" || !providerNames.has(message.provider)) return;
|
|
365
|
+
const text = message.errorMessage ?? "";
|
|
366
|
+
if (!isBudgetExceeded(text)) return;
|
|
367
|
+
const levels = (await levelsAfterError?.(ctx, message.provider).catch(() => undefined)) ?? {};
|
|
368
|
+
const explained = formatBudgetExceeded(text, levels, Date.now());
|
|
369
|
+
return { message: { ...message, errorMessage: `LiteLLM (${JSON.stringify(message.provider)}): ${explained}` } };
|
|
370
|
+
});
|
|
371
|
+
}
|
|
372
|
+
|
|
314
373
|
export function budgetDisplaySetting(budgetSettings: unknown): { display: BudgetDisplay; warning?: string } {
|
|
315
374
|
const display = obj(budgetSettings)?.display;
|
|
316
375
|
if (display === undefined || display === "all" || display === "tightest") return { display: display ?? "all" };
|
|
@@ -344,7 +403,10 @@ interface ProviderState {
|
|
|
344
403
|
inFlight?: Promise<PollOutcome>;
|
|
345
404
|
}
|
|
346
405
|
|
|
347
|
-
export function setupLiteLLMBudget(
|
|
406
|
+
export function setupLiteLLMBudget(
|
|
407
|
+
pi: ExtensionAPI,
|
|
408
|
+
options: BudgetOptions,
|
|
409
|
+
): (ctx: ExtensionContext, name: string) => Promise<BudgetLevels> {
|
|
348
410
|
const displayNames = new Map(options.providers.map(({ name, displayName }) => [name, displayName]));
|
|
349
411
|
const states = new Map<string, ProviderState>();
|
|
350
412
|
// Keyed like the MCP registration identity in index.ts, because custom headers can carry gateway credentials.
|
|
@@ -558,4 +620,11 @@ export function setupLiteLLMBudget(pi: ExtensionAPI, options: BudgetOptions): vo
|
|
|
558
620
|
}
|
|
559
621
|
},
|
|
560
622
|
});
|
|
623
|
+
|
|
624
|
+
// The levels a budget refusal is explained with. Polled for each refusal, joining a poll in flight, because only a
|
|
625
|
+
// poll notices that the credential changed since the last one.
|
|
626
|
+
return async (ctx, name) => {
|
|
627
|
+
if (ctx?.hasUI && !gated()) await poll(ctx, name);
|
|
628
|
+
return stateFor(name).levels;
|
|
629
|
+
};
|
|
561
630
|
}
|
|
@@ -0,0 +1,174 @@
|
|
|
1
|
+
import type { ClassifierAnswer, ClassifierModel, ClassifierResult, ProviderClassifier } from "@earendil-works/pi-ai";
|
|
2
|
+
import { LITELLM_DISCOVERY_VERSION } from "./backend-identity.js";
|
|
3
|
+
import { errorDetail, parseUsage, perMillion, requestHeaders, requestSignal, responseCost } from "./images.js";
|
|
4
|
+
import { defaultContextWindow, wireString } from "./model-groups.js";
|
|
5
|
+
import type { ModelInfoEntry } from "./types.js";
|
|
6
|
+
|
|
7
|
+
/** Decision models through the proxy's System One `/v1/systemone` endpoint. */
|
|
8
|
+
export const LITELLM_DECISIONS_API = "litellm-decisions" as const;
|
|
9
|
+
export type LiteLLMDecisionsApi = typeof LITELLM_DECISIONS_API;
|
|
10
|
+
|
|
11
|
+
export type DiscoveredClassifierModel = Omit<ClassifierModel<LiteLLMDecisionsApi>, "provider" | "baseUrl"> & {
|
|
12
|
+
litellmDiscoveryVersion?: typeof LITELLM_DISCOVERY_VERSION;
|
|
13
|
+
};
|
|
14
|
+
|
|
15
|
+
export type LiteLLMClassifierModel = ClassifierModel<LiteLLMDecisionsApi> & {
|
|
16
|
+
litellmDiscoveryVersion?: typeof LITELLM_DISCOVERY_VERSION;
|
|
17
|
+
};
|
|
18
|
+
|
|
19
|
+
const DECISION_MODES = new Set(["evaluation", "decisions"]);
|
|
20
|
+
const DECISION_ENDPOINTS = new Set(["/v1/systemone", "/v1/decisions"]);
|
|
21
|
+
|
|
22
|
+
/**
|
|
23
|
+
* LiteLLM's own rule: a declared `supported_endpoints` list decides alone, so a list without a
|
|
24
|
+
* decision endpoint denies even with an evaluation mode; without a list, the mode decides.
|
|
25
|
+
*/
|
|
26
|
+
export function isDecisionDeployment(entry: ModelInfoEntry): boolean {
|
|
27
|
+
const info = entry.model_info;
|
|
28
|
+
const endpoints: unknown = info?.supported_endpoints;
|
|
29
|
+
if (Array.isArray(endpoints)) {
|
|
30
|
+
return endpoints.some((endpoint) => typeof endpoint === "string" && DECISION_ENDPOINTS.has(endpoint.trim()));
|
|
31
|
+
}
|
|
32
|
+
const mode = wireString(info?.mode);
|
|
33
|
+
return mode !== undefined && DECISION_MODES.has(mode.trim().toLowerCase());
|
|
34
|
+
}
|
|
35
|
+
|
|
36
|
+
/**
|
|
37
|
+
* Reduce one `/model/info` route group to a classifier model. Every deployment behind the route
|
|
38
|
+
* must be a decision deployment: LiteLLM load-balances across them, so a mixed group could answer
|
|
39
|
+
* from a non-decision deployment.
|
|
40
|
+
*/
|
|
41
|
+
export function mapFromClassifierModelInfoGroup(
|
|
42
|
+
route: string,
|
|
43
|
+
entries: readonly ModelInfoEntry[],
|
|
44
|
+
): DiscoveredClassifierModel | undefined {
|
|
45
|
+
// A wildcard row is not addressable, and `/v1/models` expansion carries no mode evidence.
|
|
46
|
+
if (route.includes("*")) return undefined;
|
|
47
|
+
const deployments = entries.filter((entry) => wireString(entry.model_name) === route);
|
|
48
|
+
if (deployments.length === 0) return undefined;
|
|
49
|
+
if (!deployments.every(isDecisionDeployment)) return undefined;
|
|
50
|
+
// Prices are estimates for Pi's usage display; the `x-litellm-response-cost` header replaces them.
|
|
51
|
+
const rate = (read: (entry: ModelInfoEntry) => number | undefined): number => {
|
|
52
|
+
const values = deployments.map(read);
|
|
53
|
+
return values.every((value) => value !== undefined) ? Math.max(...(values as number[])) : 0;
|
|
54
|
+
};
|
|
55
|
+
// The smallest declared window; an undeclared one counts as the default so it never raises the result.
|
|
56
|
+
const windows = deployments.map((entry) => entry.model_info?.max_input_tokens);
|
|
57
|
+
const declared = windows.filter(
|
|
58
|
+
(value): value is number => typeof value === "number" && Number.isFinite(value) && value > 0,
|
|
59
|
+
);
|
|
60
|
+
const contextWindow = Math.min(...declared, ...(declared.length < windows.length ? [defaultContextWindow()] : []));
|
|
61
|
+
return {
|
|
62
|
+
type: "classifier",
|
|
63
|
+
id: route,
|
|
64
|
+
name: route,
|
|
65
|
+
api: LITELLM_DECISIONS_API,
|
|
66
|
+
input: ["text"],
|
|
67
|
+
cost: {
|
|
68
|
+
input: rate((entry) => perMillion(entry.model_info?.input_cost_per_token)),
|
|
69
|
+
output: rate((entry) => perMillion(entry.model_info?.output_cost_per_token)),
|
|
70
|
+
cacheRead: 0,
|
|
71
|
+
cacheWrite: 0,
|
|
72
|
+
},
|
|
73
|
+
contextWindow,
|
|
74
|
+
litellmDiscoveryVersion: LITELLM_DISCOVERY_VERSION,
|
|
75
|
+
};
|
|
76
|
+
}
|
|
77
|
+
|
|
78
|
+
function isRecord(value: unknown): value is Record<string, unknown> {
|
|
79
|
+
return typeof value === "object" && value !== null && !Array.isArray(value);
|
|
80
|
+
}
|
|
81
|
+
|
|
82
|
+
const isFiniteNumber = (value: unknown): value is number => typeof value === "number" && Number.isFinite(value);
|
|
83
|
+
|
|
84
|
+
function parseAnswer(model: string, id: string, type: "choice" | "score" | "bool", raw: unknown): ClassifierAnswer {
|
|
85
|
+
const wireType = type === "bool" ? "noul" : type;
|
|
86
|
+
if (!isRecord(raw) || raw.type !== wireType) {
|
|
87
|
+
throw new Error(`LiteLLM classifier ${model} did not return a ${type} answer for ${id}`);
|
|
88
|
+
}
|
|
89
|
+
const invalid = (field: string): Error =>
|
|
90
|
+
new Error(`LiteLLM classifier ${model} returned an invalid ${field} for ${id}`);
|
|
91
|
+
if (type === "bool") {
|
|
92
|
+
if (!isFiniteNumber(raw.noul)) throw invalid("noul");
|
|
93
|
+
return { type: "bool", probability: raw.noul };
|
|
94
|
+
}
|
|
95
|
+
if (!isFiniteNumber(raw.confidence)) throw invalid("confidence");
|
|
96
|
+
if (type === "score") {
|
|
97
|
+
if (!isFiniteNumber(raw.score)) throw invalid("score");
|
|
98
|
+
return { type: "score", score: raw.score, confidence: raw.confidence };
|
|
99
|
+
}
|
|
100
|
+
if (typeof raw.choice !== "string") throw invalid("choice");
|
|
101
|
+
if (!isRecord(raw.probabilities) || !Object.values(raw.probabilities).every(isFiniteNumber)) {
|
|
102
|
+
throw invalid("probabilities");
|
|
103
|
+
}
|
|
104
|
+
return {
|
|
105
|
+
type: "choice",
|
|
106
|
+
choice: raw.choice,
|
|
107
|
+
probabilities: raw.probabilities as Record<string, number>,
|
|
108
|
+
confidence: raw.confidence,
|
|
109
|
+
};
|
|
110
|
+
}
|
|
111
|
+
|
|
112
|
+
/** Classify through LiteLLM's `/v1/systemone`. Never rejects. */
|
|
113
|
+
export const classify: ProviderClassifier["classify"] = async (model, context, options) => {
|
|
114
|
+
const result: ClassifierResult = {
|
|
115
|
+
api: model.api,
|
|
116
|
+
provider: model.provider,
|
|
117
|
+
model: model.id,
|
|
118
|
+
answers: {},
|
|
119
|
+
stopReason: "stop",
|
|
120
|
+
timestamp: Date.now(),
|
|
121
|
+
};
|
|
122
|
+
try {
|
|
123
|
+
const apiKey = options?.apiKey;
|
|
124
|
+
if (!apiKey) throw new Error(`No API key for provider: ${model.provider}`);
|
|
125
|
+
if (context.images?.length) {
|
|
126
|
+
throw new Error(`LiteLLM classifier ${model.id} takes text state only; images are not sent to /v1/systemone`);
|
|
127
|
+
}
|
|
128
|
+
// System One calls Pi's `bool` question `noul`; the fields are otherwise the same.
|
|
129
|
+
const questions = Object.fromEntries(
|
|
130
|
+
Object.entries(context.questions).map(([id, question]) => [
|
|
131
|
+
id,
|
|
132
|
+
question.type === "bool" ? { ...question, type: "noul" } : question,
|
|
133
|
+
]),
|
|
134
|
+
);
|
|
135
|
+
let params: Record<string, unknown> = { model: model.id, state: context.state, questions };
|
|
136
|
+
const nextParams = await options?.onPayload?.(params, model);
|
|
137
|
+
if (nextParams !== undefined) params = nextParams as Record<string, unknown>;
|
|
138
|
+
const response = await (options?.fetch ?? fetch)(`${model.baseUrl}/systemone`, {
|
|
139
|
+
method: "POST",
|
|
140
|
+
headers: requestHeaders(apiKey, model.headers, options?.headers),
|
|
141
|
+
body: JSON.stringify(params),
|
|
142
|
+
signal: requestSignal(options),
|
|
143
|
+
});
|
|
144
|
+
const responseHeaders = Object.fromEntries(response.headers.entries());
|
|
145
|
+
await options?.onResponse?.({ status: response.status, headers: responseHeaders }, model);
|
|
146
|
+
const body = await response.text();
|
|
147
|
+
if (!response.ok) throw new Error(`LiteLLM decisions request failed (${response.status}): ${errorDetail(body)}`);
|
|
148
|
+
// The proxy billed the request whatever the body holds; token counts are added once it parses.
|
|
149
|
+
const cost = responseCost(responseHeaders);
|
|
150
|
+
const billed = parseUsage(undefined, model.cost, cost);
|
|
151
|
+
if (billed) result.usage = billed;
|
|
152
|
+
const parsed: unknown = JSON.parse(body);
|
|
153
|
+
if (!isRecord(parsed)) throw new Error(`LiteLLM classifier ${model.id} returned an unexpected response`);
|
|
154
|
+
// A malformed answer set was still billed.
|
|
155
|
+
const usage = parseUsage(isRecord(parsed.usage) ? parsed.usage : undefined, model.cost, cost);
|
|
156
|
+
if (usage) result.usage = usage;
|
|
157
|
+
const answers = isRecord(parsed.answers) ? parsed.answers : {};
|
|
158
|
+
const parsedAnswers: Record<string, ClassifierAnswer> = {};
|
|
159
|
+
for (const [id, question] of Object.entries(context.questions)) {
|
|
160
|
+
parsedAnswers[id] = parseAnswer(model.id, id, question.type, answers[id]);
|
|
161
|
+
}
|
|
162
|
+
result.answers = parsedAnswers;
|
|
163
|
+
return result;
|
|
164
|
+
} catch (error) {
|
|
165
|
+
result.answers = {};
|
|
166
|
+
result.stopReason = options?.signal?.aborted ? "aborted" : "error";
|
|
167
|
+
result.errorMessage = error instanceof Error ? error.message : String(error);
|
|
168
|
+
return result;
|
|
169
|
+
}
|
|
170
|
+
};
|
|
171
|
+
|
|
172
|
+
export function createLiteLLMClassifierApis(): Record<LiteLLMDecisionsApi, ProviderClassifier> {
|
|
173
|
+
return { [LITELLM_DECISIONS_API]: { classify } };
|
|
174
|
+
}
|
package/src/discover.ts
CHANGED
|
@@ -8,6 +8,7 @@ import {
|
|
|
8
8
|
resolveCatalogProvider,
|
|
9
9
|
routesOnlyThrough,
|
|
10
10
|
} from "./backend-identity.js";
|
|
11
|
+
import { type DiscoveredClassifierModel, mapFromClassifierModelInfoGroup } from "./classifiers.js";
|
|
11
12
|
import { type DiscoveredImageModel, mapFromImageModelInfoGroup } from "./images.js";
|
|
12
13
|
import {
|
|
13
14
|
type CatalogResolution,
|
|
@@ -50,6 +51,15 @@ import type {
|
|
|
50
51
|
} from "./types.js";
|
|
51
52
|
|
|
52
53
|
const DEFAULT_TIMEOUT_MS = 5000;
|
|
54
|
+
// Anthropic's documented 5-minute and 1-hour cache lifetimes; the values Pi's built-in Anthropic catalog uses.
|
|
55
|
+
// shortcut: `long` assumes the host offers the 1-hour TTL (Bedrock only does for Claude 4.5+), gate it on
|
|
56
|
+
// deployment routing if LiteLLM ever reports per-route TTL support.
|
|
57
|
+
const CLAUDE_PROMPT_CACHE = { short: 300, long: 3600 } as const;
|
|
58
|
+
// OpenAI's and Azure's documented 30-minute default minimum for GPT-5.6 and later. No `long`: the
|
|
59
|
+
// 12-hour lifetime needs `prompt_cache_options.ttl`, which Pi's long retention does not request.
|
|
60
|
+
const OPENAI_PROMPT_CACHE = { short: 1800 } as const;
|
|
61
|
+
// Relays (OpenRouter, Copilot, resellers) document no lifetime of their own and may bill or route differently.
|
|
62
|
+
const OPENAI_PROMPT_CACHE_ROUTES: ReadonlySet<string> = new Set(["openai", "azure", "azure_ai"]);
|
|
53
63
|
const HEALTH_DETAIL_CONCURRENCY = 8;
|
|
54
64
|
const reportedConflictingFamilyRoutes = new Set<string>();
|
|
55
65
|
const reportedWithheldRepairRoutes = new Set<string>();
|
|
@@ -127,6 +137,12 @@ export function isGpt55OrNewerModel(modelId: string): boolean {
|
|
|
127
137
|
return generation !== undefined && (generation[0] > 5 || (generation[0] === 5 && generation[1] >= 5));
|
|
128
138
|
}
|
|
129
139
|
|
|
140
|
+
// GPT-5.6 and later keep a cached prefix for at least 30 minutes by default.
|
|
141
|
+
function isGpt56OrNewerModel(modelId: string): boolean {
|
|
142
|
+
const generation = gptGeneration(modelId);
|
|
143
|
+
return generation !== undefined && (generation[0] > 5 || (generation[0] === 5 && generation[1] >= 6));
|
|
144
|
+
}
|
|
145
|
+
|
|
130
146
|
// GPT-6 also rejects an omitted effort there: it falls back to its default effort.
|
|
131
147
|
export function isGpt6OrNewerModel(modelId: string): boolean {
|
|
132
148
|
const generation = gptGeneration(modelId);
|
|
@@ -496,6 +512,7 @@ function adaptPublicCatalogRecord(
|
|
|
496
512
|
const piCost = piCatalog?.cost;
|
|
497
513
|
return {
|
|
498
514
|
provider,
|
|
515
|
+
...(resolved ? { piProvider: resolved.provider } : {}),
|
|
499
516
|
// A models.dev hit names the model canonically even when Pi's catalog does not know it yet,
|
|
500
517
|
// so two spellings of one backend model reduce to one identity instead of a conflict.
|
|
501
518
|
catalogModelId: resolved?.model.id ?? record?.modelId ?? catalogModelId,
|
|
@@ -689,10 +706,16 @@ function mapFromModelInfoGroup(
|
|
|
689
706
|
denyLevels?: boolean;
|
|
690
707
|
allowMessages?: boolean;
|
|
691
708
|
proxyVersion?: ProxyVersion;
|
|
709
|
+
modelOverrides?: DiscoveryOptions["modelOverrides"];
|
|
692
710
|
} = {},
|
|
693
711
|
): DiscoveredModel | undefined {
|
|
694
712
|
const reduced = reduceModelGroup(entries, (entry) => {
|
|
695
|
-
const
|
|
713
|
+
const resolved = resolveModelInfoCatalog(entry, publicCatalog);
|
|
714
|
+
const limitOverrides =
|
|
715
|
+
resolved?.piProvider && resolved.catalogModelId
|
|
716
|
+
? options.modelOverrides?.[resolved.piProvider]?.[resolved.catalogModelId]
|
|
717
|
+
: undefined;
|
|
718
|
+
const catalog = limitOverrides ? { ...resolved, limitOverrides } : resolved;
|
|
696
719
|
const family = deploymentFamily(entry);
|
|
697
720
|
const generations = [entry.litellm_params?.model, entry.model_info?.base_model]
|
|
698
721
|
.map((id) => wireString(id)?.trim())
|
|
@@ -714,7 +737,8 @@ function mapFromModelInfoGroup(
|
|
|
714
737
|
const protocols = entries.map((entry) => modelProtocol(reduced.id, entry, options.proxyVersion));
|
|
715
738
|
const protocol = protocols.find((candidate) => candidate.api === "openai-completions") ?? protocols[0]!;
|
|
716
739
|
const api = reduced.api === "anthropic-messages" ? reduced.api : protocol.api;
|
|
717
|
-
const
|
|
740
|
+
const identities = entries.map((entry) => resolveBackendIdentity({ ...entry, model_name: undefined }));
|
|
741
|
+
const families = new Set(identities.map((identity) => identity?.family));
|
|
718
742
|
const [family] = families;
|
|
719
743
|
const hasBackendEvidence = entries.some(hasReadableBackendEvidence);
|
|
720
744
|
const reasoningPolicy =
|
|
@@ -756,6 +780,10 @@ function mapFromModelInfoGroup(
|
|
|
756
780
|
acceptsResponsesReasoningControl: reduced.acceptsResponsesReasoningControl,
|
|
757
781
|
denyLevels: options.denyLevels,
|
|
758
782
|
});
|
|
783
|
+
const relaysAnthropicCacheMarkers =
|
|
784
|
+
api === "anthropic-messages" ||
|
|
785
|
+
(api === "openai-completions" &&
|
|
786
|
+
(policy.compat as Model<"openai-completions">["compat"])?.cacheControlFormat === "anthropic");
|
|
759
787
|
const hasConflictingFamily = reduced.deploymentFamilies.includes("conflicting");
|
|
760
788
|
const unanimousMoonshot =
|
|
761
789
|
reduced.deploymentFamilies.length > 0 && reduced.deploymentFamilies.every((family) => family === "kimi");
|
|
@@ -772,10 +800,7 @@ function mapFromModelInfoGroup(
|
|
|
772
800
|
);
|
|
773
801
|
// One GPT-5.5+ deployment is enough: a tool request routed to it fails with
|
|
774
802
|
// reasoning attached, while dropping reasoning elsewhere only loses effort.
|
|
775
|
-
const chatBackendIds =
|
|
776
|
-
api === "openai-completions"
|
|
777
|
-
? entries.flatMap((entry) => resolveBackendIdentity({ ...entry, model_name: undefined })?.modelId ?? [])
|
|
778
|
-
: [];
|
|
803
|
+
const chatBackendIds = api === "openai-completions" ? identities.flatMap((identity) => identity?.modelId ?? []) : [];
|
|
779
804
|
const modelPolicy: LiteLLMModelPolicy | undefined = chatBackendIds.some(isGpt55OrNewerModel)
|
|
780
805
|
? {
|
|
781
806
|
normalizeStrictToolMessages: false,
|
|
@@ -802,6 +827,18 @@ function mapFromModelInfoGroup(
|
|
|
802
827
|
? { litellmResponsesReasoningControl: true as const }
|
|
803
828
|
: {}),
|
|
804
829
|
...(families.size === 1 && family ? { litellmBackendFamily: family } : {}),
|
|
830
|
+
...(families.size === 1 && family === "claude" && relaysAnthropicCacheMarkers
|
|
831
|
+
? { promptCache: { ...CLAUDE_PROMPT_CACHE } }
|
|
832
|
+
: {}),
|
|
833
|
+
...(api === "openai-responses" &&
|
|
834
|
+
identities.every(
|
|
835
|
+
(identity, index) =>
|
|
836
|
+
identity?.family === "openai" &&
|
|
837
|
+
isGpt56OrNewerModel(identity.modelId) &&
|
|
838
|
+
routesOnlyThrough(entries[index]!, OPENAI_PROMPT_CACHE_ROUTES),
|
|
839
|
+
)
|
|
840
|
+
? { promptCache: { ...OPENAI_PROMPT_CACHE } }
|
|
841
|
+
: {}),
|
|
805
842
|
...(modelPolicy ? { litellmPolicy: modelPolicy } : {}),
|
|
806
843
|
// closeSerializerPolicy derives compat from `api`, a correlation its return type cannot carry.
|
|
807
844
|
} as DiscoveredModel;
|
|
@@ -930,6 +967,7 @@ async function discoverFromHealth(
|
|
|
930
967
|
denyLevels: group.some(({ denyLevels }) => denyLevels),
|
|
931
968
|
allowMessages: false,
|
|
932
969
|
proxyVersion,
|
|
970
|
+
modelOverrides: options.modelOverrides,
|
|
933
971
|
});
|
|
934
972
|
})
|
|
935
973
|
.filter((model): model is DiscoveredModel => model !== undefined);
|
|
@@ -1036,10 +1074,11 @@ function applyWildcardEvidence(
|
|
|
1036
1074
|
publishedWildcardIds: ReadonlySet<string>,
|
|
1037
1075
|
publicCatalog: PublicCatalog | undefined,
|
|
1038
1076
|
proxyVersion: ProxyVersion | undefined,
|
|
1077
|
+
modelOverrides: DiscoveryOptions["modelOverrides"],
|
|
1039
1078
|
): DiscoveredModel | undefined {
|
|
1040
1079
|
const parents = selectedWildcardRows(model.id, wildcardRows, publishedWildcardIds);
|
|
1041
1080
|
if (!parents) return undefined;
|
|
1042
|
-
const selected = mapFromModelInfoGroup(parents, publicCatalog, { proxyVersion });
|
|
1081
|
+
const selected = mapFromModelInfoGroup(parents, publicCatalog, { proxyVersion, modelOverrides });
|
|
1043
1082
|
if (!selected) return undefined;
|
|
1044
1083
|
const policies = [model.litellmPolicy, selected.litellmPolicy].filter((policy) => policy !== undefined);
|
|
1045
1084
|
const combinedPolicy =
|
|
@@ -1061,6 +1100,7 @@ function applyWildcardEvidence(
|
|
|
1061
1100
|
litellmBackendFamily: _family,
|
|
1062
1101
|
litellmResponsesReasoningControl: _control,
|
|
1063
1102
|
litellmPolicy: _policy,
|
|
1103
|
+
promptCache: _promptCache,
|
|
1064
1104
|
thinkingLevelMap: _levels,
|
|
1065
1105
|
...rest
|
|
1066
1106
|
} = model;
|
|
@@ -1071,6 +1111,7 @@ function applyWildcardEvidence(
|
|
|
1071
1111
|
compat: selected.compat,
|
|
1072
1112
|
...(selected.thinkingLevelMap ? { thinkingLevelMap: selected.thinkingLevelMap } : {}),
|
|
1073
1113
|
...(selected.litellmBackendFamily ? { litellmBackendFamily: selected.litellmBackendFamily } : {}),
|
|
1114
|
+
...(selected.promptCache ? { promptCache: selected.promptCache } : {}),
|
|
1074
1115
|
...(combinedPolicy ? { litellmPolicy: combinedPolicy } : {}),
|
|
1075
1116
|
...(selected.litellmResponsesReasoningControl ? { litellmResponsesReasoningControl: true as const } : {}),
|
|
1076
1117
|
} as DiscoveredModel;
|
|
@@ -1216,6 +1257,7 @@ export async function discoverModels(
|
|
|
1216
1257
|
withheldRepairRoutes,
|
|
1217
1258
|
defaultedContextRoutes,
|
|
1218
1259
|
proxyVersion,
|
|
1260
|
+
modelOverrides: options.modelOverrides,
|
|
1219
1261
|
}),
|
|
1220
1262
|
deploymentFamilies: group.map(deploymentFamily),
|
|
1221
1263
|
};
|
|
@@ -1272,14 +1314,26 @@ export async function discoverModels(
|
|
|
1272
1314
|
// The parents were reduced before the version was known; an expansion
|
|
1273
1315
|
// must inherit from parents reduced under the version it is published with.
|
|
1274
1316
|
sources = wildcardRoutes.flatMap(({ route, deploymentFamilies }) => {
|
|
1275
|
-
const model = mapFromModelInfoGroup(groups.get(route)!, publicCatalog, {
|
|
1317
|
+
const model = mapFromModelInfoGroup(groups.get(route)!, publicCatalog, {
|
|
1318
|
+
proxyVersion,
|
|
1319
|
+
modelOverrides: options.modelOverrides,
|
|
1320
|
+
});
|
|
1276
1321
|
return model ? [{ model, deploymentFamilies }] : [];
|
|
1277
1322
|
});
|
|
1278
1323
|
}
|
|
1279
1324
|
const expanded = candidates
|
|
1280
1325
|
.map((entry) => mapFromWildcardExpansion(entry, sources, withheldRepairRoutes))
|
|
1281
1326
|
.filter((model): model is DiscoveredModel => model !== undefined)
|
|
1282
|
-
.map((model) =>
|
|
1327
|
+
.map((model) =>
|
|
1328
|
+
applyWildcardEvidence(
|
|
1329
|
+
model,
|
|
1330
|
+
wildcardRows,
|
|
1331
|
+
publishedWildcardIds,
|
|
1332
|
+
publicCatalog,
|
|
1333
|
+
proxyVersion,
|
|
1334
|
+
options.modelOverrides,
|
|
1335
|
+
),
|
|
1336
|
+
)
|
|
1283
1337
|
.filter((model): model is DiscoveredModel => model !== undefined);
|
|
1284
1338
|
models = [...models, ...expanded];
|
|
1285
1339
|
}
|
|
@@ -1309,10 +1363,15 @@ export async function discoverModels(
|
|
|
1309
1363
|
const imageModels = [...groups.entries()]
|
|
1310
1364
|
.map(([route, group]) => mapFromImageModelInfoGroup(route, group))
|
|
1311
1365
|
.filter((model): model is DiscoveredImageModel => model !== undefined);
|
|
1366
|
+
// Decision routes are a separate operation in Pi; the chat reducer withholds their modes.
|
|
1367
|
+
const classifierModels = [...groups.entries()]
|
|
1368
|
+
.map(([route, group]) => mapFromClassifierModelInfoGroup(route, group))
|
|
1369
|
+
.filter((model): model is DiscoveredClassifierModel => model !== undefined);
|
|
1312
1370
|
return {
|
|
1313
1371
|
source: "model_info",
|
|
1314
1372
|
models,
|
|
1315
1373
|
...(imageModels.length > 0 ? { imageModels } : {}),
|
|
1374
|
+
...(classifierModels.length > 0 ? { classifierModels } : {}),
|
|
1316
1375
|
...(proxyVersion ? { proxyVersion } : {}),
|
|
1317
1376
|
};
|
|
1318
1377
|
}
|
package/src/images.ts
CHANGED
|
@@ -1,9 +1,8 @@
|
|
|
1
1
|
import type {
|
|
2
2
|
AssistantImages,
|
|
3
|
-
ImageApi,
|
|
4
3
|
ImageContent,
|
|
5
4
|
ImageModel,
|
|
6
|
-
|
|
5
|
+
ModelCost,
|
|
7
6
|
ProviderImages,
|
|
8
7
|
Usage,
|
|
9
8
|
} from "@earendil-works/pi-ai";
|
|
@@ -33,7 +32,7 @@ export function isImageGenerationMode(mode: unknown): boolean {
|
|
|
33
32
|
return value !== undefined && IMAGE_GENERATION_MODE_PATTERN.test(value.trim());
|
|
34
33
|
}
|
|
35
34
|
|
|
36
|
-
function perMillion(value: unknown): number | undefined {
|
|
35
|
+
export function perMillion(value: unknown): number | undefined {
|
|
37
36
|
return typeof value === "number" && Number.isFinite(value) && value >= 0 ? value * 1_000_000 : undefined;
|
|
38
37
|
}
|
|
39
38
|
|
|
@@ -119,18 +118,19 @@ function tokenCount(...values: unknown[]): number {
|
|
|
119
118
|
return 0;
|
|
120
119
|
}
|
|
121
120
|
|
|
122
|
-
function parseUsage(
|
|
123
|
-
|
|
124
|
-
|
|
121
|
+
export function parseUsage(
|
|
122
|
+
raw:
|
|
123
|
+
| { input_tokens?: unknown; output_tokens?: unknown; prompt_tokens?: unknown; completion_tokens?: unknown }
|
|
124
|
+
| undefined,
|
|
125
|
+
cost: ModelCost,
|
|
125
126
|
responseCost: number | undefined,
|
|
126
127
|
): Usage | undefined {
|
|
127
|
-
const raw = response.usage;
|
|
128
128
|
if (!raw && responseCost === undefined) return undefined;
|
|
129
129
|
const input = tokenCount(raw?.input_tokens, raw?.prompt_tokens);
|
|
130
130
|
const output = tokenCount(raw?.output_tokens, raw?.completion_tokens);
|
|
131
131
|
const estimated = {
|
|
132
|
-
input: (
|
|
133
|
-
output: (
|
|
132
|
+
input: (cost.input / 1_000_000) * input,
|
|
133
|
+
output: (cost.output / 1_000_000) * output,
|
|
134
134
|
};
|
|
135
135
|
// LiteLLM's own cost is authoritative: it knows per-image prices that a token rate cannot express.
|
|
136
136
|
const total = responseCost ?? estimated.input + estimated.output;
|
|
@@ -148,7 +148,13 @@ function parseUsage(
|
|
|
148
148
|
};
|
|
149
149
|
}
|
|
150
150
|
|
|
151
|
-
|
|
151
|
+
/** LiteLLM's `x-litellm-response-cost` header, when it is a usable amount. */
|
|
152
|
+
export function responseCost(headers: Record<string, string>): number | undefined {
|
|
153
|
+
const cost = Number.parseFloat(headers["x-litellm-response-cost"] ?? "");
|
|
154
|
+
return Number.isFinite(cost) && cost >= 0 ? cost : undefined;
|
|
155
|
+
}
|
|
156
|
+
|
|
157
|
+
export function requestHeaders(
|
|
152
158
|
apiKey: string,
|
|
153
159
|
...sources: (Record<string, string | null | undefined> | undefined)[]
|
|
154
160
|
): Record<string, string> {
|
|
@@ -167,7 +173,7 @@ function requestHeaders(
|
|
|
167
173
|
return headers;
|
|
168
174
|
}
|
|
169
175
|
|
|
170
|
-
function errorDetail(body: string): string {
|
|
176
|
+
export function errorDetail(body: string): string {
|
|
171
177
|
try {
|
|
172
178
|
const parsed = JSON.parse(body) as { error?: { message?: unknown } | unknown };
|
|
173
179
|
const message =
|
|
@@ -181,7 +187,9 @@ function errorDetail(body: string): string {
|
|
|
181
187
|
return body.slice(0, MAX_ERROR_DETAIL);
|
|
182
188
|
}
|
|
183
189
|
|
|
184
|
-
function requestSignal(
|
|
190
|
+
export function requestSignal(
|
|
191
|
+
options: { signal?: AbortSignal; timeoutMs?: number } | undefined,
|
|
192
|
+
): AbortSignal | undefined {
|
|
185
193
|
const signals = [
|
|
186
194
|
options?.signal,
|
|
187
195
|
options?.timeoutMs !== undefined ? AbortSignal.timeout(options.timeoutMs) : undefined,
|
|
@@ -242,8 +250,7 @@ export const generateImages: ProviderImages["generateImages"] = async (model, co
|
|
|
242
250
|
: `LiteLLM image model ${model.id} returned no images`,
|
|
243
251
|
);
|
|
244
252
|
}
|
|
245
|
-
const
|
|
246
|
-
const usage = parseUsage(parsed, model, Number.isFinite(costHeader) && costHeader >= 0 ? costHeader : undefined);
|
|
253
|
+
const usage = parseUsage(parsed.usage, model.cost, responseCost(responseHeaders));
|
|
247
254
|
if (usage) result.usage = usage;
|
|
248
255
|
return result;
|
|
249
256
|
} catch (error) {
|
package/src/index.ts
CHANGED
|
@@ -18,7 +18,8 @@ import type { ExtensionAPI, ExtensionContext, McpServerConfig } from "@earendil-
|
|
|
18
18
|
// A namespace import, so a Pi without `VERSION` reaches the version check instead of failing to link.
|
|
19
19
|
import * as piCodingAgent from "@earendil-works/pi-coding-agent";
|
|
20
20
|
import { getAgentDir, readStoredCredential } from "@earendil-works/pi-coding-agent";
|
|
21
|
-
import { budgetDisplaySetting, setupLiteLLMBudget } from "./budget.js";
|
|
21
|
+
import { budgetDisplaySetting, isBudgetExceeded, setupLiteLLMBudget, setupLiteLLMBudgetErrors } from "./budget.js";
|
|
22
|
+
import type { LiteLLMClassifierModel } from "./classifiers.js";
|
|
22
23
|
import { setupLiteLLMCostTracking } from "./cost.js";
|
|
23
24
|
import {
|
|
24
25
|
discoverModels,
|
|
@@ -34,6 +35,7 @@ import {
|
|
|
34
35
|
createLiteLLMProvider,
|
|
35
36
|
DEFAULT_LITELLM_BASE_URL,
|
|
36
37
|
isPlaceholderHost,
|
|
38
|
+
toNativeClassifierModels,
|
|
37
39
|
toNativeImageModels,
|
|
38
40
|
toNativeModels,
|
|
39
41
|
} from "./provider.js";
|
|
@@ -286,6 +288,37 @@ async function readModelsProviderSettings(): Promise<Record<string, unknown> | u
|
|
|
286
288
|
}
|
|
287
289
|
}
|
|
288
290
|
|
|
291
|
+
// Pi `modelOverrides` limits from the global models.json, keyed by provider id then model id.
|
|
292
|
+
async function readModelLimitOverrides(): Promise<NonNullable<DiscoveryOptions["modelOverrides"]>> {
|
|
293
|
+
// Null prototypes: a `__proto__` provider or model key in the user's file stays a plain key.
|
|
294
|
+
const result: Record<string, Record<string, { contextWindow?: number; maxTokens?: number }>> = Object.create(null);
|
|
295
|
+
try {
|
|
296
|
+
const raw = await readFile(join(getAgentDir(), MODELS_FILENAME), "utf8");
|
|
297
|
+
const parsed = JSON.parse(stripJsonComments(stripBom(raw))) as Record<string, unknown>;
|
|
298
|
+
if (!isPlainObject(parsed.providers)) return result;
|
|
299
|
+
const limit = (value: unknown) =>
|
|
300
|
+
typeof value === "number" && Number.isFinite(value) && value > 0 ? value : undefined;
|
|
301
|
+
for (const [providerId, provider] of Object.entries(parsed.providers)) {
|
|
302
|
+
if (!isPlainObject(provider) || !isPlainObject(provider.modelOverrides)) continue;
|
|
303
|
+
for (const [modelId, override] of Object.entries(provider.modelOverrides)) {
|
|
304
|
+
if (!isPlainObject(override)) continue;
|
|
305
|
+
const contextWindow = limit(override.contextWindow);
|
|
306
|
+
const maxTokens = limit(override.maxTokens);
|
|
307
|
+
if (contextWindow === undefined && maxTokens === undefined) continue;
|
|
308
|
+
const models = result[providerId] ?? Object.create(null);
|
|
309
|
+
result[providerId] = models;
|
|
310
|
+
models[modelId] = {
|
|
311
|
+
...(contextWindow !== undefined ? { contextWindow } : {}),
|
|
312
|
+
...(maxTokens !== undefined ? { maxTokens } : {}),
|
|
313
|
+
};
|
|
314
|
+
}
|
|
315
|
+
}
|
|
316
|
+
} catch {
|
|
317
|
+
// Missing or unparsable models.json: no overrides.
|
|
318
|
+
}
|
|
319
|
+
return result;
|
|
320
|
+
}
|
|
321
|
+
|
|
289
322
|
// A provider block may live in either file; merge per field with models.json winning. Non-provider
|
|
290
323
|
// settings.json keys pass through untouched, which is what keeps `skills`/`mcp`/`budget` working.
|
|
291
324
|
function mergeProviderSettings(
|
|
@@ -690,6 +723,19 @@ function piLiteral(value: string): string {
|
|
|
690
723
|
return escaped.startsWith("!") ? `$${escaped}` : escaped;
|
|
691
724
|
}
|
|
692
725
|
|
|
726
|
+
// The first 64 KiB of a body, as much as the MCP access check reads of a refusal.
|
|
727
|
+
async function readBodyPrefix(response: Response, limit = 64 * 1024): Promise<string> {
|
|
728
|
+
if (!response.body) return "";
|
|
729
|
+
const decoder = new TextDecoder();
|
|
730
|
+
let text = "";
|
|
731
|
+
for await (const chunk of response.body) {
|
|
732
|
+
text += decoder.decode(chunk, { stream: true });
|
|
733
|
+
// Leaving the loop cancels the rest of the stream.
|
|
734
|
+
if (text.length >= limit) break;
|
|
735
|
+
}
|
|
736
|
+
return text;
|
|
737
|
+
}
|
|
738
|
+
|
|
693
739
|
/**
|
|
694
740
|
* LiteLLM's /model/info is the metadata authority, so models.dev is an opt-in escape hatch.
|
|
695
741
|
* Without a cache path and with refresh suppressed, discovery reads neither the models.dev
|
|
@@ -2468,8 +2514,12 @@ export default async function (pi: ExtensionAPI): Promise<void> {
|
|
|
2468
2514
|
// LiteLLM 1.102.0 and later answer `initialize` with 403 when the key has no MCP servers granted,
|
|
2469
2515
|
// and Pi warns at every start about a server that fails to connect. Asking first lets such a
|
|
2470
2516
|
// server be registered disabled: Pi neither connects to it nor warns, and /mcp still lists it, so
|
|
2471
|
-
// the user can enable it once servers are granted.
|
|
2472
|
-
|
|
2517
|
+
// the user can enable it once servers are granted. A used-up budget fails every MCP request until
|
|
2518
|
+
// it resets, so it is refused the same way, with a notice.
|
|
2519
|
+
async function mcpAccessRefused(
|
|
2520
|
+
definition: ProviderDefinition,
|
|
2521
|
+
url: string,
|
|
2522
|
+
): Promise<"no-servers" | "out-of-budget" | undefined> {
|
|
2473
2523
|
try {
|
|
2474
2524
|
const stored = readStoredCredential(definition.name, join(getAgentDir(), "auth.json"));
|
|
2475
2525
|
// The check must send the credential Pi will send. Like seeding, it runs no key helper and
|
|
@@ -2482,11 +2532,11 @@ export default async function (pi: ExtensionAPI): Promise<void> {
|
|
|
2482
2532
|
definition.apiKeyConfig?.startsWith("!") ||
|
|
2483
2533
|
useGcloudToken
|
|
2484
2534
|
) {
|
|
2485
|
-
return
|
|
2535
|
+
return undefined;
|
|
2486
2536
|
}
|
|
2487
2537
|
const auth = await authForCredential(definition, stored, false);
|
|
2488
2538
|
// The credential goes only to its own root.
|
|
2489
|
-
if (`${auth.baseUrl}/mcp` !== url) return
|
|
2539
|
+
if (`${auth.baseUrl}/mcp` !== url) return undefined;
|
|
2490
2540
|
// The credential and headers Pi's client sends, so the proxy answers as it will answer Pi.
|
|
2491
2541
|
const headers = { ...auth.headers, Authorization: `Bearer ${auth.apiKey}` };
|
|
2492
2542
|
const response = await fetch(url, {
|
|
@@ -2505,7 +2555,9 @@ export default async function (pi: ExtensionAPI): Promise<void> {
|
|
|
2505
2555
|
redirect: "manual",
|
|
2506
2556
|
signal: AbortSignal.timeout(getSeedTimeoutMs()),
|
|
2507
2557
|
});
|
|
2508
|
-
|
|
2558
|
+
// Only a refusal is read, and only its start: a granted session can answer with an open event stream.
|
|
2559
|
+
const refusal = response.ok ? undefined : await readBodyPrefix(response).catch(() => "");
|
|
2560
|
+
if (refusal === undefined) await response.body?.cancel().catch(() => undefined);
|
|
2509
2561
|
// Pi opens its own session; end the one this check opened.
|
|
2510
2562
|
const session = response.headers.get("mcp-session-id");
|
|
2511
2563
|
if (session) {
|
|
@@ -2518,9 +2570,11 @@ export default async function (pi: ExtensionAPI): Promise<void> {
|
|
|
2518
2570
|
.then((closed) => closed.body?.cancel())
|
|
2519
2571
|
.catch(() => undefined);
|
|
2520
2572
|
}
|
|
2521
|
-
|
|
2573
|
+
// The budget refusal's status is configurable on the proxy, so its text is checked first.
|
|
2574
|
+
if (refusal !== undefined && isBudgetExceeded(refusal)) return "out-of-budget";
|
|
2575
|
+
return response.status === 403 ? "no-servers" : undefined;
|
|
2522
2576
|
} catch {
|
|
2523
|
-
return
|
|
2577
|
+
return undefined;
|
|
2524
2578
|
}
|
|
2525
2579
|
}
|
|
2526
2580
|
|
|
@@ -2541,6 +2595,12 @@ export default async function (pi: ExtensionAPI): Promise<void> {
|
|
|
2541
2595
|
try {
|
|
2542
2596
|
pi.registerMcpServer(definition.name, refused ? { ...config, enabled: false } : config);
|
|
2543
2597
|
attempt.registered = true;
|
|
2598
|
+
if (refused === "out-of-budget") {
|
|
2599
|
+
notifyMcp(
|
|
2600
|
+
`LiteLLM MCP (${JSON.stringify(definition.name)}): out of budget, so its MCP server is off for this ` +
|
|
2601
|
+
"session. Turn it on in /mcp once the budget resets.",
|
|
2602
|
+
);
|
|
2603
|
+
}
|
|
2544
2604
|
} catch (error) {
|
|
2545
2605
|
// The claim stays, so the refusal is reported once rather than on every turn.
|
|
2546
2606
|
notifyMcp(
|
|
@@ -2637,6 +2697,7 @@ export default async function (pi: ExtensionAPI): Promise<void> {
|
|
|
2637
2697
|
headers: auth.headers,
|
|
2638
2698
|
allowInsecureHttp: auth.allowInsecureHttp,
|
|
2639
2699
|
...modelsDevDiscoveryOptions(),
|
|
2700
|
+
modelOverrides: await readModelLimitOverrides(),
|
|
2640
2701
|
silent: !isVerboseDiscovery(),
|
|
2641
2702
|
onProgress: isVerboseDiscovery() ? (message) => process.stderr.write(`LiteLLM: ${message}\n`) : undefined,
|
|
2642
2703
|
});
|
|
@@ -2653,7 +2714,9 @@ export default async function (pi: ExtensionAPI): Promise<void> {
|
|
|
2653
2714
|
// critical path of every start -- is replaced by reading that catalog while it is fresh (models-store.ts). Nothing is
|
|
2654
2715
|
// written: Pi owns the file. The stored models are handed over rather than an empty list, so that a store Pi keeps
|
|
2655
2716
|
// somewhere else cannot leave the provider with nothing.
|
|
2656
|
-
async function seedModels(
|
|
2717
|
+
async function seedModels(
|
|
2718
|
+
definition: ProviderDefinition,
|
|
2719
|
+
): Promise<(Model<LiteLLMApi> | LiteLLMImageModel | LiteLLMClassifierModel)[]> {
|
|
2657
2720
|
const disabledReason = discoveryDisabledReason() ?? (isHostOffline() ? "PI_OFFLINE" : null);
|
|
2658
2721
|
if (disabledReason) {
|
|
2659
2722
|
if (isVerboseDiscovery()) {
|
|
@@ -2685,6 +2748,12 @@ export default async function (pi: ExtensionAPI): Promise<void> {
|
|
|
2685
2748
|
return [
|
|
2686
2749
|
...toNativeModels(definition.name, result.baseUrl, result.models, definition.allowInsecureHttp),
|
|
2687
2750
|
...toNativeImageModels(definition.name, result.baseUrl, result.imageModels ?? [], definition.allowInsecureHttp),
|
|
2751
|
+
...toNativeClassifierModels(
|
|
2752
|
+
definition.name,
|
|
2753
|
+
result.baseUrl,
|
|
2754
|
+
result.classifierModels ?? [],
|
|
2755
|
+
definition.allowInsecureHttp,
|
|
2756
|
+
),
|
|
2688
2757
|
];
|
|
2689
2758
|
} catch (error) {
|
|
2690
2759
|
// No credentials yet is the normal unconfigured case; anything else is worth one line.
|
|
@@ -2836,10 +2905,11 @@ export default async function (pi: ExtensionAPI): Promise<void> {
|
|
|
2836
2905
|
|
|
2837
2906
|
setupLiteLLMCostTracking(pi, [...providerNames]);
|
|
2838
2907
|
|
|
2908
|
+
let budgetLevels: ReturnType<typeof setupLiteLLMBudget> | undefined;
|
|
2839
2909
|
if (budgetEnabled) {
|
|
2840
2910
|
const { display, warning } = budgetDisplaySetting(settings?.budget);
|
|
2841
2911
|
const definitionNamed = (name: string) => definitions.find((definition) => definition.name === name)!;
|
|
2842
|
-
setupLiteLLMBudget(pi, {
|
|
2912
|
+
budgetLevels = setupLiteLLMBudget(pi, {
|
|
2843
2913
|
providers: definitions.map(({ name, displayName }) => ({ name, displayName })),
|
|
2844
2914
|
display,
|
|
2845
2915
|
displayWarning: warning,
|
|
@@ -2850,6 +2920,7 @@ export default async function (pi: ExtensionAPI): Promise<void> {
|
|
|
2850
2920
|
missingCredentials: (name) => missingCredentials(definitionNamed(name)),
|
|
2851
2921
|
});
|
|
2852
2922
|
}
|
|
2923
|
+
setupLiteLLMBudgetErrors(pi, providerNames, budgetLevels);
|
|
2853
2924
|
|
|
2854
2925
|
if (skillsEnabled) {
|
|
2855
2926
|
for (const tool of createSkillToolDefinitions(resolveDefaultRuntimeAuth)) {
|
package/src/model-groups.ts
CHANGED
|
@@ -54,6 +54,8 @@ export interface ReasoningPolicy {
|
|
|
54
54
|
|
|
55
55
|
export interface CatalogResolution {
|
|
56
56
|
provider?: string;
|
|
57
|
+
// The Pi provider whose catalog entry resolved catalogModelId; set only then, so the pair is Pi's providerId/model.id.
|
|
58
|
+
piProvider?: string;
|
|
57
59
|
catalogModelId?: string;
|
|
58
60
|
semanticFamily?: FamilyEvidence;
|
|
59
61
|
semanticModel?: SemanticModel;
|
|
@@ -68,6 +70,8 @@ export interface CatalogResolution {
|
|
|
68
70
|
vision?: boolean;
|
|
69
71
|
contextWindow?: number;
|
|
70
72
|
maxTokens?: number;
|
|
73
|
+
// User `modelOverrides` for the identified catalog model; read only under unanimous catalog authority.
|
|
74
|
+
limitOverrides?: { contextWindow?: number; maxTokens?: number };
|
|
71
75
|
cost?: Partial<DiscoveredModel["cost"]>;
|
|
72
76
|
}
|
|
73
77
|
|
|
@@ -657,6 +661,8 @@ function min(values: readonly number[]): number {
|
|
|
657
661
|
}
|
|
658
662
|
|
|
659
663
|
function conservativeLimit(explicit: number | undefined, catalog: number | undefined): number | undefined {
|
|
664
|
+
// LITELLM_TRUST_PROXY_LIMITS=1 lets the router's own figure win over the catalog.
|
|
665
|
+
if (process.env.LITELLM_TRUST_PROXY_LIMITS === "1") return explicitLimit(explicit) ?? explicitLimit(catalog);
|
|
660
666
|
const valid = [explicitLimit(explicit), explicitLimit(catalog)].filter(
|
|
661
667
|
(value): value is number => value !== undefined,
|
|
662
668
|
);
|
|
@@ -819,11 +825,15 @@ export function reduceModelGroup(
|
|
|
819
825
|
const visionEvidence = deployments.map(
|
|
820
826
|
(entry, index) => wireBoolean(entry.model_info?.supports_vision) ?? catalogAuthority[index]?.vision,
|
|
821
827
|
);
|
|
822
|
-
const contextWindowEvidence = deployments.map(
|
|
823
|
-
|
|
828
|
+
const contextWindowEvidence = deployments.map(
|
|
829
|
+
(entry, index) =>
|
|
830
|
+
explicitLimit(catalogAuthority[index]?.limitOverrides?.contextWindow) ??
|
|
831
|
+
conservativeLimit(entry.model_info?.max_input_tokens, catalogAuthority[index]?.contextWindow),
|
|
824
832
|
);
|
|
825
|
-
const maxTokensEvidence = deployments.map(
|
|
826
|
-
|
|
833
|
+
const maxTokensEvidence = deployments.map(
|
|
834
|
+
(entry, index) =>
|
|
835
|
+
explicitLimit(catalogAuthority[index]?.limitOverrides?.maxTokens) ??
|
|
836
|
+
conservativeLimit(entry.model_info?.max_output_tokens, catalogAuthority[index]?.maxTokens),
|
|
827
837
|
);
|
|
828
838
|
const reasoning = reasoningEvidence.every((value) => value ?? false);
|
|
829
839
|
const explicitlyUnsupported = deployments.some(
|
package/src/models-store.ts
CHANGED
|
@@ -39,10 +39,11 @@
|
|
|
39
39
|
import { readFile } from "node:fs/promises";
|
|
40
40
|
import { join } from "node:path";
|
|
41
41
|
import { LITELLM_DISCOVERY_VERSION } from "./backend-identity.js";
|
|
42
|
+
import { LITELLM_DECISIONS_API, type LiteLLMClassifierModel } from "./classifiers.js";
|
|
42
43
|
import { enrichCachedModel } from "./discover.js";
|
|
43
44
|
import { LITELLM_IMAGES_API, type LiteLLMImageModel } from "./images.js";
|
|
44
45
|
import { isLiteLLMApi, resolveModelBaseUrl } from "./protocols.js";
|
|
45
|
-
import {
|
|
46
|
+
import { proxyV1BaseUrl } from "./provider.js";
|
|
46
47
|
import type { LiteLLMModel } from "./types.js";
|
|
47
48
|
|
|
48
49
|
/**
|
|
@@ -59,17 +60,20 @@ function isRecord(value: unknown): value is Record<string, unknown> {
|
|
|
59
60
|
/** Pi's own remote catalogs skip the network for 4 hours after `checkedAt` (`REMOTE_CATALOG_REFRESH_INTERVAL_MS`). */
|
|
60
61
|
const STORE_FRESH_MS = 4 * 60 * 60 * 1000;
|
|
61
62
|
|
|
62
|
-
function
|
|
63
|
-
return
|
|
63
|
+
function isOperationModel(value: Record<string, unknown>): boolean {
|
|
64
|
+
return (
|
|
65
|
+
(value.type === "image" && value.api === LITELLM_IMAGES_API) ||
|
|
66
|
+
(value.type === "classifier" && value.api === LITELLM_DECISIONS_API)
|
|
67
|
+
);
|
|
64
68
|
}
|
|
65
69
|
|
|
66
70
|
function isUsableModel(value: unknown, provider: string, baseUrl: string, allowInsecureHttp?: boolean): boolean {
|
|
67
71
|
if (!isRecord(value) || value.provider !== provider) return false;
|
|
68
72
|
if (value.litellmDiscoveryVersion !== LITELLM_DISCOVERY_VERSION) return false;
|
|
69
73
|
try {
|
|
70
|
-
// A stored baseUrl is per protocol -- `/v1` for the OpenAI APIs and
|
|
74
|
+
// A stored baseUrl is per protocol -- `/v1` for the OpenAI APIs, images and classifiers, the bare root for Messages -- so the root
|
|
71
75
|
// goes through the same resolution rather than being compared to it.
|
|
72
|
-
if (
|
|
76
|
+
if (isOperationModel(value)) return value.baseUrl === proxyV1BaseUrl(baseUrl, allowInsecureHttp);
|
|
73
77
|
// `api` indexes LITELLM_PROTOCOLS in resolveModelBaseUrl, so an unknown one is a TypeError, not a bad model.
|
|
74
78
|
return isLiteLLMApi(value.api) && value.baseUrl === resolveModelBaseUrl(baseUrl, value.api, allowInsecureHttp);
|
|
75
79
|
} catch {
|
|
@@ -96,7 +100,7 @@ export async function readStoredModels(options: {
|
|
|
96
100
|
provider: string;
|
|
97
101
|
baseUrl: string;
|
|
98
102
|
allowInsecureHttp?: boolean;
|
|
99
|
-
}): Promise<(LiteLLMModel | LiteLLMImageModel)[] | undefined> {
|
|
103
|
+
}): Promise<(LiteLLMModel | LiteLLMImageModel | LiteLLMClassifierModel)[] | undefined> {
|
|
100
104
|
const entry = (await readStore(options.agentDir))?.[options.provider];
|
|
101
105
|
if (!isRecord(entry) || !Array.isArray(entry.models) || entry.models.length === 0) return undefined;
|
|
102
106
|
if (typeof entry.checkedAt !== "number" || Date.now() - entry.checkedAt >= STORE_FRESH_MS) return undefined;
|
|
@@ -106,10 +110,11 @@ export async function readStoredModels(options: {
|
|
|
106
110
|
if (!usable) return undefined;
|
|
107
111
|
try {
|
|
108
112
|
// Pi's restore path (provider.ts) runs the same call; when it throws, nothing is restored and the seed is the catalog.
|
|
109
|
-
// Image models carry no request policy, so that path restores them as stored.
|
|
110
|
-
return entry.models.map((model) => (
|
|
113
|
+
// Image and classifier models carry no request policy, so that path restores them as stored.
|
|
114
|
+
return entry.models.map((model) => (isOperationModel(model) ? model : enrichCachedModel(model))) as (
|
|
111
115
|
| LiteLLMModel
|
|
112
116
|
| LiteLLMImageModel
|
|
117
|
+
| LiteLLMClassifierModel
|
|
113
118
|
)[];
|
|
114
119
|
} catch {
|
|
115
120
|
return undefined;
|
package/src/provider.ts
CHANGED
|
@@ -3,6 +3,10 @@ import {
|
|
|
3
3
|
type Api,
|
|
4
4
|
type ApiStreamOptions,
|
|
5
5
|
type AssistantImages,
|
|
6
|
+
type ClassifierContext,
|
|
7
|
+
type ClassifierModel,
|
|
8
|
+
type ClassifierOptions,
|
|
9
|
+
type ClassifierResult,
|
|
6
10
|
type Credential,
|
|
7
11
|
createProvider,
|
|
8
12
|
getCurrentTools,
|
|
@@ -17,6 +21,12 @@ import {
|
|
|
17
21
|
type TranscriptContext,
|
|
18
22
|
} from "@earendil-works/pi-ai";
|
|
19
23
|
import { LITELLM_DISCOVERY_VERSION } from "./backend-identity.js";
|
|
24
|
+
import {
|
|
25
|
+
createLiteLLMClassifierApis,
|
|
26
|
+
type DiscoveredClassifierModel,
|
|
27
|
+
LITELLM_DECISIONS_API,
|
|
28
|
+
type LiteLLMClassifierModel,
|
|
29
|
+
} from "./classifiers.js";
|
|
20
30
|
import { enrichCachedModel, normalizeBaseUrl, restoreCachedModelPolicy } from "./discover.js";
|
|
21
31
|
import {
|
|
22
32
|
createLiteLLMImageApis,
|
|
@@ -34,7 +44,7 @@ export type LiteLLMProviderOptions = {
|
|
|
34
44
|
auth: ProviderAuth;
|
|
35
45
|
/** Catalog discovered during activation. Pi's startup refresh never allows network access, so
|
|
36
46
|
* without a seed the provider stays empty until the user opens /model. */
|
|
37
|
-
models?: readonly (LiteLLMModel | LiteLLMImageModel)[];
|
|
47
|
+
models?: readonly (LiteLLMModel | LiteLLMImageModel | LiteLLMClassifierModel)[];
|
|
38
48
|
allowInsecureHttp?: boolean;
|
|
39
49
|
resolveCredentialRoot: (credential?: Credential, requestBaseUrl?: string, apiKey?: string) => string | undefined;
|
|
40
50
|
discover(credential: Credential, signal?: AbortSignal): Promise<DiscoveryResult & { baseUrl?: string }>;
|
|
@@ -49,17 +59,31 @@ function isLiteLLMImageModel(model: AnyModel): model is LiteLLMImageModel {
|
|
|
49
59
|
return model.type === "image" && model.api === LITELLM_IMAGES_API;
|
|
50
60
|
}
|
|
51
61
|
|
|
52
|
-
|
|
62
|
+
function isLiteLLMClassifierModel(model: AnyModel): model is LiteLLMClassifierModel {
|
|
63
|
+
return model.type === "classifier" && model.api === LITELLM_DECISIONS_API;
|
|
64
|
+
}
|
|
65
|
+
|
|
66
|
+
/** Where image and classifier requests go: `<root>/v1`. */
|
|
67
|
+
export function proxyV1BaseUrl(root: string, allowInsecureHttp = false): string {
|
|
53
68
|
return `${normalizeBaseUrl(root, allowInsecureHttp)}/v1`;
|
|
54
69
|
}
|
|
55
70
|
|
|
71
|
+
export function toNativeClassifierModels(
|
|
72
|
+
provider: string,
|
|
73
|
+
baseUrl: string,
|
|
74
|
+
models: readonly DiscoveredClassifierModel[],
|
|
75
|
+
allowInsecureHttp = false,
|
|
76
|
+
): LiteLLMClassifierModel[] {
|
|
77
|
+
return models.map((model) => ({ ...model, provider, baseUrl: proxyV1BaseUrl(baseUrl, allowInsecureHttp) }));
|
|
78
|
+
}
|
|
79
|
+
|
|
56
80
|
export function toNativeImageModels(
|
|
57
81
|
provider: string,
|
|
58
82
|
baseUrl: string,
|
|
59
83
|
models: readonly DiscoveredImageModel[],
|
|
60
84
|
allowInsecureHttp = false,
|
|
61
85
|
): LiteLLMImageModel[] {
|
|
62
|
-
return models.map((model) => ({ ...model, provider, baseUrl:
|
|
86
|
+
return models.map((model) => ({ ...model, provider, baseUrl: proxyV1BaseUrl(baseUrl, allowInsecureHttp) }));
|
|
63
87
|
}
|
|
64
88
|
|
|
65
89
|
export function toNativeModels(
|
|
@@ -145,23 +169,24 @@ function modelRootError(
|
|
|
145
169
|
}
|
|
146
170
|
}
|
|
147
171
|
|
|
148
|
-
function
|
|
149
|
-
model:
|
|
172
|
+
function operationRootError(
|
|
173
|
+
model: { baseUrl: string },
|
|
174
|
+
subject: string,
|
|
150
175
|
active: { root: string; canonical: string; hostname: string },
|
|
151
176
|
allowInsecureHttp = false,
|
|
152
177
|
): Error | undefined {
|
|
153
178
|
let stored: { root: string; canonical: string; hostname: string };
|
|
154
179
|
try {
|
|
155
|
-
stored = proxyRoot(model.baseUrl,
|
|
180
|
+
stored = proxyRoot(model.baseUrl, subject, allowInsecureHttp);
|
|
156
181
|
} catch (error) {
|
|
157
|
-
return error instanceof Error ? error : refreshRequired(
|
|
182
|
+
return error instanceof Error ? error : refreshRequired(`${subject} has an invalid LiteLLM model URL`);
|
|
158
183
|
}
|
|
159
184
|
if (isPlaceholderHost(stored.hostname)) {
|
|
160
|
-
return refreshRequired(
|
|
185
|
+
return refreshRequired(`${subject} uses a placeholder LiteLLM model host`);
|
|
161
186
|
}
|
|
162
187
|
if (stored.canonical !== active.canonical) {
|
|
163
188
|
return refreshRequired(
|
|
164
|
-
|
|
189
|
+
`${subject} has stale LiteLLM model root ${stored.root}; active credentials use ${active.root}`,
|
|
165
190
|
);
|
|
166
191
|
}
|
|
167
192
|
}
|
|
@@ -244,6 +269,7 @@ export function createLiteLLMProvider(options: LiteLLMProviderOptions): Provider
|
|
|
244
269
|
return [
|
|
245
270
|
...toNativeModels(options.id, root, result.models, options.allowInsecureHttp),
|
|
246
271
|
...toNativeImageModels(options.id, root, result.imageModels ?? [], options.allowInsecureHttp),
|
|
272
|
+
...toNativeClassifierModels(options.id, root, result.classifierModels ?? [], options.allowInsecureHttp),
|
|
247
273
|
];
|
|
248
274
|
},
|
|
249
275
|
filterModels(models, credential) {
|
|
@@ -257,17 +283,26 @@ export function createLiteLLMProvider(options: LiteLLMProviderOptions): Provider
|
|
|
257
283
|
// Image models follow the same rule as chat: a cached root that no longer matches the
|
|
258
284
|
// active credential is hidden until a network refresh proves the new proxy's catalog.
|
|
259
285
|
const images = models.filter(isLiteLLMImageModel).flatMap((model) => {
|
|
260
|
-
const error =
|
|
286
|
+
const error = operationRootError(model, "Cached image model", active, options.allowInsecureHttp);
|
|
261
287
|
if (error) {
|
|
262
288
|
reportUnavailable(error.message);
|
|
263
289
|
return [];
|
|
264
290
|
}
|
|
265
|
-
return [{ ...model, baseUrl:
|
|
291
|
+
return [{ ...model, baseUrl: proxyV1BaseUrl(active.root, options.allowInsecureHttp) }];
|
|
266
292
|
});
|
|
267
|
-
|
|
293
|
+
const classifiers = models.filter(isLiteLLMClassifierModel).flatMap((model) => {
|
|
294
|
+
const error = operationRootError(model, "Cached classifier model", active, options.allowInsecureHttp);
|
|
295
|
+
if (error) {
|
|
296
|
+
reportUnavailable(error.message);
|
|
297
|
+
return [];
|
|
298
|
+
}
|
|
299
|
+
return [{ ...model, baseUrl: proxyV1BaseUrl(active.root, options.allowInsecureHttp) }];
|
|
300
|
+
});
|
|
301
|
+
return [...chat, ...images, ...classifiers];
|
|
268
302
|
},
|
|
269
303
|
api: createLiteLLMProtocolApis(),
|
|
270
304
|
images: createLiteLLMImageApis(),
|
|
305
|
+
classifiers: createLiteLLMClassifierApis(),
|
|
271
306
|
});
|
|
272
307
|
const refreshModels = provider.refreshModels;
|
|
273
308
|
const guardedProvider: Provider<LiteLLMApi> = {
|
|
@@ -307,10 +342,10 @@ export function createLiteLLMProvider(options: LiteLLMProviderOptions): Provider
|
|
|
307
342
|
);
|
|
308
343
|
if (!root) throw refreshRequired("Active credentials do not identify a LiteLLM model host");
|
|
309
344
|
const active = activeCredentialRoot(root, options.allowInsecureHttp);
|
|
310
|
-
const error =
|
|
345
|
+
const error = operationRootError(model, "Cached image model", active, options.allowInsecureHttp);
|
|
311
346
|
if (error) throw error;
|
|
312
347
|
return await provider.generateImages!(
|
|
313
|
-
{ ...model, provider: options.id, baseUrl:
|
|
348
|
+
{ ...model, provider: options.id, baseUrl: proxyV1BaseUrl(active.root, options.allowInsecureHttp) },
|
|
314
349
|
context,
|
|
315
350
|
requestOptions,
|
|
316
351
|
);
|
|
@@ -328,23 +363,67 @@ export function createLiteLLMProvider(options: LiteLLMProviderOptions): Provider
|
|
|
328
363
|
return failed;
|
|
329
364
|
}
|
|
330
365
|
},
|
|
366
|
+
classify: async (
|
|
367
|
+
model: ClassifierModel<string>,
|
|
368
|
+
context: ClassifierContext,
|
|
369
|
+
requestOptions?: ClassifierOptions,
|
|
370
|
+
) => {
|
|
371
|
+
try {
|
|
372
|
+
const root = options.resolveCredentialRoot(
|
|
373
|
+
undefined,
|
|
374
|
+
requestOptions?.env?.LITELLM_BASE_URL,
|
|
375
|
+
requestOptions?.apiKey,
|
|
376
|
+
);
|
|
377
|
+
if (!root) throw refreshRequired("Active credentials do not identify a LiteLLM model host");
|
|
378
|
+
const active = activeCredentialRoot(root, options.allowInsecureHttp);
|
|
379
|
+
const error = operationRootError(model, "Cached classifier model", active, options.allowInsecureHttp);
|
|
380
|
+
if (error) throw error;
|
|
381
|
+
return await provider.classify!(
|
|
382
|
+
{ ...model, provider: options.id, baseUrl: proxyV1BaseUrl(active.root, options.allowInsecureHttp) },
|
|
383
|
+
context,
|
|
384
|
+
requestOptions,
|
|
385
|
+
);
|
|
386
|
+
} catch (error) {
|
|
387
|
+
// `Provider.classify` never rejects.
|
|
388
|
+
const failed: ClassifierResult = {
|
|
389
|
+
api: model.api,
|
|
390
|
+
provider: model.provider,
|
|
391
|
+
model: model.id,
|
|
392
|
+
answers: {},
|
|
393
|
+
stopReason: requestOptions?.signal?.aborted ? "aborted" : "error",
|
|
394
|
+
errorMessage: error instanceof Error ? error.message : String(error),
|
|
395
|
+
timestamp: Date.now(),
|
|
396
|
+
};
|
|
397
|
+
return failed;
|
|
398
|
+
}
|
|
399
|
+
},
|
|
331
400
|
};
|
|
332
401
|
if (!refreshModels) return guardedProvider;
|
|
333
|
-
|
|
402
|
+
// A route can be published as both a chat model and a classifier (GPT-6 Luna lists `/v1/decisions`
|
|
403
|
+
// next to its chat endpoints), so the type keeps those seeds apart.
|
|
404
|
+
const seedKey = (model: { type?: string; id: string; baseUrl: string }) =>
|
|
405
|
+
`${model.type ?? "chat"}\0${model.id}\0${model.baseUrl}`;
|
|
334
406
|
const seeded = new Map((options.models ?? []).map((model) => [seedKey(model), model]));
|
|
335
407
|
return {
|
|
336
408
|
...guardedProvider,
|
|
337
409
|
refreshModels: async (context) => {
|
|
338
410
|
// Pi 0.99 persists models of every type; the policy helpers below read chat-only fields, so
|
|
339
|
-
// chat and
|
|
411
|
+
// chat, image and classifier entries are restored separately. Other types are not this provider's.
|
|
340
412
|
const storedModels = (context.stored?.models ?? []).filter(isChatModel);
|
|
341
|
-
|
|
342
|
-
|
|
343
|
-
|
|
344
|
-
|
|
345
|
-
|
|
346
|
-
|
|
347
|
-
|
|
413
|
+
// Image and classifier entries carry no request policy to restore; a stale one is only kept
|
|
414
|
+
// when startup discovery proved the same route, otherwise the network refresh replaces it.
|
|
415
|
+
const restoreOperationModels = <T extends AnyModel>(is: (model: AnyModel) => model is T): T[] =>
|
|
416
|
+
(context.stored?.models ?? []).filter(is).flatMap((model) => {
|
|
417
|
+
if ((model as { litellmDiscoveryVersion?: number }).litellmDiscoveryVersion === LITELLM_DISCOVERY_VERSION) {
|
|
418
|
+
return [model];
|
|
419
|
+
}
|
|
420
|
+
const fresh = seeded.get(seedKey(model));
|
|
421
|
+
return fresh && is(fresh) ? [fresh] : [];
|
|
422
|
+
});
|
|
423
|
+
const storedOperationModels = [
|
|
424
|
+
...restoreOperationModels(isLiteLLMImageModel),
|
|
425
|
+
...restoreOperationModels(isLiteLLMClassifierModel),
|
|
426
|
+
];
|
|
348
427
|
let legacyCount = 0;
|
|
349
428
|
const models = storedModels.map((model) => {
|
|
350
429
|
if ((model as LiteLLMModel).litellmDiscoveryVersion !== LITELLM_DISCOVERY_VERSION) {
|
|
@@ -362,7 +441,7 @@ export function createLiteLLMProvider(options: LiteLLMProviderOptions): Provider
|
|
|
362
441
|
...context,
|
|
363
442
|
// The current pi-ai provider has no freshness gate, but force is the documented refresh contract field.
|
|
364
443
|
force: context.force || (context.allowNetwork && legacyCount > 0),
|
|
365
|
-
stored: context.stored && { ...context.stored, models: [...models, ...
|
|
444
|
+
stored: context.stored && { ...context.stored, models: [...models, ...storedOperationModels] },
|
|
366
445
|
});
|
|
367
446
|
} catch (error) {
|
|
368
447
|
if (context.allowNetwork && legacyCount > 0) {
|
package/src/types.ts
CHANGED
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
import type { Model } from "@earendil-works/pi-ai";
|
|
2
2
|
import type { BackendFamily, LITELLM_DISCOVERY_VERSION } from "./backend-identity.js";
|
|
3
|
+
import type { DiscoveredClassifierModel } from "./classifiers.js";
|
|
3
4
|
import type { DiscoveredImageModel } from "./images.js";
|
|
4
5
|
import type { ProxyVersion } from "./proxy-version.js";
|
|
5
6
|
|
|
@@ -67,6 +68,11 @@ export interface DiscoveryResult {
|
|
|
67
68
|
models: DiscoveredModel[];
|
|
68
69
|
/** `mode: "image_generation"` routes from `/model/info`; absent when there are none. */
|
|
69
70
|
imageModels?: DiscoveredImageModel[];
|
|
71
|
+
/**
|
|
72
|
+
* Decision routes from `/model/info` (`mode: "evaluation"`/`"decisions"`, or a decision endpoint in
|
|
73
|
+
* `supported_endpoints`); absent when there are none.
|
|
74
|
+
*/
|
|
75
|
+
classifierModels?: DiscoveredClassifierModel[];
|
|
70
76
|
source: DiscoverySource;
|
|
71
77
|
// Present only when a `/model/info` discovery had to read it and the proxy
|
|
72
78
|
// answered. Absent means unknown, not old: discovery asks only when it must.
|
|
@@ -82,6 +88,9 @@ export interface DiscoveryOptions {
|
|
|
82
88
|
// path, models.dev is not consulted at all.
|
|
83
89
|
modelsDev?: boolean;
|
|
84
90
|
modelsDevCachePath?: string;
|
|
91
|
+
// Pi `models.json` `providers.<providerId>.modelOverrides.<modelId>` limits. Pi applies them to that
|
|
92
|
+
// provider's own models only, so discovery applies them to routes backed by the same catalog model.
|
|
93
|
+
modelOverrides?: Readonly<Record<string, Readonly<Record<string, { contextWindow?: number; maxTokens?: number }>>>>;
|
|
85
94
|
}
|
|
86
95
|
|
|
87
96
|
export interface ModelInfoEntry {
|