pi-provider-litellm 4.4.1 → 4.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +48 -2
- package/package.json +2 -2
- package/src/backend-identity.ts +1 -1
- package/src/classifiers.ts +174 -0
- package/src/discover.ts +45 -4
- package/src/images.ts +21 -14
- package/src/index.ts +43 -1
- package/src/model-groups.ts +14 -4
- package/src/models-store.ts +13 -8
- package/src/provider.ts +103 -24
- package/src/types.ts +9 -0
package/README.md
CHANGED
|
@@ -375,6 +375,38 @@ and groups that mix image and other modes, or wildcard routes, are withheld. The
|
|
|
375
375
|
need `/v1/images/edits`, which this provider does not call, and backends that answer with image URLs instead of
|
|
376
376
|
base64 data are reported as errors rather than fetched.
|
|
377
377
|
|
|
378
|
+
## Classification
|
|
379
|
+
|
|
380
|
+
Decision routes are published as Pi classifier models (for example `litellm/decision-1`, backed by
|
|
381
|
+
`azure_ai/Microsoft-Decision-1`, or a `typesafe/jev-latest` route). A route qualifies when every deployment behind it
|
|
382
|
+
reports `mode: "evaluation"` or `"decisions"` in `/model/info`, or lists `/v1/systemone` or `/v1/decisions` in
|
|
383
|
+
`supported_endpoints`; a declared endpoint list without them withholds the route. Like Pi's other classifiers they do
|
|
384
|
+
not appear in `/model`; scripts reach them through Pi's `codemode` tool:
|
|
385
|
+
|
|
386
|
+
```js
|
|
387
|
+
const judge = await models.getModelOfType("classifier", "litellm", "decision-1");
|
|
388
|
+
const result = await models.classify(judge, {
|
|
389
|
+
state: { text: "Please send me the report by Friday." },
|
|
390
|
+
questions: {
|
|
391
|
+
polite: {
|
|
392
|
+
type: "bool",
|
|
393
|
+
instructions: "Is the message polite?",
|
|
394
|
+
criteria: { true: "The message is polite.", false: "The message is rude." },
|
|
395
|
+
},
|
|
396
|
+
},
|
|
397
|
+
});
|
|
398
|
+
if (result.stopReason !== "stop") return result.errorMessage;
|
|
399
|
+
return result.answers.polite.probability;
|
|
400
|
+
```
|
|
401
|
+
|
|
402
|
+
Question types are `bool`, `choice` and `score`. Requests go to the proxy's `/v1/systemone` (LiteLLM v1.104.2 and
|
|
403
|
+
later) with the provider's own credential and headers, and are budgeted and logged by the proxy like chat requests. A
|
|
404
|
+
`bool` question is sent as System One's `noul`. Usage cost comes from LiteLLM's `x-litellm-response-cost` header when
|
|
405
|
+
present; otherwise it is estimated from `input_cost_per_token` and `output_cost_per_token`. Only routes found through
|
|
406
|
+
`/model/info` qualify, and groups that mix decision and other deployments, or wildcard routes, are withheld. The state
|
|
407
|
+
is text only: images in the context return an error without a request. A chat route that also lists `/v1/decisions`,
|
|
408
|
+
such as OpenAI's GPT-6 Luna, is published as both a chat model and a classifier.
|
|
409
|
+
|
|
378
410
|
## Optional environment variables
|
|
379
411
|
|
|
380
412
|
| Variable | Default | Effect |
|
|
@@ -393,6 +425,7 @@ base64 data are reported as errors rather than fetched.
|
|
|
393
425
|
| `LITELLM_CLI_JWT_EXPIRATION_HOURS` | `24` | CLI SSO token lifetime fallback for older proxies whose poll response omits `expires_in`; mirror a non-default proxy setting locally |
|
|
394
426
|
| `LITELLM_VERBOSE_DISCOVERY` | unset | If `1`, enable progress messages during model discovery (login, refresh, startup), including startup skip reasons and defaulted `/model/info` context windows. Progress messages are off by default |
|
|
395
427
|
| `LITELLM_DEFAULT_CONTEXT_WINDOW` | `128000` | Context window assumed when neither LiteLLM's `/model/info` nor the catalog reports `max_input_tokens`. Set a positive integer for proxies whose custom aliases carry no metadata; an unset or unusable value keeps 128K |
|
|
428
|
+
| `LITELLM_TRUST_PROXY_LIMITS` | unset | If `1`, a deployment's explicit `model_info.max_input_tokens` / `max_output_tokens` win over Pi's catalog instead of taking the smaller of the two; the catalog still fills missing values and the group minimum still applies. A `models.json` `modelOverrides` entry still wins first |
|
|
396
429
|
| `LITELLM_MODELS_DEV` | unset | If `1`, enrich discovered metadata (limits, prices, effort lists) from models.dev, cached for 28 days in `litellm-models-dev.json`. Off by default because LiteLLM's `/model/info` is authoritative; use it when LiteLLM's model map lacks a model's metadata |
|
|
397
430
|
|
|
398
431
|
Only use a trusted `LITELLM_API_KEY_HELPER` or `!command`. Prefer an absolute executable path, keep secrets out of command arguments, and print only the token to stdout without logging it to stderr.
|
|
@@ -512,7 +545,7 @@ Each distinct availability diagnostic is written once per session on stderr. `LI
|
|
|
512
545
|
|
|
513
546
|
Discovery selects native Messages for proven Claude groups under the [Model transport](#model-transport) rules, Responses for OpenAI-family backends, and Chat Completions for other OpenAI-compatible backends. Azure deployments stay on Chat Completions unless an explicit Responses mode or a `supported_endpoints` list containing `/v1/responses` authorizes Responses; `api_version` alone does not prove support. Azure AI (`azure_ai/` prefix or `custom_llm_provider: azure_ai`) deployments additionally depend on the proxy version. On LiteLLM before v1.103.0, or when the version is unknown, they ignore a `supported_endpoints` list that contains `/v1/responses` and need an explicit `mode: "responses"`: those releases have no native Responses config for that provider, so they bridge `/v1/responses` through Chat Completions, and the bridge fails streaming with `list index out of range` on Azure's empty-choices chunks (BerriAI/litellm#34455) while the endpoint list, copied from the model cost map, still advertises `/v1/responses`. From v1.103.0 the list is honored again. Discovery reads the version from the `x-litellm-version` header of one extra request, `GET /v1/responses/resp_version_probe`, sent only when such a deployment is published; LiteLLM answers it with an error and calls no model. A pre-release counts as older than the release it names, and a proxy that omits the header keeps these deployments on Chat Completions. Azure is detected from the adapter field, an `azure/` or `azure_ai/` model prefix, or the reported provider; other adapters follow the backend family because LiteLLM bridges `/v1/responses` to Chat Completions when the provider has no native Responses config (`litellm/responses/main.py`, `_bridges_to_chat_completions`). The family comes from the model id, not from LiteLLM's `openai/`, `openai_like/` or `custom_openai/` adapter prefix, so `openai/qwen3` stays on Chat Completions while `openai/gpt-5.5` uses Responses; a deployment opts into Responses with `mode: "responses"` or a `supported_endpoints` list containing `/v1/responses`. A route whose model prefix and `custom_llm_provider` name different providers is treated as unidentified and stays on Chat Completions, so it also does not get the local 128-tool preflight. When `/model/info` supplies `supported_endpoints`, that list takes precedence, and `mode: "responses"` remains an explicit Responses signal. An evidence-free fallback entry—a bare `/v1/models` id or a `/health` entry with no detail row—takes its protocol from the Pi catalog entry for its id; both evidence-free fallback paths also inherit presentation metadata from the bounded catalog lookup. An unknown id stays on Chat Completions. The route name alone authorizes nothing. A concrete id expanded from wildcard `/model/info` routes inherits the deployment evidence of the wildcard route LiteLLM would select for it, with the requested id substituted into the row's `model` the way LiteLLM serves it; the id is omitted when that selected route is not a published chat-style route. Chat Completions and Responses use `<root>/v1`; native Messages uses the proxy root directly. All three protocols pass through the provider's host guard. When an evidence-free cached `(no metadata)` entry can be enriched from the Pi catalog, it is restored on Responses only for a catalog Responses model and otherwise on Chat Completions, never Anthropic Messages. A model supplied only through `models.json` whose `api` this provider does not implement takes Pi's global API fallback rather than this provider's host guard; the credential-root pin (see [Model host enforcement](#model-host-enforcement)) still keeps the credential on the active proxy.
|
|
514
547
|
|
|
515
|
-
`cacheControlFormat: "anthropic"` applies only to Chat Completions, where Pi adds Anthropic `cache_control` markers. The Responses transport has a different compatibility type and uses native `prompt_cache_key` (plus supported retention fields), so Responses models must not receive `cacheControlFormat`. Freshly discovered OpenAI-family models forced onto Chat Completions are rejected locally when a request contains more than 128 tools; route them through Responses or reduce enabled extensions. Moonshot request-side reasoning suppression follows LiteLLM's routing provider, not `model_info.base_model`; response-side `<think>` normalization follows discovered Kimi-family evidence. Opaque aliases retain display normalization without sending Moonshot-only parameters to Azure- or Bedrock-hosted Kimi models. Strict tool-message repair requires every deployment to identify Kimi; Gemini effort normalization likewise requires consistent backend-family evidence.
|
|
548
|
+
`cacheControlFormat: "anthropic"` applies only to Chat Completions, where Pi adds Anthropic `cache_control` markers. The Responses transport has a different compatibility type and uses native `prompt_cache_key` (plus supported retention fields), so Responses models must not receive `cacheControlFormat`. Freshly discovered OpenAI-family models forced onto Chat Completions are rejected locally when a request contains more than 128 tools; route them through Responses or reduce enabled extensions. Moonshot request-side reasoning suppression follows LiteLLM's routing provider, not `model_info.base_model`; response-side `<think>` normalization follows discovered Kimi-family evidence. Opaque aliases retain display normalization without sending Moonshot-only parameters to Azure- or Bedrock-hosted Kimi models. Strict tool-message repair requires every deployment to identify Kimi; Gemini effort normalization likewise requires consistent backend-family evidence. Claude-backed routes on native Messages or Chat Completions declare Pi's `promptCache` lifetimes (`short: 300`, `long: 3600` seconds, the values Pi's built-in Anthropic models use), so Pi's `cacheWarming` can keep their caches warm. Responses routes, routes identified only by their name, and other backend families declare none, so Pi does not warm them. A `modelOverrides` entry for the route in the global `models.json` can change the values, and `cacheWarming: "off"` disables warming.
|
|
516
549
|
|
|
517
550
|
Discovery policy is versioned in Pi's persisted model catalog. After upgrading, an online refresh replaces legacy entries. Offline legacy entries remain usable without a diagnostic; the extension reports the discovery-version mismatch once only when an attempted online refresh fails. To force rediscovery or roll back across this policy change, delete the `litellm` entry from `~/.pi/agent/models-store.json` (or delete the file) and refresh once online.
|
|
518
551
|
|
|
@@ -522,7 +555,19 @@ LiteLLM may load-balance one public `model_name` across deployments with differe
|
|
|
522
555
|
|
|
523
556
|
- Responses is selected only when every deployment supports it under the protocol rules above; a deployment that requires Chat keeps the group on Chat.
|
|
524
557
|
- Vision and reasoning are advertised only when every routable deployment resolves them as supported. An explicit `supports_reasoning: false` suppresses all thinking controls, even when catalog or router effort metadata exists. Catalog thinking controls are intersected per level: a level is exposed only when every deployment maps it to the same value, while disagreement or absence becomes an explicit denial. Catalog maps are not complete effort lists, so absence has one exception: an omitted `minimal`, `low`, `medium` or `high` counts as Pi's default spelling of that level. An omitted `off`, `xhigh` or `max` still differs from an explicit mapping. Reasoning selectors additionally require an accepted wire carrier on every deployment. Public effort lists are intersected, explicit LiteLLM denials win, and `xhigh`/`max` need unanimous explicit support. Null or absent standard-effort flags have no opinion. A deployment that declares `model_info.reasoning_effort_levels` answers every level from that list, ahead of its per-level flags, as LiteLLM itself reads it.
|
|
525
|
-
- Context and output limits use the minimum resolved value across deployments. Pi
|
|
558
|
+
- Context and output limits use the minimum resolved value across deployments. Per deployment, a Pi `models.json` `modelOverrides` entry for the identified backend model (Pi provider id plus model id, resolved from backend identity, never from the route name; the provider id is the one whose Pi catalog entry identifies the backend, such as `openai-codex` for a `chatgpt/` deployment, not the LiteLLM provider label) sets `contextWindow` / `maxTokens` ahead of both the router figure and the catalog; it applies only under unanimous catalog authority, and the group minimum still applies. Without an override, the smaller of the router's explicit figure and the catalog figure is used, or the router's figure first under `LITELLM_TRUST_PROXY_LIMITS=1`. Overrides are read from the global `models.json` only and take effect at the next discovery (`/litellm-refresh` or cache expiry). Pi applies `modelOverrides` only to that provider's own models, so one entry covers every LiteLLM route backed by the model:
|
|
559
|
+
|
|
560
|
+
```json
|
|
561
|
+
{
|
|
562
|
+
"providers": {
|
|
563
|
+
"openai": {
|
|
564
|
+
"modelOverrides": { "gpt-5.2": { "contextWindow": 500000 } }
|
|
565
|
+
}
|
|
566
|
+
}
|
|
567
|
+
}
|
|
568
|
+
```
|
|
569
|
+
|
|
570
|
+
Pi catalogue lookup prefers the deployment's provider-specific entry before falling back to the generic vendor entry; Azure and Azure AI therefore retain Azure-specific limits rather than being capped by a different OpenAI catalogue entry. When models.dev supplies a sparse vendor record, missing fields still use the provider-specific Pi catalogue.
|
|
526
571
|
- Each displayed base-price field uses the maximum only when every deployment resolves that field; unresolved fields remain zero and the model name is suffixed with ` (incomplete metadata)`.
|
|
527
572
|
- When complete base pricing comes from the catalog under unanimous catalog authority, tiered pricing is the conservative worst-case envelope. Its usable finite non-negative thresholds are the sorted union of every deployment's thresholds, and each price field at each interval is the maximum applicable rate across all deployments, floored by each deployment's base rate. Differing tier breakpoints are therefore preserved rather than treated as incompatible. An explicit `/model/info` base-price field replaces catalog pricing for that field at every threshold; catalog tiers remain for unaffected fields and are omitted only when every base-price field is explicit.
|
|
528
573
|
- Reduced route groups omit tier ladders when complete base-pricing or catalog authority is unavailable. Wildcard expansion retains every known tier from matching groups even when another match has incomplete metadata: suppressing a complete sibling's higher tier could understate the known worst-case rate. The ` (incomplete metadata)` marker remains because the resulting envelope is only as complete as the available ladders.
|
|
@@ -563,6 +608,7 @@ The development probe runs against minimized snapshots with `npm run probe:proxy
|
|
|
563
608
|
| `LiteLLM discovery: ... route group(s) mix chat-style and explicitly incompatible deployment modes` | A public route can select both a chat-style deployment and an incompatible deployment such as an embedding model. Give the embedding deployment its own `model_name`. |
|
|
564
609
|
| `LiteLLM (...): the request for "<route>" was answered by a fallback` | A successful HTTP response reports a LiteLLM router fallback (`x-litellm-attempted-fallbacks`). `/model/info` does not expose fallbacks, so protocol and model handling follow the requested route's own deployments. If the primary and fallback groups support Chat Completions, set `model_info.supported_endpoints: ["/v1/chat/completions"]` on the primary to avoid Responses-to-Chat bridging. This warning does not change routing or fix upstream bridging defects. It names only the requested route; LiteLLM reports the group and model that answered in the `x-litellm-model-group` and `x-litellm-model-name` response headers. Warned once per provider and route until the extension reloads; shown in the UI, or stderr without a UI. |
|
|
565
610
|
| A model is marked ` (incomplete metadata)` | `/model/info` or `/health` did not provide enough authoritative metadata. Explicit fields remain usable, but unknown cost fields are shown as zero and route-name cache enrichment stays disabled. |
|
|
611
|
+
| Cache miss notices after idle, no cache warming | The model declares no `promptCache`: only Claude-backed Messages and Chat routes get one. Run `/litellm-refresh` after updating so stored models are rediscovered. |
|
|
566
612
|
| `401 Token expired` | Set `LITELLM_API_KEY_HELPER`. |
|
|
567
613
|
| No models with gcloud auth | Verify `gcloud auth application-default login` has been run or set `GOOGLE_APPLICATION_CREDENTIALS` to an `authorized_user` ADC file |
|
|
568
614
|
| Enterprise SSO waits for token insertion | The proxy returned 404/405 for `/sso/cli/start`, so Pi used the legacy flow — upgrade LiteLLM or paste the UI token |
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-provider-litellm",
|
|
3
|
-
"version": "4.
|
|
3
|
+
"version": "4.5.0",
|
|
4
4
|
"description": "LiteLLM proxy provider extension for Pi",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|
|
@@ -74,7 +74,7 @@
|
|
|
74
74
|
"@typescript/native-preview": "7.0.0-dev.20260707.2",
|
|
75
75
|
"es-module-lexer": "^3.0.2",
|
|
76
76
|
"tsx": "^4.21.0",
|
|
77
|
-
"typebox": "1.
|
|
77
|
+
"typebox": "1.4.2",
|
|
78
78
|
"vitest": "^5.0.0"
|
|
79
79
|
}
|
|
80
80
|
}
|
package/src/backend-identity.ts
CHANGED
|
@@ -0,0 +1,174 @@
|
|
|
1
|
+
import type { ClassifierAnswer, ClassifierModel, ClassifierResult, ProviderClassifier } from "@earendil-works/pi-ai";
|
|
2
|
+
import { LITELLM_DISCOVERY_VERSION } from "./backend-identity.js";
|
|
3
|
+
import { errorDetail, parseUsage, perMillion, requestHeaders, requestSignal, responseCost } from "./images.js";
|
|
4
|
+
import { defaultContextWindow, wireString } from "./model-groups.js";
|
|
5
|
+
import type { ModelInfoEntry } from "./types.js";
|
|
6
|
+
|
|
7
|
+
/** Decision models through the proxy's System One `/v1/systemone` endpoint. */
|
|
8
|
+
export const LITELLM_DECISIONS_API = "litellm-decisions" as const;
|
|
9
|
+
export type LiteLLMDecisionsApi = typeof LITELLM_DECISIONS_API;
|
|
10
|
+
|
|
11
|
+
export type DiscoveredClassifierModel = Omit<ClassifierModel<LiteLLMDecisionsApi>, "provider" | "baseUrl"> & {
|
|
12
|
+
litellmDiscoveryVersion?: typeof LITELLM_DISCOVERY_VERSION;
|
|
13
|
+
};
|
|
14
|
+
|
|
15
|
+
export type LiteLLMClassifierModel = ClassifierModel<LiteLLMDecisionsApi> & {
|
|
16
|
+
litellmDiscoveryVersion?: typeof LITELLM_DISCOVERY_VERSION;
|
|
17
|
+
};
|
|
18
|
+
|
|
19
|
+
const DECISION_MODES = new Set(["evaluation", "decisions"]);
|
|
20
|
+
const DECISION_ENDPOINTS = new Set(["/v1/systemone", "/v1/decisions"]);
|
|
21
|
+
|
|
22
|
+
/**
|
|
23
|
+
* LiteLLM's own rule: a declared `supported_endpoints` list decides alone, so a list without a
|
|
24
|
+
* decision endpoint denies even with an evaluation mode; without a list, the mode decides.
|
|
25
|
+
*/
|
|
26
|
+
export function isDecisionDeployment(entry: ModelInfoEntry): boolean {
|
|
27
|
+
const info = entry.model_info;
|
|
28
|
+
const endpoints: unknown = info?.supported_endpoints;
|
|
29
|
+
if (Array.isArray(endpoints)) {
|
|
30
|
+
return endpoints.some((endpoint) => typeof endpoint === "string" && DECISION_ENDPOINTS.has(endpoint.trim()));
|
|
31
|
+
}
|
|
32
|
+
const mode = wireString(info?.mode);
|
|
33
|
+
return mode !== undefined && DECISION_MODES.has(mode.trim().toLowerCase());
|
|
34
|
+
}
|
|
35
|
+
|
|
36
|
+
/**
|
|
37
|
+
* Reduce one `/model/info` route group to a classifier model. Every deployment behind the route
|
|
38
|
+
* must be a decision deployment: LiteLLM load-balances across them, so a mixed group could answer
|
|
39
|
+
* from a non-decision deployment.
|
|
40
|
+
*/
|
|
41
|
+
export function mapFromClassifierModelInfoGroup(
|
|
42
|
+
route: string,
|
|
43
|
+
entries: readonly ModelInfoEntry[],
|
|
44
|
+
): DiscoveredClassifierModel | undefined {
|
|
45
|
+
// A wildcard row is not addressable, and `/v1/models` expansion carries no mode evidence.
|
|
46
|
+
if (route.includes("*")) return undefined;
|
|
47
|
+
const deployments = entries.filter((entry) => wireString(entry.model_name) === route);
|
|
48
|
+
if (deployments.length === 0) return undefined;
|
|
49
|
+
if (!deployments.every(isDecisionDeployment)) return undefined;
|
|
50
|
+
// Prices are estimates for Pi's usage display; the `x-litellm-response-cost` header replaces them.
|
|
51
|
+
const rate = (read: (entry: ModelInfoEntry) => number | undefined): number => {
|
|
52
|
+
const values = deployments.map(read);
|
|
53
|
+
return values.every((value) => value !== undefined) ? Math.max(...(values as number[])) : 0;
|
|
54
|
+
};
|
|
55
|
+
// The smallest declared window; an undeclared one counts as the default so it never raises the result.
|
|
56
|
+
const windows = deployments.map((entry) => entry.model_info?.max_input_tokens);
|
|
57
|
+
const declared = windows.filter(
|
|
58
|
+
(value): value is number => typeof value === "number" && Number.isFinite(value) && value > 0,
|
|
59
|
+
);
|
|
60
|
+
const contextWindow = Math.min(...declared, ...(declared.length < windows.length ? [defaultContextWindow()] : []));
|
|
61
|
+
return {
|
|
62
|
+
type: "classifier",
|
|
63
|
+
id: route,
|
|
64
|
+
name: route,
|
|
65
|
+
api: LITELLM_DECISIONS_API,
|
|
66
|
+
input: ["text"],
|
|
67
|
+
cost: {
|
|
68
|
+
input: rate((entry) => perMillion(entry.model_info?.input_cost_per_token)),
|
|
69
|
+
output: rate((entry) => perMillion(entry.model_info?.output_cost_per_token)),
|
|
70
|
+
cacheRead: 0,
|
|
71
|
+
cacheWrite: 0,
|
|
72
|
+
},
|
|
73
|
+
contextWindow,
|
|
74
|
+
litellmDiscoveryVersion: LITELLM_DISCOVERY_VERSION,
|
|
75
|
+
};
|
|
76
|
+
}
|
|
77
|
+
|
|
78
|
+
function isRecord(value: unknown): value is Record<string, unknown> {
|
|
79
|
+
return typeof value === "object" && value !== null && !Array.isArray(value);
|
|
80
|
+
}
|
|
81
|
+
|
|
82
|
+
const isFiniteNumber = (value: unknown): value is number => typeof value === "number" && Number.isFinite(value);
|
|
83
|
+
|
|
84
|
+
function parseAnswer(model: string, id: string, type: "choice" | "score" | "bool", raw: unknown): ClassifierAnswer {
|
|
85
|
+
const wireType = type === "bool" ? "noul" : type;
|
|
86
|
+
if (!isRecord(raw) || raw.type !== wireType) {
|
|
87
|
+
throw new Error(`LiteLLM classifier ${model} did not return a ${type} answer for ${id}`);
|
|
88
|
+
}
|
|
89
|
+
const invalid = (field: string): Error =>
|
|
90
|
+
new Error(`LiteLLM classifier ${model} returned an invalid ${field} for ${id}`);
|
|
91
|
+
if (type === "bool") {
|
|
92
|
+
if (!isFiniteNumber(raw.noul)) throw invalid("noul");
|
|
93
|
+
return { type: "bool", probability: raw.noul };
|
|
94
|
+
}
|
|
95
|
+
if (!isFiniteNumber(raw.confidence)) throw invalid("confidence");
|
|
96
|
+
if (type === "score") {
|
|
97
|
+
if (!isFiniteNumber(raw.score)) throw invalid("score");
|
|
98
|
+
return { type: "score", score: raw.score, confidence: raw.confidence };
|
|
99
|
+
}
|
|
100
|
+
if (typeof raw.choice !== "string") throw invalid("choice");
|
|
101
|
+
if (!isRecord(raw.probabilities) || !Object.values(raw.probabilities).every(isFiniteNumber)) {
|
|
102
|
+
throw invalid("probabilities");
|
|
103
|
+
}
|
|
104
|
+
return {
|
|
105
|
+
type: "choice",
|
|
106
|
+
choice: raw.choice,
|
|
107
|
+
probabilities: raw.probabilities as Record<string, number>,
|
|
108
|
+
confidence: raw.confidence,
|
|
109
|
+
};
|
|
110
|
+
}
|
|
111
|
+
|
|
112
|
+
/** Classify through LiteLLM's `/v1/systemone`. Never rejects. */
|
|
113
|
+
export const classify: ProviderClassifier["classify"] = async (model, context, options) => {
|
|
114
|
+
const result: ClassifierResult = {
|
|
115
|
+
api: model.api,
|
|
116
|
+
provider: model.provider,
|
|
117
|
+
model: model.id,
|
|
118
|
+
answers: {},
|
|
119
|
+
stopReason: "stop",
|
|
120
|
+
timestamp: Date.now(),
|
|
121
|
+
};
|
|
122
|
+
try {
|
|
123
|
+
const apiKey = options?.apiKey;
|
|
124
|
+
if (!apiKey) throw new Error(`No API key for provider: ${model.provider}`);
|
|
125
|
+
if (context.images?.length) {
|
|
126
|
+
throw new Error(`LiteLLM classifier ${model.id} takes text state only; images are not sent to /v1/systemone`);
|
|
127
|
+
}
|
|
128
|
+
// System One calls Pi's `bool` question `noul`; the fields are otherwise the same.
|
|
129
|
+
const questions = Object.fromEntries(
|
|
130
|
+
Object.entries(context.questions).map(([id, question]) => [
|
|
131
|
+
id,
|
|
132
|
+
question.type === "bool" ? { ...question, type: "noul" } : question,
|
|
133
|
+
]),
|
|
134
|
+
);
|
|
135
|
+
let params: Record<string, unknown> = { model: model.id, state: context.state, questions };
|
|
136
|
+
const nextParams = await options?.onPayload?.(params, model);
|
|
137
|
+
if (nextParams !== undefined) params = nextParams as Record<string, unknown>;
|
|
138
|
+
const response = await (options?.fetch ?? fetch)(`${model.baseUrl}/systemone`, {
|
|
139
|
+
method: "POST",
|
|
140
|
+
headers: requestHeaders(apiKey, model.headers, options?.headers),
|
|
141
|
+
body: JSON.stringify(params),
|
|
142
|
+
signal: requestSignal(options),
|
|
143
|
+
});
|
|
144
|
+
const responseHeaders = Object.fromEntries(response.headers.entries());
|
|
145
|
+
await options?.onResponse?.({ status: response.status, headers: responseHeaders }, model);
|
|
146
|
+
const body = await response.text();
|
|
147
|
+
if (!response.ok) throw new Error(`LiteLLM decisions request failed (${response.status}): ${errorDetail(body)}`);
|
|
148
|
+
// The proxy billed the request whatever the body holds; token counts are added once it parses.
|
|
149
|
+
const cost = responseCost(responseHeaders);
|
|
150
|
+
const billed = parseUsage(undefined, model.cost, cost);
|
|
151
|
+
if (billed) result.usage = billed;
|
|
152
|
+
const parsed: unknown = JSON.parse(body);
|
|
153
|
+
if (!isRecord(parsed)) throw new Error(`LiteLLM classifier ${model.id} returned an unexpected response`);
|
|
154
|
+
// A malformed answer set was still billed.
|
|
155
|
+
const usage = parseUsage(isRecord(parsed.usage) ? parsed.usage : undefined, model.cost, cost);
|
|
156
|
+
if (usage) result.usage = usage;
|
|
157
|
+
const answers = isRecord(parsed.answers) ? parsed.answers : {};
|
|
158
|
+
const parsedAnswers: Record<string, ClassifierAnswer> = {};
|
|
159
|
+
for (const [id, question] of Object.entries(context.questions)) {
|
|
160
|
+
parsedAnswers[id] = parseAnswer(model.id, id, question.type, answers[id]);
|
|
161
|
+
}
|
|
162
|
+
result.answers = parsedAnswers;
|
|
163
|
+
return result;
|
|
164
|
+
} catch (error) {
|
|
165
|
+
result.answers = {};
|
|
166
|
+
result.stopReason = options?.signal?.aborted ? "aborted" : "error";
|
|
167
|
+
result.errorMessage = error instanceof Error ? error.message : String(error);
|
|
168
|
+
return result;
|
|
169
|
+
}
|
|
170
|
+
};
|
|
171
|
+
|
|
172
|
+
export function createLiteLLMClassifierApis(): Record<LiteLLMDecisionsApi, ProviderClassifier> {
|
|
173
|
+
return { [LITELLM_DECISIONS_API]: { classify } };
|
|
174
|
+
}
|
package/src/discover.ts
CHANGED
|
@@ -8,6 +8,7 @@ import {
|
|
|
8
8
|
resolveCatalogProvider,
|
|
9
9
|
routesOnlyThrough,
|
|
10
10
|
} from "./backend-identity.js";
|
|
11
|
+
import { type DiscoveredClassifierModel, mapFromClassifierModelInfoGroup } from "./classifiers.js";
|
|
11
12
|
import { type DiscoveredImageModel, mapFromImageModelInfoGroup } from "./images.js";
|
|
12
13
|
import {
|
|
13
14
|
type CatalogResolution,
|
|
@@ -50,6 +51,10 @@ import type {
|
|
|
50
51
|
} from "./types.js";
|
|
51
52
|
|
|
52
53
|
const DEFAULT_TIMEOUT_MS = 5000;
|
|
54
|
+
// Anthropic's documented 5-minute and 1-hour cache lifetimes; the values Pi's built-in Anthropic catalog uses.
|
|
55
|
+
// shortcut: `long` assumes the host offers the 1-hour TTL (Bedrock only does for Claude 4.5+), gate it on
|
|
56
|
+
// deployment routing if LiteLLM ever reports per-route TTL support.
|
|
57
|
+
const CLAUDE_PROMPT_CACHE = { short: 300, long: 3600 } as const;
|
|
53
58
|
const HEALTH_DETAIL_CONCURRENCY = 8;
|
|
54
59
|
const reportedConflictingFamilyRoutes = new Set<string>();
|
|
55
60
|
const reportedWithheldRepairRoutes = new Set<string>();
|
|
@@ -496,6 +501,7 @@ function adaptPublicCatalogRecord(
|
|
|
496
501
|
const piCost = piCatalog?.cost;
|
|
497
502
|
return {
|
|
498
503
|
provider,
|
|
504
|
+
...(resolved ? { piProvider: resolved.provider } : {}),
|
|
499
505
|
// A models.dev hit names the model canonically even when Pi's catalog does not know it yet,
|
|
500
506
|
// so two spellings of one backend model reduce to one identity instead of a conflict.
|
|
501
507
|
catalogModelId: resolved?.model.id ?? record?.modelId ?? catalogModelId,
|
|
@@ -689,10 +695,16 @@ function mapFromModelInfoGroup(
|
|
|
689
695
|
denyLevels?: boolean;
|
|
690
696
|
allowMessages?: boolean;
|
|
691
697
|
proxyVersion?: ProxyVersion;
|
|
698
|
+
modelOverrides?: DiscoveryOptions["modelOverrides"];
|
|
692
699
|
} = {},
|
|
693
700
|
): DiscoveredModel | undefined {
|
|
694
701
|
const reduced = reduceModelGroup(entries, (entry) => {
|
|
695
|
-
const
|
|
702
|
+
const resolved = resolveModelInfoCatalog(entry, publicCatalog);
|
|
703
|
+
const limitOverrides =
|
|
704
|
+
resolved?.piProvider && resolved.catalogModelId
|
|
705
|
+
? options.modelOverrides?.[resolved.piProvider]?.[resolved.catalogModelId]
|
|
706
|
+
: undefined;
|
|
707
|
+
const catalog = limitOverrides ? { ...resolved, limitOverrides } : resolved;
|
|
696
708
|
const family = deploymentFamily(entry);
|
|
697
709
|
const generations = [entry.litellm_params?.model, entry.model_info?.base_model]
|
|
698
710
|
.map((id) => wireString(id)?.trim())
|
|
@@ -756,6 +768,10 @@ function mapFromModelInfoGroup(
|
|
|
756
768
|
acceptsResponsesReasoningControl: reduced.acceptsResponsesReasoningControl,
|
|
757
769
|
denyLevels: options.denyLevels,
|
|
758
770
|
});
|
|
771
|
+
const relaysAnthropicCacheMarkers =
|
|
772
|
+
api === "anthropic-messages" ||
|
|
773
|
+
(api === "openai-completions" &&
|
|
774
|
+
(policy.compat as Model<"openai-completions">["compat"])?.cacheControlFormat === "anthropic");
|
|
759
775
|
const hasConflictingFamily = reduced.deploymentFamilies.includes("conflicting");
|
|
760
776
|
const unanimousMoonshot =
|
|
761
777
|
reduced.deploymentFamilies.length > 0 && reduced.deploymentFamilies.every((family) => family === "kimi");
|
|
@@ -802,6 +818,9 @@ function mapFromModelInfoGroup(
|
|
|
802
818
|
? { litellmResponsesReasoningControl: true as const }
|
|
803
819
|
: {}),
|
|
804
820
|
...(families.size === 1 && family ? { litellmBackendFamily: family } : {}),
|
|
821
|
+
...(families.size === 1 && family === "claude" && relaysAnthropicCacheMarkers
|
|
822
|
+
? { promptCache: { ...CLAUDE_PROMPT_CACHE } }
|
|
823
|
+
: {}),
|
|
805
824
|
...(modelPolicy ? { litellmPolicy: modelPolicy } : {}),
|
|
806
825
|
// closeSerializerPolicy derives compat from `api`, a correlation its return type cannot carry.
|
|
807
826
|
} as DiscoveredModel;
|
|
@@ -930,6 +949,7 @@ async function discoverFromHealth(
|
|
|
930
949
|
denyLevels: group.some(({ denyLevels }) => denyLevels),
|
|
931
950
|
allowMessages: false,
|
|
932
951
|
proxyVersion,
|
|
952
|
+
modelOverrides: options.modelOverrides,
|
|
933
953
|
});
|
|
934
954
|
})
|
|
935
955
|
.filter((model): model is DiscoveredModel => model !== undefined);
|
|
@@ -1036,10 +1056,11 @@ function applyWildcardEvidence(
|
|
|
1036
1056
|
publishedWildcardIds: ReadonlySet<string>,
|
|
1037
1057
|
publicCatalog: PublicCatalog | undefined,
|
|
1038
1058
|
proxyVersion: ProxyVersion | undefined,
|
|
1059
|
+
modelOverrides: DiscoveryOptions["modelOverrides"],
|
|
1039
1060
|
): DiscoveredModel | undefined {
|
|
1040
1061
|
const parents = selectedWildcardRows(model.id, wildcardRows, publishedWildcardIds);
|
|
1041
1062
|
if (!parents) return undefined;
|
|
1042
|
-
const selected = mapFromModelInfoGroup(parents, publicCatalog, { proxyVersion });
|
|
1063
|
+
const selected = mapFromModelInfoGroup(parents, publicCatalog, { proxyVersion, modelOverrides });
|
|
1043
1064
|
if (!selected) return undefined;
|
|
1044
1065
|
const policies = [model.litellmPolicy, selected.litellmPolicy].filter((policy) => policy !== undefined);
|
|
1045
1066
|
const combinedPolicy =
|
|
@@ -1061,6 +1082,7 @@ function applyWildcardEvidence(
|
|
|
1061
1082
|
litellmBackendFamily: _family,
|
|
1062
1083
|
litellmResponsesReasoningControl: _control,
|
|
1063
1084
|
litellmPolicy: _policy,
|
|
1085
|
+
promptCache: _promptCache,
|
|
1064
1086
|
thinkingLevelMap: _levels,
|
|
1065
1087
|
...rest
|
|
1066
1088
|
} = model;
|
|
@@ -1071,6 +1093,7 @@ function applyWildcardEvidence(
|
|
|
1071
1093
|
compat: selected.compat,
|
|
1072
1094
|
...(selected.thinkingLevelMap ? { thinkingLevelMap: selected.thinkingLevelMap } : {}),
|
|
1073
1095
|
...(selected.litellmBackendFamily ? { litellmBackendFamily: selected.litellmBackendFamily } : {}),
|
|
1096
|
+
...(selected.promptCache ? { promptCache: selected.promptCache } : {}),
|
|
1074
1097
|
...(combinedPolicy ? { litellmPolicy: combinedPolicy } : {}),
|
|
1075
1098
|
...(selected.litellmResponsesReasoningControl ? { litellmResponsesReasoningControl: true as const } : {}),
|
|
1076
1099
|
} as DiscoveredModel;
|
|
@@ -1216,6 +1239,7 @@ export async function discoverModels(
|
|
|
1216
1239
|
withheldRepairRoutes,
|
|
1217
1240
|
defaultedContextRoutes,
|
|
1218
1241
|
proxyVersion,
|
|
1242
|
+
modelOverrides: options.modelOverrides,
|
|
1219
1243
|
}),
|
|
1220
1244
|
deploymentFamilies: group.map(deploymentFamily),
|
|
1221
1245
|
};
|
|
@@ -1272,14 +1296,26 @@ export async function discoverModels(
|
|
|
1272
1296
|
// The parents were reduced before the version was known; an expansion
|
|
1273
1297
|
// must inherit from parents reduced under the version it is published with.
|
|
1274
1298
|
sources = wildcardRoutes.flatMap(({ route, deploymentFamilies }) => {
|
|
1275
|
-
const model = mapFromModelInfoGroup(groups.get(route)!, publicCatalog, {
|
|
1299
|
+
const model = mapFromModelInfoGroup(groups.get(route)!, publicCatalog, {
|
|
1300
|
+
proxyVersion,
|
|
1301
|
+
modelOverrides: options.modelOverrides,
|
|
1302
|
+
});
|
|
1276
1303
|
return model ? [{ model, deploymentFamilies }] : [];
|
|
1277
1304
|
});
|
|
1278
1305
|
}
|
|
1279
1306
|
const expanded = candidates
|
|
1280
1307
|
.map((entry) => mapFromWildcardExpansion(entry, sources, withheldRepairRoutes))
|
|
1281
1308
|
.filter((model): model is DiscoveredModel => model !== undefined)
|
|
1282
|
-
.map((model) =>
|
|
1309
|
+
.map((model) =>
|
|
1310
|
+
applyWildcardEvidence(
|
|
1311
|
+
model,
|
|
1312
|
+
wildcardRows,
|
|
1313
|
+
publishedWildcardIds,
|
|
1314
|
+
publicCatalog,
|
|
1315
|
+
proxyVersion,
|
|
1316
|
+
options.modelOverrides,
|
|
1317
|
+
),
|
|
1318
|
+
)
|
|
1283
1319
|
.filter((model): model is DiscoveredModel => model !== undefined);
|
|
1284
1320
|
models = [...models, ...expanded];
|
|
1285
1321
|
}
|
|
@@ -1309,10 +1345,15 @@ export async function discoverModels(
|
|
|
1309
1345
|
const imageModels = [...groups.entries()]
|
|
1310
1346
|
.map(([route, group]) => mapFromImageModelInfoGroup(route, group))
|
|
1311
1347
|
.filter((model): model is DiscoveredImageModel => model !== undefined);
|
|
1348
|
+
// Decision routes are a separate operation in Pi; the chat reducer withholds their modes.
|
|
1349
|
+
const classifierModels = [...groups.entries()]
|
|
1350
|
+
.map(([route, group]) => mapFromClassifierModelInfoGroup(route, group))
|
|
1351
|
+
.filter((model): model is DiscoveredClassifierModel => model !== undefined);
|
|
1312
1352
|
return {
|
|
1313
1353
|
source: "model_info",
|
|
1314
1354
|
models,
|
|
1315
1355
|
...(imageModels.length > 0 ? { imageModels } : {}),
|
|
1356
|
+
...(classifierModels.length > 0 ? { classifierModels } : {}),
|
|
1316
1357
|
...(proxyVersion ? { proxyVersion } : {}),
|
|
1317
1358
|
};
|
|
1318
1359
|
}
|
package/src/images.ts
CHANGED
|
@@ -1,9 +1,8 @@
|
|
|
1
1
|
import type {
|
|
2
2
|
AssistantImages,
|
|
3
|
-
ImageApi,
|
|
4
3
|
ImageContent,
|
|
5
4
|
ImageModel,
|
|
6
|
-
|
|
5
|
+
ModelCost,
|
|
7
6
|
ProviderImages,
|
|
8
7
|
Usage,
|
|
9
8
|
} from "@earendil-works/pi-ai";
|
|
@@ -33,7 +32,7 @@ export function isImageGenerationMode(mode: unknown): boolean {
|
|
|
33
32
|
return value !== undefined && IMAGE_GENERATION_MODE_PATTERN.test(value.trim());
|
|
34
33
|
}
|
|
35
34
|
|
|
36
|
-
function perMillion(value: unknown): number | undefined {
|
|
35
|
+
export function perMillion(value: unknown): number | undefined {
|
|
37
36
|
return typeof value === "number" && Number.isFinite(value) && value >= 0 ? value * 1_000_000 : undefined;
|
|
38
37
|
}
|
|
39
38
|
|
|
@@ -119,18 +118,19 @@ function tokenCount(...values: unknown[]): number {
|
|
|
119
118
|
return 0;
|
|
120
119
|
}
|
|
121
120
|
|
|
122
|
-
function parseUsage(
|
|
123
|
-
|
|
124
|
-
|
|
121
|
+
export function parseUsage(
|
|
122
|
+
raw:
|
|
123
|
+
| { input_tokens?: unknown; output_tokens?: unknown; prompt_tokens?: unknown; completion_tokens?: unknown }
|
|
124
|
+
| undefined,
|
|
125
|
+
cost: ModelCost,
|
|
125
126
|
responseCost: number | undefined,
|
|
126
127
|
): Usage | undefined {
|
|
127
|
-
const raw = response.usage;
|
|
128
128
|
if (!raw && responseCost === undefined) return undefined;
|
|
129
129
|
const input = tokenCount(raw?.input_tokens, raw?.prompt_tokens);
|
|
130
130
|
const output = tokenCount(raw?.output_tokens, raw?.completion_tokens);
|
|
131
131
|
const estimated = {
|
|
132
|
-
input: (
|
|
133
|
-
output: (
|
|
132
|
+
input: (cost.input / 1_000_000) * input,
|
|
133
|
+
output: (cost.output / 1_000_000) * output,
|
|
134
134
|
};
|
|
135
135
|
// LiteLLM's own cost is authoritative: it knows per-image prices that a token rate cannot express.
|
|
136
136
|
const total = responseCost ?? estimated.input + estimated.output;
|
|
@@ -148,7 +148,13 @@ function parseUsage(
|
|
|
148
148
|
};
|
|
149
149
|
}
|
|
150
150
|
|
|
151
|
-
|
|
151
|
+
/** LiteLLM's `x-litellm-response-cost` header, when it is a usable amount. */
|
|
152
|
+
export function responseCost(headers: Record<string, string>): number | undefined {
|
|
153
|
+
const cost = Number.parseFloat(headers["x-litellm-response-cost"] ?? "");
|
|
154
|
+
return Number.isFinite(cost) && cost >= 0 ? cost : undefined;
|
|
155
|
+
}
|
|
156
|
+
|
|
157
|
+
export function requestHeaders(
|
|
152
158
|
apiKey: string,
|
|
153
159
|
...sources: (Record<string, string | null | undefined> | undefined)[]
|
|
154
160
|
): Record<string, string> {
|
|
@@ -167,7 +173,7 @@ function requestHeaders(
|
|
|
167
173
|
return headers;
|
|
168
174
|
}
|
|
169
175
|
|
|
170
|
-
function errorDetail(body: string): string {
|
|
176
|
+
export function errorDetail(body: string): string {
|
|
171
177
|
try {
|
|
172
178
|
const parsed = JSON.parse(body) as { error?: { message?: unknown } | unknown };
|
|
173
179
|
const message =
|
|
@@ -181,7 +187,9 @@ function errorDetail(body: string): string {
|
|
|
181
187
|
return body.slice(0, MAX_ERROR_DETAIL);
|
|
182
188
|
}
|
|
183
189
|
|
|
184
|
-
function requestSignal(
|
|
190
|
+
export function requestSignal(
|
|
191
|
+
options: { signal?: AbortSignal; timeoutMs?: number } | undefined,
|
|
192
|
+
): AbortSignal | undefined {
|
|
185
193
|
const signals = [
|
|
186
194
|
options?.signal,
|
|
187
195
|
options?.timeoutMs !== undefined ? AbortSignal.timeout(options.timeoutMs) : undefined,
|
|
@@ -242,8 +250,7 @@ export const generateImages: ProviderImages["generateImages"] = async (model, co
|
|
|
242
250
|
: `LiteLLM image model ${model.id} returned no images`,
|
|
243
251
|
);
|
|
244
252
|
}
|
|
245
|
-
const
|
|
246
|
-
const usage = parseUsage(parsed, model, Number.isFinite(costHeader) && costHeader >= 0 ? costHeader : undefined);
|
|
253
|
+
const usage = parseUsage(parsed.usage, model.cost, responseCost(responseHeaders));
|
|
247
254
|
if (usage) result.usage = usage;
|
|
248
255
|
return result;
|
|
249
256
|
} catch (error) {
|
package/src/index.ts
CHANGED
|
@@ -19,6 +19,7 @@ import type { ExtensionAPI, ExtensionContext, McpServerConfig } from "@earendil-
|
|
|
19
19
|
import * as piCodingAgent from "@earendil-works/pi-coding-agent";
|
|
20
20
|
import { getAgentDir, readStoredCredential } from "@earendil-works/pi-coding-agent";
|
|
21
21
|
import { budgetDisplaySetting, setupLiteLLMBudget } from "./budget.js";
|
|
22
|
+
import type { LiteLLMClassifierModel } from "./classifiers.js";
|
|
22
23
|
import { setupLiteLLMCostTracking } from "./cost.js";
|
|
23
24
|
import {
|
|
24
25
|
discoverModels,
|
|
@@ -34,6 +35,7 @@ import {
|
|
|
34
35
|
createLiteLLMProvider,
|
|
35
36
|
DEFAULT_LITELLM_BASE_URL,
|
|
36
37
|
isPlaceholderHost,
|
|
38
|
+
toNativeClassifierModels,
|
|
37
39
|
toNativeImageModels,
|
|
38
40
|
toNativeModels,
|
|
39
41
|
} from "./provider.js";
|
|
@@ -286,6 +288,37 @@ async function readModelsProviderSettings(): Promise<Record<string, unknown> | u
|
|
|
286
288
|
}
|
|
287
289
|
}
|
|
288
290
|
|
|
291
|
+
// Pi `modelOverrides` limits from the global models.json, keyed by provider id then model id.
|
|
292
|
+
async function readModelLimitOverrides(): Promise<NonNullable<DiscoveryOptions["modelOverrides"]>> {
|
|
293
|
+
// Null prototypes: a `__proto__` provider or model key in the user's file stays a plain key.
|
|
294
|
+
const result: Record<string, Record<string, { contextWindow?: number; maxTokens?: number }>> = Object.create(null);
|
|
295
|
+
try {
|
|
296
|
+
const raw = await readFile(join(getAgentDir(), MODELS_FILENAME), "utf8");
|
|
297
|
+
const parsed = JSON.parse(stripJsonComments(stripBom(raw))) as Record<string, unknown>;
|
|
298
|
+
if (!isPlainObject(parsed.providers)) return result;
|
|
299
|
+
const limit = (value: unknown) =>
|
|
300
|
+
typeof value === "number" && Number.isFinite(value) && value > 0 ? value : undefined;
|
|
301
|
+
for (const [providerId, provider] of Object.entries(parsed.providers)) {
|
|
302
|
+
if (!isPlainObject(provider) || !isPlainObject(provider.modelOverrides)) continue;
|
|
303
|
+
for (const [modelId, override] of Object.entries(provider.modelOverrides)) {
|
|
304
|
+
if (!isPlainObject(override)) continue;
|
|
305
|
+
const contextWindow = limit(override.contextWindow);
|
|
306
|
+
const maxTokens = limit(override.maxTokens);
|
|
307
|
+
if (contextWindow === undefined && maxTokens === undefined) continue;
|
|
308
|
+
const models = result[providerId] ?? Object.create(null);
|
|
309
|
+
result[providerId] = models;
|
|
310
|
+
models[modelId] = {
|
|
311
|
+
...(contextWindow !== undefined ? { contextWindow } : {}),
|
|
312
|
+
...(maxTokens !== undefined ? { maxTokens } : {}),
|
|
313
|
+
};
|
|
314
|
+
}
|
|
315
|
+
}
|
|
316
|
+
} catch {
|
|
317
|
+
// Missing or unparsable models.json: no overrides.
|
|
318
|
+
}
|
|
319
|
+
return result;
|
|
320
|
+
}
|
|
321
|
+
|
|
289
322
|
// A provider block may live in either file; merge per field with models.json winning. Non-provider
|
|
290
323
|
// settings.json keys pass through untouched, which is what keeps `skills`/`mcp`/`budget` working.
|
|
291
324
|
function mergeProviderSettings(
|
|
@@ -2637,6 +2670,7 @@ export default async function (pi: ExtensionAPI): Promise<void> {
|
|
|
2637
2670
|
headers: auth.headers,
|
|
2638
2671
|
allowInsecureHttp: auth.allowInsecureHttp,
|
|
2639
2672
|
...modelsDevDiscoveryOptions(),
|
|
2673
|
+
modelOverrides: await readModelLimitOverrides(),
|
|
2640
2674
|
silent: !isVerboseDiscovery(),
|
|
2641
2675
|
onProgress: isVerboseDiscovery() ? (message) => process.stderr.write(`LiteLLM: ${message}\n`) : undefined,
|
|
2642
2676
|
});
|
|
@@ -2653,7 +2687,9 @@ export default async function (pi: ExtensionAPI): Promise<void> {
|
|
|
2653
2687
|
// critical path of every start -- is replaced by reading that catalog while it is fresh (models-store.ts). Nothing is
|
|
2654
2688
|
// written: Pi owns the file. The stored models are handed over rather than an empty list, so that a store Pi keeps
|
|
2655
2689
|
// somewhere else cannot leave the provider with nothing.
|
|
2656
|
-
async function seedModels(
|
|
2690
|
+
async function seedModels(
|
|
2691
|
+
definition: ProviderDefinition,
|
|
2692
|
+
): Promise<(Model<LiteLLMApi> | LiteLLMImageModel | LiteLLMClassifierModel)[]> {
|
|
2657
2693
|
const disabledReason = discoveryDisabledReason() ?? (isHostOffline() ? "PI_OFFLINE" : null);
|
|
2658
2694
|
if (disabledReason) {
|
|
2659
2695
|
if (isVerboseDiscovery()) {
|
|
@@ -2685,6 +2721,12 @@ export default async function (pi: ExtensionAPI): Promise<void> {
|
|
|
2685
2721
|
return [
|
|
2686
2722
|
...toNativeModels(definition.name, result.baseUrl, result.models, definition.allowInsecureHttp),
|
|
2687
2723
|
...toNativeImageModels(definition.name, result.baseUrl, result.imageModels ?? [], definition.allowInsecureHttp),
|
|
2724
|
+
...toNativeClassifierModels(
|
|
2725
|
+
definition.name,
|
|
2726
|
+
result.baseUrl,
|
|
2727
|
+
result.classifierModels ?? [],
|
|
2728
|
+
definition.allowInsecureHttp,
|
|
2729
|
+
),
|
|
2688
2730
|
];
|
|
2689
2731
|
} catch (error) {
|
|
2690
2732
|
// No credentials yet is the normal unconfigured case; anything else is worth one line.
|
package/src/model-groups.ts
CHANGED
|
@@ -54,6 +54,8 @@ export interface ReasoningPolicy {
|
|
|
54
54
|
|
|
55
55
|
export interface CatalogResolution {
|
|
56
56
|
provider?: string;
|
|
57
|
+
// The Pi provider whose catalog entry resolved catalogModelId; set only then, so the pair is Pi's providerId/model.id.
|
|
58
|
+
piProvider?: string;
|
|
57
59
|
catalogModelId?: string;
|
|
58
60
|
semanticFamily?: FamilyEvidence;
|
|
59
61
|
semanticModel?: SemanticModel;
|
|
@@ -68,6 +70,8 @@ export interface CatalogResolution {
|
|
|
68
70
|
vision?: boolean;
|
|
69
71
|
contextWindow?: number;
|
|
70
72
|
maxTokens?: number;
|
|
73
|
+
// User `modelOverrides` for the identified catalog model; read only under unanimous catalog authority.
|
|
74
|
+
limitOverrides?: { contextWindow?: number; maxTokens?: number };
|
|
71
75
|
cost?: Partial<DiscoveredModel["cost"]>;
|
|
72
76
|
}
|
|
73
77
|
|
|
@@ -657,6 +661,8 @@ function min(values: readonly number[]): number {
|
|
|
657
661
|
}
|
|
658
662
|
|
|
659
663
|
function conservativeLimit(explicit: number | undefined, catalog: number | undefined): number | undefined {
|
|
664
|
+
// LITELLM_TRUST_PROXY_LIMITS=1 lets the router's own figure win over the catalog.
|
|
665
|
+
if (process.env.LITELLM_TRUST_PROXY_LIMITS === "1") return explicitLimit(explicit) ?? explicitLimit(catalog);
|
|
660
666
|
const valid = [explicitLimit(explicit), explicitLimit(catalog)].filter(
|
|
661
667
|
(value): value is number => value !== undefined,
|
|
662
668
|
);
|
|
@@ -819,11 +825,15 @@ export function reduceModelGroup(
|
|
|
819
825
|
const visionEvidence = deployments.map(
|
|
820
826
|
(entry, index) => wireBoolean(entry.model_info?.supports_vision) ?? catalogAuthority[index]?.vision,
|
|
821
827
|
);
|
|
822
|
-
const contextWindowEvidence = deployments.map(
|
|
823
|
-
|
|
828
|
+
const contextWindowEvidence = deployments.map(
|
|
829
|
+
(entry, index) =>
|
|
830
|
+
explicitLimit(catalogAuthority[index]?.limitOverrides?.contextWindow) ??
|
|
831
|
+
conservativeLimit(entry.model_info?.max_input_tokens, catalogAuthority[index]?.contextWindow),
|
|
824
832
|
);
|
|
825
|
-
const maxTokensEvidence = deployments.map(
|
|
826
|
-
|
|
833
|
+
const maxTokensEvidence = deployments.map(
|
|
834
|
+
(entry, index) =>
|
|
835
|
+
explicitLimit(catalogAuthority[index]?.limitOverrides?.maxTokens) ??
|
|
836
|
+
conservativeLimit(entry.model_info?.max_output_tokens, catalogAuthority[index]?.maxTokens),
|
|
827
837
|
);
|
|
828
838
|
const reasoning = reasoningEvidence.every((value) => value ?? false);
|
|
829
839
|
const explicitlyUnsupported = deployments.some(
|
package/src/models-store.ts
CHANGED
|
@@ -39,10 +39,11 @@
|
|
|
39
39
|
import { readFile } from "node:fs/promises";
|
|
40
40
|
import { join } from "node:path";
|
|
41
41
|
import { LITELLM_DISCOVERY_VERSION } from "./backend-identity.js";
|
|
42
|
+
import { LITELLM_DECISIONS_API, type LiteLLMClassifierModel } from "./classifiers.js";
|
|
42
43
|
import { enrichCachedModel } from "./discover.js";
|
|
43
44
|
import { LITELLM_IMAGES_API, type LiteLLMImageModel } from "./images.js";
|
|
44
45
|
import { isLiteLLMApi, resolveModelBaseUrl } from "./protocols.js";
|
|
45
|
-
import {
|
|
46
|
+
import { proxyV1BaseUrl } from "./provider.js";
|
|
46
47
|
import type { LiteLLMModel } from "./types.js";
|
|
47
48
|
|
|
48
49
|
/**
|
|
@@ -59,17 +60,20 @@ function isRecord(value: unknown): value is Record<string, unknown> {
|
|
|
59
60
|
/** Pi's own remote catalogs skip the network for 4 hours after `checkedAt` (`REMOTE_CATALOG_REFRESH_INTERVAL_MS`). */
|
|
60
61
|
const STORE_FRESH_MS = 4 * 60 * 60 * 1000;
|
|
61
62
|
|
|
62
|
-
function
|
|
63
|
-
return
|
|
63
|
+
function isOperationModel(value: Record<string, unknown>): boolean {
|
|
64
|
+
return (
|
|
65
|
+
(value.type === "image" && value.api === LITELLM_IMAGES_API) ||
|
|
66
|
+
(value.type === "classifier" && value.api === LITELLM_DECISIONS_API)
|
|
67
|
+
);
|
|
64
68
|
}
|
|
65
69
|
|
|
66
70
|
function isUsableModel(value: unknown, provider: string, baseUrl: string, allowInsecureHttp?: boolean): boolean {
|
|
67
71
|
if (!isRecord(value) || value.provider !== provider) return false;
|
|
68
72
|
if (value.litellmDiscoveryVersion !== LITELLM_DISCOVERY_VERSION) return false;
|
|
69
73
|
try {
|
|
70
|
-
// A stored baseUrl is per protocol -- `/v1` for the OpenAI APIs and
|
|
74
|
+
// A stored baseUrl is per protocol -- `/v1` for the OpenAI APIs, images and classifiers, the bare root for Messages -- so the root
|
|
71
75
|
// goes through the same resolution rather than being compared to it.
|
|
72
|
-
if (
|
|
76
|
+
if (isOperationModel(value)) return value.baseUrl === proxyV1BaseUrl(baseUrl, allowInsecureHttp);
|
|
73
77
|
// `api` indexes LITELLM_PROTOCOLS in resolveModelBaseUrl, so an unknown one is a TypeError, not a bad model.
|
|
74
78
|
return isLiteLLMApi(value.api) && value.baseUrl === resolveModelBaseUrl(baseUrl, value.api, allowInsecureHttp);
|
|
75
79
|
} catch {
|
|
@@ -96,7 +100,7 @@ export async function readStoredModels(options: {
|
|
|
96
100
|
provider: string;
|
|
97
101
|
baseUrl: string;
|
|
98
102
|
allowInsecureHttp?: boolean;
|
|
99
|
-
}): Promise<(LiteLLMModel | LiteLLMImageModel)[] | undefined> {
|
|
103
|
+
}): Promise<(LiteLLMModel | LiteLLMImageModel | LiteLLMClassifierModel)[] | undefined> {
|
|
100
104
|
const entry = (await readStore(options.agentDir))?.[options.provider];
|
|
101
105
|
if (!isRecord(entry) || !Array.isArray(entry.models) || entry.models.length === 0) return undefined;
|
|
102
106
|
if (typeof entry.checkedAt !== "number" || Date.now() - entry.checkedAt >= STORE_FRESH_MS) return undefined;
|
|
@@ -106,10 +110,11 @@ export async function readStoredModels(options: {
|
|
|
106
110
|
if (!usable) return undefined;
|
|
107
111
|
try {
|
|
108
112
|
// Pi's restore path (provider.ts) runs the same call; when it throws, nothing is restored and the seed is the catalog.
|
|
109
|
-
// Image models carry no request policy, so that path restores them as stored.
|
|
110
|
-
return entry.models.map((model) => (
|
|
113
|
+
// Image and classifier models carry no request policy, so that path restores them as stored.
|
|
114
|
+
return entry.models.map((model) => (isOperationModel(model) ? model : enrichCachedModel(model))) as (
|
|
111
115
|
| LiteLLMModel
|
|
112
116
|
| LiteLLMImageModel
|
|
117
|
+
| LiteLLMClassifierModel
|
|
113
118
|
)[];
|
|
114
119
|
} catch {
|
|
115
120
|
return undefined;
|
package/src/provider.ts
CHANGED
|
@@ -3,6 +3,10 @@ import {
|
|
|
3
3
|
type Api,
|
|
4
4
|
type ApiStreamOptions,
|
|
5
5
|
type AssistantImages,
|
|
6
|
+
type ClassifierContext,
|
|
7
|
+
type ClassifierModel,
|
|
8
|
+
type ClassifierOptions,
|
|
9
|
+
type ClassifierResult,
|
|
6
10
|
type Credential,
|
|
7
11
|
createProvider,
|
|
8
12
|
getCurrentTools,
|
|
@@ -17,6 +21,12 @@ import {
|
|
|
17
21
|
type TranscriptContext,
|
|
18
22
|
} from "@earendil-works/pi-ai";
|
|
19
23
|
import { LITELLM_DISCOVERY_VERSION } from "./backend-identity.js";
|
|
24
|
+
import {
|
|
25
|
+
createLiteLLMClassifierApis,
|
|
26
|
+
type DiscoveredClassifierModel,
|
|
27
|
+
LITELLM_DECISIONS_API,
|
|
28
|
+
type LiteLLMClassifierModel,
|
|
29
|
+
} from "./classifiers.js";
|
|
20
30
|
import { enrichCachedModel, normalizeBaseUrl, restoreCachedModelPolicy } from "./discover.js";
|
|
21
31
|
import {
|
|
22
32
|
createLiteLLMImageApis,
|
|
@@ -34,7 +44,7 @@ export type LiteLLMProviderOptions = {
|
|
|
34
44
|
auth: ProviderAuth;
|
|
35
45
|
/** Catalog discovered during activation. Pi's startup refresh never allows network access, so
|
|
36
46
|
* without a seed the provider stays empty until the user opens /model. */
|
|
37
|
-
models?: readonly (LiteLLMModel | LiteLLMImageModel)[];
|
|
47
|
+
models?: readonly (LiteLLMModel | LiteLLMImageModel | LiteLLMClassifierModel)[];
|
|
38
48
|
allowInsecureHttp?: boolean;
|
|
39
49
|
resolveCredentialRoot: (credential?: Credential, requestBaseUrl?: string, apiKey?: string) => string | undefined;
|
|
40
50
|
discover(credential: Credential, signal?: AbortSignal): Promise<DiscoveryResult & { baseUrl?: string }>;
|
|
@@ -49,17 +59,31 @@ function isLiteLLMImageModel(model: AnyModel): model is LiteLLMImageModel {
|
|
|
49
59
|
return model.type === "image" && model.api === LITELLM_IMAGES_API;
|
|
50
60
|
}
|
|
51
61
|
|
|
52
|
-
|
|
62
|
+
function isLiteLLMClassifierModel(model: AnyModel): model is LiteLLMClassifierModel {
|
|
63
|
+
return model.type === "classifier" && model.api === LITELLM_DECISIONS_API;
|
|
64
|
+
}
|
|
65
|
+
|
|
66
|
+
/** Where image and classifier requests go: `<root>/v1`. */
|
|
67
|
+
export function proxyV1BaseUrl(root: string, allowInsecureHttp = false): string {
|
|
53
68
|
return `${normalizeBaseUrl(root, allowInsecureHttp)}/v1`;
|
|
54
69
|
}
|
|
55
70
|
|
|
71
|
+
export function toNativeClassifierModels(
|
|
72
|
+
provider: string,
|
|
73
|
+
baseUrl: string,
|
|
74
|
+
models: readonly DiscoveredClassifierModel[],
|
|
75
|
+
allowInsecureHttp = false,
|
|
76
|
+
): LiteLLMClassifierModel[] {
|
|
77
|
+
return models.map((model) => ({ ...model, provider, baseUrl: proxyV1BaseUrl(baseUrl, allowInsecureHttp) }));
|
|
78
|
+
}
|
|
79
|
+
|
|
56
80
|
export function toNativeImageModels(
|
|
57
81
|
provider: string,
|
|
58
82
|
baseUrl: string,
|
|
59
83
|
models: readonly DiscoveredImageModel[],
|
|
60
84
|
allowInsecureHttp = false,
|
|
61
85
|
): LiteLLMImageModel[] {
|
|
62
|
-
return models.map((model) => ({ ...model, provider, baseUrl:
|
|
86
|
+
return models.map((model) => ({ ...model, provider, baseUrl: proxyV1BaseUrl(baseUrl, allowInsecureHttp) }));
|
|
63
87
|
}
|
|
64
88
|
|
|
65
89
|
export function toNativeModels(
|
|
@@ -145,23 +169,24 @@ function modelRootError(
|
|
|
145
169
|
}
|
|
146
170
|
}
|
|
147
171
|
|
|
148
|
-
function
|
|
149
|
-
model:
|
|
172
|
+
function operationRootError(
|
|
173
|
+
model: { baseUrl: string },
|
|
174
|
+
subject: string,
|
|
150
175
|
active: { root: string; canonical: string; hostname: string },
|
|
151
176
|
allowInsecureHttp = false,
|
|
152
177
|
): Error | undefined {
|
|
153
178
|
let stored: { root: string; canonical: string; hostname: string };
|
|
154
179
|
try {
|
|
155
|
-
stored = proxyRoot(model.baseUrl,
|
|
180
|
+
stored = proxyRoot(model.baseUrl, subject, allowInsecureHttp);
|
|
156
181
|
} catch (error) {
|
|
157
|
-
return error instanceof Error ? error : refreshRequired(
|
|
182
|
+
return error instanceof Error ? error : refreshRequired(`${subject} has an invalid LiteLLM model URL`);
|
|
158
183
|
}
|
|
159
184
|
if (isPlaceholderHost(stored.hostname)) {
|
|
160
|
-
return refreshRequired(
|
|
185
|
+
return refreshRequired(`${subject} uses a placeholder LiteLLM model host`);
|
|
161
186
|
}
|
|
162
187
|
if (stored.canonical !== active.canonical) {
|
|
163
188
|
return refreshRequired(
|
|
164
|
-
|
|
189
|
+
`${subject} has stale LiteLLM model root ${stored.root}; active credentials use ${active.root}`,
|
|
165
190
|
);
|
|
166
191
|
}
|
|
167
192
|
}
|
|
@@ -244,6 +269,7 @@ export function createLiteLLMProvider(options: LiteLLMProviderOptions): Provider
|
|
|
244
269
|
return [
|
|
245
270
|
...toNativeModels(options.id, root, result.models, options.allowInsecureHttp),
|
|
246
271
|
...toNativeImageModels(options.id, root, result.imageModels ?? [], options.allowInsecureHttp),
|
|
272
|
+
...toNativeClassifierModels(options.id, root, result.classifierModels ?? [], options.allowInsecureHttp),
|
|
247
273
|
];
|
|
248
274
|
},
|
|
249
275
|
filterModels(models, credential) {
|
|
@@ -257,17 +283,26 @@ export function createLiteLLMProvider(options: LiteLLMProviderOptions): Provider
|
|
|
257
283
|
// Image models follow the same rule as chat: a cached root that no longer matches the
|
|
258
284
|
// active credential is hidden until a network refresh proves the new proxy's catalog.
|
|
259
285
|
const images = models.filter(isLiteLLMImageModel).flatMap((model) => {
|
|
260
|
-
const error =
|
|
286
|
+
const error = operationRootError(model, "Cached image model", active, options.allowInsecureHttp);
|
|
261
287
|
if (error) {
|
|
262
288
|
reportUnavailable(error.message);
|
|
263
289
|
return [];
|
|
264
290
|
}
|
|
265
|
-
return [{ ...model, baseUrl:
|
|
291
|
+
return [{ ...model, baseUrl: proxyV1BaseUrl(active.root, options.allowInsecureHttp) }];
|
|
266
292
|
});
|
|
267
|
-
|
|
293
|
+
const classifiers = models.filter(isLiteLLMClassifierModel).flatMap((model) => {
|
|
294
|
+
const error = operationRootError(model, "Cached classifier model", active, options.allowInsecureHttp);
|
|
295
|
+
if (error) {
|
|
296
|
+
reportUnavailable(error.message);
|
|
297
|
+
return [];
|
|
298
|
+
}
|
|
299
|
+
return [{ ...model, baseUrl: proxyV1BaseUrl(active.root, options.allowInsecureHttp) }];
|
|
300
|
+
});
|
|
301
|
+
return [...chat, ...images, ...classifiers];
|
|
268
302
|
},
|
|
269
303
|
api: createLiteLLMProtocolApis(),
|
|
270
304
|
images: createLiteLLMImageApis(),
|
|
305
|
+
classifiers: createLiteLLMClassifierApis(),
|
|
271
306
|
});
|
|
272
307
|
const refreshModels = provider.refreshModels;
|
|
273
308
|
const guardedProvider: Provider<LiteLLMApi> = {
|
|
@@ -307,10 +342,10 @@ export function createLiteLLMProvider(options: LiteLLMProviderOptions): Provider
|
|
|
307
342
|
);
|
|
308
343
|
if (!root) throw refreshRequired("Active credentials do not identify a LiteLLM model host");
|
|
309
344
|
const active = activeCredentialRoot(root, options.allowInsecureHttp);
|
|
310
|
-
const error =
|
|
345
|
+
const error = operationRootError(model, "Cached image model", active, options.allowInsecureHttp);
|
|
311
346
|
if (error) throw error;
|
|
312
347
|
return await provider.generateImages!(
|
|
313
|
-
{ ...model, provider: options.id, baseUrl:
|
|
348
|
+
{ ...model, provider: options.id, baseUrl: proxyV1BaseUrl(active.root, options.allowInsecureHttp) },
|
|
314
349
|
context,
|
|
315
350
|
requestOptions,
|
|
316
351
|
);
|
|
@@ -328,23 +363,67 @@ export function createLiteLLMProvider(options: LiteLLMProviderOptions): Provider
|
|
|
328
363
|
return failed;
|
|
329
364
|
}
|
|
330
365
|
},
|
|
366
|
+
classify: async (
|
|
367
|
+
model: ClassifierModel<string>,
|
|
368
|
+
context: ClassifierContext,
|
|
369
|
+
requestOptions?: ClassifierOptions,
|
|
370
|
+
) => {
|
|
371
|
+
try {
|
|
372
|
+
const root = options.resolveCredentialRoot(
|
|
373
|
+
undefined,
|
|
374
|
+
requestOptions?.env?.LITELLM_BASE_URL,
|
|
375
|
+
requestOptions?.apiKey,
|
|
376
|
+
);
|
|
377
|
+
if (!root) throw refreshRequired("Active credentials do not identify a LiteLLM model host");
|
|
378
|
+
const active = activeCredentialRoot(root, options.allowInsecureHttp);
|
|
379
|
+
const error = operationRootError(model, "Cached classifier model", active, options.allowInsecureHttp);
|
|
380
|
+
if (error) throw error;
|
|
381
|
+
return await provider.classify!(
|
|
382
|
+
{ ...model, provider: options.id, baseUrl: proxyV1BaseUrl(active.root, options.allowInsecureHttp) },
|
|
383
|
+
context,
|
|
384
|
+
requestOptions,
|
|
385
|
+
);
|
|
386
|
+
} catch (error) {
|
|
387
|
+
// `Provider.classify` never rejects.
|
|
388
|
+
const failed: ClassifierResult = {
|
|
389
|
+
api: model.api,
|
|
390
|
+
provider: model.provider,
|
|
391
|
+
model: model.id,
|
|
392
|
+
answers: {},
|
|
393
|
+
stopReason: requestOptions?.signal?.aborted ? "aborted" : "error",
|
|
394
|
+
errorMessage: error instanceof Error ? error.message : String(error),
|
|
395
|
+
timestamp: Date.now(),
|
|
396
|
+
};
|
|
397
|
+
return failed;
|
|
398
|
+
}
|
|
399
|
+
},
|
|
331
400
|
};
|
|
332
401
|
if (!refreshModels) return guardedProvider;
|
|
333
|
-
|
|
402
|
+
// A route can be published as both a chat model and a classifier (GPT-6 Luna lists `/v1/decisions`
|
|
403
|
+
// next to its chat endpoints), so the type keeps those seeds apart.
|
|
404
|
+
const seedKey = (model: { type?: string; id: string; baseUrl: string }) =>
|
|
405
|
+
`${model.type ?? "chat"}\0${model.id}\0${model.baseUrl}`;
|
|
334
406
|
const seeded = new Map((options.models ?? []).map((model) => [seedKey(model), model]));
|
|
335
407
|
return {
|
|
336
408
|
...guardedProvider,
|
|
337
409
|
refreshModels: async (context) => {
|
|
338
410
|
// Pi 0.99 persists models of every type; the policy helpers below read chat-only fields, so
|
|
339
|
-
// chat and
|
|
411
|
+
// chat, image and classifier entries are restored separately. Other types are not this provider's.
|
|
340
412
|
const storedModels = (context.stored?.models ?? []).filter(isChatModel);
|
|
341
|
-
|
|
342
|
-
|
|
343
|
-
|
|
344
|
-
|
|
345
|
-
|
|
346
|
-
|
|
347
|
-
|
|
413
|
+
// Image and classifier entries carry no request policy to restore; a stale one is only kept
|
|
414
|
+
// when startup discovery proved the same route, otherwise the network refresh replaces it.
|
|
415
|
+
const restoreOperationModels = <T extends AnyModel>(is: (model: AnyModel) => model is T): T[] =>
|
|
416
|
+
(context.stored?.models ?? []).filter(is).flatMap((model) => {
|
|
417
|
+
if ((model as { litellmDiscoveryVersion?: number }).litellmDiscoveryVersion === LITELLM_DISCOVERY_VERSION) {
|
|
418
|
+
return [model];
|
|
419
|
+
}
|
|
420
|
+
const fresh = seeded.get(seedKey(model));
|
|
421
|
+
return fresh && is(fresh) ? [fresh] : [];
|
|
422
|
+
});
|
|
423
|
+
const storedOperationModels = [
|
|
424
|
+
...restoreOperationModels(isLiteLLMImageModel),
|
|
425
|
+
...restoreOperationModels(isLiteLLMClassifierModel),
|
|
426
|
+
];
|
|
348
427
|
let legacyCount = 0;
|
|
349
428
|
const models = storedModels.map((model) => {
|
|
350
429
|
if ((model as LiteLLMModel).litellmDiscoveryVersion !== LITELLM_DISCOVERY_VERSION) {
|
|
@@ -362,7 +441,7 @@ export function createLiteLLMProvider(options: LiteLLMProviderOptions): Provider
|
|
|
362
441
|
...context,
|
|
363
442
|
// The current pi-ai provider has no freshness gate, but force is the documented refresh contract field.
|
|
364
443
|
force: context.force || (context.allowNetwork && legacyCount > 0),
|
|
365
|
-
stored: context.stored && { ...context.stored, models: [...models, ...
|
|
444
|
+
stored: context.stored && { ...context.stored, models: [...models, ...storedOperationModels] },
|
|
366
445
|
});
|
|
367
446
|
} catch (error) {
|
|
368
447
|
if (context.allowNetwork && legacyCount > 0) {
|
package/src/types.ts
CHANGED
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
import type { Model } from "@earendil-works/pi-ai";
|
|
2
2
|
import type { BackendFamily, LITELLM_DISCOVERY_VERSION } from "./backend-identity.js";
|
|
3
|
+
import type { DiscoveredClassifierModel } from "./classifiers.js";
|
|
3
4
|
import type { DiscoveredImageModel } from "./images.js";
|
|
4
5
|
import type { ProxyVersion } from "./proxy-version.js";
|
|
5
6
|
|
|
@@ -67,6 +68,11 @@ export interface DiscoveryResult {
|
|
|
67
68
|
models: DiscoveredModel[];
|
|
68
69
|
/** `mode: "image_generation"` routes from `/model/info`; absent when there are none. */
|
|
69
70
|
imageModels?: DiscoveredImageModel[];
|
|
71
|
+
/**
|
|
72
|
+
* Decision routes from `/model/info` (`mode: "evaluation"`/`"decisions"`, or a decision endpoint in
|
|
73
|
+
* `supported_endpoints`); absent when there are none.
|
|
74
|
+
*/
|
|
75
|
+
classifierModels?: DiscoveredClassifierModel[];
|
|
70
76
|
source: DiscoverySource;
|
|
71
77
|
// Present only when a `/model/info` discovery had to read it and the proxy
|
|
72
78
|
// answered. Absent means unknown, not old: discovery asks only when it must.
|
|
@@ -82,6 +88,9 @@ export interface DiscoveryOptions {
|
|
|
82
88
|
// path, models.dev is not consulted at all.
|
|
83
89
|
modelsDev?: boolean;
|
|
84
90
|
modelsDevCachePath?: string;
|
|
91
|
+
// Pi `models.json` `providers.<providerId>.modelOverrides.<modelId>` limits. Pi applies them to that
|
|
92
|
+
// provider's own models only, so discovery applies them to routes backed by the same catalog model.
|
|
93
|
+
modelOverrides?: Readonly<Record<string, Readonly<Record<string, { contextWindow?: number; maxTokens?: number }>>>>;
|
|
85
94
|
}
|
|
86
95
|
|
|
87
96
|
export interface ModelInfoEntry {
|