pi-provider-litellm 4.4.2 → 4.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -375,6 +375,38 @@ and groups that mix image and other modes, or wildcard routes, are withheld. The
375
375
  need `/v1/images/edits`, which this provider does not call, and backends that answer with image URLs instead of
376
376
  base64 data are reported as errors rather than fetched.
377
377
 
378
+ ## Classification
379
+
380
+ Decision routes are published as Pi classifier models (for example `litellm/decision-1`, backed by
381
+ `azure_ai/Microsoft-Decision-1`, or a `typesafe/jev-latest` route). A route qualifies when every deployment behind it
382
+ reports `mode: "evaluation"` or `"decisions"` in `/model/info`, or lists `/v1/systemone` or `/v1/decisions` in
383
+ `supported_endpoints`; a declared endpoint list without them withholds the route. Like Pi's other classifiers they do
384
+ not appear in `/model`; scripts reach them through Pi's `codemode` tool:
385
+
386
+ ```js
387
+ const judge = await models.getModelOfType("classifier", "litellm", "decision-1");
388
+ const result = await models.classify(judge, {
389
+ state: { text: "Please send me the report by Friday." },
390
+ questions: {
391
+ polite: {
392
+ type: "bool",
393
+ instructions: "Is the message polite?",
394
+ criteria: { true: "The message is polite.", false: "The message is rude." },
395
+ },
396
+ },
397
+ });
398
+ if (result.stopReason !== "stop") return result.errorMessage;
399
+ return result.answers.polite.probability;
400
+ ```
401
+
402
+ Question types are `bool`, `choice` and `score`. Requests go to the proxy's `/v1/systemone` (LiteLLM v1.104.2 and
403
+ later) with the provider's own credential and headers, and are budgeted and logged by the proxy like chat requests. A
404
+ `bool` question is sent as System One's `noul`. Usage cost comes from LiteLLM's `x-litellm-response-cost` header when
405
+ present; otherwise it is estimated from `input_cost_per_token` and `output_cost_per_token`. Only routes found through
406
+ `/model/info` qualify, and groups that mix decision and other deployments, or wildcard routes, are withheld. The state
407
+ is text only: images in the context return an error without a request. A chat route that also lists `/v1/decisions`,
408
+ such as OpenAI's GPT-6 Luna, is published as both a chat model and a classifier.
409
+
378
410
  ## Optional environment variables
379
411
 
380
412
  | Variable | Default | Effect |
@@ -393,6 +425,7 @@ base64 data are reported as errors rather than fetched.
393
425
  | `LITELLM_CLI_JWT_EXPIRATION_HOURS` | `24` | CLI SSO token lifetime fallback for older proxies whose poll response omits `expires_in`; mirror a non-default proxy setting locally |
394
426
  | `LITELLM_VERBOSE_DISCOVERY` | unset | If `1`, enable progress messages during model discovery (login, refresh, startup), including startup skip reasons and defaulted `/model/info` context windows. Progress messages are off by default |
395
427
  | `LITELLM_DEFAULT_CONTEXT_WINDOW` | `128000` | Context window assumed when neither LiteLLM's `/model/info` nor the catalog reports `max_input_tokens`. Set a positive integer for proxies whose custom aliases carry no metadata; an unset or unusable value keeps 128K |
428
+ | `LITELLM_TRUST_PROXY_LIMITS` | unset | If `1`, a deployment's explicit `model_info.max_input_tokens` / `max_output_tokens` win over Pi's catalog instead of taking the smaller of the two; the catalog still fills missing values and the group minimum still applies. A `models.json` `modelOverrides` entry still wins first |
396
429
  | `LITELLM_MODELS_DEV` | unset | If `1`, enrich discovered metadata (limits, prices, effort lists) from models.dev, cached for 28 days in `litellm-models-dev.json`. Off by default because LiteLLM's `/model/info` is authoritative; use it when LiteLLM's model map lacks a model's metadata |
397
430
 
398
431
  Only use a trusted `LITELLM_API_KEY_HELPER` or `!command`. Prefer an absolute executable path, keep secrets out of command arguments, and print only the token to stdout without logging it to stderr.
@@ -512,7 +545,7 @@ Each distinct availability diagnostic is written once per session on stderr. `LI
512
545
 
513
546
  Discovery selects native Messages for proven Claude groups under the [Model transport](#model-transport) rules, Responses for OpenAI-family backends, and Chat Completions for other OpenAI-compatible backends. Azure deployments stay on Chat Completions unless an explicit Responses mode or a `supported_endpoints` list containing `/v1/responses` authorizes Responses; `api_version` alone does not prove support. Azure AI (`azure_ai/` prefix or `custom_llm_provider: azure_ai`) deployments additionally depend on the proxy version. On LiteLLM before v1.103.0, or when the version is unknown, they ignore a `supported_endpoints` list that contains `/v1/responses` and need an explicit `mode: "responses"`: those releases have no native Responses config for that provider, so they bridge `/v1/responses` through Chat Completions, and the bridge fails streaming with `list index out of range` on Azure's empty-choices chunks (BerriAI/litellm#34455) while the endpoint list, copied from the model cost map, still advertises `/v1/responses`. From v1.103.0 the list is honored again. Discovery reads the version from the `x-litellm-version` header of one extra request, `GET /v1/responses/resp_version_probe`, sent only when such a deployment is published; LiteLLM answers it with an error and calls no model. A pre-release counts as older than the release it names, and a proxy that omits the header keeps these deployments on Chat Completions. Azure is detected from the adapter field, an `azure/` or `azure_ai/` model prefix, or the reported provider; other adapters follow the backend family because LiteLLM bridges `/v1/responses` to Chat Completions when the provider has no native Responses config (`litellm/responses/main.py`, `_bridges_to_chat_completions`). The family comes from the model id, not from LiteLLM's `openai/`, `openai_like/` or `custom_openai/` adapter prefix, so `openai/qwen3` stays on Chat Completions while `openai/gpt-5.5` uses Responses; a deployment opts into Responses with `mode: "responses"` or a `supported_endpoints` list containing `/v1/responses`. A route whose model prefix and `custom_llm_provider` name different providers is treated as unidentified and stays on Chat Completions, so it also does not get the local 128-tool preflight. When `/model/info` supplies `supported_endpoints`, that list takes precedence, and `mode: "responses"` remains an explicit Responses signal. An evidence-free fallback entry—a bare `/v1/models` id or a `/health` entry with no detail row—takes its protocol from the Pi catalog entry for its id; both evidence-free fallback paths also inherit presentation metadata from the bounded catalog lookup. An unknown id stays on Chat Completions. The route name alone authorizes nothing. A concrete id expanded from wildcard `/model/info` routes inherits the deployment evidence of the wildcard route LiteLLM would select for it, with the requested id substituted into the row's `model` the way LiteLLM serves it; the id is omitted when that selected route is not a published chat-style route. Chat Completions and Responses use `<root>/v1`; native Messages uses the proxy root directly. All three protocols pass through the provider's host guard. When an evidence-free cached `(no metadata)` entry can be enriched from the Pi catalog, it is restored on Responses only for a catalog Responses model and otherwise on Chat Completions, never Anthropic Messages. A model supplied only through `models.json` whose `api` this provider does not implement takes Pi's global API fallback rather than this provider's host guard; the credential-root pin (see [Model host enforcement](#model-host-enforcement)) still keeps the credential on the active proxy.
514
547
 
515
- `cacheControlFormat: "anthropic"` applies only to Chat Completions, where Pi adds Anthropic `cache_control` markers. The Responses transport has a different compatibility type and uses native `prompt_cache_key` (plus supported retention fields), so Responses models must not receive `cacheControlFormat`. Freshly discovered OpenAI-family models forced onto Chat Completions are rejected locally when a request contains more than 128 tools; route them through Responses or reduce enabled extensions. Moonshot request-side reasoning suppression follows LiteLLM's routing provider, not `model_info.base_model`; response-side `<think>` normalization follows discovered Kimi-family evidence. Opaque aliases retain display normalization without sending Moonshot-only parameters to Azure- or Bedrock-hosted Kimi models. Strict tool-message repair requires every deployment to identify Kimi; Gemini effort normalization likewise requires consistent backend-family evidence.
548
+ `cacheControlFormat: "anthropic"` applies only to Chat Completions, where Pi adds Anthropic `cache_control` markers. The Responses transport has a different compatibility type and uses native `prompt_cache_key` (plus supported retention fields), so Responses models must not receive `cacheControlFormat`. Freshly discovered OpenAI-family models forced onto Chat Completions are rejected locally when a request contains more than 128 tools; route them through Responses or reduce enabled extensions. Moonshot request-side reasoning suppression follows LiteLLM's routing provider, not `model_info.base_model`; response-side `<think>` normalization follows discovered Kimi-family evidence. Opaque aliases retain display normalization without sending Moonshot-only parameters to Azure- or Bedrock-hosted Kimi models. Strict tool-message repair requires every deployment to identify Kimi; Gemini effort normalization likewise requires consistent backend-family evidence. Claude-backed routes on native Messages or Chat Completions declare Pi's `promptCache` lifetimes (`short: 300`, `long: 3600` seconds, the values Pi's built-in Anthropic models use), so Pi's `cacheWarming` can keep their caches warm. Responses routes, routes identified only by their name, and other backend families declare none, so Pi does not warm them. A `modelOverrides` entry for the route in the global `models.json` can change the values, and `cacheWarming: "off"` disables warming.
516
549
 
517
550
  Discovery policy is versioned in Pi's persisted model catalog. After upgrading, an online refresh replaces legacy entries. Offline legacy entries remain usable without a diagnostic; the extension reports the discovery-version mismatch once only when an attempted online refresh fails. To force rediscovery or roll back across this policy change, delete the `litellm` entry from `~/.pi/agent/models-store.json` (or delete the file) and refresh once online.
518
551
 
@@ -522,7 +555,19 @@ LiteLLM may load-balance one public `model_name` across deployments with differe
522
555
 
523
556
  - Responses is selected only when every deployment supports it under the protocol rules above; a deployment that requires Chat keeps the group on Chat.
524
557
  - Vision and reasoning are advertised only when every routable deployment resolves them as supported. An explicit `supports_reasoning: false` suppresses all thinking controls, even when catalog or router effort metadata exists. Catalog thinking controls are intersected per level: a level is exposed only when every deployment maps it to the same value, while disagreement or absence becomes an explicit denial. Catalog maps are not complete effort lists, so absence has one exception: an omitted `minimal`, `low`, `medium` or `high` counts as Pi's default spelling of that level. An omitted `off`, `xhigh` or `max` still differs from an explicit mapping. Reasoning selectors additionally require an accepted wire carrier on every deployment. Public effort lists are intersected, explicit LiteLLM denials win, and `xhigh`/`max` need unanimous explicit support. Null or absent standard-effort flags have no opinion. A deployment that declares `model_info.reasoning_effort_levels` answers every level from that list, ahead of its per-level flags, as LiteLLM itself reads it.
525
- - Context and output limits use the minimum resolved value across deployments. Pi catalogue lookup prefers the deployment's provider-specific entry before falling back to the generic vendor entry; Azure and Azure AI therefore retain Azure-specific limits rather than being capped by a different OpenAI catalogue entry. When models.dev supplies a sparse vendor record, missing fields still use the provider-specific Pi catalogue.
558
+ - Context and output limits use the minimum resolved value across deployments. Per deployment, a Pi `models.json` `modelOverrides` entry for the identified backend model (Pi provider id plus model id, resolved from backend identity, never from the route name; the provider id is the one whose Pi catalog entry identifies the backend, such as `openai-codex` for a `chatgpt/` deployment, not the LiteLLM provider label) sets `contextWindow` / `maxTokens` ahead of both the router figure and the catalog; it applies only under unanimous catalog authority, and the group minimum still applies. Without an override, the smaller of the router's explicit figure and the catalog figure is used, or the router's figure first under `LITELLM_TRUST_PROXY_LIMITS=1`. Overrides are read from the global `models.json` only and take effect at the next discovery (`/litellm-refresh` or cache expiry). Pi applies `modelOverrides` only to that provider's own models, so one entry covers every LiteLLM route backed by the model:
559
+
560
+ ```json
561
+ {
562
+ "providers": {
563
+ "openai": {
564
+ "modelOverrides": { "gpt-5.2": { "contextWindow": 500000 } }
565
+ }
566
+ }
567
+ }
568
+ ```
569
+
570
+ Pi catalogue lookup prefers the deployment's provider-specific entry before falling back to the generic vendor entry; Azure and Azure AI therefore retain Azure-specific limits rather than being capped by a different OpenAI catalogue entry. When models.dev supplies a sparse vendor record, missing fields still use the provider-specific Pi catalogue.
526
571
  - Each displayed base-price field uses the maximum only when every deployment resolves that field; unresolved fields remain zero and the model name is suffixed with ` (incomplete metadata)`.
527
572
  - When complete base pricing comes from the catalog under unanimous catalog authority, tiered pricing is the conservative worst-case envelope. Its usable finite non-negative thresholds are the sorted union of every deployment's thresholds, and each price field at each interval is the maximum applicable rate across all deployments, floored by each deployment's base rate. Differing tier breakpoints are therefore preserved rather than treated as incompatible. An explicit `/model/info` base-price field replaces catalog pricing for that field at every threshold; catalog tiers remain for unaffected fields and are omitted only when every base-price field is explicit.
528
573
  - Reduced route groups omit tier ladders when complete base-pricing or catalog authority is unavailable. Wildcard expansion retains every known tier from matching groups even when another match has incomplete metadata: suppressing a complete sibling's higher tier could understate the known worst-case rate. The ` (incomplete metadata)` marker remains because the resulting envelope is only as complete as the available ladders.
@@ -563,6 +608,7 @@ The development probe runs against minimized snapshots with `npm run probe:proxy
563
608
  | `LiteLLM discovery: ... route group(s) mix chat-style and explicitly incompatible deployment modes` | A public route can select both a chat-style deployment and an incompatible deployment such as an embedding model. Give the embedding deployment its own `model_name`. |
564
609
  | `LiteLLM (...): the request for "<route>" was answered by a fallback` | A successful HTTP response reports a LiteLLM router fallback (`x-litellm-attempted-fallbacks`). `/model/info` does not expose fallbacks, so protocol and model handling follow the requested route's own deployments. If the primary and fallback groups support Chat Completions, set `model_info.supported_endpoints: ["/v1/chat/completions"]` on the primary to avoid Responses-to-Chat bridging. This warning does not change routing or fix upstream bridging defects. It names only the requested route; LiteLLM reports the group and model that answered in the `x-litellm-model-group` and `x-litellm-model-name` response headers. Warned once per provider and route until the extension reloads; shown in the UI, or stderr without a UI. |
565
610
  | A model is marked ` (incomplete metadata)` | `/model/info` or `/health` did not provide enough authoritative metadata. Explicit fields remain usable, but unknown cost fields are shown as zero and route-name cache enrichment stays disabled. |
611
+ | Cache miss notices after idle, no cache warming | The model declares no `promptCache`: only Claude-backed Messages and Chat routes get one. Run `/litellm-refresh` after updating so stored models are rediscovered. |
566
612
  | `401 Token expired` | Set `LITELLM_API_KEY_HELPER`. |
567
613
  | No models with gcloud auth | Verify `gcloud auth application-default login` has been run or set `GOOGLE_APPLICATION_CREDENTIALS` to an `authorized_user` ADC file |
568
614
  | Enterprise SSO waits for token insertion | The proxy returned 404/405 for `/sso/cli/start`, so Pi used the legacy flow — upgrade LiteLLM or paste the UI token |
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "pi-provider-litellm",
3
- "version": "4.4.2",
3
+ "version": "4.5.0",
4
4
  "description": "LiteLLM proxy provider extension for Pi",
5
5
  "type": "module",
6
6
  "license": "MIT",
@@ -1,4 +1,4 @@
1
- export const LITELLM_DISCOVERY_VERSION = 7 as const;
1
+ export const LITELLM_DISCOVERY_VERSION = 9 as const;
2
2
 
3
3
  export type BackendFamily = "claude" | "deepseek" | "gemini" | "kimi" | "openai";
4
4
 
@@ -0,0 +1,174 @@
1
+ import type { ClassifierAnswer, ClassifierModel, ClassifierResult, ProviderClassifier } from "@earendil-works/pi-ai";
2
+ import { LITELLM_DISCOVERY_VERSION } from "./backend-identity.js";
3
+ import { errorDetail, parseUsage, perMillion, requestHeaders, requestSignal, responseCost } from "./images.js";
4
+ import { defaultContextWindow, wireString } from "./model-groups.js";
5
+ import type { ModelInfoEntry } from "./types.js";
6
+
7
+ /** Decision models through the proxy's System One `/v1/systemone` endpoint. */
8
+ export const LITELLM_DECISIONS_API = "litellm-decisions" as const;
9
+ export type LiteLLMDecisionsApi = typeof LITELLM_DECISIONS_API;
10
+
11
+ export type DiscoveredClassifierModel = Omit<ClassifierModel<LiteLLMDecisionsApi>, "provider" | "baseUrl"> & {
12
+ litellmDiscoveryVersion?: typeof LITELLM_DISCOVERY_VERSION;
13
+ };
14
+
15
+ export type LiteLLMClassifierModel = ClassifierModel<LiteLLMDecisionsApi> & {
16
+ litellmDiscoveryVersion?: typeof LITELLM_DISCOVERY_VERSION;
17
+ };
18
+
19
+ const DECISION_MODES = new Set(["evaluation", "decisions"]);
20
+ const DECISION_ENDPOINTS = new Set(["/v1/systemone", "/v1/decisions"]);
21
+
22
+ /**
23
+ * LiteLLM's own rule: a declared `supported_endpoints` list decides alone, so a list without a
24
+ * decision endpoint denies even with an evaluation mode; without a list, the mode decides.
25
+ */
26
+ export function isDecisionDeployment(entry: ModelInfoEntry): boolean {
27
+ const info = entry.model_info;
28
+ const endpoints: unknown = info?.supported_endpoints;
29
+ if (Array.isArray(endpoints)) {
30
+ return endpoints.some((endpoint) => typeof endpoint === "string" && DECISION_ENDPOINTS.has(endpoint.trim()));
31
+ }
32
+ const mode = wireString(info?.mode);
33
+ return mode !== undefined && DECISION_MODES.has(mode.trim().toLowerCase());
34
+ }
35
+
36
+ /**
37
+ * Reduce one `/model/info` route group to a classifier model. Every deployment behind the route
38
+ * must be a decision deployment: LiteLLM load-balances across them, so a mixed group could answer
39
+ * from a non-decision deployment.
40
+ */
41
+ export function mapFromClassifierModelInfoGroup(
42
+ route: string,
43
+ entries: readonly ModelInfoEntry[],
44
+ ): DiscoveredClassifierModel | undefined {
45
+ // A wildcard row is not addressable, and `/v1/models` expansion carries no mode evidence.
46
+ if (route.includes("*")) return undefined;
47
+ const deployments = entries.filter((entry) => wireString(entry.model_name) === route);
48
+ if (deployments.length === 0) return undefined;
49
+ if (!deployments.every(isDecisionDeployment)) return undefined;
50
+ // Prices are estimates for Pi's usage display; the `x-litellm-response-cost` header replaces them.
51
+ const rate = (read: (entry: ModelInfoEntry) => number | undefined): number => {
52
+ const values = deployments.map(read);
53
+ return values.every((value) => value !== undefined) ? Math.max(...(values as number[])) : 0;
54
+ };
55
+ // The smallest declared window; an undeclared one counts as the default so it never raises the result.
56
+ const windows = deployments.map((entry) => entry.model_info?.max_input_tokens);
57
+ const declared = windows.filter(
58
+ (value): value is number => typeof value === "number" && Number.isFinite(value) && value > 0,
59
+ );
60
+ const contextWindow = Math.min(...declared, ...(declared.length < windows.length ? [defaultContextWindow()] : []));
61
+ return {
62
+ type: "classifier",
63
+ id: route,
64
+ name: route,
65
+ api: LITELLM_DECISIONS_API,
66
+ input: ["text"],
67
+ cost: {
68
+ input: rate((entry) => perMillion(entry.model_info?.input_cost_per_token)),
69
+ output: rate((entry) => perMillion(entry.model_info?.output_cost_per_token)),
70
+ cacheRead: 0,
71
+ cacheWrite: 0,
72
+ },
73
+ contextWindow,
74
+ litellmDiscoveryVersion: LITELLM_DISCOVERY_VERSION,
75
+ };
76
+ }
77
+
78
+ function isRecord(value: unknown): value is Record<string, unknown> {
79
+ return typeof value === "object" && value !== null && !Array.isArray(value);
80
+ }
81
+
82
+ const isFiniteNumber = (value: unknown): value is number => typeof value === "number" && Number.isFinite(value);
83
+
84
+ function parseAnswer(model: string, id: string, type: "choice" | "score" | "bool", raw: unknown): ClassifierAnswer {
85
+ const wireType = type === "bool" ? "noul" : type;
86
+ if (!isRecord(raw) || raw.type !== wireType) {
87
+ throw new Error(`LiteLLM classifier ${model} did not return a ${type} answer for ${id}`);
88
+ }
89
+ const invalid = (field: string): Error =>
90
+ new Error(`LiteLLM classifier ${model} returned an invalid ${field} for ${id}`);
91
+ if (type === "bool") {
92
+ if (!isFiniteNumber(raw.noul)) throw invalid("noul");
93
+ return { type: "bool", probability: raw.noul };
94
+ }
95
+ if (!isFiniteNumber(raw.confidence)) throw invalid("confidence");
96
+ if (type === "score") {
97
+ if (!isFiniteNumber(raw.score)) throw invalid("score");
98
+ return { type: "score", score: raw.score, confidence: raw.confidence };
99
+ }
100
+ if (typeof raw.choice !== "string") throw invalid("choice");
101
+ if (!isRecord(raw.probabilities) || !Object.values(raw.probabilities).every(isFiniteNumber)) {
102
+ throw invalid("probabilities");
103
+ }
104
+ return {
105
+ type: "choice",
106
+ choice: raw.choice,
107
+ probabilities: raw.probabilities as Record<string, number>,
108
+ confidence: raw.confidence,
109
+ };
110
+ }
111
+
112
+ /** Classify through LiteLLM's `/v1/systemone`. Never rejects. */
113
+ export const classify: ProviderClassifier["classify"] = async (model, context, options) => {
114
+ const result: ClassifierResult = {
115
+ api: model.api,
116
+ provider: model.provider,
117
+ model: model.id,
118
+ answers: {},
119
+ stopReason: "stop",
120
+ timestamp: Date.now(),
121
+ };
122
+ try {
123
+ const apiKey = options?.apiKey;
124
+ if (!apiKey) throw new Error(`No API key for provider: ${model.provider}`);
125
+ if (context.images?.length) {
126
+ throw new Error(`LiteLLM classifier ${model.id} takes text state only; images are not sent to /v1/systemone`);
127
+ }
128
+ // System One calls Pi's `bool` question `noul`; the fields are otherwise the same.
129
+ const questions = Object.fromEntries(
130
+ Object.entries(context.questions).map(([id, question]) => [
131
+ id,
132
+ question.type === "bool" ? { ...question, type: "noul" } : question,
133
+ ]),
134
+ );
135
+ let params: Record<string, unknown> = { model: model.id, state: context.state, questions };
136
+ const nextParams = await options?.onPayload?.(params, model);
137
+ if (nextParams !== undefined) params = nextParams as Record<string, unknown>;
138
+ const response = await (options?.fetch ?? fetch)(`${model.baseUrl}/systemone`, {
139
+ method: "POST",
140
+ headers: requestHeaders(apiKey, model.headers, options?.headers),
141
+ body: JSON.stringify(params),
142
+ signal: requestSignal(options),
143
+ });
144
+ const responseHeaders = Object.fromEntries(response.headers.entries());
145
+ await options?.onResponse?.({ status: response.status, headers: responseHeaders }, model);
146
+ const body = await response.text();
147
+ if (!response.ok) throw new Error(`LiteLLM decisions request failed (${response.status}): ${errorDetail(body)}`);
148
+ // The proxy billed the request whatever the body holds; token counts are added once it parses.
149
+ const cost = responseCost(responseHeaders);
150
+ const billed = parseUsage(undefined, model.cost, cost);
151
+ if (billed) result.usage = billed;
152
+ const parsed: unknown = JSON.parse(body);
153
+ if (!isRecord(parsed)) throw new Error(`LiteLLM classifier ${model.id} returned an unexpected response`);
154
+ // A malformed answer set was still billed.
155
+ const usage = parseUsage(isRecord(parsed.usage) ? parsed.usage : undefined, model.cost, cost);
156
+ if (usage) result.usage = usage;
157
+ const answers = isRecord(parsed.answers) ? parsed.answers : {};
158
+ const parsedAnswers: Record<string, ClassifierAnswer> = {};
159
+ for (const [id, question] of Object.entries(context.questions)) {
160
+ parsedAnswers[id] = parseAnswer(model.id, id, question.type, answers[id]);
161
+ }
162
+ result.answers = parsedAnswers;
163
+ return result;
164
+ } catch (error) {
165
+ result.answers = {};
166
+ result.stopReason = options?.signal?.aborted ? "aborted" : "error";
167
+ result.errorMessage = error instanceof Error ? error.message : String(error);
168
+ return result;
169
+ }
170
+ };
171
+
172
+ export function createLiteLLMClassifierApis(): Record<LiteLLMDecisionsApi, ProviderClassifier> {
173
+ return { [LITELLM_DECISIONS_API]: { classify } };
174
+ }
package/src/discover.ts CHANGED
@@ -8,6 +8,7 @@ import {
8
8
  resolveCatalogProvider,
9
9
  routesOnlyThrough,
10
10
  } from "./backend-identity.js";
11
+ import { type DiscoveredClassifierModel, mapFromClassifierModelInfoGroup } from "./classifiers.js";
11
12
  import { type DiscoveredImageModel, mapFromImageModelInfoGroup } from "./images.js";
12
13
  import {
13
14
  type CatalogResolution,
@@ -50,6 +51,10 @@ import type {
50
51
  } from "./types.js";
51
52
 
52
53
  const DEFAULT_TIMEOUT_MS = 5000;
54
+ // Anthropic's documented 5-minute and 1-hour cache lifetimes; the values Pi's built-in Anthropic catalog uses.
55
+ // shortcut: `long` assumes the host offers the 1-hour TTL (Bedrock only does for Claude 4.5+), gate it on
56
+ // deployment routing if LiteLLM ever reports per-route TTL support.
57
+ const CLAUDE_PROMPT_CACHE = { short: 300, long: 3600 } as const;
53
58
  const HEALTH_DETAIL_CONCURRENCY = 8;
54
59
  const reportedConflictingFamilyRoutes = new Set<string>();
55
60
  const reportedWithheldRepairRoutes = new Set<string>();
@@ -496,6 +501,7 @@ function adaptPublicCatalogRecord(
496
501
  const piCost = piCatalog?.cost;
497
502
  return {
498
503
  provider,
504
+ ...(resolved ? { piProvider: resolved.provider } : {}),
499
505
  // A models.dev hit names the model canonically even when Pi's catalog does not know it yet,
500
506
  // so two spellings of one backend model reduce to one identity instead of a conflict.
501
507
  catalogModelId: resolved?.model.id ?? record?.modelId ?? catalogModelId,
@@ -689,10 +695,16 @@ function mapFromModelInfoGroup(
689
695
  denyLevels?: boolean;
690
696
  allowMessages?: boolean;
691
697
  proxyVersion?: ProxyVersion;
698
+ modelOverrides?: DiscoveryOptions["modelOverrides"];
692
699
  } = {},
693
700
  ): DiscoveredModel | undefined {
694
701
  const reduced = reduceModelGroup(entries, (entry) => {
695
- const catalog = resolveModelInfoCatalog(entry, publicCatalog);
702
+ const resolved = resolveModelInfoCatalog(entry, publicCatalog);
703
+ const limitOverrides =
704
+ resolved?.piProvider && resolved.catalogModelId
705
+ ? options.modelOverrides?.[resolved.piProvider]?.[resolved.catalogModelId]
706
+ : undefined;
707
+ const catalog = limitOverrides ? { ...resolved, limitOverrides } : resolved;
696
708
  const family = deploymentFamily(entry);
697
709
  const generations = [entry.litellm_params?.model, entry.model_info?.base_model]
698
710
  .map((id) => wireString(id)?.trim())
@@ -756,6 +768,10 @@ function mapFromModelInfoGroup(
756
768
  acceptsResponsesReasoningControl: reduced.acceptsResponsesReasoningControl,
757
769
  denyLevels: options.denyLevels,
758
770
  });
771
+ const relaysAnthropicCacheMarkers =
772
+ api === "anthropic-messages" ||
773
+ (api === "openai-completions" &&
774
+ (policy.compat as Model<"openai-completions">["compat"])?.cacheControlFormat === "anthropic");
759
775
  const hasConflictingFamily = reduced.deploymentFamilies.includes("conflicting");
760
776
  const unanimousMoonshot =
761
777
  reduced.deploymentFamilies.length > 0 && reduced.deploymentFamilies.every((family) => family === "kimi");
@@ -802,6 +818,9 @@ function mapFromModelInfoGroup(
802
818
  ? { litellmResponsesReasoningControl: true as const }
803
819
  : {}),
804
820
  ...(families.size === 1 && family ? { litellmBackendFamily: family } : {}),
821
+ ...(families.size === 1 && family === "claude" && relaysAnthropicCacheMarkers
822
+ ? { promptCache: { ...CLAUDE_PROMPT_CACHE } }
823
+ : {}),
805
824
  ...(modelPolicy ? { litellmPolicy: modelPolicy } : {}),
806
825
  // closeSerializerPolicy derives compat from `api`, a correlation its return type cannot carry.
807
826
  } as DiscoveredModel;
@@ -930,6 +949,7 @@ async function discoverFromHealth(
930
949
  denyLevels: group.some(({ denyLevels }) => denyLevels),
931
950
  allowMessages: false,
932
951
  proxyVersion,
952
+ modelOverrides: options.modelOverrides,
933
953
  });
934
954
  })
935
955
  .filter((model): model is DiscoveredModel => model !== undefined);
@@ -1036,10 +1056,11 @@ function applyWildcardEvidence(
1036
1056
  publishedWildcardIds: ReadonlySet<string>,
1037
1057
  publicCatalog: PublicCatalog | undefined,
1038
1058
  proxyVersion: ProxyVersion | undefined,
1059
+ modelOverrides: DiscoveryOptions["modelOverrides"],
1039
1060
  ): DiscoveredModel | undefined {
1040
1061
  const parents = selectedWildcardRows(model.id, wildcardRows, publishedWildcardIds);
1041
1062
  if (!parents) return undefined;
1042
- const selected = mapFromModelInfoGroup(parents, publicCatalog, { proxyVersion });
1063
+ const selected = mapFromModelInfoGroup(parents, publicCatalog, { proxyVersion, modelOverrides });
1043
1064
  if (!selected) return undefined;
1044
1065
  const policies = [model.litellmPolicy, selected.litellmPolicy].filter((policy) => policy !== undefined);
1045
1066
  const combinedPolicy =
@@ -1061,6 +1082,7 @@ function applyWildcardEvidence(
1061
1082
  litellmBackendFamily: _family,
1062
1083
  litellmResponsesReasoningControl: _control,
1063
1084
  litellmPolicy: _policy,
1085
+ promptCache: _promptCache,
1064
1086
  thinkingLevelMap: _levels,
1065
1087
  ...rest
1066
1088
  } = model;
@@ -1071,6 +1093,7 @@ function applyWildcardEvidence(
1071
1093
  compat: selected.compat,
1072
1094
  ...(selected.thinkingLevelMap ? { thinkingLevelMap: selected.thinkingLevelMap } : {}),
1073
1095
  ...(selected.litellmBackendFamily ? { litellmBackendFamily: selected.litellmBackendFamily } : {}),
1096
+ ...(selected.promptCache ? { promptCache: selected.promptCache } : {}),
1074
1097
  ...(combinedPolicy ? { litellmPolicy: combinedPolicy } : {}),
1075
1098
  ...(selected.litellmResponsesReasoningControl ? { litellmResponsesReasoningControl: true as const } : {}),
1076
1099
  } as DiscoveredModel;
@@ -1216,6 +1239,7 @@ export async function discoverModels(
1216
1239
  withheldRepairRoutes,
1217
1240
  defaultedContextRoutes,
1218
1241
  proxyVersion,
1242
+ modelOverrides: options.modelOverrides,
1219
1243
  }),
1220
1244
  deploymentFamilies: group.map(deploymentFamily),
1221
1245
  };
@@ -1272,14 +1296,26 @@ export async function discoverModels(
1272
1296
  // The parents were reduced before the version was known; an expansion
1273
1297
  // must inherit from parents reduced under the version it is published with.
1274
1298
  sources = wildcardRoutes.flatMap(({ route, deploymentFamilies }) => {
1275
- const model = mapFromModelInfoGroup(groups.get(route)!, publicCatalog, { proxyVersion });
1299
+ const model = mapFromModelInfoGroup(groups.get(route)!, publicCatalog, {
1300
+ proxyVersion,
1301
+ modelOverrides: options.modelOverrides,
1302
+ });
1276
1303
  return model ? [{ model, deploymentFamilies }] : [];
1277
1304
  });
1278
1305
  }
1279
1306
  const expanded = candidates
1280
1307
  .map((entry) => mapFromWildcardExpansion(entry, sources, withheldRepairRoutes))
1281
1308
  .filter((model): model is DiscoveredModel => model !== undefined)
1282
- .map((model) => applyWildcardEvidence(model, wildcardRows, publishedWildcardIds, publicCatalog, proxyVersion))
1309
+ .map((model) =>
1310
+ applyWildcardEvidence(
1311
+ model,
1312
+ wildcardRows,
1313
+ publishedWildcardIds,
1314
+ publicCatalog,
1315
+ proxyVersion,
1316
+ options.modelOverrides,
1317
+ ),
1318
+ )
1283
1319
  .filter((model): model is DiscoveredModel => model !== undefined);
1284
1320
  models = [...models, ...expanded];
1285
1321
  }
@@ -1309,10 +1345,15 @@ export async function discoverModels(
1309
1345
  const imageModels = [...groups.entries()]
1310
1346
  .map(([route, group]) => mapFromImageModelInfoGroup(route, group))
1311
1347
  .filter((model): model is DiscoveredImageModel => model !== undefined);
1348
+ // Decision routes are a separate operation in Pi; the chat reducer withholds their modes.
1349
+ const classifierModels = [...groups.entries()]
1350
+ .map(([route, group]) => mapFromClassifierModelInfoGroup(route, group))
1351
+ .filter((model): model is DiscoveredClassifierModel => model !== undefined);
1312
1352
  return {
1313
1353
  source: "model_info",
1314
1354
  models,
1315
1355
  ...(imageModels.length > 0 ? { imageModels } : {}),
1356
+ ...(classifierModels.length > 0 ? { classifierModels } : {}),
1316
1357
  ...(proxyVersion ? { proxyVersion } : {}),
1317
1358
  };
1318
1359
  }
package/src/images.ts CHANGED
@@ -1,9 +1,8 @@
1
1
  import type {
2
2
  AssistantImages,
3
- ImageApi,
4
3
  ImageContent,
5
4
  ImageModel,
6
- ImagesOptions,
5
+ ModelCost,
7
6
  ProviderImages,
8
7
  Usage,
9
8
  } from "@earendil-works/pi-ai";
@@ -33,7 +32,7 @@ export function isImageGenerationMode(mode: unknown): boolean {
33
32
  return value !== undefined && IMAGE_GENERATION_MODE_PATTERN.test(value.trim());
34
33
  }
35
34
 
36
- function perMillion(value: unknown): number | undefined {
35
+ export function perMillion(value: unknown): number | undefined {
37
36
  return typeof value === "number" && Number.isFinite(value) && value >= 0 ? value * 1_000_000 : undefined;
38
37
  }
39
38
 
@@ -119,18 +118,19 @@ function tokenCount(...values: unknown[]): number {
119
118
  return 0;
120
119
  }
121
120
 
122
- function parseUsage(
123
- response: ImagesResponse,
124
- model: ImageModel<ImageApi>,
121
+ export function parseUsage(
122
+ raw:
123
+ | { input_tokens?: unknown; output_tokens?: unknown; prompt_tokens?: unknown; completion_tokens?: unknown }
124
+ | undefined,
125
+ cost: ModelCost,
125
126
  responseCost: number | undefined,
126
127
  ): Usage | undefined {
127
- const raw = response.usage;
128
128
  if (!raw && responseCost === undefined) return undefined;
129
129
  const input = tokenCount(raw?.input_tokens, raw?.prompt_tokens);
130
130
  const output = tokenCount(raw?.output_tokens, raw?.completion_tokens);
131
131
  const estimated = {
132
- input: (model.cost.input / 1_000_000) * input,
133
- output: (model.cost.output / 1_000_000) * output,
132
+ input: (cost.input / 1_000_000) * input,
133
+ output: (cost.output / 1_000_000) * output,
134
134
  };
135
135
  // LiteLLM's own cost is authoritative: it knows per-image prices that a token rate cannot express.
136
136
  const total = responseCost ?? estimated.input + estimated.output;
@@ -148,7 +148,13 @@ function parseUsage(
148
148
  };
149
149
  }
150
150
 
151
- function requestHeaders(
151
+ /** LiteLLM's `x-litellm-response-cost` header, when it is a usable amount. */
152
+ export function responseCost(headers: Record<string, string>): number | undefined {
153
+ const cost = Number.parseFloat(headers["x-litellm-response-cost"] ?? "");
154
+ return Number.isFinite(cost) && cost >= 0 ? cost : undefined;
155
+ }
156
+
157
+ export function requestHeaders(
152
158
  apiKey: string,
153
159
  ...sources: (Record<string, string | null | undefined> | undefined)[]
154
160
  ): Record<string, string> {
@@ -167,7 +173,7 @@ function requestHeaders(
167
173
  return headers;
168
174
  }
169
175
 
170
- function errorDetail(body: string): string {
176
+ export function errorDetail(body: string): string {
171
177
  try {
172
178
  const parsed = JSON.parse(body) as { error?: { message?: unknown } | unknown };
173
179
  const message =
@@ -181,7 +187,9 @@ function errorDetail(body: string): string {
181
187
  return body.slice(0, MAX_ERROR_DETAIL);
182
188
  }
183
189
 
184
- function requestSignal(options: ImagesOptions | undefined): AbortSignal | undefined {
190
+ export function requestSignal(
191
+ options: { signal?: AbortSignal; timeoutMs?: number } | undefined,
192
+ ): AbortSignal | undefined {
185
193
  const signals = [
186
194
  options?.signal,
187
195
  options?.timeoutMs !== undefined ? AbortSignal.timeout(options.timeoutMs) : undefined,
@@ -242,8 +250,7 @@ export const generateImages: ProviderImages["generateImages"] = async (model, co
242
250
  : `LiteLLM image model ${model.id} returned no images`,
243
251
  );
244
252
  }
245
- const costHeader = Number.parseFloat(responseHeaders["x-litellm-response-cost"] ?? "");
246
- const usage = parseUsage(parsed, model, Number.isFinite(costHeader) && costHeader >= 0 ? costHeader : undefined);
253
+ const usage = parseUsage(parsed.usage, model.cost, responseCost(responseHeaders));
247
254
  if (usage) result.usage = usage;
248
255
  return result;
249
256
  } catch (error) {
package/src/index.ts CHANGED
@@ -19,6 +19,7 @@ import type { ExtensionAPI, ExtensionContext, McpServerConfig } from "@earendil-
19
19
  import * as piCodingAgent from "@earendil-works/pi-coding-agent";
20
20
  import { getAgentDir, readStoredCredential } from "@earendil-works/pi-coding-agent";
21
21
  import { budgetDisplaySetting, setupLiteLLMBudget } from "./budget.js";
22
+ import type { LiteLLMClassifierModel } from "./classifiers.js";
22
23
  import { setupLiteLLMCostTracking } from "./cost.js";
23
24
  import {
24
25
  discoverModels,
@@ -34,6 +35,7 @@ import {
34
35
  createLiteLLMProvider,
35
36
  DEFAULT_LITELLM_BASE_URL,
36
37
  isPlaceholderHost,
38
+ toNativeClassifierModels,
37
39
  toNativeImageModels,
38
40
  toNativeModels,
39
41
  } from "./provider.js";
@@ -286,6 +288,37 @@ async function readModelsProviderSettings(): Promise<Record<string, unknown> | u
286
288
  }
287
289
  }
288
290
 
291
+ // Pi `modelOverrides` limits from the global models.json, keyed by provider id then model id.
292
+ async function readModelLimitOverrides(): Promise<NonNullable<DiscoveryOptions["modelOverrides"]>> {
293
+ // Null prototypes: a `__proto__` provider or model key in the user's file stays a plain key.
294
+ const result: Record<string, Record<string, { contextWindow?: number; maxTokens?: number }>> = Object.create(null);
295
+ try {
296
+ const raw = await readFile(join(getAgentDir(), MODELS_FILENAME), "utf8");
297
+ const parsed = JSON.parse(stripJsonComments(stripBom(raw))) as Record<string, unknown>;
298
+ if (!isPlainObject(parsed.providers)) return result;
299
+ const limit = (value: unknown) =>
300
+ typeof value === "number" && Number.isFinite(value) && value > 0 ? value : undefined;
301
+ for (const [providerId, provider] of Object.entries(parsed.providers)) {
302
+ if (!isPlainObject(provider) || !isPlainObject(provider.modelOverrides)) continue;
303
+ for (const [modelId, override] of Object.entries(provider.modelOverrides)) {
304
+ if (!isPlainObject(override)) continue;
305
+ const contextWindow = limit(override.contextWindow);
306
+ const maxTokens = limit(override.maxTokens);
307
+ if (contextWindow === undefined && maxTokens === undefined) continue;
308
+ const models = result[providerId] ?? Object.create(null);
309
+ result[providerId] = models;
310
+ models[modelId] = {
311
+ ...(contextWindow !== undefined ? { contextWindow } : {}),
312
+ ...(maxTokens !== undefined ? { maxTokens } : {}),
313
+ };
314
+ }
315
+ }
316
+ } catch {
317
+ // Missing or unparsable models.json: no overrides.
318
+ }
319
+ return result;
320
+ }
321
+
289
322
  // A provider block may live in either file; merge per field with models.json winning. Non-provider
290
323
  // settings.json keys pass through untouched, which is what keeps `skills`/`mcp`/`budget` working.
291
324
  function mergeProviderSettings(
@@ -2637,6 +2670,7 @@ export default async function (pi: ExtensionAPI): Promise<void> {
2637
2670
  headers: auth.headers,
2638
2671
  allowInsecureHttp: auth.allowInsecureHttp,
2639
2672
  ...modelsDevDiscoveryOptions(),
2673
+ modelOverrides: await readModelLimitOverrides(),
2640
2674
  silent: !isVerboseDiscovery(),
2641
2675
  onProgress: isVerboseDiscovery() ? (message) => process.stderr.write(`LiteLLM: ${message}\n`) : undefined,
2642
2676
  });
@@ -2653,7 +2687,9 @@ export default async function (pi: ExtensionAPI): Promise<void> {
2653
2687
  // critical path of every start -- is replaced by reading that catalog while it is fresh (models-store.ts). Nothing is
2654
2688
  // written: Pi owns the file. The stored models are handed over rather than an empty list, so that a store Pi keeps
2655
2689
  // somewhere else cannot leave the provider with nothing.
2656
- async function seedModels(definition: ProviderDefinition): Promise<(Model<LiteLLMApi> | LiteLLMImageModel)[]> {
2690
+ async function seedModels(
2691
+ definition: ProviderDefinition,
2692
+ ): Promise<(Model<LiteLLMApi> | LiteLLMImageModel | LiteLLMClassifierModel)[]> {
2657
2693
  const disabledReason = discoveryDisabledReason() ?? (isHostOffline() ? "PI_OFFLINE" : null);
2658
2694
  if (disabledReason) {
2659
2695
  if (isVerboseDiscovery()) {
@@ -2685,6 +2721,12 @@ export default async function (pi: ExtensionAPI): Promise<void> {
2685
2721
  return [
2686
2722
  ...toNativeModels(definition.name, result.baseUrl, result.models, definition.allowInsecureHttp),
2687
2723
  ...toNativeImageModels(definition.name, result.baseUrl, result.imageModels ?? [], definition.allowInsecureHttp),
2724
+ ...toNativeClassifierModels(
2725
+ definition.name,
2726
+ result.baseUrl,
2727
+ result.classifierModels ?? [],
2728
+ definition.allowInsecureHttp,
2729
+ ),
2688
2730
  ];
2689
2731
  } catch (error) {
2690
2732
  // No credentials yet is the normal unconfigured case; anything else is worth one line.
@@ -54,6 +54,8 @@ export interface ReasoningPolicy {
54
54
 
55
55
  export interface CatalogResolution {
56
56
  provider?: string;
57
+ // The Pi provider whose catalog entry resolved catalogModelId; set only then, so the pair is Pi's providerId/model.id.
58
+ piProvider?: string;
57
59
  catalogModelId?: string;
58
60
  semanticFamily?: FamilyEvidence;
59
61
  semanticModel?: SemanticModel;
@@ -68,6 +70,8 @@ export interface CatalogResolution {
68
70
  vision?: boolean;
69
71
  contextWindow?: number;
70
72
  maxTokens?: number;
73
+ // User `modelOverrides` for the identified catalog model; read only under unanimous catalog authority.
74
+ limitOverrides?: { contextWindow?: number; maxTokens?: number };
71
75
  cost?: Partial<DiscoveredModel["cost"]>;
72
76
  }
73
77
 
@@ -657,6 +661,8 @@ function min(values: readonly number[]): number {
657
661
  }
658
662
 
659
663
  function conservativeLimit(explicit: number | undefined, catalog: number | undefined): number | undefined {
664
+ // LITELLM_TRUST_PROXY_LIMITS=1 lets the router's own figure win over the catalog.
665
+ if (process.env.LITELLM_TRUST_PROXY_LIMITS === "1") return explicitLimit(explicit) ?? explicitLimit(catalog);
660
666
  const valid = [explicitLimit(explicit), explicitLimit(catalog)].filter(
661
667
  (value): value is number => value !== undefined,
662
668
  );
@@ -819,11 +825,15 @@ export function reduceModelGroup(
819
825
  const visionEvidence = deployments.map(
820
826
  (entry, index) => wireBoolean(entry.model_info?.supports_vision) ?? catalogAuthority[index]?.vision,
821
827
  );
822
- const contextWindowEvidence = deployments.map((entry, index) =>
823
- conservativeLimit(entry.model_info?.max_input_tokens, catalogAuthority[index]?.contextWindow),
828
+ const contextWindowEvidence = deployments.map(
829
+ (entry, index) =>
830
+ explicitLimit(catalogAuthority[index]?.limitOverrides?.contextWindow) ??
831
+ conservativeLimit(entry.model_info?.max_input_tokens, catalogAuthority[index]?.contextWindow),
824
832
  );
825
- const maxTokensEvidence = deployments.map((entry, index) =>
826
- conservativeLimit(entry.model_info?.max_output_tokens, catalogAuthority[index]?.maxTokens),
833
+ const maxTokensEvidence = deployments.map(
834
+ (entry, index) =>
835
+ explicitLimit(catalogAuthority[index]?.limitOverrides?.maxTokens) ??
836
+ conservativeLimit(entry.model_info?.max_output_tokens, catalogAuthority[index]?.maxTokens),
827
837
  );
828
838
  const reasoning = reasoningEvidence.every((value) => value ?? false);
829
839
  const explicitlyUnsupported = deployments.some(
@@ -39,10 +39,11 @@
39
39
  import { readFile } from "node:fs/promises";
40
40
  import { join } from "node:path";
41
41
  import { LITELLM_DISCOVERY_VERSION } from "./backend-identity.js";
42
+ import { LITELLM_DECISIONS_API, type LiteLLMClassifierModel } from "./classifiers.js";
42
43
  import { enrichCachedModel } from "./discover.js";
43
44
  import { LITELLM_IMAGES_API, type LiteLLMImageModel } from "./images.js";
44
45
  import { isLiteLLMApi, resolveModelBaseUrl } from "./protocols.js";
45
- import { imageBaseUrl } from "./provider.js";
46
+ import { proxyV1BaseUrl } from "./provider.js";
46
47
  import type { LiteLLMModel } from "./types.js";
47
48
 
48
49
  /**
@@ -59,17 +60,20 @@ function isRecord(value: unknown): value is Record<string, unknown> {
59
60
  /** Pi's own remote catalogs skip the network for 4 hours after `checkedAt` (`REMOTE_CATALOG_REFRESH_INTERVAL_MS`). */
60
61
  const STORE_FRESH_MS = 4 * 60 * 60 * 1000;
61
62
 
62
- function isImageModel(value: Record<string, unknown>): boolean {
63
- return value.type === "image" && value.api === LITELLM_IMAGES_API;
63
+ function isOperationModel(value: Record<string, unknown>): boolean {
64
+ return (
65
+ (value.type === "image" && value.api === LITELLM_IMAGES_API) ||
66
+ (value.type === "classifier" && value.api === LITELLM_DECISIONS_API)
67
+ );
64
68
  }
65
69
 
66
70
  function isUsableModel(value: unknown, provider: string, baseUrl: string, allowInsecureHttp?: boolean): boolean {
67
71
  if (!isRecord(value) || value.provider !== provider) return false;
68
72
  if (value.litellmDiscoveryVersion !== LITELLM_DISCOVERY_VERSION) return false;
69
73
  try {
70
- // A stored baseUrl is per protocol -- `/v1` for the OpenAI APIs and images, the bare root for Messages -- so the root
74
+ // A stored baseUrl is per protocol -- `/v1` for the OpenAI APIs, images and classifiers, the bare root for Messages -- so the root
71
75
  // goes through the same resolution rather than being compared to it.
72
- if (isImageModel(value)) return value.baseUrl === imageBaseUrl(baseUrl, allowInsecureHttp);
76
+ if (isOperationModel(value)) return value.baseUrl === proxyV1BaseUrl(baseUrl, allowInsecureHttp);
73
77
  // `api` indexes LITELLM_PROTOCOLS in resolveModelBaseUrl, so an unknown one is a TypeError, not a bad model.
74
78
  return isLiteLLMApi(value.api) && value.baseUrl === resolveModelBaseUrl(baseUrl, value.api, allowInsecureHttp);
75
79
  } catch {
@@ -96,7 +100,7 @@ export async function readStoredModels(options: {
96
100
  provider: string;
97
101
  baseUrl: string;
98
102
  allowInsecureHttp?: boolean;
99
- }): Promise<(LiteLLMModel | LiteLLMImageModel)[] | undefined> {
103
+ }): Promise<(LiteLLMModel | LiteLLMImageModel | LiteLLMClassifierModel)[] | undefined> {
100
104
  const entry = (await readStore(options.agentDir))?.[options.provider];
101
105
  if (!isRecord(entry) || !Array.isArray(entry.models) || entry.models.length === 0) return undefined;
102
106
  if (typeof entry.checkedAt !== "number" || Date.now() - entry.checkedAt >= STORE_FRESH_MS) return undefined;
@@ -106,10 +110,11 @@ export async function readStoredModels(options: {
106
110
  if (!usable) return undefined;
107
111
  try {
108
112
  // Pi's restore path (provider.ts) runs the same call; when it throws, nothing is restored and the seed is the catalog.
109
- // Image models carry no request policy, so that path restores them as stored.
110
- return entry.models.map((model) => (isImageModel(model) ? model : enrichCachedModel(model))) as (
113
+ // Image and classifier models carry no request policy, so that path restores them as stored.
114
+ return entry.models.map((model) => (isOperationModel(model) ? model : enrichCachedModel(model))) as (
111
115
  | LiteLLMModel
112
116
  | LiteLLMImageModel
117
+ | LiteLLMClassifierModel
113
118
  )[];
114
119
  } catch {
115
120
  return undefined;
package/src/provider.ts CHANGED
@@ -3,6 +3,10 @@ import {
3
3
  type Api,
4
4
  type ApiStreamOptions,
5
5
  type AssistantImages,
6
+ type ClassifierContext,
7
+ type ClassifierModel,
8
+ type ClassifierOptions,
9
+ type ClassifierResult,
6
10
  type Credential,
7
11
  createProvider,
8
12
  getCurrentTools,
@@ -17,6 +21,12 @@ import {
17
21
  type TranscriptContext,
18
22
  } from "@earendil-works/pi-ai";
19
23
  import { LITELLM_DISCOVERY_VERSION } from "./backend-identity.js";
24
+ import {
25
+ createLiteLLMClassifierApis,
26
+ type DiscoveredClassifierModel,
27
+ LITELLM_DECISIONS_API,
28
+ type LiteLLMClassifierModel,
29
+ } from "./classifiers.js";
20
30
  import { enrichCachedModel, normalizeBaseUrl, restoreCachedModelPolicy } from "./discover.js";
21
31
  import {
22
32
  createLiteLLMImageApis,
@@ -34,7 +44,7 @@ export type LiteLLMProviderOptions = {
34
44
  auth: ProviderAuth;
35
45
  /** Catalog discovered during activation. Pi's startup refresh never allows network access, so
36
46
  * without a seed the provider stays empty until the user opens /model. */
37
- models?: readonly (LiteLLMModel | LiteLLMImageModel)[];
47
+ models?: readonly (LiteLLMModel | LiteLLMImageModel | LiteLLMClassifierModel)[];
38
48
  allowInsecureHttp?: boolean;
39
49
  resolveCredentialRoot: (credential?: Credential, requestBaseUrl?: string, apiKey?: string) => string | undefined;
40
50
  discover(credential: Credential, signal?: AbortSignal): Promise<DiscoveryResult & { baseUrl?: string }>;
@@ -49,17 +59,31 @@ function isLiteLLMImageModel(model: AnyModel): model is LiteLLMImageModel {
49
59
  return model.type === "image" && model.api === LITELLM_IMAGES_API;
50
60
  }
51
61
 
52
- export function imageBaseUrl(root: string, allowInsecureHttp = false): string {
62
+ function isLiteLLMClassifierModel(model: AnyModel): model is LiteLLMClassifierModel {
63
+ return model.type === "classifier" && model.api === LITELLM_DECISIONS_API;
64
+ }
65
+
66
+ /** Where image and classifier requests go: `<root>/v1`. */
67
+ export function proxyV1BaseUrl(root: string, allowInsecureHttp = false): string {
53
68
  return `${normalizeBaseUrl(root, allowInsecureHttp)}/v1`;
54
69
  }
55
70
 
71
+ export function toNativeClassifierModels(
72
+ provider: string,
73
+ baseUrl: string,
74
+ models: readonly DiscoveredClassifierModel[],
75
+ allowInsecureHttp = false,
76
+ ): LiteLLMClassifierModel[] {
77
+ return models.map((model) => ({ ...model, provider, baseUrl: proxyV1BaseUrl(baseUrl, allowInsecureHttp) }));
78
+ }
79
+
56
80
  export function toNativeImageModels(
57
81
  provider: string,
58
82
  baseUrl: string,
59
83
  models: readonly DiscoveredImageModel[],
60
84
  allowInsecureHttp = false,
61
85
  ): LiteLLMImageModel[] {
62
- return models.map((model) => ({ ...model, provider, baseUrl: imageBaseUrl(baseUrl, allowInsecureHttp) }));
86
+ return models.map((model) => ({ ...model, provider, baseUrl: proxyV1BaseUrl(baseUrl, allowInsecureHttp) }));
63
87
  }
64
88
 
65
89
  export function toNativeModels(
@@ -145,23 +169,24 @@ function modelRootError(
145
169
  }
146
170
  }
147
171
 
148
- function imageRootError(
149
- model: ImageModel<ImageApi>,
172
+ function operationRootError(
173
+ model: { baseUrl: string },
174
+ subject: string,
150
175
  active: { root: string; canonical: string; hostname: string },
151
176
  allowInsecureHttp = false,
152
177
  ): Error | undefined {
153
178
  let stored: { root: string; canonical: string; hostname: string };
154
179
  try {
155
- stored = proxyRoot(model.baseUrl, "Cached image model", allowInsecureHttp);
180
+ stored = proxyRoot(model.baseUrl, subject, allowInsecureHttp);
156
181
  } catch (error) {
157
- return error instanceof Error ? error : refreshRequired("Cached image model has an invalid LiteLLM model URL");
182
+ return error instanceof Error ? error : refreshRequired(`${subject} has an invalid LiteLLM model URL`);
158
183
  }
159
184
  if (isPlaceholderHost(stored.hostname)) {
160
- return refreshRequired("Cached image model uses a placeholder LiteLLM model host");
185
+ return refreshRequired(`${subject} uses a placeholder LiteLLM model host`);
161
186
  }
162
187
  if (stored.canonical !== active.canonical) {
163
188
  return refreshRequired(
164
- `Cached image model has stale LiteLLM model root ${stored.root}; active credentials use ${active.root}`,
189
+ `${subject} has stale LiteLLM model root ${stored.root}; active credentials use ${active.root}`,
165
190
  );
166
191
  }
167
192
  }
@@ -244,6 +269,7 @@ export function createLiteLLMProvider(options: LiteLLMProviderOptions): Provider
244
269
  return [
245
270
  ...toNativeModels(options.id, root, result.models, options.allowInsecureHttp),
246
271
  ...toNativeImageModels(options.id, root, result.imageModels ?? [], options.allowInsecureHttp),
272
+ ...toNativeClassifierModels(options.id, root, result.classifierModels ?? [], options.allowInsecureHttp),
247
273
  ];
248
274
  },
249
275
  filterModels(models, credential) {
@@ -257,17 +283,26 @@ export function createLiteLLMProvider(options: LiteLLMProviderOptions): Provider
257
283
  // Image models follow the same rule as chat: a cached root that no longer matches the
258
284
  // active credential is hidden until a network refresh proves the new proxy's catalog.
259
285
  const images = models.filter(isLiteLLMImageModel).flatMap((model) => {
260
- const error = imageRootError(model, active, options.allowInsecureHttp);
286
+ const error = operationRootError(model, "Cached image model", active, options.allowInsecureHttp);
261
287
  if (error) {
262
288
  reportUnavailable(error.message);
263
289
  return [];
264
290
  }
265
- return [{ ...model, baseUrl: imageBaseUrl(active.root, options.allowInsecureHttp) }];
291
+ return [{ ...model, baseUrl: proxyV1BaseUrl(active.root, options.allowInsecureHttp) }];
266
292
  });
267
- return [...chat, ...images];
293
+ const classifiers = models.filter(isLiteLLMClassifierModel).flatMap((model) => {
294
+ const error = operationRootError(model, "Cached classifier model", active, options.allowInsecureHttp);
295
+ if (error) {
296
+ reportUnavailable(error.message);
297
+ return [];
298
+ }
299
+ return [{ ...model, baseUrl: proxyV1BaseUrl(active.root, options.allowInsecureHttp) }];
300
+ });
301
+ return [...chat, ...images, ...classifiers];
268
302
  },
269
303
  api: createLiteLLMProtocolApis(),
270
304
  images: createLiteLLMImageApis(),
305
+ classifiers: createLiteLLMClassifierApis(),
271
306
  });
272
307
  const refreshModels = provider.refreshModels;
273
308
  const guardedProvider: Provider<LiteLLMApi> = {
@@ -307,10 +342,10 @@ export function createLiteLLMProvider(options: LiteLLMProviderOptions): Provider
307
342
  );
308
343
  if (!root) throw refreshRequired("Active credentials do not identify a LiteLLM model host");
309
344
  const active = activeCredentialRoot(root, options.allowInsecureHttp);
310
- const error = imageRootError(model, active, options.allowInsecureHttp);
345
+ const error = operationRootError(model, "Cached image model", active, options.allowInsecureHttp);
311
346
  if (error) throw error;
312
347
  return await provider.generateImages!(
313
- { ...model, provider: options.id, baseUrl: imageBaseUrl(active.root, options.allowInsecureHttp) },
348
+ { ...model, provider: options.id, baseUrl: proxyV1BaseUrl(active.root, options.allowInsecureHttp) },
314
349
  context,
315
350
  requestOptions,
316
351
  );
@@ -328,23 +363,67 @@ export function createLiteLLMProvider(options: LiteLLMProviderOptions): Provider
328
363
  return failed;
329
364
  }
330
365
  },
366
+ classify: async (
367
+ model: ClassifierModel<string>,
368
+ context: ClassifierContext,
369
+ requestOptions?: ClassifierOptions,
370
+ ) => {
371
+ try {
372
+ const root = options.resolveCredentialRoot(
373
+ undefined,
374
+ requestOptions?.env?.LITELLM_BASE_URL,
375
+ requestOptions?.apiKey,
376
+ );
377
+ if (!root) throw refreshRequired("Active credentials do not identify a LiteLLM model host");
378
+ const active = activeCredentialRoot(root, options.allowInsecureHttp);
379
+ const error = operationRootError(model, "Cached classifier model", active, options.allowInsecureHttp);
380
+ if (error) throw error;
381
+ return await provider.classify!(
382
+ { ...model, provider: options.id, baseUrl: proxyV1BaseUrl(active.root, options.allowInsecureHttp) },
383
+ context,
384
+ requestOptions,
385
+ );
386
+ } catch (error) {
387
+ // `Provider.classify` never rejects.
388
+ const failed: ClassifierResult = {
389
+ api: model.api,
390
+ provider: model.provider,
391
+ model: model.id,
392
+ answers: {},
393
+ stopReason: requestOptions?.signal?.aborted ? "aborted" : "error",
394
+ errorMessage: error instanceof Error ? error.message : String(error),
395
+ timestamp: Date.now(),
396
+ };
397
+ return failed;
398
+ }
399
+ },
331
400
  };
332
401
  if (!refreshModels) return guardedProvider;
333
- const seedKey = (model: { id: string; baseUrl: string }) => `${model.id}\0${model.baseUrl}`;
402
+ // A route can be published as both a chat model and a classifier (GPT-6 Luna lists `/v1/decisions`
403
+ // next to its chat endpoints), so the type keeps those seeds apart.
404
+ const seedKey = (model: { type?: string; id: string; baseUrl: string }) =>
405
+ `${model.type ?? "chat"}\0${model.id}\0${model.baseUrl}`;
334
406
  const seeded = new Map((options.models ?? []).map((model) => [seedKey(model), model]));
335
407
  return {
336
408
  ...guardedProvider,
337
409
  refreshModels: async (context) => {
338
410
  // Pi 0.99 persists models of every type; the policy helpers below read chat-only fields, so
339
- // chat and image entries are restored separately. Other types are not this provider's.
411
+ // chat, image and classifier entries are restored separately. Other types are not this provider's.
340
412
  const storedModels = (context.stored?.models ?? []).filter(isChatModel);
341
- const storedImageModels = (context.stored?.models ?? []).filter(isLiteLLMImageModel).flatMap((model) => {
342
- if (model.litellmDiscoveryVersion === LITELLM_DISCOVERY_VERSION) return [model];
343
- // An image entry carries no request policy to restore; a stale one is only kept when
344
- // startup discovery proved the same route, otherwise the network refresh replaces it.
345
- const fresh = seeded.get(seedKey(model));
346
- return fresh && isLiteLLMImageModel(fresh) ? [fresh] : [];
347
- });
413
+ // Image and classifier entries carry no request policy to restore; a stale one is only kept
414
+ // when startup discovery proved the same route, otherwise the network refresh replaces it.
415
+ const restoreOperationModels = <T extends AnyModel>(is: (model: AnyModel) => model is T): T[] =>
416
+ (context.stored?.models ?? []).filter(is).flatMap((model) => {
417
+ if ((model as { litellmDiscoveryVersion?: number }).litellmDiscoveryVersion === LITELLM_DISCOVERY_VERSION) {
418
+ return [model];
419
+ }
420
+ const fresh = seeded.get(seedKey(model));
421
+ return fresh && is(fresh) ? [fresh] : [];
422
+ });
423
+ const storedOperationModels = [
424
+ ...restoreOperationModels(isLiteLLMImageModel),
425
+ ...restoreOperationModels(isLiteLLMClassifierModel),
426
+ ];
348
427
  let legacyCount = 0;
349
428
  const models = storedModels.map((model) => {
350
429
  if ((model as LiteLLMModel).litellmDiscoveryVersion !== LITELLM_DISCOVERY_VERSION) {
@@ -362,7 +441,7 @@ export function createLiteLLMProvider(options: LiteLLMProviderOptions): Provider
362
441
  ...context,
363
442
  // The current pi-ai provider has no freshness gate, but force is the documented refresh contract field.
364
443
  force: context.force || (context.allowNetwork && legacyCount > 0),
365
- stored: context.stored && { ...context.stored, models: [...models, ...storedImageModels] },
444
+ stored: context.stored && { ...context.stored, models: [...models, ...storedOperationModels] },
366
445
  });
367
446
  } catch (error) {
368
447
  if (context.allowNetwork && legacyCount > 0) {
package/src/types.ts CHANGED
@@ -1,5 +1,6 @@
1
1
  import type { Model } from "@earendil-works/pi-ai";
2
2
  import type { BackendFamily, LITELLM_DISCOVERY_VERSION } from "./backend-identity.js";
3
+ import type { DiscoveredClassifierModel } from "./classifiers.js";
3
4
  import type { DiscoveredImageModel } from "./images.js";
4
5
  import type { ProxyVersion } from "./proxy-version.js";
5
6
 
@@ -67,6 +68,11 @@ export interface DiscoveryResult {
67
68
  models: DiscoveredModel[];
68
69
  /** `mode: "image_generation"` routes from `/model/info`; absent when there are none. */
69
70
  imageModels?: DiscoveredImageModel[];
71
+ /**
72
+ * Decision routes from `/model/info` (`mode: "evaluation"`/`"decisions"`, or a decision endpoint in
73
+ * `supported_endpoints`); absent when there are none.
74
+ */
75
+ classifierModels?: DiscoveredClassifierModel[];
70
76
  source: DiscoverySource;
71
77
  // Present only when a `/model/info` discovery had to read it and the proxy
72
78
  // answered. Absent means unknown, not old: discovery asks only when it must.
@@ -82,6 +88,9 @@ export interface DiscoveryOptions {
82
88
  // path, models.dev is not consulted at all.
83
89
  modelsDev?: boolean;
84
90
  modelsDevCachePath?: string;
91
+ // Pi `models.json` `providers.<providerId>.modelOverrides.<modelId>` limits. Pi applies them to that
92
+ // provider's own models only, so discovery applies them to routes backed by the same catalog model.
93
+ modelOverrides?: Readonly<Record<string, Readonly<Record<string, { contextWindow?: number; maxTokens?: number }>>>>;
85
94
  }
86
95
 
87
96
  export interface ModelInfoEntry {