pi-provider-litellm 3.3.0 → 3.4.0

package/README.md

@@ -205,7 +205,7 @@ adding transport-specific fields to request bodies.
 
 ## Model transport
 
-A `/model/info` route group uses native Anthropic `/v1/messages` only when every deployment explicitly reports Chat mode, identifies a Claude backend through an Anthropic-, Bedrock-, or Vertex-capable adapter, and every declared backend identity resolves the same Anthropic compatibility policy. A `claude-<name>-<major>-<minor>` backend id resolves through its `claude-<name>-<major>` entry when the exact id is absent; an unknown generation stays on Chat Completions. When `supported_endpoints` is present, it must include `/v1/messages`; an explicit endpoint list without it keeps the group on Chat Completions. An unresolved routing or base-model identity denies native Messages rather than being discarded from the unanimity check. Mixed-generation Claude groups can use Messages when their serializer policies match, with reasoning levels restricted to their common supported set. Groups with incompatible policies, mixed families, or unknown backend evidence remain on Chat Completions; unanimous explicit Responses mode takes precedence. Routes discovered through `/health` never select native Messages: Messages candidates are downgraded to Chat Completions with reasoning controls closed, while correlated Responses detail remains on Responses. Evidence-free `/v1/models` and wholly health-only `/health` groups take transport and presentation metadata from the bounded Pi catalog lookup: `openai-responses` when the catalog says so, otherwise Chat Completions. They expose no speculative reasoning selector and never select native Messages. Catalog identity such as Amazon Bedrock remains separate from the selected wire protocol.
+A `/model/info` route group uses native Anthropic `/v1/messages` only when every deployment explicitly reports Chat mode, identifies a Claude backend through an Anthropic-, Bedrock-, or Vertex-capable adapter, and every declared backend identity resolves the same Anthropic compatibility policy. A `claude-<name>-<major>-<minor>` backend id resolves through its `claude-<name>-<major>` entry when the exact id is absent; an unknown generation stays on Chat Completions. When `supported_endpoints` is a list, it must include `/v1/messages`; an explicit endpoint list without it keeps the group on Chat Completions. A `null` value, which LiteLLM v1.100 and later report for models absent from their model map, counts as absent. An unresolved routing or base-model identity denies native Messages rather than being discarded from the unanimity check. Mixed-generation Claude groups can use Messages when their serializer policies match, with reasoning levels restricted to their common supported set. Groups with incompatible policies, mixed families, or unknown backend evidence remain on Chat Completions; unanimous explicit Responses mode takes precedence. Routes discovered through `/health` never select native Messages: Messages candidates are downgraded to Chat Completions with reasoning controls closed, while correlated Responses detail remains on Responses. Evidence-free `/v1/models` and wholly health-only `/health` groups take transport and presentation metadata from the bounded Pi catalog lookup: `openai-responses` when the catalog says so, otherwise Chat Completions. They expose no speculative reasoning selector and never select native Messages. Catalog identity such as Amazon Bedrock remains separate from the selected wire protocol.
 
 Native Messages authenticates with `x-api-key`; every transport carries the `x-litellm-session-id` header described above. Strict JSON-schema tools are enabled separately from Messages transport and thinking compatibility: every deployment must affirm support for the actual route/API. Generic `supports_response_schema` and `supports_native_structured_output` metadata does not grant strict tools. Only deployments that LiteLLM reports with the `anthropic` adapter carry that evidence today. Claude served through Bedrock or Vertex therefore uses ordinary tool serialization, and a tool that requires strict sampling fails before the request is sent.
 
@@ -405,6 +405,8 @@ Reasoning levels require deployment evidence for the parameter Pi sends: `reason
 
 LiteLLM omits `supported_openai_params` for a deployment its model map does not describe. Such a deployment keeps the `reasoning_effort` carrier when its `model_info` explicitly sets `supports_reasoning: true`, which is the operator's opt-in. A `supported_openai_params` list that omits `reasoning_effort` still denies effort levels unless `litellm_params.allowed_openai_params` adds it. Kimi and DeepSeek generations are excluded because their contracts name their own carriers. Evidence-free fallback models retain catalog presentation metadata but expose no speculative selector.
 
+GPT-5.5 and later reject `reasoning_effort` together with function tools on `/v1/chat/completions`. On a Chat route where any deployment's `model_info.base_model` or `litellm_params.model` names such a backend, tool requests turn reasoning off instead of failing or being answered by a router fallback: `reasoning_effort` is set to the model's declared off value, or removed when it declares none. GPT-6 and later fall back to their default effort when the field is omitted, so they get `none` either way. A model with no backend evidence falls back to its route name. Requests without tools keep their reasoning level. A model that cannot turn reasoning off, such as GPT-6 Astra, cannot call tools on Chat Completions at all. To keep reasoning on tool turns, serve the route through `/v1/responses` (see [Protocols and prompt caching](#protocols-and-prompt-caching)).
+
 Kimi K2.5/K2.6 use an on/off thinking switch; K2.7 Code/Highspeed is always thinking. Kimi K3 and DeepSeek V4 use public effort evidence with LiteLLM overrides. Responses levels are translated to valid `reasoning.effort` values; Chat-only compatibility fields never reach Responses models. Legacy cached level maps remain usable while discovery refreshes, and stored Moonshot compatibility can restore safe display normalization.
 
 The development probe runs against minimized snapshots with `npm run probe:proxy -- --snapshot tests/fixtures/proxy/prod-model-info-2026-09-04.json`. A bounded live matrix is available with `--live --base-url <proxy-url> --max-requests 100`; its request bodies use the discovered carrier and request policy.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-provider-litellm",
-  "version": "3.3.0",
+  "version": "3.4.0",
   "description": "LiteLLM proxy provider extension for Pi",
   "type": "module",
   "license": "MIT",

package/src/backend-identity.ts

@@ -1,4 +1,4 @@
-export const LITELLM_DISCOVERY_VERSION = 3 as const;
+export const LITELLM_DISCOVERY_VERSION = 4 as const;
 
 export type BackendFamily = "claude" | "deepseek" | "gemini" | "kimi" | "openai";
 

package/src/discover.ts

@@ -93,17 +93,31 @@ export function normalizeBaseUrl(input: string, allowInsecureHttp = false): stri
 const ANTHROPIC_MODEL_PATTERN = /(?:^|[-_/.:])(?:anthropic\/|(?:claude|fable|opus|sonnet|haiku)(?=$|[-_/.:]))/i;
 const MOONSHOT_MODEL_PATTERN = /^(moonshotai\/|moonshot\/|kimi[-/])/i;
 const FORCED_THINKING_MODEL_PATTERN = /(?:^|[-/])thinking(?:[-/]|$)/i;
-// Deployments expose the gpt-5.5 route under varying names (`llm-gateway/gpt-5.5`,
-// bare `gpt-5.5`, dated ids like `gpt-5.5-20260504143601`); match them all so the
-// tool+reasoning workaround survives route renames.
-const GPT55_MODEL_PATTERN = /(?:^|\/)gpt-5\.5(?:$|[-.])/i;
+// GPT ids appear bare, prefixed, or dated (`gpt-6-sol`, `llm-gateway/gpt-5.5`,
+// `gpt-5.5-20260504143601`); match them all so the tool+reasoning workaround
+// survives renames. `gpt-4o` and `gpt-oss` carry no generation number here.
+const GPT_GENERATION_PATTERN = /(?:^|\/)gpt-(\d+)(?:\.(\d+))?(?=$|[-.])/i;
 
 export function isMoonshotModel(modelId: string): boolean {
   return MOONSHOT_MODEL_PATTERN.test(modelId);
 }
 
-export function isGpt55Model(modelId: string): boolean {
-  return GPT55_MODEL_PATTERN.test(modelId);
+function gptGeneration(modelId: string): [major: number, minor: number] | undefined {
+  const match = GPT_GENERATION_PATTERN.exec(modelId);
+  return match ? [Number(match[1]), Number(match[2] ?? 0)] : undefined;
+}
+
+// GPT-5.5 and later reject `reasoning_effort` alongside function tools on
+// /v1/chat/completions ("use /v1/responses").
+export function isGpt55OrNewerModel(modelId: string): boolean {
+  const generation = gptGeneration(modelId);
+  return generation !== undefined && (generation[0] > 5 || (generation[0] === 5 && generation[1] >= 5));
+}
+
+// GPT-6 also rejects an omitted effort there: it falls back to its default effort.
+export function isGpt6OrNewerModel(modelId: string): boolean {
+  const generation = gptGeneration(modelId);
+  return generation !== undefined && generation[0] >= 6;
 }
 
 function shouldSuppressReasoningContent(modelId: string): boolean {
@@ -736,13 +750,29 @@ function mapFromModelInfoGroup(
     !hasConflictingFamily &&
     (reduced.deploymentFamilies.includes("kimi") || (unlabeled && isMoonshotModel(reduced.id)));
   if (moonshotEvidence && !unanimousMoonshot) options.withheldRepairRoutes?.push(reduced.id);
-  const modelPolicy = requestPolicy(
+  const familyPolicy = requestPolicy(
     reduced.id,
     reduced.deploymentFamilies,
     reduced.normalizeThinkTags,
     reduced.suppressReasoningVisibility,
     unanimousMoonshot,
   );
+  // One GPT-5.5+ deployment is enough: a tool request routed to it fails with
+  // reasoning attached, while dropping reasoning elsewhere only loses effort.
+  const chatBackendIds =
+    api === "openai-completions"
+      ? entries.flatMap((entry) => resolveBackendIdentity({ ...entry, model_name: undefined })?.modelId ?? [])
+      : [];
+  const modelPolicy: LiteLLMModelPolicy | undefined = chatBackendIds.some(isGpt55OrNewerModel)
+    ? {
+        normalizeStrictToolMessages: false,
+        normalizeThinkTags: false,
+        suppressReasoningVisibility: false,
+        ...familyPolicy,
+        dropToolReasoning: true,
+        ...(chatBackendIds.some(isGpt6OrNewerModel) ? { explicitToolReasoningOff: true } : {}),
+      }
+    : familyPolicy;
   return {
     id: reduced.id,
     // Reduced groups never borrow the ` (no metadata)` sentinel, which authorizes
@@ -1008,6 +1038,10 @@ function applyWildcardEvidence(
           ...(policies.every((policy) => policy.normalizeGeminiReasoningEffort)
             ? { normalizeGeminiReasoningEffort: true as const }
             : {}),
+          ...(policies.some((policy) => policy.dropToolReasoning) ? { dropToolReasoning: true as const } : {}),
+          ...(policies.some((policy) => policy.explicitToolReasoningOff)
+            ? { explicitToolReasoningOff: true as const }
+            : {}),
         }
       : undefined;
   const {
@@ -1062,6 +1096,12 @@ function mapFromWildcardExpansion(
         ...(modelPolicies.every((policy) => policy?.normalizeGeminiReasoningEffort === true)
           ? { normalizeGeminiReasoningEffort: true as const }
           : {}),
+        ...(api === "openai-completions" && modelPolicies.some((policy) => policy?.dropToolReasoning === true)
+          ? { dropToolReasoning: true as const }
+          : {}),
+        ...(api === "openai-completions" && modelPolicies.some((policy) => policy?.explicitToolReasoningOff === true)
+          ? { explicitToolReasoningOff: true as const }
+          : {}),
       }
     : !hasFamilyEvidence && isMoonshotModel(id)
       ? moonshotPolicy(id)
@@ -1329,7 +1369,26 @@ export function restoreCachedModelPolicy(model: Model<Api>): Model<Api> {
     // re-proven offline, so fail closed until the forced network refresh below.
     restored = { ...cached, compat: Object.keys(compat).length > 0 ? compat : undefined };
   }
-  if (restored.litellmPolicy || !hasMoonshotCompatEvidence(restored.compat)) return restored;
+  if (restored.litellmPolicy) return restored;
+  // The cache no longer holds the deployment rows that prove a GPT-5.5+ backend, so an
+  // OpenAI-family Chat entry fails toward the tool-reasoning workaround until refreshed.
+  if (
+    restored.api === "openai-completions" &&
+    restored.litellmBackendFamily === "openai" &&
+    isGpt55OrNewerModel(restored.id)
+  ) {
+    return {
+      ...restored,
+      litellmPolicy: {
+        normalizeStrictToolMessages: false,
+        normalizeThinkTags: false,
+        suppressReasoningVisibility: false,
+        dropToolReasoning: true,
+        ...(isGpt6OrNewerModel(restored.id) ? { explicitToolReasoningOff: true } : {}),
+      },
+    } as LiteLLMModel;
+  }
+  if (!hasMoonshotCompatEvidence(restored.compat)) return restored;
   return { ...restored, litellmPolicy: moonshotPolicy(restored.id) } as LiteLLMModel;
 }
 

package/src/index.ts

@@ -19,7 +19,7 @@ import { Type } from "@earendil-works/pi-ai";
 import type { ExtensionAPI, ExtensionContext } from "@earendil-works/pi-coding-agent";
 import { getAgentDir, readStoredCredential } from "@earendil-works/pi-coding-agent";
 import { setupLiteLLMCostTracking } from "./cost.js";
-import { discoverModels, isGpt55Model, normalizeBaseUrl } from "./discover.js";
+import { discoverModels, isGpt6OrNewerModel, isGpt55OrNewerModel, normalizeBaseUrl } from "./discover.js";
 import { getGcloudToken, hasGcloudAdcCredentials, isGcloudTokenAuthEnabled } from "./gcloud-token.js";
 import {
   createMcpToolDefinitions,
@@ -1360,7 +1360,8 @@ async function refreshWithBackoff(
     return credentials;
   }
   pkceTransientRefreshBackoff.delete(credentials.refresh);
-  if (!result.ok) throw new Error(`${result.message}; run /login litellm again`);
+  // A transient failure leaves the stored credential intact, and a new login would need the same endpoint.
+  if (!result.ok) throw new Error(result.transient ? result.message : `${result.message}; run /login litellm again`);
   return { ...credentials, ...result.token };
 }
 
@@ -1713,12 +1714,13 @@ export function prepareLiteLLMRequestPayload(
     for (const [key, value] of Object.entries(REASONING_VISIBILITY_DEFAULTS)) update(key, value);
   }
 
-  // LiteLLM still routes gpt-5.5 tool+reasoning requests through chat completions.
-  // Drop reasoning until the gateway honors /v1/responses for this route.
+  // GPT-5.5 and later reject reasoning alongside function tools on Chat Completions.
+  // Drop it until the gateway serves /v1/responses for the route. Discovery decides
+  // from deployment evidence; only a model with no backend evidence uses its route name.
+  const routeOnlyId = model?.litellmBackendFamily === undefined ? modelId : undefined;
   if (
     openAIApi === "openai-completions" &&
-    modelId &&
-    isGpt55Model(modelId) &&
+    (modelPolicy?.dropToolReasoning === true || (routeOnlyId !== undefined && isGpt55OrNewerModel(routeOnlyId))) &&
     Array.isArray(payload.tools) &&
     payload.tools.length > 0
   ) {
@@ -1727,6 +1729,17 @@ export function prepareLiteLLMRequestPayload(
       next ??= { ...payload };
       delete next[key];
     }
+    // Send the model's own off effort when it declares one. GPT-6 also rejects an
+    // omitted effort (it falls back to its default), and its error names `none` as
+    // the fix, so it gets `none` even without carrier evidence for an off level.
+    const declaredOff = model?.thinkingLevelMap?.off;
+    const explicitOff =
+      modelPolicy?.explicitToolReasoningOff === true || (routeOnlyId !== undefined && isGpt6OrNewerModel(routeOnlyId));
+    const off = typeof declaredOff === "string" ? declaredOff : explicitOff ? "none" : undefined;
+    if (off !== undefined && (next ?? payload).reasoning_effort !== off) {
+      next ??= { ...payload };
+      next.reasoning_effort = off;
+    }
     const include = (next ?? payload).include;
     if (Array.isArray(include) && include.includes("reasoning.encrypted_content")) {
       next ??= { ...payload };

package/src/model-groups.ts

@@ -798,9 +798,12 @@ export function reduceModelGroup(
   // Strict tools require affirmative evidence from every routable deployment.
   const messagesStrictTools =
     catalogs.length > 0 && catalogs.every((catalog) => catalog?.messagesStrictTools === true) ? true : undefined;
+  // LiteLLM v1.100+ reports `supported_endpoints: null` for every model its model map does not
+  // list, so null means unknown, like an absent field. Only a list without Messages, or a
+  // malformed non-null value, withholds the transport.
   const messagesEndpointAllowed = deployments.every((entry) => {
     const endpoints = entry.model_info?.supported_endpoints;
-    return endpoints === undefined || (Array.isArray(endpoints) && endpoints.includes("/v1/messages"));
+    return endpoints == null || (Array.isArray(endpoints) && endpoints.includes("/v1/messages"));
   });
   const api = candidateModes.every((mode) => mode === "responses")
     ? "openai-responses"

package/src/types.ts

@@ -28,6 +28,14 @@ export interface LiteLLMModelPolicy {
   // Gemini-backed deployments require lowercase effort values. Discovery carries
   // this evidence so the request hook does not infer a backend from the route name.
   normalizeGeminiReasoningEffort?: boolean;
+  // GPT-5.5 and later reject `reasoning_effort` alongside function tools on
+  // /v1/chat/completions. Discovery sets this for Chat routes from deployment
+  // evidence so tool requests drop reasoning instead of failing over.
+  dropToolReasoning?: boolean;
+  // GPT-6 falls back to its default effort when `reasoning_effort` is omitted, and
+  // that is rejected alongside tools too, so these tool requests send `none` unless
+  // the model declares its own off effort.
+  explicitToolReasoningOff?: boolean;
 }
 
 export type DiscoveredModelFor<TApi extends LiteLLMApi> = Omit<Model<TApi>, "provider" | "baseUrl"> & {
@@ -86,7 +94,7 @@ export interface ModelInfoEntry {
     mode?: string | null;
     base_model?: string;
     litellm_provider?: string;
-    supported_endpoints?: string[];
+    supported_endpoints?: string[] | null;
     supported_openai_params?: string[];
     input_cost_per_token?: number;
     output_cost_per_token?: number;