pi-lilac-provider 1.2.1 → 1.4.0

package/README.md

@@ -103,13 +103,33 @@ pi --provider lilac --model moonshotai/kimi-k2.6
 
 ### Thinking Mode
 
-All Lilac models support chain-of-thought reasoning via `chat_template_kwargs`. Pi uses the `qwen-chat-template` thinking format to send both `thinking` and `enable_thinking` keys, which works across all model families:
-
-- **Kimi K2.6**: Honors `thinking` key (Moonshot template)
-- **GLM 5.1**: Honors `enable_thinking` key (Z.ai template)
-- **Gemma 4**: Honors `enable_thinking` key (Google template)
-
-In pi, reasoning models automatically use the appropriate thinking format. Use Shift+Tab to control thinking level.
+All Lilac models toggle reasoning via `chat_template_kwargs`, but the key each
+model's chat template honors differs per family. The provider uses pi's
+`chat-template` thinkingFormat with per-model `chatTemplateKwargs` (configured in
+`patch.json`) so the right key reaches each template:
+
+| Model | Reasoning key | Default |
+|-------|---------------|---------|
+| Kimi K2.6 | `thinking` (bool) | on |
+| GLM 5.1 | `enable_thinking` (bool) | on |
+| GLM 5.2 | `enable_thinking` (bool) + `reasoning_effort` (`max`\|`high`) | on (`max`) |
+| Gemma 4 | `enable_thinking` (bool) | off |
+| MiniMax M2.7 | `thinking` + `enable_thinking` (bool) | on |
+| MiniMax M3 | `thinking_mode` (`disabled`\|`adaptive`\|`enabled`) | adaptive (server) |
+
+Kimi K2.6, GLM 5.1, Gemma 4, and MiniMax M2.7 use the forward-compatible form
+that sends **both** `thinking` and `enable_thinking`, so whichever key the
+template honors is set. GLM 5.2 additionally maps pi's thinking levels to
+`reasoning_effort` (`high` = lower-latency, `xhigh` = `max`). MiniMax M3 uses
+the `thinking_mode` enum, exposed as three pi thinking levels: `off` →
+`disabled` (never think), `minimal` → `adaptive` (the model decides), `high` →
+`enabled` (always think). Pi starts at `off` (`disabled`); cycle to `minimal`
+for M3's adaptive "model decides" mode. (The selector/footer show pi's level
+names — `minimal`/`high` — not the `thinking_mode` values; pi has no per-model
+level-relabel hook.)
+
+In pi, reasoning models automatically use the appropriate thinking format. Use
+Shift+Tab to control thinking level.
 
 ### Vision
 
@@ -153,7 +173,7 @@ Add to your pi configuration for automatic loading:
 
 Lilac's API is OpenAI-compatible with these specifics:
 
-- **`thinkingFormat: "qwen-chat-template"`** — All reasoning models. Lilac uses `chat_template_kwargs` (with `thinking` and `enable_thinking` keys) to toggle reasoning. Pi sends both keys for forward compatibility.
+- **`thinkingFormat: "chat-template"`** — All reasoning models. Lilac's vLLM backend toggles reasoning via `chat_template_kwargs`, but the honored key differs per model family. Per-model `chatTemplateKwargs` in `patch.json` send the right key(s): `thinking`+`enable_thinking` (bool) for Kimi K2.6, GLM 5.1, Gemma 4, and MiniMax M2.7; `enable_thinking` + `reasoning_effort` for GLM 5.2; `thinking_mode` (adaptive|enabled|disabled) for MiniMax M3.
 - **`maxTokensField: "max_completion_tokens"`** — All models. Lilac supports `max_completion_tokens` (preferred for reasoning models as it includes reasoning tokens).
 - **`supportsDeveloperRole: true`** — All models. Lilac's vLLM backend maps the developer role to system.
 - **`supportsStore: false`** — All models. Lilac doesn't support the `store` parameter.

package/index.ts

@@ -5,14 +5,28 @@
  * Base URL: https://api.getlilac.com/v1
  *
  * Lilac serves models via a customized fork of vLLM tuned for idle-GPU scheduling
- * and shared warm endpoints. All models use chat_template_kwargs to toggle reasoning:
+ * and shared warm endpoints. Reasoning is toggled via chat_template_kwargs, but
+ * the key each model's chat template honors differs per family, so per-model
+ * chatTemplateKwargs are configured in patch.json:
  *
- *   - Kimi K2.6: reasoning ON by default, honors `thinking` key
- *   - GLM 5.1:   reasoning ON by default, honors `enable_thinking` key
- *   - Gemma 4:   reasoning OFF by default, honors `enable_thinking` key
+ *   - Kimi K2.6:    honors `thinking` (bool); `enable_thinking` ignored. ON by default.
+ *   - GLM 5.1:      honors `enable_thinking` (bool). ON by default.
+ *   - GLM 5.2:      honors `enable_thinking` (bool) + `reasoning_effort` (max|high). ON by default.
+ *   - Gemma 4:      honors `enable_thinking` (bool). OFF by default.
+ *   - MiniMax M2.7: forward-compatible `thinking` + `enable_thinking` (bool).
+ *   - MiniMax M3:   honors `thinking_mode` (disabled|adaptive|enabled); bool keys ignored.
  *
- * The forward-compatible approach is to send both `thinking` and `enable_thinking`
- * in chat_template_kwargs — pi's `qwen-chat-template` thinkingFormat does this.
+ * We use pi's `chat-template` thinkingFormat (NOT `qwen-chat-template`, which
+ * sends only `enable_thinking` + `preserve_thinking` and is ignored by Kimi and
+ * MiniMax M3). The forward-compatible form sends BOTH `thinking` and
+ * `enable_thinking` so whichever key the template honors is set:
+ *   { chat_template_kwargs: { thinking: <bool>, enable_thinking: <bool> } }
+ * GLM 5.2 adds `reasoning_effort` (high = lower-latency, xhigh = max) via a
+ * thinkingLevelMap. MiniMax M3 maps to the `thinking_mode` enum as three pi
+ * thinking levels — off→disabled, minimal→adaptive (model decides), high→enabled
+ * — so adaptive is selectable via pi's Shift+Tab cycle (off→minimal→high). Pi
+ * shows the pi level names (minimal/high) in the selector/footer, not the
+ * thinking_mode values; there's no per-model level-relabel hook.
  *
  * Key API notes:
  *   - Uses `max_completion_tokens` (preferred for reasoning models)
@@ -99,6 +113,17 @@ type ThinkingLevelMap = {
   xhigh?: string | null;
 };
 
+// A chat_template_kwargs value, mirroring pi-ai's ChatTemplateKwargSchema. Scalar
+// values are passed through verbatim; { $var } values are resolved by pi-ai from
+// the turn's thinking state ("thinking.enabled" → bool, "thinking.effort" → the
+// mapped effort string). omitWhenOff drops the key entirely when thinking is off.
+type ChatTemplateKwargValue =
+  | string
+  | number
+  | boolean
+  | null
+  | { $var: "thinking.enabled" | "thinking.effort"; omitWhenOff?: boolean };
+
 interface JsonModel {
   id: string;
   name: string;
@@ -117,7 +142,19 @@ interface JsonModel {
     supportsDeveloperRole?: boolean;
     supportsStore?: boolean;
     maxTokensField?: "max_completion_tokens" | "max_tokens";
-    thinkingFormat?: "openai" | "zai" | "qwen" | "qwen-chat-template";
+    thinkingFormat?:
+      | "openai"
+      | "openrouter"
+      | "together"
+      | "deepseek"
+      | "zai"
+      | "qwen"
+      | "chat-template"
+      | "qwen-chat-template"
+      | "string-thinking"
+      | "ant-ling";
+    chatTemplateKwargs?: Record<string, ChatTemplateKwargValue>;
+    zaiToolStream?: boolean;
     supportsReasoningEffort?: boolean;
   };
   discount?: JsonDiscount;
@@ -252,14 +289,21 @@ function transformApiModel(apiModel: any): JsonModel | null {
     maxTokens: apiModel.top_provider?.max_completion_tokens || apiModel.context_length || 131072,
   };
 
-  // All Lilac models use chat_template_kwargs for reasoning toggle
+  // All Lilac models toggle reasoning via chat_template_kwargs, but the key each
+  // model's chat template honors differs per family. Default newly discovered
+  // models to the forward-compatible both-keys form (works across all current
+  // Lilac templates); per-model overrides in patch.json refine this — e.g. GLM
+  // 5.2 adds reasoning_effort, MiniMax M3 uses the thinking_mode enum.
   if (features.includes("reasoning")) {
     model.compat = {
       supportsDeveloperRole: true,
       supportsStore: false,
       maxTokensField: "max_completion_tokens",
-      thinkingFormat: "qwen-chat-template",
-      supportsReasoningEffort: true,
+      thinkingFormat: "chat-template",
+      chatTemplateKwargs: {
+        thinking: { $var: "thinking.enabled" },
+        enable_thinking: { $var: "thinking.enabled" },
+      },
     };
   } else {
     model.compat = {
@@ -538,7 +582,19 @@ let cachedApiKey: string | undefined;
 let revalidateAbort: AbortController | null = null;
 let latestDiscounts: Map<string, JsonDiscount> | null = null;
 let lastDiscountFetchTime = 0;
-const STATUS_CACHE_TTL_MS = 30000;
+// Turn-initiated /status fetches are throttled to once per TTL window so a burst
+// of messages doesn't hammer the endpoint. Background polling (see
+// STATUS_POLL_INTERVAL_MS) and turn fetches both stamp lastDiscountFetchTime, so
+// they cooperate: a poll that just ran lets the next turn skip its own fetch
+// within the TTL.
+const STATUS_CACHE_TTL_MS = 60000;
+// Lilac refreshes discounts ~every 10 minutes (per their docs: "Discounts refresh
+// approximately every 10 minutes and are locked in when a request starts"). Poll
+// on that cadence during idle so a long-idle session still catches supply/sub
+// changes without waiting for the user to send a message — turn fetches alone
+// only refresh on a user message and are TTL-throttled to 1/min.
+const STATUS_POLL_INTERVAL_MS = 10 * 60 * 1000;
+let pollInterval: ReturnType<typeof setInterval> | null = null;
 // List-price (patch-applied, pre-discount) models, cached until the base set
 // changes. Reset in cacheModels() so the next getListModels() rebuilds from the
 // refreshed disk cache / embedded set.
@@ -599,11 +655,49 @@ export default function (pi: ExtensionAPI) {
     }
   }
 
+  /**
+   * Background /status poll, fired every STATUS_POLL_INTERVAL_MS (10 min) from
+   * session_start to cover idle sessions. Mirrors the discount half of
+   * before_provider_request, but without an in-flight turn model to mutate: it
+   * only refreshes latestDiscounts, re-registers (so the next turn's models carry
+   * the new price), and re-paints the footer from the LIVE model. Passes the
+   * session AbortSignal so the fetch dies on session_shutdown or a subsequent
+   * session_start; bails on a missing API key or an aborted signal.
+   */
+  function pollStatusDiscounts(ctx: any, signal: AbortSignal): void {
+    if (signal.aborted || !cachedApiKey) return;
+    fetchStatusDiscounts(cachedApiKey, signal).then(discounts => {
+      if (signal.aborted || !discounts) return;
+      lastDiscountFetchTime = Date.now();
+      if (!discountsChanged(latestDiscounts, discounts)) {
+        syncStatus(ctx);
+        return;
+      }
+      cacheDiscounts(discounts);
+      latestDiscounts = discounts;
+      const freshList = getListModels();
+      pi.registerProvider("lilac", {
+        baseUrl: BASE_URL,
+        apiKey: "$LILAC_API_KEY",
+        api: "openai-completions",
+        models: applyDiscounts(freshList, discounts),
+      });
+      syncStatus(ctx);
+    }).catch(() => { /* network errors are non-fatal */ });
+  }
+
   pi.on("session_start", async (_event, ctx) => {
     revalidateAbort?.abort();
     revalidateAbort = new AbortController();
     const signal = revalidateAbort.signal;
 
+    // Tear down any poll interval left from a prior session before starting a
+    // fresh one (defensive; session_shutdown normally handles this).
+    if (pollInterval) {
+      clearInterval(pollInterval);
+      pollInterval = null;
+    }
+
     // Replay persisted discount state from session JSONL (synchronous, zero-latency)
     replayDiscountEvents(ctx);
 
@@ -653,6 +747,15 @@ export default function (pi: ExtensionAPI) {
         syncStatus(ctx);
       }).catch(() => { /* network errors are non-fatal */ });
     });
+
+    // Background poll for idle sessions: Lilac refreshes discounts ~every 10
+    // minutes, so poll on that cadence to catch supply/sub changes while the
+    // user is idle (turn fetches only run when a message is sent). The callback
+    // bails on a missing API key or an aborted/shut-down session. Cleared in
+    // session_shutdown and at the top of the next session_start.
+    pollInterval = setInterval(() => pollStatusDiscounts(ctx, signal), STATUS_POLL_INTERVAL_MS);
+    // Don't keep the process alive solely for discount polling.
+    pollInterval.unref?.();
   });
 
   pi.on("turn_end", async (_event, ctx) => {
@@ -776,6 +879,10 @@ export default function (pi: ExtensionAPI) {
 
   pi.on("session_shutdown", () => {
     revalidateAbort?.abort();
+    if (pollInterval) {
+      clearInterval(pollInterval);
+      pollInterval = null;
+    }
   });
 }
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-lilac-provider",
-  "version": "1.2.1",
+  "version": "1.4.0",
   "description": "Lilac provider extension for pi - Access Kimi K2.6, GLM 5.1, and Gemma 4 models through Lilac's OpenAI-compatible API on idle GPUs",
   "type": "module",
   "main": "index.ts",

package/patch.json

@@ -7,11 +7,14 @@
       "cacheRead": 0.20
     },
     "compat": {
-      "thinkingFormat": "qwen-chat-template",
+      "thinkingFormat": "chat-template",
+      "chatTemplateKwargs": {
+        "thinking": { "$var": "thinking.enabled" },
+        "enable_thinking": { "$var": "thinking.enabled" }
+      },
       "maxTokensField": "max_completion_tokens",
       "supportsDeveloperRole": false,
-      "supportsStore": false,
-      "supportsReasoningEffort": true
+      "supportsStore": false
     }
   },
   "zai-org/glm-5.1": {
@@ -23,20 +26,29 @@
       "cacheRead": 0.27
     },
     "compat": {
-      "thinkingFormat": "qwen-chat-template",
+      "thinkingFormat": "chat-template",
+      "chatTemplateKwargs": {
+        "thinking": { "$var": "thinking.enabled" },
+        "enable_thinking": { "$var": "thinking.enabled" }
+      },
       "maxTokensField": "max_completion_tokens",
       "supportsDeveloperRole": false,
       "supportsStore": false,
-      "zaiToolStream": true,
-      "supportsReasoningEffort": true
+      "zaiToolStream": true
     }
   },
   "zai-org/glm-5.2": {
+    "compat": {
+      "thinkingFormat": "chat-template",
+      "chatTemplateKwargs": {
+        "enable_thinking": { "$var": "thinking.enabled" },
+        "reasoning_effort": { "$var": "thinking.effort", "omitWhenOff": true }
+      }
+    },
     "thinkingLevelMap": {
-      "off": "minimal",
       "minimal": null,
-      "low": null,
-      "medium": null,
+      "low": "high",
+      "medium": "high",
       "high": "high",
       "xhigh": "max"
     }
@@ -49,16 +61,40 @@
       "cacheRead": 0
     },
     "compat": {
-      "thinkingFormat": "qwen-chat-template",
+      "thinkingFormat": "chat-template",
+      "chatTemplateKwargs": {
+        "thinking": { "$var": "thinking.enabled" },
+        "enable_thinking": { "$var": "thinking.enabled" }
+      },
       "maxTokensField": "max_completion_tokens",
       "supportsDeveloperRole": true,
-      "supportsStore": false,
-      "supportsReasoningEffort": true
+      "supportsStore": false
     }
   },
   "minimaxai/minimax-m2.7": {
     "compat": {
+      "thinkingFormat": "chat-template",
+      "chatTemplateKwargs": {
+        "thinking": { "$var": "thinking.enabled" },
+        "enable_thinking": { "$var": "thinking.enabled" }
+      },
       "supportsDeveloperRole": false
     }
+  },
+  "minimaxai/minimax-m3": {
+    "compat": {
+      "thinkingFormat": "chat-template",
+      "chatTemplateKwargs": {
+        "thinking_mode": { "$var": "thinking.effort" }
+      }
+    },
+    "thinkingLevelMap": {
+      "off": "disabled",
+      "minimal": "adaptive",
+      "low": null,
+      "medium": null,
+      "high": "enabled",
+      "xhigh": null
+    }
   }
 }

package/scripts/test-discounts.ts

@@ -11,12 +11,14 @@
  * 5. session_start replays persisted discount events and sets footer status.
  * 6. model_select sets/clears footer status for lilac/non-lilac models.
  * 7. turn_end appends discount entry to session JSONL.
- * 8. before_provider_request refreshes discounts with a 30s cache.
+ * 8. before_provider_request refreshes discounts with a 60s cache.
  * 9. formatDiscountStatus returns fallbacks when data is missing.
  * 10. applyDiscountInPlace mutates the in-flight model's cost in place,
  *     recomputed from list price so re-applied discounts never compound.
  * 11. before_provider_request mutates the bound (in-flight) model object so the
  *     current turn's cost calc sees the discount in real time.
+ * 12. session_start schedules a 10-minute background /status poll to cover idle
+ *     sessions; session_shutdown clears it.
  */
 
 import type { ExtensionAPI, ModelRegistry } from "@earendil-works/pi-coding-agent";
@@ -596,15 +598,16 @@ assert(
 
 console.log("\n--- Test 15: before_provider_request clears status after switch ---");
 
-// Regression: every 30s (when the discount TTL expires) before_provider_request
+// Regression: every 60s (when the discount TTL expires) before_provider_request
 // awaits a fresh /status fetch. If the user switches to a non-lilac model during
 // that await, the post-await paint must clear (live model) instead of re-painting
 // the stale captured in-flight lilac model. Cost mutation still targets the
 // captured in-flight model (Tests 12-13); only the DISPLAY follows the live model.
 
-// Force the 30s TTL to look expired so the fetch path executes.
+// Force the 60s TTL to look expired so the fetch path executes. Offset 120s for
+// a clear 2x margin over the 60s TTL (the original used 60s over a 30s TTL).
 const realDateNow = Date.now;
-Date.now = () => realDateNow.call(Date) + 60000;
+Date.now = () => realDateNow.call(Date) + 120000;
 try {
   globalThis.fetch = mockFetch({
     "/status": {
@@ -697,6 +700,86 @@ assert(
   "deferred session_start paints the NEW lilac model's discount (glm), not the stale kimi capture",
 );
 
+// ─── Test 17: session_start schedules a 10-min idle poll; shutdown clears it ─
+
+console.log("\n--- Test 17: session_start schedules idle poll; session_shutdown clears it ---");
+
+// Lilac refreshes discounts ~every 10 minutes. session_start must schedule a
+// background /status poll at that cadence to cover idle sessions (turn fetches
+// only run when the user sends a message), and session_shutdown must clear it so
+// it neither leaks nor keeps the process alive. Wrap the global timer APIs to
+// capture the scheduled delay + handle, then confirm shutdown clears it.
+
+const realSetInterval = globalThis.setInterval.bind(globalThis);
+const realClearInterval = globalThis.clearInterval.bind(globalThis);
+let scheduledDelay: number | null = null;
+let scheduledHandle: ReturnType<typeof setInterval> | null = null;
+const clearedHandles = new Set<ReturnType<typeof setInterval>>();
+
+globalThis.setInterval = ((fn: (...args: any[]) => void, delay?: number, ...rest: any[]) => {
+  scheduledDelay = delay ?? null;
+  const h = realSetInterval(fn, delay as any, ...rest);
+  scheduledHandle = h;
+  return h;
+}) as any;
+globalThis.clearInterval = ((handle: ReturnType<typeof setInterval>) => {
+  clearedHandles.add(handle);
+  return realClearInterval(handle);
+}) as any;
+
+try {
+  // Benign fetch mock so the session_start fire-and-forget /models + /status
+  // fetch doesn't hit the network. (The poll itself never fires — 10 min — so
+  // only the startup fetch needs mocking here.)
+  globalThis.fetch = mockFetch({
+    "/models": { body: { data: [] } },
+    "/status": {
+      body: {
+        models: [
+          {
+            id: "moonshotai/kimi-k2.6",
+            current_subscription_supply_state: "healthy",
+            current_subscription_discount_percent: 25,
+            current_subscription_credit_multiplier: "0.75",
+          },
+        ],
+      },
+    },
+  }) as any;
+
+  const pollCtx: any = {
+    modelRegistry: mockRegistry,
+    ui: mockUi,
+    model: { id: "moonshotai/kimi-k2.6", provider: "lilac" },
+    sessionManager: { getBranch: () => [] },
+  };
+
+  for (const handler of handlers.get("session_start") || []) {
+    await handler({}, pollCtx);
+  }
+
+  assert(
+    scheduledDelay === 10 * 60 * 1000,
+    "session_start schedules a 10-minute (600000ms) /status poll for idle sessions",
+  );
+  assert(scheduledHandle !== null, "poll interval handle was captured");
+  const capturedHandle = scheduledHandle;
+
+  for (const handler of handlers.get("session_shutdown") || []) {
+    await handler({}, pollCtx);
+  }
+
+  assert(
+    capturedHandle !== null && clearedHandles.has(capturedHandle),
+    "session_shutdown clears the scheduled poll interval (no leak / process-hang)",
+  );
+} finally {
+  // Restore globals; clear any interval this test scheduled before it can fire.
+  if (scheduledHandle) realClearInterval(scheduledHandle);
+  globalThis.setInterval = realSetInterval as any;
+  globalThis.clearInterval = realClearInterval as any;
+}
+
 // ─── Cleanup ──────────────────────────────────────────────────────────────────
 
 globalThis.fetch = originalFetch;

package/scripts/update-models.js

@@ -147,7 +147,11 @@ function transformApiModel(apiModel, existingModelsMap) {
     maxTokens: apiModel.top_provider?.max_completion_tokens || apiModel.context_length || 131072,
   };
 
-  // Add compat — all Lilac models use chat_template_kwargs for reasoning toggle
+  // Add compat — all Lilac models toggle reasoning via chat_template_kwargs, but
+  // the honored key differs per model family. Default newly discovered models to
+  // the forward-compatible both-keys form (works across all current Lilac
+  // templates); refine per-model in patch.json (e.g. GLM 5.2 adds reasoning_effort,
+  // MiniMax M3 uses the thinking_mode enum).
   const compat = {
     supportsDeveloperRole: true,
     supportsStore: false,
@@ -155,7 +159,11 @@ function transformApiModel(apiModel, existingModelsMap) {
   };
 
   if (hasReasoning) {
-    compat.thinkingFormat = 'qwen-chat-template';
+    compat.thinkingFormat = 'chat-template';
+    compat.chatTemplateKwargs = {
+      thinking: { $var: 'thinking.enabled' },
+      enable_thinking: { $var: 'thinking.enabled' },
+    };
   }
 
   model.compat = compat;