pi-cache-optimizer 2.8.22 → 2.8.23

package/README.md

@@ -21,6 +21,7 @@ Pi extension for improving provider-side KV / prompt cache hit rates. It keeps s
 - [Opt-in deterministic tool ordering](#opt-in-deterministic-tool-ordering)
 - [Footer cache stats mode](#footer-cache-stats-mode)
 - [OpenAI-compatible proxy setup](#openai-compatible-proxy-setup)
+- [Prompt cache warming (Pi 0.86+)](#prompt-cache-warming-pi-086)
 - [Adaptive thinking models](#adaptive-thinking-models)
 - [Auto-repair with `/cache-optimizer fix`](#auto-repair-with-cache-optimizer-fix)
 - [DeepSeek protocol safety and rollback](#deepseek-protocol-safety-and-rollback)
@@ -71,7 +72,7 @@ This extension requires Pi 0.82+ and is validated against Pi 1.1.0. It uses the
 | `/cache-optimizer` | Interactive menu when UI supports it; otherwise prints help and current state. |
 | `/cache-optimizer enable` | Enables runtime optimizations for the current Pi process, resets local footer stats, and starts a fresh “enabled” measurement. |
 | `/cache-optimizer disable` | Disables optimization for the current Pi process, resets local footer stats, and keeps collecting footer stats in disabled comparison mode. Run `/reload` or restart Pi to return to startup behavior. |
-| `/cache-optimizer doctor` | Shows active model/provider/API/base URL/compat, whether the skill list was compressed on the last prompt (and why not, e.g. an unrecognised Pi format), plus low-hit diagnosis. |
+| `/cache-optimizer doctor` | Shows active model/provider/API/base URL/compat, whether the skill list was compressed on the last prompt (and why not, e.g. an unrecognised Pi format), plus Pi cache-warming eligibility and low-hit diagnosis. |
 | `/cache-optimizer compat` | Shows copyable compat advice for the active model, if applicable. |
 | `/cache-optimizer stats` | Shows detailed counters for every cache-adapter-matched model used by the current conversation session today. |
 | `/cache-optimizer stats all` | Shows detailed per-model totals across all valid local sessions/shards today, including request and token counts. |
@@ -203,6 +204,14 @@ Notes:
 - Do not add `thinkingFormat: "deepseek"` merely because a model id contains `deepseek`; `openai`, `qwen`, `openrouter`, `together`, or no explicit format are all valid catalog cases.
 - This extension's `doctor` and `compat` commands only advise; they do not modify `models.json`.
 
+## Prompt cache warming (Pi 0.86+)
+
+Pi itself can keep a prompt cache alive by re-sending the last request with a one-token output cap shortly before the entry expires (the global `cacheWarming` setting: `off`, `streaming` (default), or `idle`; change it with `/settings` → Cache warming or `"cacheWarming"` in `~/.pi/agent/settings.json`). This extension does not run a keepalive timer of its own: a second warmer would pay for duplicate refreshes and cannot replay Pi's exact request.
+
+Pi only warms a model that declares a cache lifetime for the retention tier in use, has input/cache prices, and is replayable (Anthropic budget-based thinking is not). Custom proxy models in `models.json` often lack the first two, so warming never starts. `/cache-optimizer doctor` now reports the mode, the tier, and what is missing, with a snippet such as `"promptCache": {"short":300,"long":3600}` for `anthropic-messages` models. Put it on the model entry (custom models) or under the provider's `modelOverrides` (built-in models). It is advice only; the extension does not write it for you.
+
+Streaming warming stops 60 minutes after the real request and idle warming after 30 minutes. With this extension's 1-hour retention, the refresh would fire at 54 minutes, so `idle` adds nothing; the 1-hour lifetime already covers breaks under an hour. Keep the default `streaming` unless you use 5-minute caches and often return within half an hour.
+
 ## Anthropic cache TTL compatibility
 
 Anthropic processes cache breakpoints in `tools → system → messages` order and rejects a `ttl: "1h"` breakpoint that appears after a 5-minute breakpoint. An ephemeral `cache_control` without `ttl` uses the default 5-minute retention.

package/README.zh-CN.md

@@ -21,6 +21,7 @@
 - [Opt-in 确定性工具排序](#opt-in-确定性工具排序)
 - [Footer 缓存统计模式](#footer-缓存统计模式)
 - [OpenAI-compatible 代理配置](#openai-compatible-代理配置)
+- [Prompt cache 保温（Pi 0.86+）](#prompt-cache-保温pi-086)
 - [Adaptive thinking 模型](#adaptive-thinking-模型)
 - [使用 `/cache-optimizer fix` 自动修复](#使用-cache-optimizer-fix-自动修复)
 - [DeepSeek 协议安全与回滚](#deepseek-协议安全与回滚)
@@ -71,7 +72,7 @@ Pi 0.79.7 及之后，`pi update` 默认只更新 Pi 本体。若要更新已安
 | `/cache-optimizer` | UI 支持时打开交互菜单；否则打印帮助和当前状态。 |
 | `/cache-optimizer enable` | 在当前 Pi 进程中开启运行时优化，清零本地 footer 统计，并开始新的“开启状态”测量。 |
 | `/cache-optimizer disable` | 在当前 Pi 进程中关闭优化，清零本地 footer 统计，并继续以 disabled 对比模式采集 footer 统计。运行 `/reload` 或重启 Pi 后回到启动时行为。 |
-| `/cache-optimizer doctor` | 显示当前模型 / provider / API / base URL / compat、上一次提示中 skill 列表是否被压缩（未压缩时给出原因，例如无法识别 Pi 的格式）以及低命中诊断。 |
+| `/cache-optimizer doctor` | 显示当前模型 / provider / API / base URL / compat、上一次提示中 skill 列表是否被压缩（未压缩时给出原因，例如无法识别 Pi 的格式）、Pi 缓存保温是否生效以及低命中诊断。 |
 | `/cache-optimizer compat` | 对当前模型显示可复制的 compat 建议（如适用）。 |
 | `/cache-optimizer stats` | 显示当前 conversation session 今天使用过的各 cache-adapter-matched 模型详细统计。 |
 | `/cache-optimizer stats all` | 显示所有有效本地 session/shard 今天的逐模型详细总计，包括请求数与 token 数。 |
@@ -203,6 +204,14 @@ Pi 0.84.1 还修复了内置 Fireworks 渠道对拒绝 `prompt_cache_retention`
 - 不要因为模型 id 含有 `deepseek` 就添加 `thinkingFormat: "deepseek"`；catalog 中也存在 `openai`、`qwen`、`openrouter`、`together` 或没有显式 format 的情况。
 - 本扩展的 `doctor` 和 `compat` 命令只给建议，不会修改 `models.json`。
 
+## Prompt cache 保温（Pi 0.86+）
+
+Pi 自带缓存保温：在缓存条目过期前，用一个 token 的输出上限原样重放上一次请求（全局设置 `cacheWarming`：`off`、`streaming`（默认）或 `idle`；在 `/settings` → Cache warming 中修改，或在 `~/.pi/agent/settings.json` 中设置 `"cacheWarming"`）。本扩展不自行运行保活定时器：第二个保温器会重复计费，而且无法精确重放 Pi 的请求。
+
+Pi 只会为满足以下条件的模型保温：为当前保留档位声明了缓存时长、有输入/缓存价格、请求可重放（Anthropic 基于 budget 的 thinking 不可重放）。`models.json` 里的自定义代理模型常常缺前两项，保温根本不会启动。`/cache-optimizer doctor` 现在会显示保温模式、档位和缺失项，并给出片段，例如 `anthropic-messages` 模型的 `"promptCache": {"short":300,"long":3600}`。自定义模型写在该模型条目中，内置模型写在 provider 的 `modelOverrides` 下。这只是建议，扩展不会替你写入。
+
+streaming 保温在真实请求后 60 分钟停止，idle 保温在 30 分钟后停止。本扩展使用 1 小时保留时，刷新会在第 54 分钟才触发，所以 `idle` 没有额外作用；1 小时的缓存本身已覆盖一小时内的中断。除非你使用 5 分钟缓存且经常在半小时内回来，否则保持默认的 `streaming` 即可。
+
 ## Anthropic 缓存 TTL 兼容
 
 Anthropic 按 `tools → system → messages` 顺序处理 cache breakpoint，并拒绝位于 5 分钟 breakpoint 之后的 `ttl: "1h"` breakpoint。省略 `ttl` 的 ephemeral `cache_control` 使用默认 5 分钟保留时间。

package/index.ts

@@ -16,6 +16,7 @@ import { analyzeModelsJsonForMissingEntry, applyModelsJsonFixTransaction, choose
 import { type CacheProviderId, type CacheStats, LEGACY_STATE_FILE_PATH, type PersistedRoutedModelRef, type PersistedStatsShardV7, SHARD_FILES_DIR, SHARD_GLOBAL_EPOCH_PATH, SHARD_STATE_DIR, STATE_FILE_PATH, type ShardAggregate, type UsageSnapshot, addUsageToCacheStats, advanceGlobalStatsEpoch, advanceModelStatsEpoch, aggregateStatsShardsV7, cleanupStatsShardsV7, cloneCacheStats, currentLocalDay, emptyAllCacheStats, emptyCacheStats, initialEpoch, loadStatsShardAggregateV7, maybeCleanupStatsShardsV7, modelEpochPath, parseCacheStats, parsePersistedRoutedModelRef, parsePersistedStatsShardV7, readGlobalStatsEpoch, readModelStatsEpoch, readValidStatsShardsV7, removeLegacyStatsFiles, writeStatsShardV7 } from "./src/stats-store.ts";
 import { PI_VIRTUAL_MODEL_API, getCompat, isAnthropicMessagesApi, isKimiCodingEmptySignatureModel, isKnownThirdPartyOpenAIEndpoint, isMistralConversationsApi, isNativeVirtualModel, isOfficialOpenAIBaseUrl, isOpenAICompatibleApi, isOpenAICompatibleProxyApi, isOpenAIFamilyAssistantMessage, isOpenAIFamilyModel, isOpenAIFamilyToken, isPiBuiltInLlamaCppModel, isResponsesPromptRewriteBypassApi, isRoutedFallbackModel, isValidModelsConfigForEffectiveCompat, modelKey, readEffectiveCompatConfig } from "./src/model-identity.ts";
 import { CONFIG_FILE_PATH, VIRTUAL_REWRITE_ENV, FOOTER_MODE_ENV, TOOL_ORDER_ENV, featureEnabled, footerStatsMode, isEnabledEnv, isToolOrderEnabled, parseFooterStatsMode, parsePersistedCacheOptimizerConfig, readPersistedCacheOptimizerConfig, resolveFooterStatsMode, runtimeOptimizerEnabled, setPersistedCacheOptimizerConfig, shouldInjectOpenAIPromptCacheKey, writePersistedCacheOptimizerConfig, writePersistedFeature, writePersistedFooterMode } from "./src/config.ts";
+import { readCacheWarmingMode } from "./src/cache-warming.ts";
 import { LONG_CACHE_RETENTION_VALUE, PI_CACHE_RETENTION_BASELINE_SYMBOL, PI_CACHE_RETENTION_ENV, STARTUP_CACHE_RETENTION_ENV, captureCacheRetentionEnv, getOrCaptureCacheRetentionBaseline, decidePromptCacheRetention, recordPromptCacheRetentionDecision, requestLongCacheRetention, restoreCacheRetentionEnv, userRequestedLongCacheRetention } from "./src/retention.ts";
 import { isRuntimeOptimizerEnabled, setRuntimeOptimizerEnabled } from "./src/config.ts";
 import { compareToolOrderEntries, getToolNameForPayload, isKnownToolOrderApi, isToolOrderingEligibleModel, isVerifiedToolForApi, normalizeToolsInPayload, sortToolsInPayload } from "./src/tool-ordering.ts";
@@ -1611,6 +1612,7 @@ export default function (pi: ExtensionAPI) {
     getCurrentSessionHash: () => currentSessionHash,
     getLastStatusText: () => lastStatusText,
     clearLastStatusText: () => { lastStatusText = undefined; },
+    getCacheWarmingMode: () => readCacheWarmingMode(typeof pi.getSettings === "function" ? () => pi.getSettings() : undefined),
   });
 
   pi.registerCommand("cache-optimizer", {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-cache-optimizer",
-  "version": "2.8.22",
+  "version": "2.8.23",
   "description": "Improve Pi prompt/KV cache hit rates with stable prompts, OpenAI-compatible cache keys, proxy compat warnings, and footer cache stats.",
   "keywords": [
     "pi-package",

package/src/cache-warming.ts

@@ -0,0 +1,150 @@
+import { type PiModel, asRecord } from "./common.ts";
+
+/**
+ * Diagnostics for Pi's native prompt-cache warming (Pi 0.86+, `cacheWarming`
+ * setting). Pi re-sends the last request with a one-token output cap at 90% of
+ * the cache lifetime, but only when:
+ *   - the global `cacheWarming` setting is not "off";
+ *   - the model declares `promptCache.<tier>` for the retention tier in use;
+ *   - the model's prices let Pi estimate the savings (cost.cacheRead/cacheWrite/input);
+ *   - the request is replayable (Anthropic budget-based thinking is not).
+ * Streaming warming stops 60 minutes after the real request and idle warming
+ * after 30 minutes, so idle mode never refreshes a 1-hour cache entry (its
+ * refresh would fire at 54 minutes).
+ *
+ * This module only reads state and explains it; it never changes settings or
+ * models.json. The extension does not run its own keepalive timer, because a
+ * second warmer would double-bill refreshes and could not replay Pi's exact
+ * request context.
+ */
+
+export type CacheWarmingMode = "off" | "streaming" | "idle";
+
+export const CACHE_WARMING_DEFAULT_MODE: CacheWarmingMode = "streaming";
+export const CACHE_WARMING_STREAMING_MAX_MS = 60 * 60_000;
+export const CACHE_WARMING_IDLE_MAX_MS = 30 * 60_000;
+
+/** Same schedule Pi uses: 90% of the TTL, keeping at least ten seconds of margin. */
+export function cacheWarmingRefreshDelayMs(ttlMs: number): number | undefined {
+  if (ttlMs <= 10_000) return undefined;
+  return Math.max(1, Math.floor(Math.min(ttlMs * 0.9, ttlMs - 10_000)));
+}
+
+export type CacheWarmingSettingsReader = () => unknown;
+
+/**
+ * Read the `cacheWarming` mode from `pi.getSettings()` (Pi 0.86+). Returns
+ * undefined when the host has no settings API or no warming support.
+ */
+export function readCacheWarmingMode(getSettings: CacheWarmingSettingsReader | undefined): CacheWarmingMode | "unsupported" {
+  if (typeof getSettings !== "function") return "unsupported";
+  let settings: unknown;
+  try {
+    settings = getSettings();
+  } catch {
+    return "unsupported";
+  }
+  const value = asRecord(settings)?.cacheWarming;
+  if (value === "off" || value === "streaming" || value === "idle") return value;
+  return CACHE_WARMING_DEFAULT_MODE;
+}
+
+export function suggestedPromptCacheLifetimes(model: PiModel): { short: number; long?: number } {
+  // Anthropic documents a 5-minute default and an optional 1-hour TTL. For
+  // other APIs only the conservative short lifetime is suggested; extended
+  // retention differs per provider and deployment.
+  if (model.api === "anthropic-messages") return { short: 300, long: 3600 };
+  return { short: 300 };
+}
+
+function readPromptCacheTier(model: PiModel, tier: "short" | "long"): number | undefined {
+  const promptCache = asRecord((model as unknown as Record<string, unknown>).promptCache);
+  const value = promptCache?.[tier];
+  return typeof value === "number" && Number.isFinite(value) && value > 0 ? value : undefined;
+}
+
+function hasPricing(model: PiModel): boolean {
+  const cost = asRecord((model as unknown as Record<string, unknown>).cost);
+  if (!cost) return false;
+  return ["input", "cacheRead", "cacheWrite"].some((key) => typeof cost[key] === "number" && (cost[key] as number) > 0);
+}
+
+function isReplayableForWarming(model: PiModel): boolean {
+  if (!model.reasoning || model.api !== "anthropic-messages") return true;
+  return asRecord(asRecord(model as unknown)?.compat)?.forceAdaptiveThinking === true;
+}
+
+export type CacheWarmingDiagnosisInput = {
+  mode: CacheWarmingMode | "unsupported";
+  /** Retention tier Pi uses for this request ("long" when PI_CACHE_RETENTION=long). */
+  tier: "short" | "long";
+};
+
+export type CacheWarmingDiagnosis = {
+  eligible: boolean;
+  issues: string[];
+  lines: string[];
+};
+
+function formatSeconds(seconds: number): string {
+  if (seconds % 3600 === 0) return `${seconds / 3600}h`;
+  if (seconds % 60 === 0) return `${seconds / 60}m`;
+  return `${seconds}s`;
+}
+
+export function buildCacheWarmingDiagnosis(model: PiModel, input: CacheWarmingDiagnosisInput): CacheWarmingDiagnosis {
+  const lines: string[] = ["", "🔥 Prompt cache warming (Pi cacheWarming):"];
+  const issues: string[] = [];
+
+  if (input.mode === "unsupported") {
+    lines.push("- This Pi host does not expose cache warming settings (requires Pi 0.86+). Nothing to check.");
+    return { eligible: false, issues: ["unsupported"], lines };
+  }
+
+  lines.push(`- Mode: ${input.mode}${input.mode === CACHE_WARMING_DEFAULT_MODE ? " (default)" : ""} — global setting, change with /settings → Cache warming or ~/.pi/agent/settings.json`);
+  lines.push(`- Retention tier in use: ${input.tier}`);
+
+  if (input.mode === "off") {
+    issues.push("mode_off");
+    lines.push("- Warming is off: long tool runs can outlive the cache entry and the next request rewrites it.");
+  }
+
+  const ttl = readPromptCacheTier(model, input.tier);
+  if (ttl === undefined) {
+    issues.push("prompt_cache_missing");
+    const suggested = suggestedPromptCacheLifetimes(model);
+    const snippet: Record<string, number> = { short: suggested.short };
+    if (suggested.long !== undefined) snippet.long = suggested.long;
+    lines.push(`- ⚠️ The model declares no promptCache.${input.tier} lifetime, so Pi never warms its cache.`);
+    lines.push(`  Add to this model's entry in models.json (custom models) or under the provider's modelOverrides.${JSON.stringify(model.id)} (built-in models):`);
+    lines.push(`  "promptCache": ${JSON.stringify(snippet)}`);
+    if (input.tier === "long" && suggested.long === undefined) {
+      lines.push("  Only add a long lifetime your provider documents; use the conservative end of any range.");
+    }
+  } else {
+    lines.push(`- promptCache.${input.tier}: ${formatSeconds(ttl)}`);
+  }
+
+  if (!hasPricing(model)) {
+    issues.push("pricing_missing");
+    lines.push("- ⚠️ The model has no input/cache prices, so Pi cannot estimate savings and skips warming. Add a cost block to the model entry.");
+  }
+
+  if (!isReplayableForWarming(model)) {
+    issues.push("not_replayable");
+    lines.push("- ⚠️ Anthropic budget-based thinking cannot be replayed with a one-token cap, so Pi skips warming. Set compat.forceAdaptiveThinking only if the model supports adaptive thinking.");
+  }
+
+  if (ttl !== undefined && input.mode === "idle") {
+    const delay = cacheWarmingRefreshDelayMs(ttl * 1000);
+    if (delay !== undefined && delay > CACHE_WARMING_IDLE_MAX_MS) {
+      lines.push(`- Note: idle warming stops after 30m, but a ${formatSeconds(ttl)} entry would refresh at ${Math.round(delay / 60_000)}m, so idle mode adds nothing for this tier. The ${formatSeconds(ttl)} lifetime already covers shorter breaks.`);
+    }
+  }
+
+  const eligible = input.mode !== "off" && issues.length === 0;
+  if (eligible) {
+    lines.push(`- ✓ Eligible: Pi keeps this cache warm ${input.mode === "idle" ? "during runs and briefly between runs" : "during agent runs"} when the expected savings are at least $0.05.`);
+  }
+  return { eligible, issues, lines };
+}

package/src/command.ts

@@ -21,7 +21,8 @@ import { type ProviderRequestState } from "./request-state.ts";
 import { type CacheStats, type ShardAggregate } from "./stats-store.ts";
 import { type CacheUsageSample } from "./stats-report.ts";
 import { describeSkillCompressionOutcome } from "./prompt-rewrite.ts";
-import { describePromptCacheRetentionDecision, getLastPromptCacheRetentionDecision } from "./retention.ts";
+import { describePromptCacheRetentionDecision, getLastPromptCacheRetentionDecision, LONG_CACHE_RETENTION_VALUE, PI_CACHE_RETENTION_ENV } from "./retention.ts";
+import { buildCacheWarmingDiagnosis, type CacheWarmingMode } from "./cache-warming.ts";
 
 /**
  * Everything the command needs from the extension instance. The extension keeps this state in closures
@@ -50,6 +51,8 @@ export type CommandRuntime = {
   getCurrentSessionHash(): string;
   getLastStatusText(): string | undefined;
   clearLastStatusText(): void;
+  /** Pi `cacheWarming` mode, or "unsupported" on hosts without the settings API. */
+  getCacheWarmingMode?(): CacheWarmingMode | "unsupported";
 };
 
 export function createCacheOptimizerCommandHandler(runtime: CommandRuntime) {
@@ -110,7 +113,13 @@ export function createCacheOptimizerCommandHandler(runtime: CommandRuntime) {
         const samples = sk ? getRecentSamples(sk) : [];
         const lowHitLines = buildLowHitDiagnosis(model, adapter, statsState, samples);
         const routeNote = describeNativeVirtualRouteNote(selectedModel, model);
-        const fullDiagnosis = [routeNote, diagnosis, describeSkillCompressionOutcome(), describePromptCacheRetentionDecision(getLastPromptCacheRetentionDecision()), ...lowHitLines].filter((line) => line !== undefined).join("\n");
+        const warmingLines = runtime.getCacheWarmingMode
+          ? buildCacheWarmingDiagnosis(model, {
+            mode: runtime.getCacheWarmingMode(),
+            tier: process.env[PI_CACHE_RETENTION_ENV] === LONG_CACHE_RETENTION_VALUE ? "long" : "short",
+          }).lines
+          : [];
+        const fullDiagnosis = [routeNote, diagnosis, describeSkillCompressionOutcome(), describePromptCacheRetentionDecision(getLastPromptCacheRetentionDecision()), ...warmingLines, ...lowHitLines].filter((line) => line !== undefined).join("\n");
         cmdCtx.ui.notify(fullDiagnosis, "info");
       } else if (subcommand === "stats") {
         const aggregate = await refreshShardAggregate();