pi-ccswitch-auto-switch 0.3.5 → 0.3.7

package/README.md

@@ -130,7 +130,7 @@ CCS v0.3.5 ✓70/131 · ⏳61 · ⛔0 · 🔄3 · provider/model-id
 - `⛔0`: no manually disabled models.
 - `🔄3`: **本次 pi session 成功切换的模型次数**（衡量扩展有效程度）。`/new`、`/fork`、`/resume` 等新 session 开始时归零；但**失败记录与冷却状态不重置**（它们是物理事实，跨 session 保留）。累计切换数（`state.switches`）与最近 20 条切换日志会持久化到状态文件，可在 `/ccswitch` 面板和 `/ccswitch-test` 中查看。
 - `provider/model-id`: 当前实际生效的模型（`provider/id`，切换后立即更新，长名自动截断）。
-- During a switch, `CCS ↻2/5 provider/model` means the second of at most five attempts is being made.
+- During a switch, `CCS ↻2 provider/model` means this is the second switch attempt of the current round. Attempts are only bounded by the round time window, not a fixed count; a round fails only once every candidate has been tried.
 
 四个状态（健康/冷却/禁用/切换）即使为 0 也始终显示，便于确认扩展处于监控中。All-healthy still shows zero counters, e.g. `CCS ✓131/131 · ⏳0 · ⛔0 · 🔄0 · provider/model-id`. Use `/ccswitch` or `/ccswitch-test` to see Pi's raw scope entry count, the deduplicated model count, affected model counts, and the underlying breaker-record count.
 
@@ -151,6 +151,20 @@ The state machine waits for Pi's native retry cycle to settle before switching.
 
 Cooldowns grow exponentially within bounded limits. Learned policy-constrained families are excluded only after a content-policy rejection; they remain eligible during ordinary rate-limit, network, and model-configuration failovers. Context-overflow retries only consider models with a larger context window, and image requests do not move to a model that explicitly supports text only. Unknown errors are isolated to the current model and still fail over normally. A user cancellation is the only aborted turn that does not trigger failover; watchdog cancellations are recorded as timeouts.
 
+### Modality precheck (automatic switch to a multimodal model for images)
+
+When building a provider request, Pi silently handles images according to the current model's `input` capabilities: if the model does not support images (its `input` does not include `image`), Pi replaces the image with a text placeholder `(image omitted: model does not support images)` — **the request does not fail**, so the reactive failover path never fires and the model just answers "I can't see the image", which is useless for OCR tasks.
+
+CCSwitch therefore prechecks **before the request is sent**, instead of waiting for a failure:
+
+- When user input carries images (`input` event with `images`), CCSwitch immediately switches to a healthy multimodal candidate (`input` explicitly includes `image`) before the request is processed, so the images are preserved;
+- When a tool execution returns images (e.g. the `read` tool loading an image file, whose result content includes `image` parts), CCSwitch also switches to a multimodal model before the next LLM call, so the tool-result images are not stripped;
+- Only models that **explicitly** support images (`input` includes `image`) are considered; models with missing metadata are never assumed to support images;
+- The switch reason is recorded as `modality` and counted in both the session and lifetime switch counters;
+- If no multimodal candidate is available, CCSwitch notifies the user and keeps the current model (Pi will still strip the image and add its notice).
+
+The modality precheck applies only in interactive TUI/RPC sessions (the same scope as failover); print/json modes only monitor and never switch.
+
 ## Data and privacy
 
 Health state is stored in Pi's agent directory as `ccswitch-auto-switch-state.json`. The extension stores counters, timestamps, cooldowns, learned model-family policy constraints, and redacted/truncated error summaries. It does not access credentials, authorization headers, CC Switch's database, or Pi's `auth.json`.

package/README.zh-CN.md

@@ -118,7 +118,7 @@ CCS v0.3.5 ✓70/131 · ⏳61 · ⛔0 · 🔄3 · provider/model-id
 - `⛔0`：没有被手动禁用的模型。
 - `🔄3`：**本次 pi session 成功切换的模型次数**（衡量扩展有效程度，0 时也显示图标）。`/new`、`/fork`、`/resume` 等新 session 开始时归零；但**失败记录与冷却状态不重置**（它们是物理事实，跨 session 保留）。累计切换数（`state.switches`）与最近 20 条切换日志会持久化到状态文件，可在 `/ccswitch` 面板和 `/ccswitch-test` 中查看。
 - `provider/model-id`：当前实际生效的模型（`provider/id`，切换后立即更新，长名自动截断）。
-- 切换中出现 `CCS ↻2/5 provider/model`，表示正在进行最多 5 次尝试中的第 2 次。
+- 切换中出现 `CCS ↻2 provider/model`，表示这是本轮的第 2 次切换尝试。切换次数只受本轮时间窗口限制，不再有固定次数上限；只有当所有候选都尝试过仍无可用模型时才会判定本轮失败。
 
 四个状态（健康/冷却/禁用/切换）即使为 0 也始终显示，便于确认扩展处于监控中；全部健康时仍显示零计数，例如 `CCS ✓131/131 · ⏳0 · ⛔0 · 🔄0 · provider/model-id`。通过 `/ccswitch` 或 `/ccswitch-test` 可同时查看 Pi 原始 scope 条目数、去重模型数、受影响模型数、底层熔断记录数、本 session 切换数与累计切换数。
 
@@ -139,6 +139,20 @@ CCS v0.3.5 ✓70/131 · ⏳61 · ⛔0 · 🔄3 · provider/model-id
 
 冷却时间会指数增长但有上限。发生内容审查时，插件只在审查故障转移链中避开已标记系列，普通限流、网络或模型配置故障仍可选择这些模型。上下文溢出时只会选择上下文窗口更大的模型；带图片的请求不会切到明确仅支持文本的模型。无法归类的异常按单模型故障处理并正常切换。用户主动取消是唯一不会触发故障转移的异常终止；看门狗取消会记录为超时。
 
+### 模态预检（OCR / 图片请求自动切换多模态模型）
+
+Pi 在把请求发给 Provider 时，会根据当前模型的 `input` 能力静默处理图片：如果模型不支持图片（`input` 不含 `image`，即非多模态模型），Pi 会把图片替换成一段文本提示 `(image omitted: model does not support images)` 再发送——**请求不会报错**，因此普通的“失败后切换”永远不会触发，模型只会回答“看不到图片”，OCR 结果拿不到。
+
+插件因此在**发送前主动预检**，不等失败：
+
+- **用户输入带图片时**（`input` 事件含 `images`），如果当前模型不支持图片，立即切换到健康的多模态候选（`input` 明确含 `image`）再处理，图片被保留；
+- **工具执行返回图片时**（如 `read` 工具读取图片文件，工具结果含 `image` content），同样在下一轮 LLM 调用前切到多模态模型，确保图片不被剥除；
+- 候选只选**明确声明**支持图片的模型（`input` 含 `image`）；元数据缺失的模型不冒险选择；
+- 切换原因记为 `modality`，计入本 session 切换数与累计切换；
+- 如果没有可用的多模态候选，则通知用户并保持原模型继续（Pi 会照常剥图并附带提示）。
+
+模态预检只在 TUI/RPC 交互模式下生效（与故障转移一致）；print/json 模式只监控不切换。
+
 ## 数据与隐私
 
 健康状态保存在 Pi agent 目录的 `ccswitch-auto-switch-state.json`。其中只有计数、时间、冷却信息、模型系列审查约束和脱敏/截断的错误摘要；插件不会访问凭据、Authorization 请求头、CC Switch 数据库或 Pi 的 `auth.json`。

package/candidates.ts

@@ -75,6 +75,17 @@ export function summarizeCandidateHealth(models: ModelRef[], health: HealthState
   return { total: models.length, healthy, cooling, disabled, breakerRecords }
 }
 
+/**
+ * 模态预检候选：明确声明支持图片输入的健康模型，按优先级排序。
+ * 用于当前模型不支持图片但输入/工具结果带图时的主动切换。
+ * 只选择 input 元数据明确包含 'image' 的模型，元数据缺失的模型不冒险选择。
+ */
+export function multimodalCandidates(candidates: ModelRef[], current: ModelRef | undefined, health: HealthState, now = Date.now()): ModelRef[] {
+  return candidates
+    .filter(model => model.input?.includes('image') && !blocked(model, health, now) && !(current && modelKey(model) === modelKey(current)))
+    .sort((a, b) => score(a, current, health) - score(b, current, health))
+}
+
 export function chooseCandidate(models: ModelRef[], options: CandidateOptions): ModelRef | undefined {
   const current = options.current
   const candidates = models.filter(model => {

package/index.ts

@@ -5,15 +5,14 @@ import { dirname, join } from 'node:path'
 import { fileURLToPath } from 'node:url'
 import type { ExtensionAPI, ExtensionContext, FailureObservation, ModelRef } from './types.ts'
 import { classifyFailure, parseRetryAfter } from './classify.ts'
-import { candidateSnapshot, effectiveCandidates, chooseCandidate, modelFamily, summarizeCandidateHealth } from './candidates.ts'
+import { candidateSnapshot, effectiveCandidates, chooseCandidate, multimodalCandidates, modelFamily, summarizeCandidateHealth } from './candidates.ts'
 import { HealthStore, endpointKey, modelKey, type HealthState } from './health.ts'
 
 const FIRST_RESPONSE_TIMEOUT = 90_000
 const STREAM_IDLE_TIMEOUT = 120_000
-const MAX_ATTEMPTS = 5
 const ROUND_LIMIT = 8 * 60_000
 const RPC_PROTOCOL_VERSION = 1
-const EXTENSION_VERSION = '0.3.5'
+const EXTENSION_VERSION = '0.3.6'
 // 同端点（BaseURL 相同）连续失败达到该次数即隔离该端点，避免同一个平台的多个模型逐个试错耗尽本轮切换
 const ENDPOINT_FAIL_THRESHOLD = 3
 
@@ -41,7 +40,7 @@ interface Round {
   endpointFails?: EndpointFailTracker
   /** 本轮内容审查故障转移中必须避开的模型系列（包含持久化学到的约束）。 */
   avoidFamilies?: Set<string>
-  /** 本轮内最后一次成功切换的时间；用于刷新 ROUND_LIMIT 窗口，避免供应商内部重试耗时导致误判“达到上限” */
+  /** 本轮内最后一次成功切换的时间；用于刷新 ROUND_LIMIT 窗口，避免供应商内部重试耗时导致误判“超过本轮时间限制” */
   lastSwitchAt?: number
 }
 
@@ -60,6 +59,19 @@ function resolveModel(ctx: ExtensionContext, provider: string, id: string, previ
   return refs.find(model => model?.provider === provider && model.id === id) ?? { provider, id }
 }
 
+/**
+ * 判断工具执行结果是否包含图片内容（如 read 工具读取图片文件后返回的
+ * { content: [{ type: 'image', ... }] } 结构）。递归查找，兼容各种 result 形状。
+ */
+function resultContainsImage(result: unknown): boolean {
+  if (Array.isArray(result)) return result.some(part => resultContainsImage(part))
+  if (!result || typeof result !== 'object') return false
+  const record = result as Record<string, unknown>
+  if (record.type === 'image') return true
+  if (Array.isArray(record.content)) return record.content.some(part => resultContainsImage(part))
+  return false
+}
+
 export default function (pi: ExtensionAPI) {
   const health = new HealthStore()
   let round: Round | undefined
@@ -91,7 +103,7 @@ export default function (pi: ExtensionAPI) {
     const activeModel = round?.model ?? ctx.model
     // 恒显完整状态：健康/总数 · 冷却 · 禁用 · 本session切换 · 当前模型（均为 0 时也显示，便于确认扩展在监控中）
     const prefix = `CCS v${EXTENSION_VERSION}`
-    const plain = round?.phase === 'switching' ? `${prefix} ↻${round.attempts}/${MAX_ATTEMPTS} ${modelTag(round.model)}` :
+    const plain = round?.phase === 'switching' ? `${prefix} ↻${round.attempts} ${modelTag(round.model)}` :
       round?.phase === 'exhausted' ? `${prefix} ⏸切换停止 ${modelTag(round.model ?? ctx.model)}` :
       `${prefix} ✓${counts.healthy}/${counts.total} · ⏳${counts.cooling} · ⛔${counts.disabled}${switchSuffix()} · ${modelTag(activeModel)}`
     const theme = ctx.ui.theme
@@ -185,11 +197,55 @@ export default function (pi: ExtensionAPI) {
     notify(ctx, `CCSwitch：自动切换停止（${reason}），请用 /ccswitch 查看详情`, 'error')
     status(ctx)
   }
+  /**
+   * 模态预检主动切换：当前模型不支持图片但本轮输入/工具结果带图时，
+   * 主动切到一个健康的多模态候选（不等失败）。无候选时通知并保持原模型。
+   */
+  const proactiveModalitySwitch = async (ctx: ExtensionContext, round: Round): Promise<void> => {
+    const current = round.model ?? ctx.model
+    if (!current || !canRetry(ctx) || round.phase !== 'monitoring') return
+    if (current.input?.includes('image')) return
+    const candidates = effectiveCandidates(ctx.scopedModels, ctx.modelRegistry.getAvailable())
+    const next = multimodalCandidates(candidates, current, health.snapshot)[0]
+    if (!next) {
+      await health.log(`modality precheck: no multimodal candidate, staying on ${modelKey(current)}`)
+      notify(ctx, 'CCSwitch：当前模型不支持图片，且没有可用的多模态候选模型，已保持原模型', 'warning')
+      status(ctx)
+      return
+    }
+    const previousModel = current
+    const set = await pi.setModel(next).catch(() => false)
+    if (!set) {
+      await health.log(`modality precheck: Pi refused model selection ${modelKey(next)}`)
+      notify(ctx, `CCSwitch：多模态候选 ${modelKey(next)} 切换失败，已保持原模型`, 'warning')
+      status(ctx)
+      return
+    }
+    round.model = next
+    round.lastSwitchAt = Date.now()
+    sessionSwitches += 1
+    health.recordSwitch(modelKey(previousModel), modelKey(next), 'modality')
+    await health.flush()
+    try {
+      pi.appendEntry?.('ccswitch-switch', {
+        protocolVersion: RPC_PROTOCOL_VERSION,
+        roundId: round.id,
+        from: modelKey(previousModel),
+        to: modelKey(next),
+        reason: 'modality',
+        sessionSwitches,
+      })
+    } catch { /* appendEntry 失败不影响切换 */ }
+    await health.log(`modality precheck switch: ${modelKey(previousModel)} -> ${modelKey(next)}`)
+    notify(ctx, `CCSwitch：检测到图片输入，已切换至多模态模型 ${modelKey(next)}`, 'info')
+    status(ctx)
+  }
+
   const failover = async (ctx: ExtensionContext) => {
     if (!round || !round.observation || !round.model || !canRetry(ctx)) return
     // 窗口从上一次成功切换（或本轮开始）起算：供应商内部重试耗时不应消耗整轮限额
     const windowStart = Math.max(round.startedAt, round.lastSwitchAt ?? 0)
-    if (round.attempts >= MAX_ATTEMPTS || Date.now() - windowStart >= ROUND_LIMIT) return exhaust(ctx, '达到本轮切换上限')
+    if (Date.now() - windowStart >= ROUND_LIMIT) return exhaust(ctx, '超过本轮时间限制')
     const classification = classifyFailure(round.observation)
     if (round.observation.aborted && !round.observation.watchdog) { round.phase = 'idle'; clearWatchdog(); status(ctx); return }
     round.phase = 'switching'
@@ -237,14 +293,14 @@ export default function (pi: ExtensionAPI) {
       }
       round.model = next
       round.phase = 'redispatching'
-      // 刷新本轮切换时间窗：成功切换后重新起算 ROUND_LIMIT，避免长重试轮被误判为“达到上限”
+      // 刷新本轮切换时间窗：成功切换后重新起算 ROUND_LIMIT，避免长重试轮被误判为“超过时间限制”
       round.lastSwitchAt = Date.now()
       // 本次 session 成功切换计数 + 持久化累计/日志（衡量扩展有效程度）
       sessionSwitches += 1
       const fromKey = key(previousModel ?? ctx.model)
       health.recordSwitch(fromKey ?? '', modelKey(next), classification.kind)
       await health.flush()
-      notify(ctx, `CCSwitch：已切换至 ${modelKey(next)}（${round.attempts}/${MAX_ATTEMPTS}）`, 'info')
+      notify(ctx, `CCSwitch：已切换至 ${modelKey(next)}（第${round.attempts}次切换）`, 'info')
       // 触发 TUI 底栏/界面重绘：appendEntry 会发出 entry_appended 事件进入 session.subscribe 流，
       // interactive-mode 收到后执行 footer.invalidate() + requestRender()，footer 从 session.state.model 重新读取，
       // 从而让右下角模型名同步显示新模型（setModel 只改 state，不直接触发 footer 刷新）。
@@ -284,12 +340,15 @@ export default function (pi: ExtensionAPI) {
     await health.log('extension started')
   })
   pi.on('session_shutdown', async (_event, ctx) => { clearWatchdog(); ctx.ui.setWorkingMessage(); if (sessionSwitches > 0) await health.log(`session ended with ${sessionSwitches} successful switches`); await health.flush() })
-  pi.on('input', (event, ctx) => {
+  pi.on('input', async (event, ctx) => {
     if (event.source === 'extension') return { action: 'continue' }
     clearWatchdog()
     lastStatus = {}
     round = { id: (round?.id ?? 0) + 1, phase: 'monitoring', startedAt: Date.now(), text: event.text, images: event.images, tried: new Set(), attempts: 0, hadTool: false, inTool: false, hadOutput: false, watchdog: false, cleanRetry: false, model: ctx.model, endpointFails: undefined, avoidFamilies: undefined }
     status(ctx)
+    // 模态预检：输入带图片但当前模型不支持图片（非多模态）→ 主动切换到多模态模型，
+    // 避免 Pi 静默剥图后模型只回答“看不到图片”（此类情况不会触发 failover）。
+    if (event.images?.length && round.phase === 'monitoring') await proactiveModalitySwitch(ctx, round)
     return { action: 'continue' }
   })
   pi.on('before_provider_request', (_event, ctx) => { lastStatus = {}; if (round) { round.inTool = false; armWatchdog(ctx, FIRST_RESPONSE_TIMEOUT, round.id) } })
@@ -310,7 +369,13 @@ export default function (pi: ExtensionAPI) {
     }
   })
   pi.on('tool_execution_start', () => { if (round) { round.hadTool = true; round.inTool = true; clearWatchdog() } })
-  pi.on('tool_execution_end', () => { if (round) round.inTool = false })
+  pi.on('tool_execution_end', async (_event, ctx) => {
+    if (!round) return
+    round.inTool = false
+    // 工具结果含图片（如 read 图片文件）且当前模型不支持 → 主动切换多模态模型，
+    // 确保下一轮 LLM 调用使用新模型，工具结果中的图片不被 Pi 静默剥除。
+    if (round.phase === 'monitoring' && resultContainsImage(_event.result)) await proactiveModalitySwitch(ctx, round)
+  })
   pi.on('turn_end', async (event, ctx) => {
     const message = event.message
     if (message?.role !== 'assistant' || !round) return

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-ccswitch-auto-switch",
-  "version": "0.3.5",
+  "version": "0.3.7",
   "description": "Provider-first automatic model failover extension for Pi and CC Switch",
   "license": "MIT",
   "keywords": [

package/runner.mjs

@@ -189,7 +189,7 @@ export async function run(options, runtime = {}) {
     if (entry.data?.protocolVersion !== 1) return
     if (entry.type === 'ccswitch-switch') {
       retryPending = true
-      log(`switch ${entry.data.from || '?'} → ${entry.data.to || '?'} (${entry.data.attempts ?? '?'}/5)`)
+      log(`switch ${entry.data.from || '?'} → ${entry.data.to || '?'} (${entry.data.attempts ?? '?'})`)
     } else if (entry.type === 'ccswitch-complete') {
       terminal = { kind: 'complete', data: entry.data }
     } else if (entry.type === 'ccswitch-exhausted') {