dsh-vision-router 1.3.0 → 1.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -15,7 +15,7 @@
15
15
  </p>
16
16
 
17
17
  <p align="center">
18
- <a href="https://github.com/ysr666/dsh-vision-router/releases/tag/v1.3.0"><img src="https://img.shields.io/badge/release-v1.3.0-5B4CF0?style=flat-square" alt="Release v1.3.0" /></a>
18
+ <a href="https://github.com/ysr666/dsh-vision-router/releases/tag/v1.4.0"><img src="https://img.shields.io/badge/release-v1.4.0-5B4CF0?style=flat-square" alt="Release v1.4.0" /></a>
19
19
  <a href="tests"><img src="https://img.shields.io/badge/verified-149%20tests-2EA44F?style=flat-square" alt="Verified: 149 tests" /></a>
20
20
  <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-2EA44F?style=flat-square" alt="License: MIT" /></a>
21
21
  <a href="package.json"><img src="https://img.shields.io/badge/Node.js-%3E%3D22-339933?style=flat-square&amp;logo=nodedotjs&amp;logoColor=white" alt="Node.js >=22" /></a>
@@ -28,20 +28,26 @@
28
28
  <p align="center">💬 <strong>QQ community group: 1105463028</strong></p>
29
29
 
30
30
  > [!WARNING]
31
- > 📌 **Announcement (v1.3.0)**
31
+ > 📌 **Announcement (v1.4.0)**
32
32
  >
33
- > **v1.3.0** makes long sessions and mixed-plugin setups safer: the vision-tool schema is stable by default to protect prefix/KV caches, diagnostics can be persisted and opened from Settings, custom vision-model filtering explains missing image metadata and supports re-detection, host-provided DSH capability packages use the correct peer contract, and Vision Router tool cards can coexist with other vision plugins while keeping Vision Router rendering precedence.
34
- >
35
- > v1.2.3 fixes DSH Desktop's re-appearing first-run dialog: the onboarding "seen" flag and the model-guide step now persist in the profile settings file instead of origin-scoped `localStorage`, which a random per-launch port (`--port 0`) wiped on every boot (issue #78).
36
- >
37
- > v1.2.2 closed the last attachment-id gap — ids announced for images the host persisted itself (e.g. `read_image` re-uploads, `sha256:…`) now resolve in `vision_describe` and every pixel tool (issue #72) — stopped `vision_present` and other tool-result image blocks from ever locking a text-model session with `UNSUPPORTED_CONTENT` (issue #74; already-locked sessions heal after upgrading), and warned loudly when a stale sharp left over from a pre-v1.2 upgrade would break the pixel tools with `colourspace: parameter space not set` (issue #75).
38
- >
39
- > v1.2.1 hardened the pixel loop: all eleven pixel tools now accept uploaded-image attachment ids directly (no more `cannot read …/sha256:…` round trips), artifact filenames carry collision-free fingerprints, `vision_ground` retries degenerate boxes, the model guide replays fully from step 1 (leaving the settings first), and the settings card scrolls smoothly even with hundreds of models per provider.
33
+ > **v1.4.0 now supports** automatic recognition and direct-channel bridging of undeclared vision models, doctor repair of stale version-pinned exemptions, and a spotlight-guided onboarding walkthrough — plus hardened settings-save verification and vision-backend compatibility.
40
34
 
41
35
  <p align="center">
42
36
  <img src="assets/vision-demo.gif" width="640" alt="Demo: paste an image, the agent locates the send button with vision_ground / vision_crop / vision_pixel_diff and answers with coordinates" />
43
37
  </p>
44
38
 
39
+ ## Contents
40
+
41
+ - [Why this exists](#why-this-exists)
42
+ - [How it compares](#how-it-compares)
43
+ - [Quick start](#quick-start)
44
+ - [Highlights](#highlights)
45
+ - [How it works](#how-it-works)
46
+ - [Tools](#tools)
47
+ - [Configuration](#configuration)
48
+ - [Install and lifecycle](#install-and-lifecycle)
49
+ - [Troubleshooting](#troubleshooting)
50
+
45
51
  ## Why this exists
46
52
 
47
53
  Most DSH vision plugins bridge images to DeepSeek as *text descriptions* — lossy, one-shot, and blind to pixels. This plugin keeps the **original pixels on the vision model's side** and DeepSeek on the reasoning side, and makes looking at an image an **ordinary tool call**:
@@ -72,24 +78,25 @@ Both are MIT-licensed and one command away. Pick this plugin when you want image
72
78
 
73
79
  ## Quick start
74
80
 
75
- ### 1. Install and load the plugin
81
+ ### 1. Install the plugin
76
82
 
77
- Recommended for normal npm/npx installs (the same launch style used by the DSH README):
83
+ For normal npm/npx installs, installation is a single command:
78
84
 
79
85
  ```sh
80
86
  npx @deepseek-ai/dsh plugin --profile web add dsh-vision-router
81
- npx @deepseek-ai/dsh web
82
87
  ```
83
88
 
89
+ > [!NOTE]
90
+ > Third-party `dsh-web-plugin-manager` / `dshpm` **v0.4.2+** is also compatible: its quality gate now correctly allows `@deepseek-ai/schemastery` as a runtime dependency. The official DSH CLI above remains the recommended install path.
91
+
84
92
  If you run DeepSeek Harness from a source checkout with pnpm, use the workspace script instead — `dsh` is not necessarily on your shell `PATH`:
85
93
 
86
94
  ```sh
87
95
  cd deepseek-harness
88
96
  pnpm dsh plugin --profile web add dsh-vision-router
89
- pnpm dsh web
90
97
  ```
91
98
 
92
- If you already installed the DSH CLI globally and `dsh` is on `PATH`, the shorter `dsh ...` form works too.
99
+ If you already installed the DSH CLI globally and `dsh` is on `PATH`, the shorter `dsh ...` form works too. After installation, start or reload DSH Web as you normally do.
93
100
 
94
101
  > [!NOTE]
95
102
  > If you install the plugin **into a Web process that was already running long-term**, let that DSH Web process reload once so the plugin bundle itself is discovered. After the plugin is loaded, adding/removing models or changing wrapper scope **hot-updates without further DSH restarts**.
@@ -288,18 +295,23 @@ Everything is optional; defaults work out of the box. Edit via the Web card or a
288
295
 
289
296
  ### Install
290
297
 
291
- Normal npm/npx install:
298
+ Normal npm/npx install — one command:
292
299
 
293
300
  ```sh
294
301
  npx @deepseek-ai/dsh plugin --profile web add dsh-vision-router
295
- npx @deepseek-ai/dsh --profile web --dump-config | grep vision-router
296
302
  ```
297
303
 
298
304
  From a DeepSeek Harness source checkout:
299
305
 
300
306
  ```sh
301
307
  pnpm dsh plugin --profile web add dsh-vision-router
302
- pnpm dsh --profile web --dump-config | grep vision-router
308
+ ```
309
+
310
+ Optional verification:
311
+
312
+ ```sh
313
+ npx @deepseek-ai/dsh --profile web --dump-config | grep vision-router
314
+ # source checkout: pnpm dsh --profile web --dump-config | grep vision-router
303
315
  ```
304
316
 
305
317
  When first adding the plugin to an already long-lived Web profile, let that Web process reload the plugin bundle; the host discovers the browser bundle through `dsh.client` at startup. **After the plugin is loaded, model-catalog and wrapper-scope changes hot-update and do not require a restart.**
@@ -325,6 +337,8 @@ pnpm dsh plugin --profile web update dsh-vision-router
325
337
 
326
338
  Settings live in the profile's settings provider and survive upgrades.
327
339
 
340
+ > **A fresh release does not take effect (`downloaded 0` / `added 0`):** pnpm v11 holds versions younger than 24h back; `npx dsh-vision-router repair` fixes the stale version-pinned profile exemption so updates take effect immediately.
341
+
328
342
  > **Upgrading from a pre-bundle-patch install (v0.x):** the package now mounts
329
343
  > itself through its own bundle patch, so a leftover manual row in
330
344
  > `~/.dsh/profiles/<profile>/cordis.patch.yml` duplicates it and `dsh web`
package/README.zh.md CHANGED
@@ -15,7 +15,7 @@
15
15
  </p>
16
16
 
17
17
  <p align="center">
18
- <a href="https://github.com/ysr666/dsh-vision-router/releases/tag/v1.3.0"><img src="https://img.shields.io/badge/release-v1.3.0-5B4CF0?style=flat-square" alt="Release v1.3.0" /></a>
18
+ <a href="https://github.com/ysr666/dsh-vision-router/releases/tag/v1.4.0"><img src="https://img.shields.io/badge/release-v1.4.0-5B4CF0?style=flat-square" alt="Release v1.4.0" /></a>
19
19
  <a href="tests"><img src="https://img.shields.io/badge/verified-149%20tests-2EA44F?style=flat-square" alt="Verified: 149 tests" /></a>
20
20
  <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-2EA44F?style=flat-square" alt="License: MIT" /></a>
21
21
  <a href="package.json"><img src="https://img.shields.io/badge/Node.js-%3E%3D22-339933?style=flat-square&amp;logo=nodedotjs&amp;logoColor=white" alt="Node.js >=22" /></a>
@@ -28,20 +28,26 @@
28
28
  <p align="center">💬 <strong>QQ 用户交流群:1105463028</strong></p>
29
29
 
30
30
  > [!WARNING]
31
- > 📌 **公告(v1.3.0)**
31
+ > 📌 **公告(v1.4.0)**
32
32
  >
33
- > **v1.3.0** 重点加固长会话稳定性与插件共存:视觉工具 schema 默认从会话开始保持稳定,减少 prefix/KV cache 失效;诊断日志可持久化并从设置页直接打开;自定义视觉模型缺失 image 元数据时会明确解释并支持重新检测;宿主提供的 DSH capability 包使用正确的 peer 依赖契约;与其他视觉插件共装时,同名工具卡不再导致启动失败,同时保留 Vision Router 自己的渲染优先级。
34
- >
35
- > v1.2.3 现已支持:修复 DSH Desktop 每次启动都重复弹出首次引导的问题——引导「已读」标记与模型引导步骤改存 profile 设置文件(不再依赖按 origin 隔离、每次随机端口重启即清零的 `localStorage`)(issue #78)。
36
- >
37
- > v1.2.2 现已支持:补上最后一处附件 ID 缺口——宿主 `read_image` 回挂图片公布的 `sha256:…` ID 现在可被 `vision_describe` 与全部像素工具解析(issue #72);`vision_present` 等工具结果里的图像块不再把文本模型会话锁死在 `UNSUPPORTED_CONTENT`(issue #74,已锁死的历史会话升级后自动修复);检测到 v1.1.x 升级残留的旧版 sharp 时明确告警,把玄学的 `colourspace` 报错变成一眼可见的修复指引(issue #75)。
38
- >
39
- > v1.2.1 加固像素闭环:十一个像素工具可直接接受上传图片的附件 ID(告别 `cannot read …/sha256:…` 的绕路),产物文件名带指纹不再互相覆盖,`vision_ground` 对退化框自动重试,模型引导支持从第 1 步完整重放(先退出设置页),设置卡片在数百模型目录下滚动依然流畅。
33
+ > **v1.4.0 现已支持**:未声明视觉模型的自动识别与直连桥接、doctor 修复过期版本钉住豁免、引导流程聚光灯高亮——并增强设置保存校验与视觉后端兼容性。
40
34
 
41
35
  <p align="center">
42
36
  <img src="assets/vision-demo.gif" width="640" alt="演示:粘贴图片,Agent 用 vision_ground / vision_crop / vision_pixel_diff 定位发送按钮并给出坐标" />
43
37
  </p>
44
38
 
39
+ ## 目录
40
+
41
+ - [为什么做这个](#为什么做这个)
42
+ - [对比同类插件](#对比同类插件)
43
+ - [快速开始](#快速开始)
44
+ - [亮点](#亮点)
45
+ - [工作原理](#工作原理)
46
+ - [工具](#工具)
47
+ - [配置项](#配置项)
48
+ - [安装与生命周期](#安装与生命周期)
49
+ - [故障排查](#故障排查)
50
+
45
51
  ## 为什么做这个
46
52
 
47
53
  大多数 DSH 视觉插件把图片“翻译”成一段文字描述再喂给 DeepSeek——有损、一次性、看不见像素。本插件把**原图像素留在视觉模型侧**、把推理留在 DeepSeek 侧,并把“看图”变成一次**普通的工具调用**:
@@ -72,24 +78,25 @@
72
78
 
73
79
  ## 快速开始
74
80
 
75
- ### 1. 安装并让插件加载
81
+ ### 1. 安装插件
76
82
 
77
- 普通 npm / npx 安装方式推荐这样用(与 DSH 官方 README 的启动方式一致):
83
+ 普通 npm / npx 安装只需要一条命令:
78
84
 
79
85
  ```sh
80
86
  npx @deepseek-ai/dsh plugin --profile web add dsh-vision-router
81
- npx @deepseek-ai/dsh web
82
87
  ```
83
88
 
89
+ > [!NOTE]
90
+ > 第三方 `dsh-web-plugin-manager` / `dshpm` **v0.4.2+** 现已兼容:其质量门已正确放行作为运行时依赖的 `@deepseek-ai/schemastery`。上面的官方 DSH CLI 仍是推荐安装方式。
91
+
84
92
  如果你是从 DeepSeek Harness 源码仓库通过 pnpm 运行,`dsh` 不一定在系统 `PATH` 里,请改用工作区脚本:
85
93
 
86
94
  ```sh
87
95
  cd deepseek-harness
88
96
  pnpm dsh plugin --profile web add dsh-vision-router
89
- pnpm dsh web
90
97
  ```
91
98
 
92
- 如果你已经全局安装 DSH CLI,并且终端里能直接执行 `dsh`,也可以继续使用较短的 `dsh ...` 写法。
99
+ 如果你已经全局安装 DSH CLI,并且终端里能直接执行 `dsh`,也可以继续使用较短的 `dsh ...` 写法。安装完成后,按你平时的方式启动或重新加载 DSH Web 即可。
93
100
 
94
101
  > [!NOTE]
95
102
  > 如果你是把插件**首次安装进一个已经长期运行的 Web 进程**,需要让 DSH Web 进程重新加载一次插件本体。插件加载完成后,新增/删除模型、修改自动识图包装范围都会**热更新,无需再重启 DSH**。
@@ -288,18 +295,23 @@ Web 配置页在 **设置 → 插件 → 插件配置** 下注册「视觉路由
288
295
 
289
296
  ### 安装
290
297
 
291
- 普通 npm / npx 安装:
298
+ 普通 npm / npx 安装——一条命令:
292
299
 
293
300
  ```sh
294
301
  npx @deepseek-ai/dsh plugin --profile web add dsh-vision-router
295
- npx @deepseek-ai/dsh --profile web --dump-config | grep vision-router
296
302
  ```
297
303
 
298
304
  从 DeepSeek Harness 源码仓库运行:
299
305
 
300
306
  ```sh
301
307
  pnpm dsh plugin --profile web add dsh-vision-router
302
- pnpm dsh --profile web --dump-config | grep vision-router
308
+ ```
309
+
310
+ 可选验证:
311
+
312
+ ```sh
313
+ npx @deepseek-ai/dsh --profile web --dump-config | grep vision-router
314
+ # 源码仓库:pnpm dsh --profile web --dump-config | grep vision-router
303
315
  ```
304
316
 
305
317
  首次把插件装进已经长期运行的 Web profile 时,需要让 Web 进程重新加载插件本体;宿主在启动时通过 `dsh.client` 声明发现浏览器端包。**插件加载完成后,模型目录与包装范围的变化会热更新,不需要为这些变化重启。**
@@ -325,6 +337,8 @@ pnpm dsh plugin --profile web update dsh-vision-router
325
337
 
326
338
  设置存放在 profile 的设置提供方里,升级不丢失。
327
339
 
340
+ > **新版本一直不生效(`downloaded 0` / `added 0`):** pnpm v11 会拦下发布不足 24 小时的版本;运行 `npx dsh-vision-router repair` 修复过期的带版本号豁免条目后,更新立即生效。
341
+
328
342
  > **从 bundle 补丁之前(v0.x)升级:** 现在插件由自带的 bundle 补丁自动挂载,
329
343
  > 若 `~/.dsh/profiles/<profile>/cordis.patch.yml` 里还残留旧版手动行,会与之
330
344
  > 重复,`dsh web` 启动即报 `duplicate loader entry id: vision-router`。删除
package/docs/doctor.md CHANGED
@@ -32,7 +32,7 @@ To inspect only the Web profile:
32
32
  npx dsh-vision-router doctor --profile web
33
33
  ```
34
34
 
35
- The command locates the DSH home (`$DSH_HOME` when set, otherwise `~/.dsh`), scans profile `package.json` files, reports UTF-8 BOM bytes, validates the JSON after ignoring a leading BOM for diagnosis, and reports whether `dsh-vision-router` is present as a profile dependency and bundle layer.
35
+ The command locates the DSH home (`$DSH_HOME` when set, otherwise `~/.dsh`), scans profile `package.json` files, reports UTF-8 BOM bytes, validates the JSON after ignoring a leading BOM for diagnosis, reports whether `dsh-vision-router` is present as a profile dependency and bundle layer, and flags version-pinned `minimumReleaseAgeExclude` entries in the profile's `pnpm-workspace.yaml` that would hold back the next release.
36
36
 
37
37
  ## Repair the UTF-8 BOM startup failure
38
38
 
@@ -50,3 +50,21 @@ npx dsh-vision-router repair --profile web
50
50
  ```
51
51
 
52
52
  `repair` removes only the three-byte UTF-8 BOM prefix (`EF BB BF`) when it is present, then validates the remaining JSON. It does not reformat, regenerate, or otherwise rewrite the profile contents. If JSON is still invalid for another reason, the command reports that and stops rather than guessing a repair.
53
+
54
+ ## Repair a stale release-age exemption (the "update does nothing" gate)
55
+
56
+ pnpm v11 defaults `minimumReleaseAge` to 1440 minutes: a version published less than 24 hours ago is not resolved, so `dsh plugin update` silently keeps the previous version and prints `downloaded 0 / added 0`. An exemption entry that pins a version — `dsh-vision-router@1.2.0` — only exempts that one version and goes stale on the next release, which is why "a new release is out but the update does nothing" keeps recurring.
57
+
58
+ The doctor flags version-pinned entries for `dsh-vision-router` and the `@deepseek-ai/*` host packages:
59
+
60
+ ```text
61
+ ✗ web — … — release-age exemption version-pinned (dsh-vision-router@1.2.0) — releases younger than 24h will not be picked up
62
+ ```
63
+
64
+ Run:
65
+
66
+ ```sh
67
+ npx dsh-vision-router repair --profile web
68
+ ```
69
+
70
+ to rewrite them to bare names (`dsh-vision-router`, `@deepseek-ai/*`), which exempt every future version, so upgrades take effect immediately again. Unrelated entries and the rest of the file are left untouched.
package/index.js CHANGED
@@ -233,6 +233,12 @@ export const Config = z.object({
233
233
  tool: z.boolean().default(true),
234
234
  progressiveTools: z.boolean().default(true),
235
235
  autoActivateOnImage: z.boolean().default(true),
236
+ // User feedback (Zhipu official channel): some channels expose vision
237
+ // models whose catalog metadata does not declare image input. Models the
238
+ // built-in name inference does not recognize can be forced here — one model
239
+ // id (or "provider/model") per entry. Only consulted for vision BACKEND
240
+ // capability (the session-side admission stays host-owned).
241
+ extraVisionModels: z.array(z.string()).default([]),
236
242
  // Client-persisted UI state (issue #78): DSH Desktop serves the Web UI from
237
243
  // a random port on every launch, so origin-scoped localStorage forgets the
238
244
  // first-run onboarding dialog and it re-appeared on every boot. These keys
@@ -2005,6 +2011,158 @@ export function modelInfoAcceptsImages(info) {
2005
2011
  return Array.isArray(info && info.inputModalities) && info.inputModalities.includes('image')
2006
2012
  }
2007
2013
 
2014
+ // User feedback: channels like the Zhipu official one (open.bigmodel.cn,
2015
+ // configured with a custom model list) expose vision models whose catalog
2016
+ // metadata does NOT declare image input, even though the models accept images
2017
+ // (e.g. glm-4.6v). DSH's Web settings do not write the `input: [text, image]`
2018
+ // declaration for custom channels either, so a strict metadata check hides
2019
+ // perfectly usable vision backends. The conservative, curated name patterns
2020
+ // below recognize well-known multimodal model families as a fallback; models
2021
+ // that still do not match can be forced via the `extraVisionModels` setting.
2022
+ // A vision-looking name does not necessarily identify a generative chat model.
2023
+ // Embedding and reranker endpoints often share the same VL family prefix but
2024
+ // cannot answer vision_describe. Keep them out of the automatic candidate
2025
+ // set; an explicit extraVisionModels override remains the expert escape hatch.
2026
+ const NON_GENERATIVE_VISION_MODEL_HINTS = [
2027
+ /(^|[\/_.-])(embedding|embeddings|embed)(?=$|[\/_.-])/i,
2028
+ /(^|[\/_.-])(rerank|reranker|reranking)(?=$|[\/_.-])/i,
2029
+ ]
2030
+
2031
+ export function looksLikeNonGenerativeVisionModel(modelId) {
2032
+ const id = String(modelId ?? '').trim()
2033
+ if (id === '') return false
2034
+ return NON_GENERATIVE_VISION_MODEL_HINTS.some((pattern) => pattern.test(id))
2035
+ }
2036
+
2037
+ const VISION_MODEL_NAME_HINTS = [
2038
+ // Zhipu VLM family: glm-4.6v, glm-4.6v-flash, glm-4v-plus, glm-4.5v(-plus)…
2039
+ /(^|\/)glm-4[\w.-]*v(?=$|[-/])/i,
2040
+ /(^|\/)glm-4v(?=$|[-/])/i,
2041
+ // Qwen VL / QVQ vision-reasoning family (excludes plain qwen3-14b etc.).
2042
+ /(^|\/)qwen[\w.-]*(vl|vision)/i,
2043
+ /(^|\/)qvq(?=$|[-.])/i,
2044
+ // OpenAI multimodal line (gpt-4o*, gpt-4.1*, gpt-5*, gpt-oss*).
2045
+ /(^|\/)gpt-(4o|4\.1|5|oss)(?=$|[-.])/i,
2046
+ /(^|\/)gemini/i,
2047
+ // Claude 3+ / Sonnet/Opus/Haiku are multimodal (claude-2 is not).
2048
+ /(^|\/)(claude-(3|4)(?=$|[-.])|claude[\w.-]*(sonnet|opus|haiku))/i,
2049
+ /(^|\/)(internvl|cogvlm|llava|pixtral)/i,
2050
+ /(^|\/)(doubao|hunyuan|minimax|ernie)[\w.-]*(vl|vision)/i,
2051
+ /(^|\/)ernie-4\.5/i,
2052
+ /(^|\/)(yi-vision|kimi[\w.-]*vision|moonshot[\w.-]*vision)/i,
2053
+ /(^|\/)step[\w.-]*(v|vision)(?=$|[-/])/i,
2054
+ /(^|\/)grok[\w.-]*vision/i,
2055
+ /(^|\/)grok-4(?=$|[-.])/i,
2056
+ /(^|\/)llama[\w.-]*vision/i,
2057
+ /(^|\/)mistral[\w.-]*pixtral/i,
2058
+ /(^|\/)(phi[\w.-]*vision|florence[\w.-]*)/i,
2059
+ ]
2060
+
2061
+ /**
2062
+ * Conservative name-based inference for vision capability: true only when the
2063
+ * model id matches a well-known multimodal naming pattern. Used as a fallback
2064
+ * when catalog metadata does not declare image input; never overrides an
2065
+ * explicit text-only declaration on the session/twin paths.
2066
+ */
2067
+ export function looksLikeVisionModel(modelId) {
2068
+ const id = String(modelId ?? '').trim()
2069
+ if (id === '' || looksLikeNonGenerativeVisionModel(id)) return false
2070
+ return VISION_MODEL_NAME_HINTS.some((pattern) => pattern.test(id))
2071
+ }
2072
+
2073
+ /**
2074
+ * Pure capability decision for a vision backend: an explicit user override
2075
+ * wins first, known non-generative endpoint roles are excluded next, then
2076
+ * declared image metadata and conservative name inference are considered.
2077
+ *
2078
+ * @param info - resolved model metadata (may be undefined when the lookup failed).
2079
+ * @param provider - provider id, used to match "provider/model" override entries.
2080
+ * @param model - model id.
2081
+ * @param extraVisionModels - user-configured model ids (or "provider/model") forced vision-capable.
2082
+ * @returns { image, inputModalities, inferred, reason } where `inferred` is
2083
+ * false for declared image input, 'override' for the user list, 'name' for the
2084
+ * naming heuristic, and `reason` explains a text-only verdict.
2085
+ */
2086
+ export function decideVisionBackendCapability(info, provider, model, extraVisionModels) {
2087
+ const inputModalities = Array.isArray(info && info.inputModalities)
2088
+ ? info.inputModalities.filter((item) => typeof item === 'string')
2089
+ : []
2090
+ const modelId = String(model ?? '').trim()
2091
+ const providerId = String(provider ?? '').trim()
2092
+ const extras = Array.isArray(extraVisionModels)
2093
+ ? extraVisionModels.map((entry) => String(entry ?? '').trim()).filter((entry) => entry !== '')
2094
+ : []
2095
+ const forced =
2096
+ modelId !== '' &&
2097
+ extras.some((entry) => entry === modelId || (providerId !== '' && entry === `${providerId}/${modelId}`))
2098
+ // Manual override is deliberately strongest: advanced users can still
2099
+ // force an unusual endpoint that our role/name heuristics reject.
2100
+ if (forced) {
2101
+ return { image: true, inputModalities: [...new Set([...inputModalities, 'image'])], inferred: 'override', reason: undefined }
2102
+ }
2103
+ // A model can consume images and still be the wrong KIND of endpoint
2104
+ // for this plugin: embedding/reranking produces no assistant answer.
2105
+ if (modelId !== '' && looksLikeNonGenerativeVisionModel(modelId)) {
2106
+ return {
2107
+ image: false,
2108
+ inputModalities,
2109
+ inferred: false,
2110
+ reason: 'model name indicates an embedding/reranker endpoint, not a generative vision backend',
2111
+ }
2112
+ }
2113
+ if (inputModalities.includes('image')) {
2114
+ return { image: true, inputModalities, inferred: false, reason: undefined }
2115
+ }
2116
+ if (modelId !== '' && looksLikeVisionModel(modelId)) {
2117
+ return { image: true, inputModalities: [...new Set([...inputModalities, 'image'])], inferred: 'name', reason: undefined }
2118
+ }
2119
+ return {
2120
+ image: false,
2121
+ inputModalities,
2122
+ inferred: false,
2123
+ reason: 'model metadata does not declare image input',
2124
+ }
2125
+ }
2126
+
2127
+ /**
2128
+ * Resolve transport facts for the direct channel compatibility bridge.
2129
+ * Raw llm-pi-ai settings commonly omit baseURL/api for built-in catalog
2130
+ * providers; the materialized pi-ai model carries the effective values.
2131
+ */
2132
+ export function resolveChannelBridgeTransport(rawProfile, resolvedProfile, modelId) {
2133
+ let resolvedModel
2134
+ try {
2135
+ const getModels = resolvedProfile && resolvedProfile.piProvider && resolvedProfile.piProvider.getModels
2136
+ const models = typeof getModels === 'function'
2137
+ ? getModels.call(resolvedProfile.piProvider)
2138
+ : []
2139
+ resolvedModel = Array.isArray(models)
2140
+ ? models.find((entry) => entry && String(entry.id) === String(modelId))
2141
+ : undefined
2142
+ } catch {
2143
+ resolvedModel = undefined
2144
+ }
2145
+ const firstString = (...values) =>
2146
+ values.find((value) => typeof value === 'string' && value.trim() !== '')
2147
+ return {
2148
+ baseURL: firstString(
2149
+ resolvedModel && resolvedModel.baseUrl,
2150
+ rawProfile && rawProfile.baseURL,
2151
+ resolvedProfile && resolvedProfile.baseURL,
2152
+ resolvedProfile && resolvedProfile.piProvider && resolvedProfile.piProvider.baseUrl,
2153
+ ),
2154
+ api: firstString(
2155
+ resolvedModel && resolvedModel.api,
2156
+ rawProfile && rawProfile.api,
2157
+ resolvedProfile && resolvedProfile.api,
2158
+ ),
2159
+ apiKeyEnv: firstString(
2160
+ rawProfile && rawProfile.apiKeyEnv,
2161
+ resolvedProfile && resolvedProfile.apiKeyEnv,
2162
+ ),
2163
+ }
2164
+ }
2165
+
2008
2166
  export function apply(ctx, config = {}) {
2009
2167
  // Route sharp version diagnostics (issue #75) through the harness logger
2010
2168
  // instead of console.warn, so the warning lands in the server log.
@@ -2640,15 +2798,13 @@ export function apply(ctx, config = {}) {
2640
2798
  }
2641
2799
  try {
2642
2800
  const info = await ctx.llm.resolveModelInfo(provider, model)
2643
- const inputModalities = Array.isArray(info && info.inputModalities)
2644
- ? info.inputModalities.filter((item) => typeof item === 'string')
2645
- : []
2646
- return {
2647
- image: modelInfoAcceptsImages(info),
2648
- inputModalities,
2649
- reason: modelInfoAcceptsImages(info) ? undefined : 'model metadata does not declare image input',
2650
- }
2801
+ return decideVisionBackendCapability(info, provider, model, current().extraVisionModels)
2651
2802
  } catch (error) {
2803
+ // Metadata lookup failed (custom model lists often do): fall back to
2804
+ // the user override list and the name heuristic before declaring the
2805
+ // model text-only.
2806
+ const fallback = decideVisionBackendCapability(undefined, provider, model, current().extraVisionModels)
2807
+ if (fallback.image) return fallback
2652
2808
  return {
2653
2809
  image: false,
2654
2810
  inputModalities: [],
@@ -2657,6 +2813,123 @@ export function apply(ctx, config = {}) {
2657
2813
  }
2658
2814
  }
2659
2815
 
2816
+ // ── direct OpenAI-compatible bridge for undeclared vision channels ─────────
2817
+ //
2818
+ // User feedback (Zhipu official channel, open.bigmodel.cn): some channels
2819
+ // expose vision models whose catalog metadata does NOT declare image input,
2820
+ // so the channel adapter refuses image requests at the wire
2821
+ // (UNSUPPORTED_CONTENT: model "x" does not support image input) even though
2822
+ // the models accept images. For backends recognized only through the name
2823
+ // inference or the extraVisionModels override, fall back to calling the
2824
+ // channel's OpenAI-compatible endpoint directly with the channel's own
2825
+ // baseURL and credential — no hand-edited settings.yaml needed. Defensive
2826
+ // reads only: if the channel settings section, the baseURL, or the
2827
+ // credential cannot be resolved, the bridge is simply unavailable and the
2828
+ // adapter's own error is reported.
2829
+ const rawChannelProfileOf = (provider) => {
2830
+ try {
2831
+ const settings = ctx.get('settings')
2832
+ const section =
2833
+ settings && typeof settings.get === 'function' ? settings.get('llm-pi-ai') : undefined
2834
+ return section && section.providers ? section.providers[provider] : undefined
2835
+ } catch {
2836
+ return undefined
2837
+ }
2838
+ }
2839
+ // DSH's public model metadata intentionally omits endpoint/protocol
2840
+ // details. PiAiAdapter has already materialized those facts in its
2841
+ // resolved profile, so feature-detect that shape as a compatibility
2842
+ // shim. If upstream changes it, this fails closed to the normal chain.
2843
+ const resolvedPiAiProfileOf = (provider) => {
2844
+ try {
2845
+ const registration = ctx.llm.registration(provider)
2846
+ const adapter = registration && registration.adapter
2847
+ const config = adapter && adapter.config
2848
+ const profiles = config && typeof config.profiles === 'function' ? config.profiles() : undefined
2849
+ return profiles && typeof profiles.get === 'function' ? profiles.get(provider) : undefined
2850
+ } catch {
2851
+ return undefined
2852
+ }
2853
+ }
2854
+ const channelBridgePlan = (provider, model) => {
2855
+ const rawProfile = rawChannelProfileOf(provider)
2856
+ const resolvedProfile = resolvedPiAiProfileOf(provider)
2857
+ const transport = resolveChannelBridgeTransport(rawProfile, resolvedProfile, model)
2858
+ if (!transport.baseURL) {
2859
+ return { ok: false, reason: 'no resolved channel baseURL', rawProfile, resolvedProfile, transport }
2860
+ }
2861
+ // callOpenAICompatible speaks Chat Completions. Never send another
2862
+ // provider protocol through this bridge just because its name looks visual.
2863
+ if (transport.api !== 'openai-completions') {
2864
+ return {
2865
+ ok: false,
2866
+ reason: `channel protocol ${transport.api || 'unknown'} is not OpenAI Chat Completions`,
2867
+ rawProfile,
2868
+ resolvedProfile,
2869
+ transport,
2870
+ }
2871
+ }
2872
+ return { ok: true, rawProfile, resolvedProfile, transport }
2873
+ }
2874
+ const resolveChannelApiKey = async (plan) => {
2875
+ const ref = plan && plan.transport && plan.transport.apiKeyEnv
2876
+ if (typeof ref === 'string' && ref !== '') {
2877
+ try {
2878
+ const credentials = ctx.get('credentials')
2879
+ if (credentials !== undefined) {
2880
+ const hit = await credentials.resolve(ref)
2881
+ if (hit && typeof hit.value === 'string' && hit.value.length > 0) return hit.value
2882
+ }
2883
+ } catch {
2884
+ /* fall through to the ambient environment */
2885
+ }
2886
+ if (typeof process !== 'undefined' && process.env && typeof process.env[ref] === 'string') {
2887
+ return process.env[ref]
2888
+ }
2889
+ }
2890
+ // Catalog routes may use provider-native environment discovery and
2891
+ // therefore carry no explicit Harness credential reference.
2892
+ try {
2893
+ const auth = plan && plan.resolvedProfile && plan.resolvedProfile.piProvider
2894
+ && plan.resolvedProfile.piProvider.auth && plan.resolvedProfile.piProvider.auth.apiKey
2895
+ if (auth && typeof auth.resolve === 'function') {
2896
+ const hit = await auth.resolve({ credential: undefined })
2897
+ const value = hit && hit.auth && hit.auth.apiKey
2898
+ if (typeof value === 'string' && value.length > 0) return value
2899
+ }
2900
+ } catch {
2901
+ /* unavailable native auth */
2902
+ }
2903
+ return undefined
2904
+ }
2905
+ const directChannelVisionAnswer = async (provider, model, blocks, instruction, signal) => {
2906
+ const plan = channelBridgePlan(provider, model)
2907
+ if (!plan.ok) throw new Error(`vision bridge unavailable: ${plan.reason}`)
2908
+ const apiKey = await resolveChannelApiKey(plan)
2909
+ if (apiKey === undefined || apiKey === '') {
2910
+ throw new Error('vision bridge unavailable: channel credential could not be resolved')
2911
+ }
2912
+ const attachments = ctx.get('attachments')
2913
+ if (attachments === undefined) {
2914
+ throw new Error('vision bridge unavailable: attachment service is not registered')
2915
+ }
2916
+ const content = []
2917
+ for (const block of blocks) {
2918
+ const stored = await attachments.readImage(block.attachment)
2919
+ content.push(...toOpenAIContent([block], () => stored.data))
2920
+ }
2921
+ return callOpenAICompatible(
2922
+ {
2923
+ name: provider,
2924
+ baseURL: plan.transport.baseURL,
2925
+ model,
2926
+ apiKeyEnv: '__vision-router-channel__',
2927
+ },
2928
+ [{ role: 'user', content: [...content, { type: 'text', text: instruction }] }],
2929
+ { maxTokens: 4096, signal, resolveCredential: () => apiKey },
2930
+ )
2931
+ }
2932
+
2660
2933
  const collectVisionBackendCapabilities = async () => {
2661
2934
  const capabilities = {}
2662
2935
  if (typeof ctx.llm.listProviders !== 'function') return capabilities
@@ -3768,6 +4041,7 @@ export function apply(ctx, config = {}) {
3768
4041
  const block = await visionBlocksFromBytes(imageBytes, mediaType)
3769
4042
  const signal = AbortSignal.timeout(timeoutMs())
3770
4043
  const usablePairs = await resolveToolVisionPairs()
4044
+ const pairCapabilities = new Map()
3771
4045
  for (const pair of pairs()) {
3772
4046
  if (!pair || pair.provider === HTTP_ROUTE) continue
3773
4047
  if (!adapterAvailable(ctx.llm, pair.provider)) {
@@ -3775,6 +4049,7 @@ export function apply(ctx, config = {}) {
3775
4049
  continue
3776
4050
  }
3777
4051
  const capability = await resolveVisionBackendCapability(pair.provider, pair.model)
4052
+ pairCapabilities.set(`${pair.provider}/${pair.model}`, capability)
3778
4053
  if (!capability.image) {
3779
4054
  errors.push(
3780
4055
  `${pair.provider}/${pair.model}: not an image-capable backend (${capability.reason ?? 'unknown capability'})`,
@@ -3782,6 +4057,16 @@ export function apply(ctx, config = {}) {
3782
4057
  }
3783
4058
  }
3784
4059
  for (const pair of usablePairs) {
4060
+ // usablePairs also contains auto-discovered models. Before this fix the
4061
+ // map was populated only from explicit config rows, so inferred
4062
+ // SiliconFlow models failed pi-ai image admission and never reached
4063
+ // the direct channel bridge that was meant to rescue them.
4064
+ const pairKey = `${pair.provider}/${pair.model}`
4065
+ let pairCapability = pairCapabilities.get(pairKey)
4066
+ if (pairCapability === undefined) {
4067
+ pairCapability = await resolveVisionBackendCapability(pair.provider, pair.model)
4068
+ pairCapabilities.set(pairKey, pairCapability)
4069
+ }
3785
4070
  try {
3786
4071
  const text = await visionAnswer(ctx.llm, {
3787
4072
  provider: pair.provider,
@@ -3794,6 +4079,30 @@ export function apply(ctx, config = {}) {
3794
4079
  })
3795
4080
  if (text && text.trim() !== '') return { text: text.trim() }
3796
4081
  } catch (error) {
4082
+ // Channels whose catalog does not declare image input reject images
4083
+ // at the adapter wire. When the backend was recognized by the name
4084
+ // inference or the extraVisionModels override, call the channel's
4085
+ // OpenAI-compatible endpoint directly with its own baseURL and
4086
+ // credential before giving up on this pair.
4087
+ const capability = pairCapabilities.get(`${pair.provider}/${pair.model}`)
4088
+ if (capability && capability.inferred) {
4089
+ try {
4090
+ const direct = await directChannelVisionAnswer(
4091
+ pair.provider,
4092
+ pair.model,
4093
+ [block],
4094
+ instruction,
4095
+ signal,
4096
+ )
4097
+ if (direct && direct.trim() !== '') return { text: direct.trim() }
4098
+ } catch (bridgeError) {
4099
+ errors.push(
4100
+ `${pair.provider}/${pair.model}: direct channel fallback failed (${
4101
+ bridgeError && bridgeError.message ? bridgeError.message : String(bridgeError)
4102
+ })`,
4103
+ )
4104
+ }
4105
+ }
3797
4106
  errors.push(`${pair.provider}/${pair.model}: ${error && error.message ? error.message : String(error)}`)
3798
4107
  }
3799
4108
  }