dsh-vision-router 1.3.0 → 1.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +31 -17
- package/README.zh.md +31 -17
- package/docs/doctor.md +19 -1
- package/index.js +317 -8
- package/lib/client.js +750 -96
- package/lib/doctor-cli.js +35 -6
- package/lib/doctor.js +106 -2
- package/lib/file-logger.js +81 -2
- package/package.json +2 -2
package/README.md
CHANGED
|
@@ -15,7 +15,7 @@
|
|
|
15
15
|
</p>
|
|
16
16
|
|
|
17
17
|
<p align="center">
|
|
18
|
-
<a href="https://github.com/ysr666/dsh-vision-router/releases/tag/v1.
|
|
18
|
+
<a href="https://github.com/ysr666/dsh-vision-router/releases/tag/v1.4.0"><img src="https://img.shields.io/badge/release-v1.4.0-5B4CF0?style=flat-square" alt="Release v1.4.0" /></a>
|
|
19
19
|
<a href="tests"><img src="https://img.shields.io/badge/verified-149%20tests-2EA44F?style=flat-square" alt="Verified: 149 tests" /></a>
|
|
20
20
|
<a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-2EA44F?style=flat-square" alt="License: MIT" /></a>
|
|
21
21
|
<a href="package.json"><img src="https://img.shields.io/badge/Node.js-%3E%3D22-339933?style=flat-square&logo=nodedotjs&logoColor=white" alt="Node.js >=22" /></a>
|
|
@@ -28,20 +28,26 @@
|
|
|
28
28
|
<p align="center">💬 <strong>QQ community group: 1105463028</strong></p>
|
|
29
29
|
|
|
30
30
|
> [!WARNING]
|
|
31
|
-
> 📌 **Announcement (v1.
|
|
31
|
+
> 📌 **Announcement (v1.4.0)**
|
|
32
32
|
>
|
|
33
|
-
> **v1.
|
|
34
|
-
>
|
|
35
|
-
> v1.2.3 fixes DSH Desktop's re-appearing first-run dialog: the onboarding "seen" flag and the model-guide step now persist in the profile settings file instead of origin-scoped `localStorage`, which a random per-launch port (`--port 0`) wiped on every boot (issue #78).
|
|
36
|
-
>
|
|
37
|
-
> v1.2.2 closed the last attachment-id gap — ids announced for images the host persisted itself (e.g. `read_image` re-uploads, `sha256:…`) now resolve in `vision_describe` and every pixel tool (issue #72) — stopped `vision_present` and other tool-result image blocks from ever locking a text-model session with `UNSUPPORTED_CONTENT` (issue #74; already-locked sessions heal after upgrading), and warned loudly when a stale sharp left over from a pre-v1.2 upgrade would break the pixel tools with `colourspace: parameter space not set` (issue #75).
|
|
38
|
-
>
|
|
39
|
-
> v1.2.1 hardened the pixel loop: all eleven pixel tools now accept uploaded-image attachment ids directly (no more `cannot read …/sha256:…` round trips), artifact filenames carry collision-free fingerprints, `vision_ground` retries degenerate boxes, the model guide replays fully from step 1 (leaving the settings first), and the settings card scrolls smoothly even with hundreds of models per provider.
|
|
33
|
+
> **v1.4.0 now supports** automatic recognition and direct-channel bridging of undeclared vision models, doctor repair of stale version-pinned exemptions, and a spotlight-guided onboarding walkthrough — plus hardened settings-save verification and vision-backend compatibility.
|
|
40
34
|
|
|
41
35
|
<p align="center">
|
|
42
36
|
<img src="assets/vision-demo.gif" width="640" alt="Demo: paste an image, the agent locates the send button with vision_ground / vision_crop / vision_pixel_diff and answers with coordinates" />
|
|
43
37
|
</p>
|
|
44
38
|
|
|
39
|
+
## Contents
|
|
40
|
+
|
|
41
|
+
- [Why this exists](#why-this-exists)
|
|
42
|
+
- [How it compares](#how-it-compares)
|
|
43
|
+
- [Quick start](#quick-start)
|
|
44
|
+
- [Highlights](#highlights)
|
|
45
|
+
- [How it works](#how-it-works)
|
|
46
|
+
- [Tools](#tools)
|
|
47
|
+
- [Configuration](#configuration)
|
|
48
|
+
- [Install and lifecycle](#install-and-lifecycle)
|
|
49
|
+
- [Troubleshooting](#troubleshooting)
|
|
50
|
+
|
|
45
51
|
## Why this exists
|
|
46
52
|
|
|
47
53
|
Most DSH vision plugins bridge images to DeepSeek as *text descriptions* — lossy, one-shot, and blind to pixels. This plugin keeps the **original pixels on the vision model's side** and DeepSeek on the reasoning side, and makes looking at an image an **ordinary tool call**:
|
|
@@ -72,24 +78,25 @@ Both are MIT-licensed and one command away. Pick this plugin when you want image
|
|
|
72
78
|
|
|
73
79
|
## Quick start
|
|
74
80
|
|
|
75
|
-
### 1. Install
|
|
81
|
+
### 1. Install the plugin
|
|
76
82
|
|
|
77
|
-
|
|
83
|
+
For normal npm/npx installs, installation is a single command:
|
|
78
84
|
|
|
79
85
|
```sh
|
|
80
86
|
npx @deepseek-ai/dsh plugin --profile web add dsh-vision-router
|
|
81
|
-
npx @deepseek-ai/dsh web
|
|
82
87
|
```
|
|
83
88
|
|
|
89
|
+
> [!NOTE]
|
|
90
|
+
> Third-party `dsh-web-plugin-manager` / `dshpm` **v0.4.2+** is also compatible: its quality gate now correctly allows `@deepseek-ai/schemastery` as a runtime dependency. The official DSH CLI above remains the recommended install path.
|
|
91
|
+
|
|
84
92
|
If you run DeepSeek Harness from a source checkout with pnpm, use the workspace script instead — `dsh` is not necessarily on your shell `PATH`:
|
|
85
93
|
|
|
86
94
|
```sh
|
|
87
95
|
cd deepseek-harness
|
|
88
96
|
pnpm dsh plugin --profile web add dsh-vision-router
|
|
89
|
-
pnpm dsh web
|
|
90
97
|
```
|
|
91
98
|
|
|
92
|
-
If you already installed the DSH CLI globally and `dsh` is on `PATH`, the shorter `dsh ...` form works too.
|
|
99
|
+
If you already installed the DSH CLI globally and `dsh` is on `PATH`, the shorter `dsh ...` form works too. After installation, start or reload DSH Web as you normally do.
|
|
93
100
|
|
|
94
101
|
> [!NOTE]
|
|
95
102
|
> If you install the plugin **into a Web process that was already running long-term**, let that DSH Web process reload once so the plugin bundle itself is discovered. After the plugin is loaded, adding/removing models or changing wrapper scope **hot-updates without further DSH restarts**.
|
|
@@ -288,18 +295,23 @@ Everything is optional; defaults work out of the box. Edit via the Web card or a
|
|
|
288
295
|
|
|
289
296
|
### Install
|
|
290
297
|
|
|
291
|
-
Normal npm/npx install:
|
|
298
|
+
Normal npm/npx install — one command:
|
|
292
299
|
|
|
293
300
|
```sh
|
|
294
301
|
npx @deepseek-ai/dsh plugin --profile web add dsh-vision-router
|
|
295
|
-
npx @deepseek-ai/dsh --profile web --dump-config | grep vision-router
|
|
296
302
|
```
|
|
297
303
|
|
|
298
304
|
From a DeepSeek Harness source checkout:
|
|
299
305
|
|
|
300
306
|
```sh
|
|
301
307
|
pnpm dsh plugin --profile web add dsh-vision-router
|
|
302
|
-
|
|
308
|
+
```
|
|
309
|
+
|
|
310
|
+
Optional verification:
|
|
311
|
+
|
|
312
|
+
```sh
|
|
313
|
+
npx @deepseek-ai/dsh --profile web --dump-config | grep vision-router
|
|
314
|
+
# source checkout: pnpm dsh --profile web --dump-config | grep vision-router
|
|
303
315
|
```
|
|
304
316
|
|
|
305
317
|
When first adding the plugin to an already long-lived Web profile, let that Web process reload the plugin bundle; the host discovers the browser bundle through `dsh.client` at startup. **After the plugin is loaded, model-catalog and wrapper-scope changes hot-update and do not require a restart.**
|
|
@@ -325,6 +337,8 @@ pnpm dsh plugin --profile web update dsh-vision-router
|
|
|
325
337
|
|
|
326
338
|
Settings live in the profile's settings provider and survive upgrades.
|
|
327
339
|
|
|
340
|
+
> **A fresh release does not take effect (`downloaded 0` / `added 0`):** pnpm v11 holds versions younger than 24h back; `npx dsh-vision-router repair` fixes the stale version-pinned profile exemption so updates take effect immediately.
|
|
341
|
+
|
|
328
342
|
> **Upgrading from a pre-bundle-patch install (v0.x):** the package now mounts
|
|
329
343
|
> itself through its own bundle patch, so a leftover manual row in
|
|
330
344
|
> `~/.dsh/profiles/<profile>/cordis.patch.yml` duplicates it and `dsh web`
|
package/README.zh.md
CHANGED
|
@@ -15,7 +15,7 @@
|
|
|
15
15
|
</p>
|
|
16
16
|
|
|
17
17
|
<p align="center">
|
|
18
|
-
<a href="https://github.com/ysr666/dsh-vision-router/releases/tag/v1.
|
|
18
|
+
<a href="https://github.com/ysr666/dsh-vision-router/releases/tag/v1.4.0"><img src="https://img.shields.io/badge/release-v1.4.0-5B4CF0?style=flat-square" alt="Release v1.4.0" /></a>
|
|
19
19
|
<a href="tests"><img src="https://img.shields.io/badge/verified-149%20tests-2EA44F?style=flat-square" alt="Verified: 149 tests" /></a>
|
|
20
20
|
<a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-2EA44F?style=flat-square" alt="License: MIT" /></a>
|
|
21
21
|
<a href="package.json"><img src="https://img.shields.io/badge/Node.js-%3E%3D22-339933?style=flat-square&logo=nodedotjs&logoColor=white" alt="Node.js >=22" /></a>
|
|
@@ -28,20 +28,26 @@
|
|
|
28
28
|
<p align="center">💬 <strong>QQ 用户交流群:1105463028</strong></p>
|
|
29
29
|
|
|
30
30
|
> [!WARNING]
|
|
31
|
-
> 📌 **公告(v1.
|
|
31
|
+
> 📌 **公告(v1.4.0)**
|
|
32
32
|
>
|
|
33
|
-
> **v1.
|
|
34
|
-
>
|
|
35
|
-
> v1.2.3 现已支持:修复 DSH Desktop 每次启动都重复弹出首次引导的问题——引导「已读」标记与模型引导步骤改存 profile 设置文件(不再依赖按 origin 隔离、每次随机端口重启即清零的 `localStorage`)(issue #78)。
|
|
36
|
-
>
|
|
37
|
-
> v1.2.2 现已支持:补上最后一处附件 ID 缺口——宿主 `read_image` 回挂图片公布的 `sha256:…` ID 现在可被 `vision_describe` 与全部像素工具解析(issue #72);`vision_present` 等工具结果里的图像块不再把文本模型会话锁死在 `UNSUPPORTED_CONTENT`(issue #74,已锁死的历史会话升级后自动修复);检测到 v1.1.x 升级残留的旧版 sharp 时明确告警,把玄学的 `colourspace` 报错变成一眼可见的修复指引(issue #75)。
|
|
38
|
-
>
|
|
39
|
-
> v1.2.1 加固像素闭环:十一个像素工具可直接接受上传图片的附件 ID(告别 `cannot read …/sha256:…` 的绕路),产物文件名带指纹不再互相覆盖,`vision_ground` 对退化框自动重试,模型引导支持从第 1 步完整重放(先退出设置页),设置卡片在数百模型目录下滚动依然流畅。
|
|
33
|
+
> **v1.4.0 现已支持**:未声明视觉模型的自动识别与直连桥接、doctor 修复过期版本钉住豁免、引导流程聚光灯高亮——并增强设置保存校验与视觉后端兼容性。
|
|
40
34
|
|
|
41
35
|
<p align="center">
|
|
42
36
|
<img src="assets/vision-demo.gif" width="640" alt="演示:粘贴图片,Agent 用 vision_ground / vision_crop / vision_pixel_diff 定位发送按钮并给出坐标" />
|
|
43
37
|
</p>
|
|
44
38
|
|
|
39
|
+
## 目录
|
|
40
|
+
|
|
41
|
+
- [为什么做这个](#为什么做这个)
|
|
42
|
+
- [对比同类插件](#对比同类插件)
|
|
43
|
+
- [快速开始](#快速开始)
|
|
44
|
+
- [亮点](#亮点)
|
|
45
|
+
- [工作原理](#工作原理)
|
|
46
|
+
- [工具](#工具)
|
|
47
|
+
- [配置项](#配置项)
|
|
48
|
+
- [安装与生命周期](#安装与生命周期)
|
|
49
|
+
- [故障排查](#故障排查)
|
|
50
|
+
|
|
45
51
|
## 为什么做这个
|
|
46
52
|
|
|
47
53
|
大多数 DSH 视觉插件把图片“翻译”成一段文字描述再喂给 DeepSeek——有损、一次性、看不见像素。本插件把**原图像素留在视觉模型侧**、把推理留在 DeepSeek 侧,并把“看图”变成一次**普通的工具调用**:
|
|
@@ -72,24 +78,25 @@
|
|
|
72
78
|
|
|
73
79
|
## 快速开始
|
|
74
80
|
|
|
75
|
-
### 1.
|
|
81
|
+
### 1. 安装插件
|
|
76
82
|
|
|
77
|
-
普通 npm / npx
|
|
83
|
+
普通 npm / npx 安装只需要一条命令:
|
|
78
84
|
|
|
79
85
|
```sh
|
|
80
86
|
npx @deepseek-ai/dsh plugin --profile web add dsh-vision-router
|
|
81
|
-
npx @deepseek-ai/dsh web
|
|
82
87
|
```
|
|
83
88
|
|
|
89
|
+
> [!NOTE]
|
|
90
|
+
> 第三方 `dsh-web-plugin-manager` / `dshpm` **v0.4.2+** 现已兼容:其质量门已正确放行作为运行时依赖的 `@deepseek-ai/schemastery`。上面的官方 DSH CLI 仍是推荐安装方式。
|
|
91
|
+
|
|
84
92
|
如果你是从 DeepSeek Harness 源码仓库通过 pnpm 运行,`dsh` 不一定在系统 `PATH` 里,请改用工作区脚本:
|
|
85
93
|
|
|
86
94
|
```sh
|
|
87
95
|
cd deepseek-harness
|
|
88
96
|
pnpm dsh plugin --profile web add dsh-vision-router
|
|
89
|
-
pnpm dsh web
|
|
90
97
|
```
|
|
91
98
|
|
|
92
|
-
如果你已经全局安装 DSH CLI,并且终端里能直接执行 `dsh`,也可以继续使用较短的 `dsh ...`
|
|
99
|
+
如果你已经全局安装 DSH CLI,并且终端里能直接执行 `dsh`,也可以继续使用较短的 `dsh ...` 写法。安装完成后,按你平时的方式启动或重新加载 DSH Web 即可。
|
|
93
100
|
|
|
94
101
|
> [!NOTE]
|
|
95
102
|
> 如果你是把插件**首次安装进一个已经长期运行的 Web 进程**,需要让 DSH Web 进程重新加载一次插件本体。插件加载完成后,新增/删除模型、修改自动识图包装范围都会**热更新,无需再重启 DSH**。
|
|
@@ -288,18 +295,23 @@ Web 配置页在 **设置 → 插件 → 插件配置** 下注册「视觉路由
|
|
|
288
295
|
|
|
289
296
|
### 安装
|
|
290
297
|
|
|
291
|
-
普通 npm / npx
|
|
298
|
+
普通 npm / npx 安装——一条命令:
|
|
292
299
|
|
|
293
300
|
```sh
|
|
294
301
|
npx @deepseek-ai/dsh plugin --profile web add dsh-vision-router
|
|
295
|
-
npx @deepseek-ai/dsh --profile web --dump-config | grep vision-router
|
|
296
302
|
```
|
|
297
303
|
|
|
298
304
|
从 DeepSeek Harness 源码仓库运行:
|
|
299
305
|
|
|
300
306
|
```sh
|
|
301
307
|
pnpm dsh plugin --profile web add dsh-vision-router
|
|
302
|
-
|
|
308
|
+
```
|
|
309
|
+
|
|
310
|
+
可选验证:
|
|
311
|
+
|
|
312
|
+
```sh
|
|
313
|
+
npx @deepseek-ai/dsh --profile web --dump-config | grep vision-router
|
|
314
|
+
# 源码仓库:pnpm dsh --profile web --dump-config | grep vision-router
|
|
303
315
|
```
|
|
304
316
|
|
|
305
317
|
首次把插件装进已经长期运行的 Web profile 时,需要让 Web 进程重新加载插件本体;宿主在启动时通过 `dsh.client` 声明发现浏览器端包。**插件加载完成后,模型目录与包装范围的变化会热更新,不需要为这些变化重启。**
|
|
@@ -325,6 +337,8 @@ pnpm dsh plugin --profile web update dsh-vision-router
|
|
|
325
337
|
|
|
326
338
|
设置存放在 profile 的设置提供方里,升级不丢失。
|
|
327
339
|
|
|
340
|
+
> **新版本一直不生效(`downloaded 0` / `added 0`):** pnpm v11 会拦下发布不足 24 小时的版本;运行 `npx dsh-vision-router repair` 修复过期的带版本号豁免条目后,更新立即生效。
|
|
341
|
+
|
|
328
342
|
> **从 bundle 补丁之前(v0.x)升级:** 现在插件由自带的 bundle 补丁自动挂载,
|
|
329
343
|
> 若 `~/.dsh/profiles/<profile>/cordis.patch.yml` 里还残留旧版手动行,会与之
|
|
330
344
|
> 重复,`dsh web` 启动即报 `duplicate loader entry id: vision-router`。删除
|
package/docs/doctor.md
CHANGED
|
@@ -32,7 +32,7 @@ To inspect only the Web profile:
|
|
|
32
32
|
npx dsh-vision-router doctor --profile web
|
|
33
33
|
```
|
|
34
34
|
|
|
35
|
-
The command locates the DSH home (`$DSH_HOME` when set, otherwise `~/.dsh`), scans profile `package.json` files, reports UTF-8 BOM bytes, validates the JSON after ignoring a leading BOM for diagnosis,
|
|
35
|
+
The command locates the DSH home (`$DSH_HOME` when set, otherwise `~/.dsh`), scans profile `package.json` files, reports UTF-8 BOM bytes, validates the JSON after ignoring a leading BOM for diagnosis, reports whether `dsh-vision-router` is present as a profile dependency and bundle layer, and flags version-pinned `minimumReleaseAgeExclude` entries in the profile's `pnpm-workspace.yaml` that would hold back the next release.
|
|
36
36
|
|
|
37
37
|
## Repair the UTF-8 BOM startup failure
|
|
38
38
|
|
|
@@ -50,3 +50,21 @@ npx dsh-vision-router repair --profile web
|
|
|
50
50
|
```
|
|
51
51
|
|
|
52
52
|
`repair` removes only the three-byte UTF-8 BOM prefix (`EF BB BF`) when it is present, then validates the remaining JSON. It does not reformat, regenerate, or otherwise rewrite the profile contents. If JSON is still invalid for another reason, the command reports that and stops rather than guessing a repair.
|
|
53
|
+
|
|
54
|
+
## Repair a stale release-age exemption (the "update does nothing" gate)
|
|
55
|
+
|
|
56
|
+
pnpm v11 defaults `minimumReleaseAge` to 1440 minutes: a version published less than 24 hours ago is not resolved, so `dsh plugin update` silently keeps the previous version and prints `downloaded 0 / added 0`. An exemption entry that pins a version — `dsh-vision-router@1.2.0` — only exempts that one version and goes stale on the next release, which is why "a new release is out but the update does nothing" keeps recurring.
|
|
57
|
+
|
|
58
|
+
The doctor flags version-pinned entries for `dsh-vision-router` and the `@deepseek-ai/*` host packages:
|
|
59
|
+
|
|
60
|
+
```text
|
|
61
|
+
✗ web — … — release-age exemption version-pinned (dsh-vision-router@1.2.0) — releases younger than 24h will not be picked up
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
Run:
|
|
65
|
+
|
|
66
|
+
```sh
|
|
67
|
+
npx dsh-vision-router repair --profile web
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
to rewrite them to bare names (`dsh-vision-router`, `@deepseek-ai/*`), which exempt every future version, so upgrades take effect immediately again. Unrelated entries and the rest of the file are left untouched.
|
package/index.js
CHANGED
|
@@ -233,6 +233,12 @@ export const Config = z.object({
|
|
|
233
233
|
tool: z.boolean().default(true),
|
|
234
234
|
progressiveTools: z.boolean().default(true),
|
|
235
235
|
autoActivateOnImage: z.boolean().default(true),
|
|
236
|
+
// User feedback (Zhipu official channel): some channels expose vision
|
|
237
|
+
// models whose catalog metadata does not declare image input. Models the
|
|
238
|
+
// built-in name inference does not recognize can be forced here — one model
|
|
239
|
+
// id (or "provider/model") per entry. Only consulted for vision BACKEND
|
|
240
|
+
// capability (the session-side admission stays host-owned).
|
|
241
|
+
extraVisionModels: z.array(z.string()).default([]),
|
|
236
242
|
// Client-persisted UI state (issue #78): DSH Desktop serves the Web UI from
|
|
237
243
|
// a random port on every launch, so origin-scoped localStorage forgets the
|
|
238
244
|
// first-run onboarding dialog and it re-appeared on every boot. These keys
|
|
@@ -2005,6 +2011,158 @@ export function modelInfoAcceptsImages(info) {
|
|
|
2005
2011
|
return Array.isArray(info && info.inputModalities) && info.inputModalities.includes('image')
|
|
2006
2012
|
}
|
|
2007
2013
|
|
|
2014
|
+
// User feedback: channels like the Zhipu official one (open.bigmodel.cn,
|
|
2015
|
+
// configured with a custom model list) expose vision models whose catalog
|
|
2016
|
+
// metadata does NOT declare image input, even though the models accept images
|
|
2017
|
+
// (e.g. glm-4.6v). DSH's Web settings do not write the `input: [text, image]`
|
|
2018
|
+
// declaration for custom channels either, so a strict metadata check hides
|
|
2019
|
+
// perfectly usable vision backends. The conservative, curated name patterns
|
|
2020
|
+
// below recognize well-known multimodal model families as a fallback; models
|
|
2021
|
+
// that still do not match can be forced via the `extraVisionModels` setting.
|
|
2022
|
+
// A vision-looking name does not necessarily identify a generative chat model.
|
|
2023
|
+
// Embedding and reranker endpoints often share the same VL family prefix but
|
|
2024
|
+
// cannot answer vision_describe. Keep them out of the automatic candidate
|
|
2025
|
+
// set; an explicit extraVisionModels override remains the expert escape hatch.
|
|
2026
|
+
const NON_GENERATIVE_VISION_MODEL_HINTS = [
|
|
2027
|
+
/(^|[\/_.-])(embedding|embeddings|embed)(?=$|[\/_.-])/i,
|
|
2028
|
+
/(^|[\/_.-])(rerank|reranker|reranking)(?=$|[\/_.-])/i,
|
|
2029
|
+
]
|
|
2030
|
+
|
|
2031
|
+
export function looksLikeNonGenerativeVisionModel(modelId) {
|
|
2032
|
+
const id = String(modelId ?? '').trim()
|
|
2033
|
+
if (id === '') return false
|
|
2034
|
+
return NON_GENERATIVE_VISION_MODEL_HINTS.some((pattern) => pattern.test(id))
|
|
2035
|
+
}
|
|
2036
|
+
|
|
2037
|
+
const VISION_MODEL_NAME_HINTS = [
|
|
2038
|
+
// Zhipu VLM family: glm-4.6v, glm-4.6v-flash, glm-4v-plus, glm-4.5v(-plus)…
|
|
2039
|
+
/(^|\/)glm-4[\w.-]*v(?=$|[-/])/i,
|
|
2040
|
+
/(^|\/)glm-4v(?=$|[-/])/i,
|
|
2041
|
+
// Qwen VL / QVQ vision-reasoning family (excludes plain qwen3-14b etc.).
|
|
2042
|
+
/(^|\/)qwen[\w.-]*(vl|vision)/i,
|
|
2043
|
+
/(^|\/)qvq(?=$|[-.])/i,
|
|
2044
|
+
// OpenAI multimodal line (gpt-4o*, gpt-4.1*, gpt-5*, gpt-oss*).
|
|
2045
|
+
/(^|\/)gpt-(4o|4\.1|5|oss)(?=$|[-.])/i,
|
|
2046
|
+
/(^|\/)gemini/i,
|
|
2047
|
+
// Claude 3+ / Sonnet/Opus/Haiku are multimodal (claude-2 is not).
|
|
2048
|
+
/(^|\/)(claude-(3|4)(?=$|[-.])|claude[\w.-]*(sonnet|opus|haiku))/i,
|
|
2049
|
+
/(^|\/)(internvl|cogvlm|llava|pixtral)/i,
|
|
2050
|
+
/(^|\/)(doubao|hunyuan|minimax|ernie)[\w.-]*(vl|vision)/i,
|
|
2051
|
+
/(^|\/)ernie-4\.5/i,
|
|
2052
|
+
/(^|\/)(yi-vision|kimi[\w.-]*vision|moonshot[\w.-]*vision)/i,
|
|
2053
|
+
/(^|\/)step[\w.-]*(v|vision)(?=$|[-/])/i,
|
|
2054
|
+
/(^|\/)grok[\w.-]*vision/i,
|
|
2055
|
+
/(^|\/)grok-4(?=$|[-.])/i,
|
|
2056
|
+
/(^|\/)llama[\w.-]*vision/i,
|
|
2057
|
+
/(^|\/)mistral[\w.-]*pixtral/i,
|
|
2058
|
+
/(^|\/)(phi[\w.-]*vision|florence[\w.-]*)/i,
|
|
2059
|
+
]
|
|
2060
|
+
|
|
2061
|
+
/**
|
|
2062
|
+
* Conservative name-based inference for vision capability: true only when the
|
|
2063
|
+
* model id matches a well-known multimodal naming pattern. Used as a fallback
|
|
2064
|
+
* when catalog metadata does not declare image input; never overrides an
|
|
2065
|
+
* explicit text-only declaration on the session/twin paths.
|
|
2066
|
+
*/
|
|
2067
|
+
export function looksLikeVisionModel(modelId) {
|
|
2068
|
+
const id = String(modelId ?? '').trim()
|
|
2069
|
+
if (id === '' || looksLikeNonGenerativeVisionModel(id)) return false
|
|
2070
|
+
return VISION_MODEL_NAME_HINTS.some((pattern) => pattern.test(id))
|
|
2071
|
+
}
|
|
2072
|
+
|
|
2073
|
+
/**
|
|
2074
|
+
* Pure capability decision for a vision backend: an explicit user override
|
|
2075
|
+
* wins first, known non-generative endpoint roles are excluded next, then
|
|
2076
|
+
* declared image metadata and conservative name inference are considered.
|
|
2077
|
+
*
|
|
2078
|
+
* @param info - resolved model metadata (may be undefined when the lookup failed).
|
|
2079
|
+
* @param provider - provider id, used to match "provider/model" override entries.
|
|
2080
|
+
* @param model - model id.
|
|
2081
|
+
* @param extraVisionModels - user-configured model ids (or "provider/model") forced vision-capable.
|
|
2082
|
+
* @returns { image, inputModalities, inferred, reason } where `inferred` is
|
|
2083
|
+
* false for declared image input, 'override' for the user list, 'name' for the
|
|
2084
|
+
* naming heuristic, and `reason` explains a text-only verdict.
|
|
2085
|
+
*/
|
|
2086
|
+
export function decideVisionBackendCapability(info, provider, model, extraVisionModels) {
|
|
2087
|
+
const inputModalities = Array.isArray(info && info.inputModalities)
|
|
2088
|
+
? info.inputModalities.filter((item) => typeof item === 'string')
|
|
2089
|
+
: []
|
|
2090
|
+
const modelId = String(model ?? '').trim()
|
|
2091
|
+
const providerId = String(provider ?? '').trim()
|
|
2092
|
+
const extras = Array.isArray(extraVisionModels)
|
|
2093
|
+
? extraVisionModels.map((entry) => String(entry ?? '').trim()).filter((entry) => entry !== '')
|
|
2094
|
+
: []
|
|
2095
|
+
const forced =
|
|
2096
|
+
modelId !== '' &&
|
|
2097
|
+
extras.some((entry) => entry === modelId || (providerId !== '' && entry === `${providerId}/${modelId}`))
|
|
2098
|
+
// Manual override is deliberately strongest: advanced users can still
|
|
2099
|
+
// force an unusual endpoint that our role/name heuristics reject.
|
|
2100
|
+
if (forced) {
|
|
2101
|
+
return { image: true, inputModalities: [...new Set([...inputModalities, 'image'])], inferred: 'override', reason: undefined }
|
|
2102
|
+
}
|
|
2103
|
+
// A model can consume images and still be the wrong KIND of endpoint
|
|
2104
|
+
// for this plugin: embedding/reranking produces no assistant answer.
|
|
2105
|
+
if (modelId !== '' && looksLikeNonGenerativeVisionModel(modelId)) {
|
|
2106
|
+
return {
|
|
2107
|
+
image: false,
|
|
2108
|
+
inputModalities,
|
|
2109
|
+
inferred: false,
|
|
2110
|
+
reason: 'model name indicates an embedding/reranker endpoint, not a generative vision backend',
|
|
2111
|
+
}
|
|
2112
|
+
}
|
|
2113
|
+
if (inputModalities.includes('image')) {
|
|
2114
|
+
return { image: true, inputModalities, inferred: false, reason: undefined }
|
|
2115
|
+
}
|
|
2116
|
+
if (modelId !== '' && looksLikeVisionModel(modelId)) {
|
|
2117
|
+
return { image: true, inputModalities: [...new Set([...inputModalities, 'image'])], inferred: 'name', reason: undefined }
|
|
2118
|
+
}
|
|
2119
|
+
return {
|
|
2120
|
+
image: false,
|
|
2121
|
+
inputModalities,
|
|
2122
|
+
inferred: false,
|
|
2123
|
+
reason: 'model metadata does not declare image input',
|
|
2124
|
+
}
|
|
2125
|
+
}
|
|
2126
|
+
|
|
2127
|
+
/**
|
|
2128
|
+
* Resolve transport facts for the direct channel compatibility bridge.
|
|
2129
|
+
* Raw llm-pi-ai settings commonly omit baseURL/api for built-in catalog
|
|
2130
|
+
* providers; the materialized pi-ai model carries the effective values.
|
|
2131
|
+
*/
|
|
2132
|
+
export function resolveChannelBridgeTransport(rawProfile, resolvedProfile, modelId) {
|
|
2133
|
+
let resolvedModel
|
|
2134
|
+
try {
|
|
2135
|
+
const getModels = resolvedProfile && resolvedProfile.piProvider && resolvedProfile.piProvider.getModels
|
|
2136
|
+
const models = typeof getModels === 'function'
|
|
2137
|
+
? getModels.call(resolvedProfile.piProvider)
|
|
2138
|
+
: []
|
|
2139
|
+
resolvedModel = Array.isArray(models)
|
|
2140
|
+
? models.find((entry) => entry && String(entry.id) === String(modelId))
|
|
2141
|
+
: undefined
|
|
2142
|
+
} catch {
|
|
2143
|
+
resolvedModel = undefined
|
|
2144
|
+
}
|
|
2145
|
+
const firstString = (...values) =>
|
|
2146
|
+
values.find((value) => typeof value === 'string' && value.trim() !== '')
|
|
2147
|
+
return {
|
|
2148
|
+
baseURL: firstString(
|
|
2149
|
+
resolvedModel && resolvedModel.baseUrl,
|
|
2150
|
+
rawProfile && rawProfile.baseURL,
|
|
2151
|
+
resolvedProfile && resolvedProfile.baseURL,
|
|
2152
|
+
resolvedProfile && resolvedProfile.piProvider && resolvedProfile.piProvider.baseUrl,
|
|
2153
|
+
),
|
|
2154
|
+
api: firstString(
|
|
2155
|
+
resolvedModel && resolvedModel.api,
|
|
2156
|
+
rawProfile && rawProfile.api,
|
|
2157
|
+
resolvedProfile && resolvedProfile.api,
|
|
2158
|
+
),
|
|
2159
|
+
apiKeyEnv: firstString(
|
|
2160
|
+
rawProfile && rawProfile.apiKeyEnv,
|
|
2161
|
+
resolvedProfile && resolvedProfile.apiKeyEnv,
|
|
2162
|
+
),
|
|
2163
|
+
}
|
|
2164
|
+
}
|
|
2165
|
+
|
|
2008
2166
|
export function apply(ctx, config = {}) {
|
|
2009
2167
|
// Route sharp version diagnostics (issue #75) through the harness logger
|
|
2010
2168
|
// instead of console.warn, so the warning lands in the server log.
|
|
@@ -2640,15 +2798,13 @@ export function apply(ctx, config = {}) {
|
|
|
2640
2798
|
}
|
|
2641
2799
|
try {
|
|
2642
2800
|
const info = await ctx.llm.resolveModelInfo(provider, model)
|
|
2643
|
-
|
|
2644
|
-
? info.inputModalities.filter((item) => typeof item === 'string')
|
|
2645
|
-
: []
|
|
2646
|
-
return {
|
|
2647
|
-
image: modelInfoAcceptsImages(info),
|
|
2648
|
-
inputModalities,
|
|
2649
|
-
reason: modelInfoAcceptsImages(info) ? undefined : 'model metadata does not declare image input',
|
|
2650
|
-
}
|
|
2801
|
+
return decideVisionBackendCapability(info, provider, model, current().extraVisionModels)
|
|
2651
2802
|
} catch (error) {
|
|
2803
|
+
// Metadata lookup failed (custom model lists often do): fall back to
|
|
2804
|
+
// the user override list and the name heuristic before declaring the
|
|
2805
|
+
// model text-only.
|
|
2806
|
+
const fallback = decideVisionBackendCapability(undefined, provider, model, current().extraVisionModels)
|
|
2807
|
+
if (fallback.image) return fallback
|
|
2652
2808
|
return {
|
|
2653
2809
|
image: false,
|
|
2654
2810
|
inputModalities: [],
|
|
@@ -2657,6 +2813,123 @@ export function apply(ctx, config = {}) {
|
|
|
2657
2813
|
}
|
|
2658
2814
|
}
|
|
2659
2815
|
|
|
2816
|
+
// ── direct OpenAI-compatible bridge for undeclared vision channels ─────────
|
|
2817
|
+
//
|
|
2818
|
+
// User feedback (Zhipu official channel, open.bigmodel.cn): some channels
|
|
2819
|
+
// expose vision models whose catalog metadata does NOT declare image input,
|
|
2820
|
+
// so the channel adapter refuses image requests at the wire
|
|
2821
|
+
// (UNSUPPORTED_CONTENT: model "x" does not support image input) even though
|
|
2822
|
+
// the models accept images. For backends recognized only through the name
|
|
2823
|
+
// inference or the extraVisionModels override, fall back to calling the
|
|
2824
|
+
// channel's OpenAI-compatible endpoint directly with the channel's own
|
|
2825
|
+
// baseURL and credential — no hand-edited settings.yaml needed. Defensive
|
|
2826
|
+
// reads only: if the channel settings section, the baseURL, or the
|
|
2827
|
+
// credential cannot be resolved, the bridge is simply unavailable and the
|
|
2828
|
+
// adapter's own error is reported.
|
|
2829
|
+
const rawChannelProfileOf = (provider) => {
|
|
2830
|
+
try {
|
|
2831
|
+
const settings = ctx.get('settings')
|
|
2832
|
+
const section =
|
|
2833
|
+
settings && typeof settings.get === 'function' ? settings.get('llm-pi-ai') : undefined
|
|
2834
|
+
return section && section.providers ? section.providers[provider] : undefined
|
|
2835
|
+
} catch {
|
|
2836
|
+
return undefined
|
|
2837
|
+
}
|
|
2838
|
+
}
|
|
2839
|
+
// DSH's public model metadata intentionally omits endpoint/protocol
|
|
2840
|
+
// details. PiAiAdapter has already materialized those facts in its
|
|
2841
|
+
// resolved profile, so feature-detect that shape as a compatibility
|
|
2842
|
+
// shim. If upstream changes it, this fails closed to the normal chain.
|
|
2843
|
+
const resolvedPiAiProfileOf = (provider) => {
|
|
2844
|
+
try {
|
|
2845
|
+
const registration = ctx.llm.registration(provider)
|
|
2846
|
+
const adapter = registration && registration.adapter
|
|
2847
|
+
const config = adapter && adapter.config
|
|
2848
|
+
const profiles = config && typeof config.profiles === 'function' ? config.profiles() : undefined
|
|
2849
|
+
return profiles && typeof profiles.get === 'function' ? profiles.get(provider) : undefined
|
|
2850
|
+
} catch {
|
|
2851
|
+
return undefined
|
|
2852
|
+
}
|
|
2853
|
+
}
|
|
2854
|
+
const channelBridgePlan = (provider, model) => {
|
|
2855
|
+
const rawProfile = rawChannelProfileOf(provider)
|
|
2856
|
+
const resolvedProfile = resolvedPiAiProfileOf(provider)
|
|
2857
|
+
const transport = resolveChannelBridgeTransport(rawProfile, resolvedProfile, model)
|
|
2858
|
+
if (!transport.baseURL) {
|
|
2859
|
+
return { ok: false, reason: 'no resolved channel baseURL', rawProfile, resolvedProfile, transport }
|
|
2860
|
+
}
|
|
2861
|
+
// callOpenAICompatible speaks Chat Completions. Never send another
|
|
2862
|
+
// provider protocol through this bridge just because its name looks visual.
|
|
2863
|
+
if (transport.api !== 'openai-completions') {
|
|
2864
|
+
return {
|
|
2865
|
+
ok: false,
|
|
2866
|
+
reason: `channel protocol ${transport.api || 'unknown'} is not OpenAI Chat Completions`,
|
|
2867
|
+
rawProfile,
|
|
2868
|
+
resolvedProfile,
|
|
2869
|
+
transport,
|
|
2870
|
+
}
|
|
2871
|
+
}
|
|
2872
|
+
return { ok: true, rawProfile, resolvedProfile, transport }
|
|
2873
|
+
}
|
|
2874
|
+
const resolveChannelApiKey = async (plan) => {
|
|
2875
|
+
const ref = plan && plan.transport && plan.transport.apiKeyEnv
|
|
2876
|
+
if (typeof ref === 'string' && ref !== '') {
|
|
2877
|
+
try {
|
|
2878
|
+
const credentials = ctx.get('credentials')
|
|
2879
|
+
if (credentials !== undefined) {
|
|
2880
|
+
const hit = await credentials.resolve(ref)
|
|
2881
|
+
if (hit && typeof hit.value === 'string' && hit.value.length > 0) return hit.value
|
|
2882
|
+
}
|
|
2883
|
+
} catch {
|
|
2884
|
+
/* fall through to the ambient environment */
|
|
2885
|
+
}
|
|
2886
|
+
if (typeof process !== 'undefined' && process.env && typeof process.env[ref] === 'string') {
|
|
2887
|
+
return process.env[ref]
|
|
2888
|
+
}
|
|
2889
|
+
}
|
|
2890
|
+
// Catalog routes may use provider-native environment discovery and
|
|
2891
|
+
// therefore carry no explicit Harness credential reference.
|
|
2892
|
+
try {
|
|
2893
|
+
const auth = plan && plan.resolvedProfile && plan.resolvedProfile.piProvider
|
|
2894
|
+
&& plan.resolvedProfile.piProvider.auth && plan.resolvedProfile.piProvider.auth.apiKey
|
|
2895
|
+
if (auth && typeof auth.resolve === 'function') {
|
|
2896
|
+
const hit = await auth.resolve({ credential: undefined })
|
|
2897
|
+
const value = hit && hit.auth && hit.auth.apiKey
|
|
2898
|
+
if (typeof value === 'string' && value.length > 0) return value
|
|
2899
|
+
}
|
|
2900
|
+
} catch {
|
|
2901
|
+
/* unavailable native auth */
|
|
2902
|
+
}
|
|
2903
|
+
return undefined
|
|
2904
|
+
}
|
|
2905
|
+
const directChannelVisionAnswer = async (provider, model, blocks, instruction, signal) => {
|
|
2906
|
+
const plan = channelBridgePlan(provider, model)
|
|
2907
|
+
if (!plan.ok) throw new Error(`vision bridge unavailable: ${plan.reason}`)
|
|
2908
|
+
const apiKey = await resolveChannelApiKey(plan)
|
|
2909
|
+
if (apiKey === undefined || apiKey === '') {
|
|
2910
|
+
throw new Error('vision bridge unavailable: channel credential could not be resolved')
|
|
2911
|
+
}
|
|
2912
|
+
const attachments = ctx.get('attachments')
|
|
2913
|
+
if (attachments === undefined) {
|
|
2914
|
+
throw new Error('vision bridge unavailable: attachment service is not registered')
|
|
2915
|
+
}
|
|
2916
|
+
const content = []
|
|
2917
|
+
for (const block of blocks) {
|
|
2918
|
+
const stored = await attachments.readImage(block.attachment)
|
|
2919
|
+
content.push(...toOpenAIContent([block], () => stored.data))
|
|
2920
|
+
}
|
|
2921
|
+
return callOpenAICompatible(
|
|
2922
|
+
{
|
|
2923
|
+
name: provider,
|
|
2924
|
+
baseURL: plan.transport.baseURL,
|
|
2925
|
+
model,
|
|
2926
|
+
apiKeyEnv: '__vision-router-channel__',
|
|
2927
|
+
},
|
|
2928
|
+
[{ role: 'user', content: [...content, { type: 'text', text: instruction }] }],
|
|
2929
|
+
{ maxTokens: 4096, signal, resolveCredential: () => apiKey },
|
|
2930
|
+
)
|
|
2931
|
+
}
|
|
2932
|
+
|
|
2660
2933
|
const collectVisionBackendCapabilities = async () => {
|
|
2661
2934
|
const capabilities = {}
|
|
2662
2935
|
if (typeof ctx.llm.listProviders !== 'function') return capabilities
|
|
@@ -3768,6 +4041,7 @@ export function apply(ctx, config = {}) {
|
|
|
3768
4041
|
const block = await visionBlocksFromBytes(imageBytes, mediaType)
|
|
3769
4042
|
const signal = AbortSignal.timeout(timeoutMs())
|
|
3770
4043
|
const usablePairs = await resolveToolVisionPairs()
|
|
4044
|
+
const pairCapabilities = new Map()
|
|
3771
4045
|
for (const pair of pairs()) {
|
|
3772
4046
|
if (!pair || pair.provider === HTTP_ROUTE) continue
|
|
3773
4047
|
if (!adapterAvailable(ctx.llm, pair.provider)) {
|
|
@@ -3775,6 +4049,7 @@ export function apply(ctx, config = {}) {
|
|
|
3775
4049
|
continue
|
|
3776
4050
|
}
|
|
3777
4051
|
const capability = await resolveVisionBackendCapability(pair.provider, pair.model)
|
|
4052
|
+
pairCapabilities.set(`${pair.provider}/${pair.model}`, capability)
|
|
3778
4053
|
if (!capability.image) {
|
|
3779
4054
|
errors.push(
|
|
3780
4055
|
`${pair.provider}/${pair.model}: not an image-capable backend (${capability.reason ?? 'unknown capability'})`,
|
|
@@ -3782,6 +4057,16 @@ export function apply(ctx, config = {}) {
|
|
|
3782
4057
|
}
|
|
3783
4058
|
}
|
|
3784
4059
|
for (const pair of usablePairs) {
|
|
4060
|
+
// usablePairs also contains auto-discovered models. Before this fix the
|
|
4061
|
+
// map was populated only from explicit config rows, so inferred
|
|
4062
|
+
// SiliconFlow models failed pi-ai image admission and never reached
|
|
4063
|
+
// the direct channel bridge that was meant to rescue them.
|
|
4064
|
+
const pairKey = `${pair.provider}/${pair.model}`
|
|
4065
|
+
let pairCapability = pairCapabilities.get(pairKey)
|
|
4066
|
+
if (pairCapability === undefined) {
|
|
4067
|
+
pairCapability = await resolveVisionBackendCapability(pair.provider, pair.model)
|
|
4068
|
+
pairCapabilities.set(pairKey, pairCapability)
|
|
4069
|
+
}
|
|
3785
4070
|
try {
|
|
3786
4071
|
const text = await visionAnswer(ctx.llm, {
|
|
3787
4072
|
provider: pair.provider,
|
|
@@ -3794,6 +4079,30 @@ export function apply(ctx, config = {}) {
|
|
|
3794
4079
|
})
|
|
3795
4080
|
if (text && text.trim() !== '') return { text: text.trim() }
|
|
3796
4081
|
} catch (error) {
|
|
4082
|
+
// Channels whose catalog does not declare image input reject images
|
|
4083
|
+
// at the adapter wire. When the backend was recognized by the name
|
|
4084
|
+
// inference or the extraVisionModels override, call the channel's
|
|
4085
|
+
// OpenAI-compatible endpoint directly with its own baseURL and
|
|
4086
|
+
// credential before giving up on this pair.
|
|
4087
|
+
const capability = pairCapabilities.get(`${pair.provider}/${pair.model}`)
|
|
4088
|
+
if (capability && capability.inferred) {
|
|
4089
|
+
try {
|
|
4090
|
+
const direct = await directChannelVisionAnswer(
|
|
4091
|
+
pair.provider,
|
|
4092
|
+
pair.model,
|
|
4093
|
+
[block],
|
|
4094
|
+
instruction,
|
|
4095
|
+
signal,
|
|
4096
|
+
)
|
|
4097
|
+
if (direct && direct.trim() !== '') return { text: direct.trim() }
|
|
4098
|
+
} catch (bridgeError) {
|
|
4099
|
+
errors.push(
|
|
4100
|
+
`${pair.provider}/${pair.model}: direct channel fallback failed (${
|
|
4101
|
+
bridgeError && bridgeError.message ? bridgeError.message : String(bridgeError)
|
|
4102
|
+
})`,
|
|
4103
|
+
)
|
|
4104
|
+
}
|
|
4105
|
+
}
|
|
3797
4106
|
errors.push(`${pair.provider}/${pair.model}: ${error && error.message ? error.message : String(error)}`)
|
|
3798
4107
|
}
|
|
3799
4108
|
}
|