dsh-tool-vision 0.10.4 → 0.10.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.en.md +48 -0
- package/README.md +30 -532
- package/README.zh.md +3 -379
- package/index.js +5 -4
- package/package.json +12 -9
package/README.en.md
ADDED
|
@@ -0,0 +1,48 @@
|
|
|
1
|
+
# DSH Tool Vision
|
|
2
|
+
|
|
3
|
+
[简体中文](README.md) | English
|
|
4
|
+
|
|
5
|
+
Connect a separate OpenAI-compatible vision model to [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness), with an image bridge and image tools. A text-only conversation model can ask the vision model to inspect an image and use its description.
|
|
6
|
+
|
|
7
|
+
## Features
|
|
8
|
+
|
|
9
|
+
- **Vision calls:** `inspect_image` sends a local or HTTP(S) image to the configured endpoint, optionally with a question. The model must support compatible image input.
|
|
10
|
+
- **Image bridge:** for models that do not receive images directly, converts images into durable text instructions so the Agent can call a vision tool as needed; conversation thumbnails and previews are supported.
|
|
11
|
+
- **Detection and overrides:** combines route declarations, probe results and user lists to choose between direct image input and bridging; allowlists and blocklists can correct inaccurate declarations.
|
|
12
|
+
- **Image tools:** includes image Q&A, grounding, OCR, crops, pixel differences, colors, vectorization, HTML screenshots and attachment materialization. Some operations require optional dependencies; desktop screenshots are disabled until explicitly enabled.
|
|
13
|
+
|
|
14
|
+
Bridging gives a text model the vision model's description. The result depends on that model and does not turn the text model into a native multimodal model.
|
|
15
|
+
|
|
16
|
+
## Compatibility and installation
|
|
17
|
+
|
|
18
|
+
Supports DSH Web / Desktop. The repository declares compatibility with `0.2.0-rc.2`, `0.2.1-rc.1` / `rc.2` / `rc.3` / `rc.4` / `rc.5` and versions matching `>=0.2.0-rc.2 <0.3.0`; see [package.json](package.json) for the complete prerelease range. Node.js `>=20.18` is required.
|
|
19
|
+
|
|
20
|
+
For Desktop, install its `dsh` command through **Application → Manage dsh Command** first:
|
|
21
|
+
|
|
22
|
+
```sh
|
|
23
|
+
dsh plugin --profile desktop add dsh-tool-vision
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
Replace `desktop` with `web` for Web. Restart DSH and open **Plugins → Installed → dsh-tool-vision**. You can also install it through the [Enhancement Suite](https://github.com/Scorp1o117/dsh-enhancement-suite).
|
|
27
|
+
|
|
28
|
+
## Quick setup
|
|
29
|
+
|
|
30
|
+
1. Enter the vision service's **API address, key and model ID** in the plugin page. A compatible local service can also be used; supply a key if the service requires one.
|
|
31
|
+
2. Choose **Automatic images and previews** for bridging and previews, or **Vision tools only** for explicit calls. Existing custom combinations are preserved.
|
|
32
|
+
3. If image capability detection is inaccurate, adjust the model list, list mode or automatic detection under Advanced. Save to apply.
|
|
33
|
+
4. Send DSH an image and a question, or ask the Agent to inspect a workspace image with `inspect_image`.
|
|
34
|
+
|
|
35
|
+
The form writes edited fields only and retains an unchanged key. The interface follows the DSH language setting.
|
|
36
|
+
|
|
37
|
+
## Usage and limits
|
|
38
|
+
|
|
39
|
+
- Models with image support can receive images directly. Bridged images enter the conversation as instructions and are not automatically restored to image blocks when switching models later.
|
|
40
|
+
- `vision_probe_model` can test image capability. An explicit user list takes precedence over probe results, which in turn take precedence over route declarations.
|
|
41
|
+
- `vision_screenshot` requires enabling `desktopScreenshot`. `sharp`, `potrace` and `puppeteer-core` are optional dependencies; affected tools report missing dependencies without disabling other tools.
|
|
42
|
+
- Vision requests send images to the configured service. Choose a trusted local or remote endpoint for the content. Input-size and provider limits apply.
|
|
43
|
+
|
|
44
|
+
## Development, attribution and feedback
|
|
45
|
+
|
|
46
|
+
- `npm test` runs the project tests. The [historical detailed documentation](https://github.com/Scorp1o117/dsh-tool-vision/blob/00d818b47bd91c330b91207d2be0e50caf8cd87a/README.md) retains configuration fields, tool parameters and release notes; its install commands and compatibility information may be outdated.
|
|
47
|
+
- Pixel-level tools were ported from [dsh-vision-router](https://github.com/ysr666/dsh-vision-router) (MIT).
|
|
48
|
+
- [npm](https://www.npmjs.com/package/dsh-tool-vision) · [Issues](https://github.com/Scorp1o117/dsh-tool-vision/issues) · [MIT License](LICENSE)
|
package/README.md
CHANGED
|
@@ -1,550 +1,48 @@
|
|
|
1
|
-
#
|
|
1
|
+
# DSH Tool Vision
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
简体中文 | [English](README.en.md)
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
为 [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) 接入独立的 OpenAI 兼容视觉模型,并提供图片桥接和图像工具。即使当前对话模型只能处理文本,也可以通过视觉模型查看图片并把描述带回对话。
|
|
6
6
|
|
|
7
|
-
##
|
|
7
|
+
## 功能
|
|
8
8
|
|
|
9
|
-
|
|
9
|
+
- **视觉模型调用**:`inspect_image` 将本地图片或 HTTP(S) 图片发送到配置的视觉端点,可附带问题;模型需要支持兼容的图片输入。
|
|
10
|
+
- **图片桥**:对不直接接收图片的模型,将图片转换为可持久化的文本指引,由 Agent 按需调用视觉工具;支持对话气泡缩略图和预览。
|
|
11
|
+
- **模型识别与手动覆盖**:结合路由声明、实测结果和用户名单决定直发图片还是桥接;可用白名单 / 黑名单修正错误声明。
|
|
12
|
+
- **图像工具**:包含看图问答、目标定位、OCR、裁剪、像素差异、取色、矢量化、HTML 截图和附件落盘等。部分操作需要可选依赖;桌面截图默认关闭,需在设置中主动启用。
|
|
10
13
|
|
|
11
|
-
|
|
14
|
+
图片桥让文本模型获取的是视觉模型返回的描述,效果取决于所选视觉模型,不能把文本模型变成原生多模态模型。
|
|
12
15
|
|
|
13
|
-
|
|
16
|
+
## 兼容性与安装
|
|
14
17
|
|
|
15
|
-
|
|
18
|
+
支持 DSH Web / Desktop;仓库声明兼容 `0.2.0-rc.2`、`0.2.1-rc.1` / `rc.2` / `rc.3` / `rc.4` / `rc.5` 和符合 `>=0.2.0-rc.2 <0.3.0` 的版本,预发布版本范围见 [package.json](package.json)。要求 Node.js `>=20.18`。
|
|
16
19
|
|
|
17
|
-
|
|
20
|
+
Desktop 先在 **应用 → 管理 dsh 命令** 安装桌面版 `dsh`:
|
|
18
21
|
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
Choose **Automatic images and previews** to enable image bridging, automatic admission and thumbnails together, or **Vision tools only** for explicit tool calls. Enter the endpoint, key and model on the main page; model overrides and request tuning live under **Advanced**. Existing mixed switch values appear as **Custom combination** and are retained until you change them. Selecting a mode edits a draft; click Save to apply it. Only edited fields are written, blank keys are preserved, and rejected writes keep the draft.
|
|
22
|
-
|
|
23
|
-
[](README.zh.md)
|
|
24
|
-
|
|
25
|
-
**GitHub**: [Scorp1o117/dsh-tool-vision](https://github.com/Scorp1o117/dsh-tool-vision) · **npm**: [dsh-tool-vision](https://www.npmjs.com/package/dsh-tool-vision)
|
|
26
|
-
|
|
27
|
-
[](https://github.com/Scorp1o117/dsh-enhancement-suite) [](https://www.npmjs.com/package/dsh-enhancement-suite)
|
|
28
|
-
|
|
29
|
-
Part of the [DeepSeek Harness Enhancement Suite](https://github.com/Scorp1o117/dsh-enhancement-suite) — Vision · Soul/Persona · Long-term Memory · Plugin Marketplace.
|
|
30
|
-
|
|
31
|
-
External vision model for [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness).
|
|
32
|
-
|
|
33
|
-
DSH 0.1.1 adds native image input for DeepSeek's vision catalog. This plugin
|
|
34
|
-
remains useful when you want a separate OpenAI-compatible vision endpoint,
|
|
35
|
-
pixel-level image tools, screenshots, or a text-model bridge. The harness
|
|
36
|
-
derives every model request strictly from the session log (`llm/stream`
|
|
37
|
-
requests must equal the durable derivation — the agent-loop invariant), so the
|
|
38
|
-
bridge keeps its conversion inside that durable path:
|
|
39
|
-
|
|
40
|
-
1. **`inspect_image` tool** — sends an image (local file, or http(s) URL) to
|
|
41
|
-
**any OpenAI-compatible** `/chat/completions` endpoint that supports
|
|
42
|
-
`image_url` content parts, and returns the vision model's textual answer
|
|
43
|
-
into the agent loop.
|
|
44
|
-
2. **Image bridge (v0.2.1)** — pasted images are turned into `inspect_image`
|
|
45
|
-
hints *before they enter the durable log*, on the `agent/pre-step`
|
|
46
|
-
waterfall (the one seam where the harness lets a plugin replace the
|
|
47
|
-
messages of a proposed step). Images already logged by an older version
|
|
48
|
-
are repaired lazily with a surface `replace` on the session's first
|
|
49
|
-
pre-step. Only models listed in `multimodalModels` receive image blocks
|
|
50
|
-
directly; a model's declared `inputModalities` are never consulted,
|
|
51
|
-
because profiles routinely declare `input: [text, image]` on text-only
|
|
52
|
-
models just to pass the harness's prompt-admission check.
|
|
53
|
-
|
|
54
|
-
- Zero dependencies beyond the dsh SDK — works with any compatible endpoint:
|
|
55
|
-
OpenAI GPT-4o, Qwen-VL (DashScope), GLM-4V (Zhipu), Moonshot, Gemini
|
|
56
|
-
compatible endpoints, local Ollama, etc.
|
|
57
|
-
- Registered on the **global tools layer**: every agent in the process can
|
|
58
|
-
call `inspect_image`.
|
|
59
|
-
- **Web UI settings section (v0.3.0)**: Settings → 视觉模型 edits the
|
|
60
|
-
`tool-vision` namespace (API endpoint, write-only key, model, bridge
|
|
61
|
-
options) in the active Profile patch; changes hot-apply without a restart. The API
|
|
62
|
-
key lives in that patch. Mount by package name
|
|
63
|
-
(`name: 'dsh-tool-vision'`) so the web client bundle is discovered.
|
|
64
|
-
|
|
65
|
-
## Compatibility (v0.9.7)
|
|
66
|
-
|
|
67
|
-
Verified with DSH `0.1.7-rc.2` (Web) and `0.2.0-rc.2` (Desktop runtime) in isolated profiles. The Desktop app uses its own `desktop` profile. Other DSH prereleases remain unverified.
|
|
68
|
-
|
|
69
|
-
## Desktop install
|
|
70
|
-
|
|
71
|
-
Use the Desktop-installed `dsh` command (Application → Manage dsh Command), or the app’s Plugins page. Then install into the Desktop profile:
|
|
72
|
-
|
|
73
|
-
```powershell
|
|
74
|
-
dsh plugin --profile desktop add dsh-tool-vision@0.9.7
|
|
22
|
+
```sh
|
|
23
|
+
dsh plugin --profile desktop add dsh-tool-vision
|
|
75
24
|
```
|
|
76
25
|
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
## v0.9.4: Windows bridge export filenames
|
|
81
|
-
|
|
82
|
-
- Export filenames now derive from a stable hash of the complete attachment ID. IDs such as `sha256:...` no longer create NTFS alternate data streams and zero-byte visible files.
|
|
83
|
-
- The export directory is created before writing; the in-process cache now respects changes to `bridgeExportDir`.
|
|
84
|
-
- Passed install, Web boot, homepage and client-bundle HTTP checks, and uninstall in a disposable DSH `0.1.5-rc.3` Profile.
|
|
85
|
-
|
|
86
|
-
## Install
|
|
87
|
-
|
|
88
|
-
Mount in a profile patch (`$DSH_HOME/profiles/<name>/cordis.patch.yml`):
|
|
89
|
-
|
|
90
|
-
```yaml
|
|
91
|
-
- insert:
|
|
92
|
-
- id: tool-vision
|
|
93
|
-
name: 'dsh-tool-vision' # after: pnpm add dsh-tool-vision in the profile
|
|
94
|
-
config:
|
|
95
|
-
baseURL: 'https://api.openai.com/v1'
|
|
96
|
-
apiKeyEnv: 'VISION_API_KEY'
|
|
97
|
-
model: 'gpt-4o-mini'
|
|
98
|
-
```
|
|
99
|
-
|
|
100
|
-
Or load it from a local path without npm:
|
|
101
|
-
|
|
102
|
-
```yaml
|
|
103
|
-
- id: tool-vision
|
|
104
|
-
name: './plugins/dsh-tool-vision/index.js'
|
|
105
|
-
```
|
|
106
|
-
|
|
107
|
-
## Config
|
|
108
|
-
|
|
109
|
-
| Field | Default | Meaning |
|
|
110
|
-
|---|---|---|
|
|
111
|
-
| `enabled` | `true` | **Master switch (v0.8.0).** Off unregisters everything this plugin contributes — `inspect_image`, the 14 `vision_*` tools, the image bridge, the preview route and the image-capability declaration. The settings section stays mounted so the switch can turn it back on. Hot-applies; no dsh restart. |
|
|
112
|
-
| `baseURL` | `https://api.openai.com/v1` | OpenAI-compatible API base URL. |
|
|
113
|
-
| `apiKey` | `''` | API key (takes precedence over env). |
|
|
114
|
-
| `apiKeyEnv` | `VISION_API_KEY` | Env var holding the key. |
|
|
115
|
-
| `model` | `gpt-4o-mini` | Vision model id. |
|
|
116
|
-
| `maxTokens` | `1024` | Max output tokens. |
|
|
117
|
-
| `timeoutMs` | `60000` | Per-request timeout. |
|
|
118
|
-
| `maxImageBytes` | `10MB` | Largest accepted local image. |
|
|
119
|
-
| `description` | default | Tool description shown to the model. |
|
|
120
|
-
| `bridgeTextOnly` | `true` | Bridge pasted images to text hints on models that cannot see images. |
|
|
121
|
-
| `bridgeExportDir` | temp | Export dir for bridged images (`os.tmpdir()/dsh-vision-bridge`). Filenames use a portable hash of the attachment ID. |
|
|
122
|
-
| `multimodalModels` | `[]` | Model list (comma-separated). Each entry is matched case-insensitively against the full id, its bare id after the last `/`, and `provider/id`, with `*` / `?` globs (`*vl*`, `deepseek/*`). What the list *means* is set by the mode below. |
|
|
123
|
-
| `multimodalListMode` | `whitelist` | **List mode (v0.9.0).** `whitelist`: listed models receive image blocks directly (the historical behaviour). `blacklist`: listed models are forced through the bridge — the correction layer for a model that claims image support it does not have. `off`: the list is ignored. An unknown value falls back to `whitelist`. |
|
|
124
|
-
| `autoDetectMultimodal` | `true` | **Auto-detect (v0.9.0).** Decide from the current route's own declared `inputModalities`, then combine with the list (whitelist unions, blacklist subtracts). On by default: a text-only route is bridged, a multimodal one is treated like a whitelist member and gets images directly. The declaration is always read *before* this plugin's admission wrap, so `bridgeAutoImage` can never feed its own claim back in as evidence. Set false for the hand-maintained "list only" behaviour. |
|
|
125
|
-
| `probeResults` | `{}` | **Measured verdicts (v0.9.0):** `"provider/model" → "yes"`/`"no"`, written by the `vision_probe_model` tool — do not edit by hand. A measurement outranks a **declaration** (a real request beats a claim) but not `multimodalModels` (explicit human intent has the last word). |
|
|
126
|
-
| `bridgePreview` | `true` | Inline preview for bridged images: thumbnail above the hint text in the user bubble (click to zoom). |
|
|
127
|
-
| `bridgePreviewScanIntervalMs` | `2000` | Fallback scan interval for the preview scanner (ms); `0` disables the fallback. |
|
|
128
|
-
| `bridgePreviewHideHint` | `true` | Hide the bridged hint text once the preview image has loaded (kept on failure — safe degradation). |
|
|
129
|
-
| `bridgeAutoImage` | `true` | While the bridge is on, report image input capability for **every** model to the host admission gate, so pasted images are accepted on text-only models without hand-editing provider configs. |
|
|
130
|
-
| `sendSessionHeader` | `true` | Send a stable session-id header on vision requests. OpenCode Go and similar gateways require `x-opencode-session` (one stable id per conversation); requests without it may error from 2026-09-06. |
|
|
131
|
-
| `sessionHeaderName` | `x-opencode-session` | Header name carrying the session id. |
|
|
132
|
-
| `sessionId` | `''` | Fixed session id for calls without a dsh session context; empty = auto (current dsh session id, else a stable per-process random id). |
|
|
133
|
-
|
|
134
|
-
## Image bridge setup
|
|
135
|
-
|
|
136
|
-
1. (Optional, usually not needed) If `bridgeAutoImage` is disabled, declare
|
|
137
|
-
image input on the models you paste images onto, so the harness admits
|
|
138
|
-
image messages (pi-ai style):
|
|
139
|
-
```yaml
|
|
140
|
-
llm-pi-ai:
|
|
141
|
-
providers:
|
|
142
|
-
your-provider:
|
|
143
|
-
models:
|
|
144
|
-
- id: deepseek-v4-flash
|
|
145
|
-
input: [text, image]
|
|
146
|
-
```
|
|
147
|
-
2. List genuinely multimodal models in the plugin config so they receive
|
|
148
|
-
image blocks untouched (see the next section for the list modes):
|
|
149
|
-
```yaml
|
|
150
|
-
- id: tool-vision
|
|
151
|
-
name: 'dsh-tool-vision'
|
|
152
|
-
config:
|
|
153
|
-
multimodalListMode: whitelist # default: listed models get images directly
|
|
154
|
-
multimodalModels: ['mimo-v2.5', 'grok-4.5']
|
|
155
|
-
```
|
|
156
|
-
|
|
157
|
-
Then pasting an image while on a text-only model stores a hint like
|
|
158
|
-
`[User sent an image, exported to: <path>. Inspect it with the inspect_image tool...]`
|
|
159
|
-
in the transcript (the pasted image no longer renders as pixels in that
|
|
160
|
-
message), and the agent inspects it through the configured vision endpoint.
|
|
161
|
-
|
|
162
|
-
> Why not `llm/stream`? The harness freezes every request and the agent-loop
|
|
163
|
-
> invariant fails any request whose messages diverge from the session-log
|
|
164
|
-
> derivation (`log-reconstruction desync`), and this cordis waterfall's
|
|
165
|
-
> `next()` cannot replace request arguments. The `agent/pre-step` waterfall is
|
|
166
|
-
> the supported seam: its decision messages *become* the durable log, so the
|
|
167
|
-
> invariant stays satisfied.
|
|
168
|
-
|
|
169
|
-
Key resolution order: `config.apiKey` → `process.env[apiKeyEnv]` →
|
|
170
|
-
`process.env.OPENAI_API_KEY`.
|
|
171
|
-
|
|
172
|
-
## Bridge image preview (v0.4.0)
|
|
173
|
-
|
|
174
|
-
On text-only models, pasted images become `[User sent an image...]` hint
|
|
175
|
-
text in the transcript. With `bridgePreview` enabled (default), the browser
|
|
176
|
-
half renders those hints as inline thumbnails **in the display layer only**:
|
|
177
|
-
|
|
178
|
-
- **Thumbnail + lightbox**: click to zoom full-screen; click anywhere or
|
|
179
|
-
press `Esc` to close;
|
|
180
|
-
- **Immediate + fallback**: new messages are handled by a MutationObserver;
|
|
181
|
-
history is back-filled by a periodic scan (interval via
|
|
182
|
-
`bridgePreviewScanIntervalMs`);
|
|
183
|
-
- **Hide the hint (P2)**: with `bridgePreviewHideHint` on, the hint text is
|
|
184
|
-
hidden once the image has loaded, leaving just the image; on load failure
|
|
185
|
-
the text stays (safe degradation — never "no image AND no text");
|
|
186
|
-
- **Precise identification**: bridged hints carry an invisible prefix marker
|
|
187
|
-
(`\u200b[bridge]`), so ordinary user text that happens to contain
|
|
188
|
-
"exported to:" is never misidentified;
|
|
189
|
-
- **Display-layer red line**: persisted messages, the transcript, the
|
|
190
|
-
model-facing text and the `inspect_image` chain are untouched.
|
|
191
|
-
|
|
192
|
-
Preview images are served by the same-origin loopback route
|
|
193
|
-
`/plugins/dsh-tool-vision/image`: read-only access to the bridge export
|
|
194
|
-
directory, localhost-only Host, image extensions only, ≤ 20MB per file,
|
|
195
|
-
path-traversal protected.
|
|
196
|
-
|
|
197
|
-
## Tool: `inspect_image`
|
|
198
|
-
|
|
199
|
-
| Arg | Required | Meaning |
|
|
200
|
-
|---|---|---|
|
|
201
|
-
| `path` | ✅ | Image path (absolute, or relative to the current workspace) or http(s) URL. |
|
|
202
|
-
| `question` | – | Optional specific question about the image. |
|
|
203
|
-
| `detail` | – | `auto` / `low` / `high` resolution hint. |
|
|
204
|
-
|
|
205
|
-
Example endpoints (`baseURL`):
|
|
206
|
-
|
|
207
|
-
- **OpenAI**: `https://api.openai.com/v1` — `gpt-4o`, `gpt-4o-mini`
|
|
208
|
-
- **Alibaba DashScope (Qwen-VL)**: `https://dashscope.aliyuncs.com/compatible-mode/v1` — `qwen-vl-plus`, `qwen-vl-max`
|
|
209
|
-
- **Zhipu (GLM-4V)**: `https://open.bigmodel.cn/api/paas/v4` — `glm-4v-flash` (free tier), `glm-4v-plus`
|
|
210
|
-
- **Moonshot (Kimi)**: `https://api.moonshot.cn/v1` — `moonshot-v1-8k-vision-preview`
|
|
211
|
-
- **Ollama local**: `http://localhost:11434/v1` — `llama3.2-vision` (no key)
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
> **Note for users**
|
|
215
|
-
> - This plugin is a standard **profile bundle** (`dsh.bundle.patch`):
|
|
216
|
-
> `dsh plugin --profile web add dsh-tool-vision` installs and mounts it in
|
|
217
|
-
> one step — no manual `cordis.patch.yml` edits needed.
|
|
218
|
-
> - Settings changes hot-apply (no restart needed).
|
|
219
|
-
> - Version 0.6.3 and newer require DSH `0.1.0-rc.7` or newer and are tested
|
|
220
|
-
> against `0.1.0-rc.7`, `0.1.0-rc.8`, and `0.1.1-rc.1`.
|
|
221
|
-
> - DSH `0.1.0-rc.6` users must pin `dsh-tool-vision@0.6.1`, the last release
|
|
222
|
-
> carrying the legacy settings-allowlist compatibility patch.
|
|
223
|
-
|
|
224
|
-
## Pixel-level vision tools (v0.6.0, ported from dsh-vision-router)
|
|
225
|
-
|
|
226
|
-
14 `vision_*` tools driven by the **same** configured endpoint as
|
|
227
|
-
`inspect_image` (baseURL/apiKey/model) — no provider chain, no local models,
|
|
228
|
-
no extra settings:
|
|
229
|
-
|
|
230
|
-
| Tool | Purpose |
|
|
231
|
-
|---|---|
|
|
232
|
-
| `vision_describe` | Image Q&A / multi-image comparison (optional structured JSON) |
|
|
233
|
-
| `vision_ground` | Locate a target and return its ORIGINAL-pixel bounding box |
|
|
234
|
-
| `vision_detect` | Enumerate elements (buttons, inputs, icons…) with numbered boxes |
|
|
235
|
-
| `vision_crop` | Crop a pixel region to a PNG artifact |
|
|
236
|
-
| `vision_pixel_diff` | Per-pixel comparison: ratio, worst regions, heatmap, report |
|
|
237
|
-
| `vision_colors` | Dominant-color quantization for palette matching |
|
|
238
|
-
| `vision_ocr` | Verbatim text transcription (letters only — not scene analysis) |
|
|
239
|
-
| `vision_long_screenshot_ocr` | Chunked long-screenshot transcription into Markdown |
|
|
240
|
-
| `vision_trace` | Potrace vectorization into colored SVG (worker-thread, safe) |
|
|
241
|
-
| `vision_extract_foreground` | Solid-background removal → transparent PNG |
|
|
242
|
-
| `vision_html_screenshot` | Headless render of a local .html (network blocked) |
|
|
243
|
-
| `vision_screenshot` | Desktop capture (privacy-gated: enable `desktopScreenshot` in settings; Win: PowerShell / macOS: screencapture / Linux: import/scrot) |
|
|
244
|
-
| `vision_present` | Publish a generated image to the user via the host attachment store |
|
|
245
|
-
| `vision_materialize` | Copy an attachment/local image into the workspace as a real path |
|
|
246
|
-
|
|
247
|
-
Quality & safety details:
|
|
248
|
-
|
|
249
|
-
- **Content-hash cache** keyed by endpoint+model+image+question (no stale
|
|
250
|
-
answers across model switches, failures are never cached).
|
|
251
|
-
- **Uniform 4MP downscale** before every model call; oversized inputs are
|
|
252
|
-
rejected with a clear error (stat pre-check, 20MB cap on both file and
|
|
253
|
-
attachment paths).
|
|
254
|
-
- **Rate-limit / 5xx auto-retry** with Retry-After-aware backoff; endpoint
|
|
255
|
-
**content-safety rejections** are surfaced as `VISION_CONTENT_FILTERED`
|
|
256
|
-
instead of a generic backend error.
|
|
257
|
-
- **Long-OCR bounds**: 120s total budget, 40-chunk cap, cancellation checks,
|
|
258
|
-
stop-on-first-backend-failure.
|
|
259
|
-
- **Path containment** for relative inputs; artifacts land in
|
|
260
|
-
`<workspace>/.dsh-tool-vision/`.
|
|
261
|
-
|
|
262
|
-
Requires `sharp` / `potrace` / `puppeteer-core` (declared as optional
|
|
263
|
-
dependencies: a failed platform install never blocks the plugin; missing ones
|
|
264
|
-
degrade lazily with an install hint and never break other tools).
|
|
265
|
-
|
|
266
|
-
`vision_screenshot` is privacy-sensitive and therefore **not registered by
|
|
267
|
-
default** — set `desktopScreenshot: true` in the tool-vision settings to
|
|
268
|
-
enable desktop capture.
|
|
269
|
-
|
|
270
|
-
## v0.9.0: list modes, and auto-detection
|
|
271
|
-
|
|
272
|
-
The bridge answers one question: **can the current model see images directly?**
|
|
273
|
-
v0.9.0 splits it into two independent inputs.
|
|
274
|
-
|
|
275
|
-
```
|
|
276
|
-
base = autoDetectMultimodal ? (route declares image) : {}
|
|
277
|
-
off → direct = base the list takes no part
|
|
278
|
-
whitelist → direct = base ∪ list the list only adds
|
|
279
|
-
blacklist → direct = base \ list the list only subtracts
|
|
280
|
-
```
|
|
281
|
-
|
|
282
|
-
**A list hit always wins**: the list is explicit user intent, so it outranks the
|
|
283
|
-
model's own declaration — which is what makes it a usable correction layer.
|
|
284
|
-
|
|
285
|
-
**A blacklist never degrades into "everything unlisted is direct".** Its base set
|
|
286
|
-
is the auto-detected one; with auto-detection off that base set is empty, so an
|
|
287
|
-
unlisted model is still bridged. That is deliberate: the alternative lets one
|
|
288
|
-
typo push images at a text-only endpoint.
|
|
289
|
-
|
|
290
|
-
**Matching**: `mimo-v2.5`, `xiaomi/mimo-v2.5` and `commandcode/xiaomi/mimo-v2.5`
|
|
291
|
-
all address the same route; `*` / `?` are globs; matching is case-insensitive.
|
|
292
|
-
v0.8.1 compared ids literally, so this README's own `mimo-v2.5` example silently
|
|
293
|
-
did nothing on a route spelled `xiaomi/mimo-v2.5` — fixed here, and the fix only
|
|
294
|
-
ever *adds* models to the direct set (no entry that used to force a model direct
|
|
295
|
-
stops doing so).
|
|
296
|
-
|
|
297
|
-
**In the panel** (Plugins → dsh-tool-vision Model):
|
|
298
|
-
|
|
299
|
-
- the list field is followed by a **clickable list of the models dsh actually has
|
|
300
|
-
configured** (`llm.listProviders()` + `llm.listModels()`), grouped by provider
|
|
301
|
-
and labelled with whether the route *declares* image input. Tick to add,
|
|
302
|
-
untick to remove; the text field above still takes globs by hand. Both edit the
|
|
303
|
-
*same* draft, persisted by Save — the picker head flags it as unsaved until then.
|
|
304
|
-
- a **current route** readout shows `provider / model`, whether images go direct
|
|
305
|
-
or through the bridge, and why (list hit / auto-detect / default).
|
|
306
|
-
|
|
307
|
-
> Why not just a `<datalist>`: a native datalist stays invisible until the user
|
|
308
|
-
> focuses the field and types, which made v0.9.0's first cut look like a dead
|
|
309
|
-
> panel. The list is now always visible, with the datalist kept as a typing aid.
|
|
310
|
-
>
|
|
311
|
-
> Unticking removes **the entry that actually matched** — if `mimo-v2.5` in the
|
|
312
|
-
> list is what covers `xiaomi/mimo-v2.5`, unticking drops `mimo-v2.5` rather than
|
|
313
|
-
> inventing a full id. Which entry hit is computed server-side with the *same*
|
|
314
|
-
> matcher the bridge uses (`matchedEntries` in the payload), so the panel can
|
|
315
|
-
> never display a state that disagrees with the decision.
|
|
316
|
-
|
|
317
|
-
**Read path (the easiest thing to get wrong here)**: `autoDetectMultimodal` MUST
|
|
318
|
-
read the value from *before* `resolveModelInfo` was wrapped, or the "image
|
|
319
|
-
support" that `bridgeAutoImage` stamps onto every model becomes evidence for
|
|
320
|
-
itself. `unwrappedResolveModelInfo()` enforces that, with a dedicated regression
|
|
321
|
-
test.
|
|
322
|
-
|
|
323
|
-
Candidates come from the plugin's own loopback route:
|
|
324
|
-
`GET /plugins/dsh-tool-vision/models` (loopback Host only, read-only, `no-store`).
|
|
325
|
-
It returns provider/model ids and one declared-capability boolean — **no keys and
|
|
326
|
-
no endpoint addresses**. It is registered on the plugin fiber rather than the
|
|
327
|
-
master switch's child fiber, so the panel keeps working while the plugin is off.
|
|
328
|
-
|
|
329
|
-
> ⚠️ `inputModalities` is a **declaration**, not a guarantee — profiles commonly
|
|
330
|
-
> set `input: [text, image]` on text-only models just to pass the admission gate.
|
|
331
|
-
> Upstream `dsh-llm-pi-ai` makes the same call for undeclared models, and its
|
|
332
|
-
> source says why: the two wrong answers **do not cost the same**. Under-claiming
|
|
333
|
-
> refuses the image before it is attached and names the model; over-claiming
|
|
334
|
-
> admits one the provider rejects mid-turn, after the message is already durable.
|
|
335
|
-
>
|
|
336
|
-
> That is why detection is on by default *with* three backstops: (1) the first
|
|
337
|
-
> time a route is promoted purely by its own declaration, the log says so and
|
|
338
|
-
> names the fix; (2) the panel always shows the current route, the decision and
|
|
339
|
-
> the reason; (3) listing that model under `blacklist` mode forces the bridge
|
|
340
|
-
> back on.
|
|
341
|
-
|
|
342
|
-
## Tests
|
|
343
|
-
|
|
344
|
-
```bash
|
|
345
|
-
npm test # server-side unit tests (no extra dependencies)
|
|
346
|
-
npm run test:render # panel render test (needs devDependencies)
|
|
347
|
-
```
|
|
348
|
-
|
|
349
|
-
`npm run test:render` loads the **real client bundle** in jsdom, drives the
|
|
350
|
-
**real registration path** (`apply` → `slots.register` → the component), feeds it
|
|
351
|
-
from the **real server route handler**, and asserts on the real DOM and the real
|
|
352
|
-
settings writes.
|
|
353
|
-
|
|
354
|
-
It is a separate command and deliberately **not part of `npm test`**: it needs
|
|
355
|
-
`react` / `react-dom` / `jsdom`, and a DOM test that silently skips when a
|
|
356
|
-
dependency is missing is a false comfort. Install with
|
|
357
|
-
`npm i -D react@18 react-dom@18 jsdom`.
|
|
358
|
-
|
|
359
|
-
Its reason to exist is specific: v0.9.0's first cut rendered the model list only
|
|
360
|
-
into a native `<datalist>` — every server-side unit test passed while the panel
|
|
361
|
-
looked completely dead. Nothing below the DOM can catch that class of bug.
|
|
362
|
-
|
|
363
|
-
## Capability probe: `vision_probe_model` (v0.9.0)
|
|
364
|
-
|
|
365
|
-
Each request defaults to a 60-second deadline and 2048 output tokens. Response
|
|
366
|
-
bodies are limited to 1 MiB while streaming; an oversized body stops the read
|
|
367
|
-
and yields `unknown`. Authentication, rate-limit and server errors remain
|
|
368
|
-
`unknown` even if their messages mention image input. Only HTTP 400, 415 or
|
|
369
|
-
422 image-rejection errors yield a negative verdict from an error response.
|
|
370
|
-
|
|
371
|
-
Every other signal here rests on what a model **says** about itself. This tool
|
|
372
|
-
sends a real image to the route and reports what it **does** — the only
|
|
373
|
-
ground truth in the plugin.
|
|
374
|
-
|
|
375
|
-
```
|
|
376
|
-
base = autoDetect ? route declares image : {}
|
|
377
|
-
a measured verdict (if any) overrides base measurement > claim
|
|
378
|
-
a list hit (if any) overrides everything human intent > measurement
|
|
379
|
-
```
|
|
380
|
-
|
|
381
|
-
**Why one request is not enough** (each of these was learned the hard way):
|
|
382
|
-
|
|
383
|
-
- **Models guess.** One may answer "blue" to an orange square, so a probe asks
|
|
384
|
-
about **two different colors** and only passes a model that reads both.
|
|
385
|
-
- **Reasoning models return empty `content`** when the thinking budget eats
|
|
386
|
-
`max_tokens`; the default is 2048 and the reader falls back to
|
|
387
|
-
`reasoning_content`.
|
|
388
|
-
- **A control group is mandatory.** A text-only request runs first to prove the
|
|
389
|
-
route works at all; without it a 401 or a timeout would be misread as "cannot
|
|
390
|
-
see images" and push a perfectly good multimodal route back onto the bridge
|
|
391
|
-
**permanently**.
|
|
392
|
-
- **An accepting endpoint is not an image-reading model.** Measured against a
|
|
393
|
-
real gateway: `meituan/LongCat-2.0:free` returned HTTP 200 for the image
|
|
394
|
-
request and answered `"I can't see any image."` Only an endpoint that
|
|
395
|
-
*actively rejects* the image part is a conclusive negative.
|
|
396
|
-
|
|
397
|
-
So a probe ends in one of three verdicts: `yes` (control passed, both colors
|
|
398
|
-
correct), `no` (the endpoint rejected the image part, or answered without
|
|
399
|
-
reading it), or `unknown` (network, auth or protocol trouble — **never** turned
|
|
400
|
-
into a capability claim).
|
|
401
|
-
|
|
402
|
-
**How it gets a route's endpoint and credential** (no new configuration):
|
|
403
|
-
`llm.listConfigurableProviders()` names the provider's settings namespace and
|
|
404
|
-
path → `settings.get(ns)` resolves `baseURL`/`apiKeyEnv`/`api` →
|
|
405
|
-
`credentials.resolve(apiKeyEnv)` yields the secret (`.value`) — the same path
|
|
406
|
-
`dsh-llm-pi-ai` uses for a real call. Read-only, and neither the endpoint nor
|
|
407
|
-
the key ever appears in a probe result.
|
|
408
|
-
|
|
409
|
-
**Linking**: a verdict is written to `probeResults` and takes effect in the
|
|
410
|
-
bridge decision **immediately** — `yes` sends images directly, `no` forces the
|
|
411
|
-
bridge back on — regardless of the list mode. (That is also why it does not
|
|
412
|
-
silently write into `multimodalModels`: a `blacklist` list means the *opposite*,
|
|
413
|
-
so an automatic entry there would produce exactly the wrong result.) Each row in
|
|
414
|
-
the panel's picker carries a badge:
|
|
415
|
-
|
|
416
|
-
- `measured: reads images` / `measured: no image reading` (blue / red), shown in
|
|
417
|
-
preference to the declaration;
|
|
418
|
-
- a never-probed route falls back to the `declares image` badge;
|
|
419
|
-
- the current-route readout reports `measured (a real image was read / NOT read)`.
|
|
420
|
-
|
|
421
|
-
> A route on an unknown protocol (e.g. `openai-responses`) returns `unknown`
|
|
422
|
-
> with a reason, rather than guessing confidently with `chat/completions`.
|
|
423
|
-
|
|
424
|
-
## v0.9.3: adapter snapshot model admission (pi-ai guard)
|
|
425
|
-
|
|
426
|
-
**The bug.** `v0.9.2`'s `installDispatchImageAdmission` satisfied DSH core's
|
|
427
|
-
`LlmService.generate` by declaring `image` on `adapterCall.model.inputModalities`.
|
|
428
|
-
However, `dsh-llm-pi-ai`'s adapter executes a second internal guard inside its
|
|
429
|
-
streaming path (`streamWithSnapshot`):
|
|
430
|
-
|
|
431
|
-
```js
|
|
432
|
-
const model = this.modelOf(snapshot, options.provider, options.model);
|
|
433
|
-
if (containsImage && !model.input.includes("image"))
|
|
434
|
-
throw new LlmError(`pi-ai model "${model.id}" does not support image input`, "UNSUPPORTED_CONTENT");
|
|
435
|
-
```
|
|
436
|
-
|
|
437
|
-
`this.modelOf` resolves from the adapter's own `snapshot.models` catalog, which
|
|
438
|
-
defaults to `input: ["text"]` unless the user explicitly declared
|
|
439
|
-
`input: [text, image]` in the Profile patch. When admitted by `v0.9.2`, the raw image
|
|
440
|
-
reached `streamWithSnapshot` and triggered `UNSUPPORTED_CONTENT`. Since the image
|
|
441
|
-
was already in the session's durable transcript, every subsequent turn failed.
|
|
442
|
-
|
|
443
|
-
**The fix.** On direct routes, `installDispatchImageAdmission` now additionally:
|
|
444
|
-
1. Wraps `adapter.modelOf` (when present) to include `"image"` in `resolved.input`;
|
|
445
|
-
2. Enriches the live snapshot model (`snapshot.models.getModel(route, model).input`)
|
|
446
|
-
with `"image"`;
|
|
447
|
-
3. Fully restores `adapter.modelOf` and reverts modified input arrays upon dispose.
|
|
448
|
-
|
|
449
|
-
## v0.9.2: admission is not dispatch
|
|
450
|
-
|
|
451
|
-
**The bug.** Every capability signal this plugin collects — `probeResults`,
|
|
452
|
-
`multimodalModels`, `autoDetectMultimodal` — decided whether the *bridge*
|
|
453
|
-
intercepts an image. None of them decided whether the model *receives* it.
|
|
454
|
-
|
|
455
|
-
`LlmService.generate` resolves modalities from the adapter, not from the plugin:
|
|
456
|
-
|
|
457
|
-
```js
|
|
458
|
-
const adapterCall = await adapter.prepareCall(provider, model, signal);
|
|
459
|
-
modelInfo = this.normalizeModelInfo(registration, model, adapterCall.model);
|
|
460
|
-
if (modelInfo.inputModalities !== undefined
|
|
461
|
-
&& !modelInfo.inputModalities.includes("image")
|
|
462
|
-
&& projectedMessages.some((message) => contentHasImage(message.content)))
|
|
463
|
-
projectedMessages = projectImagesForTextModel(projectedMessages);
|
|
464
|
-
```
|
|
465
|
-
|
|
466
|
-
`projectImagesForTextModel` rewrites every image block into
|
|
467
|
-
`[image omitted because this model accepts text only; attachment sha256:…]`
|
|
468
|
-
**before the adapter is called**. So on a route the plugin had measured as
|
|
469
|
-
image-capable — where the bridge therefore stepped aside and let the image
|
|
470
|
-
through — the image was still destroyed, by a check that reads the adapter's
|
|
471
|
-
declaration and nothing else. The `resolveModelInfo` wrap (`bridgeAutoImage`) is
|
|
472
|
-
*admission*: it decides who may offer an image, and it never changes what the
|
|
473
|
-
adapter streams. Nothing in the plugin reached the layer that does.
|
|
474
|
-
|
|
475
|
-
The symptom is a route that probes `yes`, is listed in `multimodalModels`, and
|
|
476
|
-
still answers `[image omitted …]`.
|
|
477
|
-
|
|
478
|
-
**The fix.** `installDispatchImageAdmission` wraps `llm.registration` — the
|
|
479
|
-
accessor the core itself calls — so every adapter, including one registered
|
|
480
|
-
later, answers `prepareCall` through the same `routeDirectDecision` the bridge
|
|
481
|
-
uses. One precedence rule, two seams, so the two can never disagree about a
|
|
482
|
-
route. When the decision is "bridge", nothing changes: the bridge already turned
|
|
483
|
-
the image into an `inspect_image` hint, and the core's projection stays as the
|
|
484
|
-
correct fallback for any image that still reaches dispatch.
|
|
485
|
-
|
|
486
|
-
Three properties worth stating, because each one is a test:
|
|
487
|
-
|
|
488
|
-
- **Fail closed.** A decision that cannot be made (a throwing config) leaves the
|
|
489
|
-
call untouched, rather than sending pixels a route may reject.
|
|
490
|
-
- **Reversible.** Dispose restores every adapter it patched *and* the accessor
|
|
491
|
-
itself; a re-install after dispose wraps again.
|
|
492
|
-
- **Late providers count.** The wrap is on the accessor rather than on a snapshot
|
|
493
|
-
of adapters, so a provider registered after install is covered too.
|
|
494
|
-
|
|
495
|
-
`routeDirectDecision` is now the single implementation of the precedence rule,
|
|
496
|
-
shared by the bridge and the dispatch path — two copies of it would drift, and
|
|
497
|
-
this one has three inputs.
|
|
498
|
-
|
|
499
|
-
## v0.8.0: master switch, and the save-path fix
|
|
500
|
-
|
|
501
|
-
**Master switch.** `enabled`, plus a one-click button at the top of the section
|
|
502
|
-
(`Disable all` / `Re-enable`). Registrations are effects on the cordis fiber
|
|
503
|
-
that makes them, so the plugin now puts every tool, the image bridge, the
|
|
504
|
-
preview route and the image-capability declaration in a **child fiber**: turning
|
|
505
|
-
the switch off disposes it, and all 15 tools leave the model's tool list
|
|
506
|
-
together. The settings section stays on the parent fiber, so the switch can turn
|
|
507
|
-
the plugin back on. No dsh restart.
|
|
508
|
-
|
|
509
|
-
**Save-path fix.** The form used to submit its 18 fields as **parallel**
|
|
510
|
-
`scope.set()/unset()` calls. Each write carries its own revision fence, a fence
|
|
511
|
-
behind the Host document is refused with `settings/conflict`, and **a refused
|
|
512
|
-
write still resolves** — the scope's contract is "settle after the write and any
|
|
513
|
-
recovery read", not "throw on refusal". The section therefore reported "Saved"
|
|
514
|
-
while the edits silently reverted, which reads as "settings cannot be saved at
|
|
515
|
-
all".
|
|
26
|
+
Web 将 `desktop` 改为 `web`。安装后重启 DSH,打开 **插件 → 已安装 → dsh-tool-vision**。也可通过 [Enhancement Suite](https://github.com/Scorp1o117/dsh-enhancement-suite) 安装。
|
|
516
27
|
|
|
517
|
-
|
|
518
|
-
one persistence decision, and the section is inspected after the write settles:
|
|
519
|
-
"Saved" only when the change is really there, otherwise "Write did not take
|
|
520
|
-
effect" plus a reload of the form. Hosts without `mutate()` fall back to
|
|
521
|
-
**sequential** writes (each waits for its predecessor, keeping the revision chain
|
|
522
|
-
intact) — never parallel.
|
|
28
|
+
## 快速配置
|
|
523
29
|
|
|
524
|
-
|
|
525
|
-
|
|
526
|
-
|
|
527
|
-
|
|
528
|
-
code that read like a refresh which never happened, and they made the missing
|
|
529
|
-
write verification look intentional.
|
|
30
|
+
1. 在插件页填写视觉服务的 **API 地址、密钥和模型 ID**。也可使用兼容图片输入的本地服务;密钥按服务要求填写。
|
|
31
|
+
2. 选择**自动图片与预览**以启用桥接等功能,或选择**仅视觉工具**进行显式调用;已有自定义组合不会自动覆盖。
|
|
32
|
+
3. 如模型的图片能力识别不准确,在高级设置中调整模型名单、名单模式或自动识别。保存后生效。
|
|
33
|
+
4. 向 DSH 发送图片并提问,或让 Agent 使用 `inspect_image` 查看工作区图片。
|
|
530
34
|
|
|
531
|
-
|
|
35
|
+
设置表单只写入编辑过的字段;未修改的密钥保留。语言跟随 DSH。
|
|
532
36
|
|
|
533
|
-
|
|
534
|
-
pixels) — pixel-precise in-context reasoning is not available to text-only
|
|
535
|
-
models; the vision model's description comes back through `inspect_image`.
|
|
536
|
-
- The bridge is a **one-way door**: an image pasted on a text-only model is
|
|
537
|
-
rewritten into the durable log at `agent/pre-step`, so switching to a
|
|
538
|
-
multimodal model later does not turn it back into an image block. (The other
|
|
539
|
-
direction — multimodal to text-only — is repaired automatically by
|
|
540
|
-
`repairLoggedImages`.)
|
|
541
|
-
- Images are base64-transferred; mind privacy and size limits.
|
|
542
|
-
- Independent of the dsh-llm routing/retry system; failures return clear
|
|
543
|
-
errors to the agent.
|
|
37
|
+
## 使用与限制
|
|
544
38
|
|
|
545
|
-
|
|
39
|
+
- 支持图片的当前模型可直接接收图片;走桥接的图片以指引进入会话,之后切换模型不会自动还原原始图片块。
|
|
40
|
+
- `vision_probe_model` 可实测模型图片能力;手动名单优先于实测,实测优先于路由声明。
|
|
41
|
+
- `vision_screenshot` 需在设置中开启 `desktopScreenshot`。`sharp`、`potrace` 和 `puppeteer-core` 是可选依赖;缺失时受影响的工具会提示,其他工具仍可使用。
|
|
42
|
+
- 视觉请求会将图片发送给配置的服务,请根据内容选择可信的本地或远程端点。输入大小与服务端能力均有限制。
|
|
546
43
|
|
|
547
|
-
|
|
548
|
-
from [dsh-vision-router](https://github.com/ysr666/dsh-vision-router)
|
|
549
|
-
(© ysr666, MIT) with gratitude.
|
|
44
|
+
## 开发、来源与反馈
|
|
550
45
|
|
|
46
|
+
- `npm test` 运行项目测试;[历史详细文档](https://github.com/Scorp1o117/dsh-tool-vision/blob/00d818b47bd91c330b91207d2be0e50caf8cd87a/README.zh.md)保留配置字段、工具参数和旧版本说明,其中安装命令与兼容性信息可能已过时。
|
|
47
|
+
- 像素级视觉工具移植自 [dsh-vision-router](https://github.com/ysr666/dsh-vision-router)(MIT)。
|
|
48
|
+
- [npm](https://www.npmjs.com/package/dsh-tool-vision) · [问题反馈](https://github.com/Scorp1o117/dsh-tool-vision/issues) · [MIT License](LICENSE)
|
package/README.zh.md
CHANGED
|
@@ -1,381 +1,5 @@
|
|
|
1
|
-
#
|
|
1
|
+
# DSH Tool Vision(简体中文)
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
中文文档已迁移至仓库首页:[阅读中文版](README.md)。
|
|
4
4
|
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
## DSH 0.2.1-rc.2
|
|
8
|
-
|
|
9
|
-
适配 DSH 0.2.1-rc.2,明确声明该预发布版本的 engines.dsh 与 peer 范围。宿主新增 DeepSeek V4.1 Flash 1017 及 Search/Fetch API,本插件使用的接口保持不变。保留已有推理配置及自动补齐显式开启策略。
|
|
10
|
-
|
|
11
|
-
## DSH 0.2.1-rc.1
|
|
12
|
-
|
|
13
|
-
支持 DSH 0.2.1-rc.1,通过 `engines.dsh` 和一致的 peer 范围声明兼容,明确包含该预发布版本,并保留已有 0.2.0-rc.2 支持。
|
|
14
|
-
|
|
15
|
-
插件跟随 DSH 的语言设置(DSH 0.2.0-rc.2 内置中文和 English),配置页面、状态提示和插件列表名称/简介同步切换。扩展语言使用宿主的回退链。切换语言保留未保存的设置,无需单独选择插件语言。
|
|
16
|
-
|
|
17
|
-
## 配置入口(DSH 0.2.0-rc.2 起)
|
|
18
|
-
|
|
19
|
-
在首页侧边栏打开 **插件 → 已安装 → dsh-tool-vision**,直接在插件详情页配置并保存。配置页注册到官方的 `plugins.bundle.config` 接口;全局设置页不再重复显示配置入口。Web 与桌面版使用相同界面,本版要求 DSH 0.2.0-rc.2 或更新的 0.2.x 版本。现有配置无需迁移。
|
|
20
|
-
|
|
21
|
-
选择 **自动接图与预览**,一次开启图片桥接、自动接图和缩略图;选择 **仅使用视觉工具**,保留手动工具调用。主页面填写端点、密钥和模型,模型覆盖与请求参数在 **高级设置** 中。已有开关组合显示为 **自定义组合**,不会自动改写。选择模式只修改草稿,点击保存后才生效;只写入编辑过的字段,密钥留空保持原值,写入被拒绝时保留草稿。
|
|
22
|
-
|
|
23
|
-
**GitHub**: [Scorp1o117/dsh-tool-vision](https://github.com/Scorp1o117/dsh-tool-vision) · **npm**: [dsh-tool-vision](https://www.npmjs.com/package/dsh-tool-vision) · [English](README.md)
|
|
24
|
-
|
|
25
|
-
[](https://github.com/Scorp1o117/dsh-enhancement-suite) [](https://www.npmjs.com/package/dsh-enhancement-suite)
|
|
26
|
-
|
|
27
|
-
属于 [DeepSeek Harness Enhancement Suite](https://github.com/Scorp1o117/dsh-enhancement-suite) —— Vision · Soul/Persona · 长期记忆 · 插件市场。
|
|
28
|
-
|
|
29
|
-
给 [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) 外接**视觉模型**的插件。
|
|
30
|
-
|
|
31
|
-
DSH 0.1.1 已为 DeepSeek 视觉模型目录加入原生图片输入。本插件继续提供独立的 OpenAI 兼容视觉端点、像素级图片工具、截图,以及文本模型图片桥。Harness 的每次模型请求都**严格从会话日志推导**(`llm/stream` 请求必须与持久化推导一致,否则 agent-loop invariant 会报 `log-reconstruction desync`),因此图片桥仍在可持久重建的路径内完成转换:
|
|
32
|
-
|
|
33
|
-
1. **`inspect_image` 工具** —— 把图片(本地文件或 http(s) URL)发给任意支持 `image_url` 内容块的 OpenAI 兼容 `/chat/completions` 端点,把视觉模型的文字回答带回对话。
|
|
34
|
-
2. **图片桥(v0.2.1)** —— 粘贴的图片在**进入持久化日志之前**就被转换成 `inspect_image` 指引文本,拦截点是 `agent/pre-step` waterfall(这是 harness 唯一允许插件替换"进入某一步的消息"的缝;替换后的消息会**成为**持久化的 `user/message` 日志,所以请求重建 invariant 天然满足)。旧版本已经写进日志的图片消息,会在该会话下一次 pre-step 时用 surface `replace` 惰性修复。只有 `multimodalModels` 白名单内的模型直收图片块;**不参考模型的 `inputModalities` 声明**——因为很多配置为了通过 prompt 准入检查,会给纯文本模型声明 `input: [text, image]`(那只是声明,不代表上游真的能吃 `image_url`)。
|
|
35
|
-
|
|
36
|
-
- 除 dsh SDK 外零依赖 —— 兼容任意端点:OpenAI GPT-4o、Qwen-VL(DashScope)、GLM-4V(智谱)、Moonshot、Gemini 兼容端点、本地 Ollama 等。
|
|
37
|
-
- 注册在**全局工具层**:进程内所有 Agent 都能调用 `inspect_image`。
|
|
38
|
-
- **Web UI 设置栏(v0.3.0)**:插件 → dsh-tool-vision 编辑 `tool-vision` 配置(API 地址、只写密钥、模型、桥接选项),写入当前 Profile patch,**改动即时生效无需重启**。插件按包名挂载(`name: 'dsh-tool-vision'`)以便 web 端发现客户端 bundle。
|
|
39
|
-
|
|
40
|
-
## 兼容性(v0.9.7)
|
|
41
|
-
|
|
42
|
-
已在 DSH `0.1.7-rc.2`(Web)与 `0.2.0-rc.2`(桌面端运行时)的一次性 Profile 验证。桌面端使用独立的 `desktop` Profile;其他预发布版本暂未验证。
|
|
43
|
-
|
|
44
|
-
## 桌面端安装
|
|
45
|
-
|
|
46
|
-
在桌面端的“插件”页面安装,或使用桌面端“应用 → 管理 dsh 命令”注册的命令:
|
|
47
|
-
|
|
48
|
-
```powershell
|
|
49
|
-
dsh plugin --profile desktop add dsh-tool-vision@0.9.7
|
|
50
|
-
```
|
|
51
|
-
|
|
52
|
-
重启桌面端以加载客户端插件。配置位于 `$DSH_HOME/profiles/desktop`。
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
## v0.9.4:修复 Windows 图片桥导出文件名
|
|
56
|
-
|
|
57
|
-
- 导出文件名改为从完整附件 ID 生成稳定哈希;`sha256:...` 等 ID 不再写入 NTFS 备用数据流,避免出现可见主文件为 0 字节的情况。
|
|
58
|
-
- 写入前确保导出目录存在;进程内缓存按 `bridgeExportDir` 区分。
|
|
59
|
-
- 已通过 DSH `0.1.5-rc.3` 一次性 Profile 的安装、Web 启动、首页与客户端 Bundle HTTP 检查,以及卸载验证。
|
|
60
|
-
|
|
61
|
-
## 安装
|
|
62
|
-
|
|
63
|
-
在 profile patch(`$DSH_HOME/profiles/<name>/cordis.patch.yml`)里挂载:
|
|
64
|
-
|
|
65
|
-
```yaml
|
|
66
|
-
- insert:
|
|
67
|
-
- id: tool-vision
|
|
68
|
-
name: 'dsh-tool-vision' # 前置:在 profile 里 pnpm add dsh-tool-vision
|
|
69
|
-
config:
|
|
70
|
-
baseURL: 'https://api.openai.com/v1'
|
|
71
|
-
apiKeyEnv: 'VISION_API_KEY'
|
|
72
|
-
model: 'gpt-4o-mini'
|
|
73
|
-
```
|
|
74
|
-
|
|
75
|
-
不装 npm 包、直接加载本地路径:
|
|
76
|
-
|
|
77
|
-
```yaml
|
|
78
|
-
- id: tool-vision
|
|
79
|
-
name: './plugins/dsh-tool-vision/index.js'
|
|
80
|
-
```
|
|
81
|
-
|
|
82
|
-
## 配置
|
|
83
|
-
|
|
84
|
-
| 字段 | 默认值 | 含义 |
|
|
85
|
-
|---|---|---|
|
|
86
|
-
| `enabled` | `true` | **总开关(v0.8.0)**。关闭后插件不注册任何东西:`inspect_image`、14 个 `vision_*` 工具、图片桥、预览路由、图片能力声明统统下线;设置栏本身保留,所以随时能再打开。热生效,无需重启 dsh |
|
|
87
|
-
| `baseURL` | `https://api.openai.com/v1` | OpenAI 兼容 API 基地址 |
|
|
88
|
-
| `apiKey` | `''` | API 密钥(优先于环境变量) |
|
|
89
|
-
| `apiKeyEnv` | `VISION_API_KEY` | 存放密钥的环境变量名 |
|
|
90
|
-
| `model` | `gpt-4o-mini` | 视觉模型 id |
|
|
91
|
-
| `maxTokens` | `1024` | 视觉调用最大输出 token |
|
|
92
|
-
| `timeoutMs` | `60000` | 单次请求超时 |
|
|
93
|
-
| `maxImageBytes` | `10MB` | 本地图片大小上限 |
|
|
94
|
-
| `description` | 默认描述 | 工具描述(模型可见) |
|
|
95
|
-
| `bridgeTextOnly` | `true` | 把粘贴图片转成文本指引(发给看不懂图片的模型时) |
|
|
96
|
-
| `bridgeExportDir` | 临时目录 | 桥接图片导出目录(`os.tmpdir()/dsh-vision-bridge`);文件名使用附件 ID 的安全哈希 |
|
|
97
|
-
| `multimodalModels` | `[]` | 模型名单(逗号分隔)。每项按「完整 id / 末段裸 id / `provider/id`」三种写法匹配,大小写不敏感,支持 `*` `?` 通配(如 `*vl*`、`deepseek/*`)。含义由下面的模式决定 |
|
|
98
|
-
| `multimodalListMode` | `whitelist` | **名单模式(v0.9.0)**:`whitelist` 名单内模型直收图片块(旧行为);`blacklist` 名单内模型强制走桥接(用来纠正"声明支持图片但实际不支持"的模型);`off` 忽略名单。未知值一律回退 `whitelist` |
|
|
99
|
-
| `autoDetectMultimodal` | `true` | **自动识别(v0.9.0)**:按当前路由自己声明的 `inputModalities` 判定,再与名单合成(白名单取并集、黑名单取差集)。默认开启=识别到纯文本就交给桥接、识别到多模态就等同白名单成员直发图片;声明永远读"包装前"的真值,不会被 `bridgeAutoImage` 的假声明污染。设为 `false` 可退回"只认名单"的纯手工行为 |
|
|
100
|
-
| `probeResults` | `{}` | **实测结果(v0.9.0)**:`"provider/model" → "yes"`/`"no"`,由 `vision_probe_model` 工具写入,不要手改。实测优先级**高于声明**(真实请求 > 自称),但**低于** `multimodalModels`(人的明确意图有最终话语权) |
|
|
101
|
-
| `bridgePreview` | `true` | 桥接图片内联预览:用户气泡内显示缩略图,点击放大 |
|
|
102
|
-
| `bridgePreviewScanIntervalMs` | `2000` | 预览兜底扫描间隔(毫秒);`0` 关闭兜底 |
|
|
103
|
-
| `bridgePreviewHideHint` | `true` | 图片加载成功后隐藏桥接提示文本(失败时保留,安全降级) |
|
|
104
|
-
| `bridgeAutoImage` | `true` | 桥接开启时向宿主准入检查报告**所有**模型都支持图片输入,纯文本模型也能直接粘贴图片,无需手动改 provider 配置 |
|
|
105
|
-
| `sendSessionHeader` | `true` | 给视觉请求发送稳定的会话标识头。OpenCode Go 等网关要求每请求带 `x-opencode-session`(一个对话一个稳定 ID),缺失的请求在 2026-09-06 起可能报错 |
|
|
106
|
-
| `sessionHeaderName` | `x-opencode-session` | 承载会话 ID 的头名 |
|
|
107
|
-
| `sessionId` | `''` | 固定会话 ID(后台/无 dsh 会话调用时用);留空则自动:优先取当前 dsh 会话 ID,否则用进程级稳定随机 ID |
|
|
108
|
-
|
|
109
|
-
## 图片桥配置
|
|
110
|
-
|
|
111
|
-
1. (一般不需要)只有关闭 `bridgeAutoImage` 时才需要手动给模型声明图片输入(pi-ai 风格),让 harness 放行图片消息:
|
|
112
|
-
```yaml
|
|
113
|
-
llm-pi-ai:
|
|
114
|
-
providers:
|
|
115
|
-
your-provider:
|
|
116
|
-
models:
|
|
117
|
-
- id: deepseek-v4-flash
|
|
118
|
-
input: [text, image]
|
|
119
|
-
```
|
|
120
|
-
2. 在插件配置里列出真正多模态的模型,让它们直收图片块(名单模式见下一节):
|
|
121
|
-
```yaml
|
|
122
|
-
- id: tool-vision
|
|
123
|
-
name: 'dsh-tool-vision'
|
|
124
|
-
config:
|
|
125
|
-
multimodalListMode: whitelist # 默认,名单内直发图片
|
|
126
|
-
multimodalModels: ['mimo-v2.5', 'grok-4.5']
|
|
127
|
-
```
|
|
128
|
-
|
|
129
|
-
之后在文本模型下贴图,转录里会留下一条指引:
|
|
130
|
-
`[User sent an image, exported to: <path>. Inspect it with the inspect_image tool...]`
|
|
131
|
-
(该消息不再以像素图形式渲染),Agent 会调用视觉端点查看并把结果带回对话。
|
|
132
|
-
|
|
133
|
-
> 为什么不用 `llm/stream`?harness 会冻结每个请求,且 agent-loop invariant 会拒绝任何与会话日志推导不一致的请求;这个 cordis 版本的 waterfall `next()` 也无法替换请求参数。`agent/pre-step` 才是受支持的缝:它的决策消息**会成为**持久化日志,invariant 天然成立。
|
|
134
|
-
|
|
135
|
-
密钥解析顺序:`config.apiKey` → `process.env[apiKeyEnv]` → `process.env.OPENAI_API_KEY`。
|
|
136
|
-
|
|
137
|
-
## 桥接图片预览(v0.4.0)
|
|
138
|
-
|
|
139
|
-
纯文本模型下,被桥接的图片在对话里只显示一段 `[User sent an image...]` 文本指引。开启 `bridgePreview`(默认开)后,浏览器端会在**展示层**把指引渲染成气泡内缩略图:
|
|
140
|
-
|
|
141
|
-
- **缩略图 + 灯箱**:点击缩略图全屏放大,点击任意处或按 `Esc` 关闭;
|
|
142
|
-
- **即时 + 兜底**:新消息由 MutationObserver 即时处理,历史消息由周期兜底扫描补齐(间隔见 `bridgePreviewScanIntervalMs`);
|
|
143
|
-
- **隐藏提示文本(P2)**:`bridgePreviewHideHint` 开启时,图片加载成功后桥接文本自动隐藏,气泡只留图片;加载失败则保留文本(安全降级,绝不出现"既无图又无字");
|
|
144
|
-
- **识别机制**:桥接文本带不可见前缀标记(`\u200b[bridge]`),客户端据此精确识别桥接块——用户正常发言中出现"exported to:"字样不会被误伤;
|
|
145
|
-
- **纯展示层红线**:不修改持久化消息、不修改转录、不修改模型侧文本、不碰 `inspect_image` 调用链。
|
|
146
|
-
|
|
147
|
-
预览图片由同源回环路由 `/plugins/dsh-tool-vision/image` 提供,只读桥接导出目录、仅本机 Host、仅图片扩展名、单文件 ≤ 20MB、防目录穿越。
|
|
148
|
-
|
|
149
|
-
## 工具:`inspect_image`
|
|
150
|
-
|
|
151
|
-
| 参数 | 必填 | 含义 |
|
|
152
|
-
|---|---|---|
|
|
153
|
-
| `path` | ✅ | 图片路径(绝对路径,或相对当前工作区)或 http(s) URL |
|
|
154
|
-
| `question` | – | 可选的具体问题 |
|
|
155
|
-
| `detail` | – | `auto` / `low` / `high` 分辨率提示 |
|
|
156
|
-
|
|
157
|
-
示例端点(`baseURL`):
|
|
158
|
-
|
|
159
|
-
- **OpenAI**:`https://api.openai.com/v1` —— `gpt-4o`、`gpt-4o-mini`
|
|
160
|
-
- **阿里云 DashScope(Qwen-VL)**:`https://dashscope.aliyuncs.com/compatible-mode/v1` —— `qwen-vl-plus`、`qwen-vl-max`
|
|
161
|
-
- **智谱(GLM-4V)**:`https://open.bigmodel.cn/api/paas/v4` —— `glm-4v-flash`(免费档)、`glm-4v-plus`
|
|
162
|
-
- **Moonshot(Kimi)**:`https://api.moonshot.cn/v1` —— `moonshot-v1-8k-vision-preview`
|
|
163
|
-
- **Ollama 本地**:`http://localhost:11434/v1` —— `llama3.2-vision`(无需密钥)
|
|
164
|
-
|
|
165
|
-
## 像素级视觉工具(v0.6.0,移植自 dsh-vision-router)
|
|
166
|
-
|
|
167
|
-
14 个 `vision_*` 工具由**同一个** `inspect_image` 配置的端点驱动
|
|
168
|
-
(baseURL/apiKey/model)——无 provider 链、无本地模型、零新增配置:
|
|
169
|
-
|
|
170
|
-
| 工具 | 用途 |
|
|
171
|
-
|---|---|
|
|
172
|
-
| `vision_describe` | 看图问答 / 多图对比(可选结构化 JSON) |
|
|
173
|
-
| `vision_ground` | 定位目标,返回原图像素坐标框 |
|
|
174
|
-
| `vision_detect` | 枚举元素(按钮/输入框/图标…),带编号框 |
|
|
175
|
-
| `vision_crop` | 按像素区域裁剪出 PNG 产物 |
|
|
176
|
-
| `vision_pixel_diff` | 逐像素对比:差异比例、最差区域、热图、报告 |
|
|
177
|
-
| `vision_colors` | 主色量化,还原 UI 调色板 |
|
|
178
|
-
| `vision_ocr` | 逐字转写文字(只读字,不做场景识别) |
|
|
179
|
-
| `vision_long_screenshot_ocr` | 长截图分块转写为 Markdown |
|
|
180
|
-
| `vision_trace` | potrace 矢量化输出彩色 SVG(worker 线程,安全) |
|
|
181
|
-
| `vision_extract_foreground` | 纯色背景抠图 → 透明 PNG |
|
|
182
|
-
| `vision_html_screenshot` | 本地 HTML 无头渲染截图(禁网) |
|
|
183
|
-
| `vision_screenshot` | 桌面截屏(隐私门控:需在设置中开启 `desktopScreenshot`;Win: PowerShell / macOS: screencapture / Linux: import/scrot) |
|
|
184
|
-
| `vision_present` | 通过宿主附件库把生成的图片正式展示给用户 |
|
|
185
|
-
| `vision_materialize` | 把附件/本地图片落盘为工作区真实路径 |
|
|
186
|
-
|
|
187
|
-
质量与安全细节:
|
|
188
|
-
|
|
189
|
-
- **内容哈希缓存**按 端点+模型+图片+问题 取键(切模型不吃旧答案,失败结果
|
|
190
|
-
不入缓存);
|
|
191
|
-
- **统一 4MP 降采样**后再调用模型;超大输入 stat 预检直接拒绝(文件与
|
|
192
|
-
附件路径统一 20MB 上限);
|
|
193
|
-
- **限流/5xx 自动重试**(感知 Retry-After 退避);端点**内容安全拒绝**明确
|
|
194
|
-
返回 `VISION_CONTENT_FILTERED`,不再误报后端不可用;
|
|
195
|
-
- **长截图 OCR 边界**:120s 总预算、40 块上限、取消检查、首块失败即停;
|
|
196
|
-
- **路径 containment**(相对输入禁止逃逸工作区);产物写入
|
|
197
|
-
`<工作区>/.dsh-tool-vision/`。
|
|
198
|
-
|
|
199
|
-
依赖 `sharp` / `potrace` / `puppeteer-core`(声明为**可选依赖**:平台安装失败
|
|
200
|
-
不会阻断插件安装;缺失时懒加载降级并给出安装提示,不影响其他工具)。
|
|
201
|
-
|
|
202
|
-
`vision_screenshot` 属于隐私敏感能力,**默认不注册**——在 tool-vision 设置中
|
|
203
|
-
开启 `desktopScreenshot: true` 后才会注册桌面截屏工具。
|
|
204
|
-
|
|
205
|
-
## v0.9.0:模型名单模式与自动识别
|
|
206
|
-
|
|
207
|
-
桥接要回答一个问题:**当前这个模型能不能直接看图片?** v0.9.0 把它拆成两个相互独立的输入。
|
|
208
|
-
|
|
209
|
-
```
|
|
210
|
-
base = autoDetectMultimodal ? (路由声明含 image) : 空集
|
|
211
|
-
off → direct = base 名单不参与
|
|
212
|
-
whitelist → direct = base ∪ 名单 名单只做"加"
|
|
213
|
-
blacklist → direct = base \ 名单 名单只做"减"
|
|
214
|
-
```
|
|
215
|
-
|
|
216
|
-
**命中名单时以名单为准**:名单是用户明确的意图,优先级高于模型自己的声明,所以永远能纠正误判。
|
|
217
|
-
|
|
218
|
-
**黑名单绝不会退化成"未列出的一律直发"**:黑名单的基准是自动识别集;没开自动识别时基准是空集,未列出的模型照样走桥接。这是刻意设计的——否则一次误配就能把图片硬塞给纯文本端点。
|
|
219
|
-
|
|
220
|
-
**匹配规则**:`mimo-v2.5`、`xiaomi/mimo-v2.5`、`commandcode/xiaomi/mimo-v2.5` 指向同一个路由;`*` / `?` 为通配符;大小写不敏感。v0.8.1 只做字符串全等,README 自己举的 `mimo-v2.5` 例子在 `xiaomi/mimo-v2.5` 这类路由上其实**静默无效**,v0.9.0 修好了(只会多放行,不会收回任何旧配置已经放行的模型)。
|
|
221
|
-
|
|
222
|
-
**面板**(插件 → dsh-tool-vision):
|
|
223
|
-
|
|
224
|
-
- 名单字段下面是一份**可直接勾选的模型清单**:候选来自 dsh 已配置的模型(`llm.listProviders()` + `llm.listModels()`),按 provider 分组,标注哪些**声明**支持图片;勾选加入、取消勾选移出,上方输入框仍可手打通配符(两者共用同一份草稿,改完按「保存」生效,未保存时清单头部会标"未保存");
|
|
225
|
-
- 顶部常驻**当前路由判定**:`provider / model`、直发还是桥接、依据是什么(名单命中 / 自动识别 / 默认)。
|
|
226
|
-
|
|
227
|
-
> 为什么不只用 `<datalist>`:原生 datalist **必须聚焦输入框并敲字才会弹出**,v0.9.0 第一版因此看起来像"面板没反应"。现在清单常驻可见,datalist 只作打字辅助。
|
|
228
|
-
>
|
|
229
|
-
> 取消勾选移除的是**命中的那条记录本身**(名单里写 `mimo-v2.5` 命中了 `xiaomi/mimo-v2.5`,取消勾选就移除 `mimo-v2.5`,不会凭空虚增一个完整 id)。"哪条命中了"由服务端用与桥接**同一套匹配器**算出(payload 的 `matchedEntries`),客户端不复制规则,所以面板显示的状态永远不会和实际判定打架。
|
|
230
|
-
|
|
231
|
-
**判定依据的读取路径**(这是本版最容易做错的地方):`autoDetectMultimodal` **必须**读 `resolveModelInfo` 被包装之前的真值,否则 `bridgeAutoImage` 给所有模型贴上的"支持图片"就成了自证。代码里由 `unwrappedResolveModelInfo()` 保证,并有专门的回归测试。
|
|
232
|
-
|
|
233
|
-
候选项由插件自己的回环路由提供:`GET /plugins/dsh-tool-vision/models`(仅本机 Host、只读、`no-store`),只返回 provider/model id 与声明能力,**不含任何密钥或端点地址**。它挂在插件主 fiber 而非总开关的子 fiber 上,所以插件关闭时面板依然可用。
|
|
234
|
-
|
|
235
|
-
> ⚠️ `inputModalities` 是**声明**,不是保证——profile 里常见"为了过准入检查而写 `input: [text, image]`"的纯文本模型。上游 `dsh-llm-pi-ai` 自己也把"未声明"一律当作纯文本,理由写在源码注释里:**两种误判的代价不对等**——少声明会在贴图前就拒绝并点名模型,多声明则会放行一张"上游中途拒收、而消息已经落盘"的图片。
|
|
236
|
-
>
|
|
237
|
-
> 所以自动识别是默认开启的,同时带三层兜底:① 某个路由**首次**因"仅凭自己的声明"被放行时,日志会打一条明确提示并点名怎么改;② 面板顶部常驻当前判定与依据;③ 把该模型写进 `multimodalModels` 并切到 `blacklist` 模式即可强制回到桥接。
|
|
238
|
-
|
|
239
|
-
## 测试
|
|
240
|
-
|
|
241
|
-
```bash
|
|
242
|
-
npm test # 服务端单测(不需要额外依赖)
|
|
243
|
-
npm run test:render # 面板渲染测试(需要 devDependencies)
|
|
244
|
-
```
|
|
245
|
-
|
|
246
|
-
`npm run test:render` 在 jsdom 里加载**真实客户端 bundle**、走**真实注册路径**(`apply` → `slots.register` → 组件),数据由**真实的服务端路由处理器**产出,最后断言真实 DOM 与真实的设置写入。
|
|
247
|
-
|
|
248
|
-
它单独成命令、**不并入 `npm test`**:它需要 `react` / `react-dom` / `jsdom`,而一个"依赖缺失就静默跳过"的 DOM 测试只会带来虚假的安全感。需要时先 `npm i -D react@18 react-dom@18 jsdom`。
|
|
249
|
-
|
|
250
|
-
它存在的理由很具体:v0.9.0 第一版把模型清单只渲染进原生 `<datalist>` —— 服务端单测**全绿**,面板却看起来完全没反应。这类 bug 在 DOM 之下根本抓不到。
|
|
251
|
-
|
|
252
|
-
## 实测探测:`vision_probe_model`(v0.9.0)
|
|
253
|
-
|
|
254
|
-
每次请求默认超时 60 秒、输出预算 2048 token。响应在流式读取时限制为 1 MiB,
|
|
255
|
-
超限即停止读取并返回 `unknown`。鉴权、限流和服务器故障即使提到图片输入,
|
|
256
|
-
也保持 `unknown`;错误响应仅在 HTTP 400、415、422 且明确拒绝图片时判为不支持。
|
|
257
|
-
|
|
258
|
-
前面所有判定都建立在"模型**说自己**能不能看图"之上。这个工具改为**真的发一张图过去试**,是插件里唯一的地面真相。
|
|
259
|
-
|
|
260
|
-
```
|
|
261
|
-
base = autoDetect ? 路由声明含 image : ∅
|
|
262
|
-
探测结果(若有)覆盖 base 实测 > 自称
|
|
263
|
-
名单命中(若有)覆盖一切 人的意图 > 实测
|
|
264
|
-
```
|
|
265
|
-
|
|
266
|
-
**为什么一次请求不够**(都是实测踩出来的):
|
|
267
|
-
|
|
268
|
-
- **会瞎猜**:模型可能对橙色方块回答"blue"。所以探测跑**两个不同颜色**,两个都答对才算通过;
|
|
269
|
-
- **推理模型会返回空 `content`**:思考预算不够时 `max_tokens` 全被烧光。默认给 2048,并回退读 `reasoning_content`;
|
|
270
|
-
- **必须要有对照组**:先发一条纯文本请求确认"这条路本身是通的",否则一个 401/超时会被误读成"它不能看图"——那会把一条好端端的多模态路由**永久**推回桥接;
|
|
271
|
-
- **端点不报错 ≠ 能看图**。真实网关实测:`meituan/LongCat-2.0:free` 收到图片请求返回 200,但回答是 `"I can't see any image."`。只有"端点主动拒绝图片部分"才是确定性的负面信号。
|
|
272
|
-
|
|
273
|
-
结论分三档:`yes`(对照组通过 + 两色全对)、`no`(端点拒绝图片,或答了但没读图)、`unknown`(网络/鉴权/协议问题——**绝不**当作能力结论)。
|
|
274
|
-
|
|
275
|
-
**它怎么拿到主模型的端点与凭据**(不新增任何配置):`llm.listConfigurableProviders()` 给出 provider 的 settings 命名空间与路径 → `settings.get(ns)` 读出 `baseURL`/`apiKeyEnv`/`api` → `credentials.resolve(apiKeyEnv)` 取出密钥(`.value`),与 `dsh-llm-pi-ai` 真实调用同一条路。只读,且**端点与密钥绝不出现在探测结果里**。
|
|
276
|
-
|
|
277
|
-
**联动**:探测结果写入 `probeResults` 并**立即参与桥接判定**——`yes` 直接放行图片、`no` 强制回到桥接,且**与名单模式无关**。(这比"自动往名单里写一笔"更正确:名单在 `blacklist` 模式下含义是反的,自动写入会得到完全相反的效果。)面板清单里每个模型带徽章:
|
|
278
|
-
|
|
279
|
-
- `已实测可看图` / `已实测不可看图`(蓝 / 红)——实测优先展示;
|
|
280
|
-
- 没探测过的才回退显示声明的 `声明支持图片`;
|
|
281
|
-
- 顶部"依据"栏显示 `实测(真实发图验证通过/未通过)`。
|
|
282
|
-
|
|
283
|
-
> 未知协议的路由(如 `openai-responses`)会直接返回 `unknown` 并说明原因,而不是用 `chat/completions` 去猜一个自信的错误答案。
|
|
284
|
-
|
|
285
|
-
## v0.9.3:适配器快照模型准入(pi-ai 门禁放行)
|
|
286
|
-
|
|
287
|
-
**问题**。`v0.9.2` 的 `installDispatchImageAdmission` 成功满足了 DSH 核心层 `LlmService.generate` 的模态检查,将图片放行给底层适配器。但在以 `dsh-llm-pi-ai` 作为适配器时,其内部在 `streamWithSnapshot` 中执行了第二道门禁校验:
|
|
288
|
-
|
|
289
|
-
```js
|
|
290
|
-
const model = this.modelOf(snapshot, options.provider, options.model);
|
|
291
|
-
if (containsImage && !model.input.includes("image"))
|
|
292
|
-
throw new LlmError(`pi-ai model "${model.id}" does not support image input`, "UNSUPPORTED_CONTENT");
|
|
293
|
-
```
|
|
294
|
-
|
|
295
|
-
`this.modelOf` 直接读取适配器自身维护的 `snapshot.models` 目录,若用户未在 `settings.yaml` 中显式声明 `input: [text, image]`,此处默认只有 `["text"]`。导致未声明的模型在放行图片后,在流式发起前被 `pi-ai` 适配器抛出 `UNSUPPORTED_CONTENT`,且由于该图片已被写进会话历史,后续所有轮次(哪怕发纯文本)都会彻底报错锁死。
|
|
296
|
-
|
|
297
|
-
**修复**。在判定为直通路由(`direct: true`)时,`installDispatchImageAdmission` 增加了双重深度注入:
|
|
298
|
-
1. 包装 `adapter.modelOf`(若存在),使其对直通路由返回携带 `"image"` 模态的模型对象;
|
|
299
|
-
2. 动态向当前快照模型(`snapshot.models.getModel(route, model).input`)追加 `"image"` 模态;
|
|
300
|
-
3. `dispose` 时完整还原 `adapter.modelOf` 并清理被追加的模态数组。
|
|
301
|
-
|
|
302
|
-
## v0.9.2:准入不等于派发
|
|
303
|
-
|
|
304
|
-
**问题**。插件收集的每一个能力信号 —— `probeResults`、`multimodalModels`、
|
|
305
|
-
`autoDetectMultimodal` —— 都只决定**桥接**要不要拦截图片,没有一个决定模型**收不收得到**它。
|
|
306
|
-
|
|
307
|
-
`LlmService.generate` 的能力来源是适配器,不是插件:
|
|
308
|
-
|
|
309
|
-
```js
|
|
310
|
-
const adapterCall = await adapter.prepareCall(provider, model, signal);
|
|
311
|
-
modelInfo = this.normalizeModelInfo(registration, model, adapterCall.model);
|
|
312
|
-
if (modelInfo.inputModalities !== undefined
|
|
313
|
-
&& !modelInfo.inputModalities.includes("image")
|
|
314
|
-
&& projectedMessages.some((message) => contentHasImage(message.content)))
|
|
315
|
-
projectedMessages = projectImagesForTextModel(projectedMessages);
|
|
316
|
-
```
|
|
317
|
-
|
|
318
|
-
`projectImagesForTextModel` 会在**调用适配器之前**把每个图像块改写成
|
|
319
|
-
`[image omitted because this model accepts text only; attachment sha256:…]`。
|
|
320
|
-
于是在一条插件已实测为「能读图」的路由上 —— 桥接因此让开放行 —— 图片依然被销毁,
|
|
321
|
-
而销毁它的那个判断只读适配器的声明,不读任何别的东西。`resolveModelInfo` 那层 wrap
|
|
322
|
-
(`bridgeAutoImage`)是**准入**:它决定谁可以「提交」图片,从不改变适配器实际流出的内容。
|
|
323
|
-
插件里没有任何东西够到真正做决定的那一层。
|
|
324
|
-
|
|
325
|
-
症状就是:一条探测结果为 `yes`、也列在 `multimodalModels` 里的路由,依然回
|
|
326
|
-
`[image omitted …]`。
|
|
327
|
-
|
|
328
|
-
**修复**。`installDispatchImageAdmission` 包装 `llm.registration` —— 也就是核心自己
|
|
329
|
-
调用的那个访问器 —— 于是每个适配器(包括安装之后才注册的)都会经由桥接所用的**同一个**
|
|
330
|
-
`routeDirectDecision` 来回答 `prepareCall`。一份优先级规则、两条缝,两者永不可能对同一
|
|
331
|
-
条路由给出不同答案。判定为「走桥接」时行为不变:桥接早已把图片变成 `inspect_image`
|
|
332
|
-
提示,核心的投影仍然是任何漏到派发层的图片的正确兜底。
|
|
333
|
-
|
|
334
|
-
三条性质值得写下来,因为每一条都有测试:
|
|
335
|
-
|
|
336
|
-
- **失败即保守**。判定无法作出时(配置抛错)保持调用原样,而不是把像素发给一条可能拒绝它的路由。
|
|
337
|
-
- **可逆**。dispose 会还原它改过的每个适配器**以及**访问器本身;dispose 后再安装会重新包装。
|
|
338
|
-
- **后注册的 provider 也算**。包装挂在访问器上而不是适配器快照上,所以安装之后注册的 provider 同样覆盖。
|
|
339
|
-
|
|
340
|
-
`routeDirectDecision` 现在是这条优先级规则的唯一实现,桥接与派发共用 —— 两份实现必然
|
|
341
|
-
漂移,而这份有三个输入。
|
|
342
|
-
|
|
343
|
-
## v0.8.0:总开关,以及保存修复
|
|
344
|
-
|
|
345
|
-
**总开关。** `enabled` 字段加设置栏顶部的一键按钮(`一键关闭` / `重新启用`)。
|
|
346
|
-
注册本身就是绑定在「发起注册的 cordis fiber」上的副作用,所以插件把所有工具、
|
|
347
|
-
图片桥、预览路由和图片能力声明都放进一个**子 fiber**:关掉开关就是销毁这个子
|
|
348
|
-
fiber,15 个工具一并从模型侧消失。设置栏留在父 fiber 上,因此关掉之后还能再打开。
|
|
349
|
-
无需重启 dsh。
|
|
350
|
-
|
|
351
|
-
**保存修复。** 此前表单把 18 个字段作为**并行** `scope.set()/unset()` 提交。每个
|
|
352
|
-
写入各自携带一个 revision 栅栏,而栅栏落后于宿主文档的写入会被 `settings/conflict`
|
|
353
|
-
拒绝——**且被拒绝的写入仍然 resolve**(scope 的契约是「完成写入与恢复读取后结算」,
|
|
354
|
-
不是「被拒就抛错」)。于是设置栏显示「已保存」,而编辑内容静默回退,看起来就是
|
|
355
|
-
「完全存不了」。
|
|
356
|
-
|
|
357
|
-
现在改为**一次原子 `mutate()`**,整批修改共用一个栅栏和一次持久化决策;写入结算后
|
|
358
|
-
再回读命名空间section做校验,只有确认生效才报「已保存」,否则弹出「写入未生效」并
|
|
359
|
-
重新载入表单。没有 `mutate()` 的旧宿主回退为**串行**写入(每个等前一个完成,revision
|
|
360
|
-
链条依然正确),绝不并行。
|
|
361
|
-
|
|
362
|
-
另外删掉了 3 处 `if (typeof scope.load === "function") scope.load()`:`SettingsScope` 接口
|
|
363
|
-
从来没有 `load()`(完整 seam 只有 getSnapshot / subscribe / mutate / set / unset,读走的是
|
|
364
|
-
共享 describe 镜像,由宿主 `settings/document-updated` 驱动刷新)。这些守卫是照臆测 API 写的
|
|
365
|
-
死代码,读起来像"已经刷新过了",反而掩盖了缺失的写入校验。
|
|
366
|
-
|
|
367
|
-
## 限制
|
|
368
|
-
|
|
369
|
-
- 从 `0.6.3` 起最低支持 DSH `0.1.0-rc.7`,已针对 `0.1.0-rc.7`、
|
|
370
|
-
`0.1.0-rc.8` 和 `0.1.1-rc.1` 测试。仍使用 DSH `0.1.0-rc.6` 的用户请锁定
|
|
371
|
-
`dsh-tool-vision@0.6.1`;这是最后一个包含旧 settings 白名单兼容补丁的版本。
|
|
372
|
-
- 被桥接的图片以文本指引进入对话(转录而非像素)——文本模型无法做像素级上下文推理;视觉模型的描述通过 `inspect_image` 回传。
|
|
373
|
-
- 桥接是**单向门**:文本模型下贴的图会在 `agent/pre-step` 被写进持久化日志,之后切到多模态模型也不会还原成图片块(反方向——多模态切到文本模型——会由 `repairLoggedImages` 自动补上桥接)。
|
|
374
|
-
- 图片以 base64 传输;注意隐私与大小限制。
|
|
375
|
-
- 独立于 dsh-llm 的路由/重试体系;失败会向 Agent 返回明确错误。
|
|
376
|
-
|
|
377
|
-
## License
|
|
378
|
-
|
|
379
|
-
MIT —— 桥接预览与整合:xing666173。像素级视觉工具移植自
|
|
380
|
-
[dsh-vision-router](https://github.com/ysr666/dsh-vision-router)(© ysr666,MIT),
|
|
381
|
-
在此致谢。
|
|
5
|
+
[English](README.en.md)
|
package/index.js
CHANGED
|
@@ -975,11 +975,12 @@ async function routeDirectDecision(provider, model, config, llm) {
|
|
|
975
975
|
/**
|
|
976
976
|
* Whether the session's current model may receive image blocks directly.
|
|
977
977
|
*
|
|
978
|
-
* {@link routeDirectDecision} keyed by the
|
|
978
|
+
* {@link routeDirectDecision} keyed by the step's fixed route when available,
|
|
979
|
+
* otherwise by the session's route on older hosts, with the answer
|
|
979
980
|
* recorded in {@link lastModelDecision} for the settings panel.
|
|
980
981
|
*/
|
|
981
|
-
async function currentModelAcceptsImage(agent, config, llm) {
|
|
982
|
-
const { provider, model } = currentRoute(agent);
|
|
982
|
+
async function currentModelAcceptsImage(agent, config, llm, modelSelection) {
|
|
983
|
+
const { provider, model } = modelSelection ?? currentRoute(agent);
|
|
983
984
|
const decision = await routeDirectDecision(provider, model, config, llm);
|
|
984
985
|
Object.assign(lastModelDecision, { provider, model, ...decision });
|
|
985
986
|
return decision.direct;
|
|
@@ -1038,7 +1039,7 @@ function attachPreStepBridge(ctx, getConfig, exportDir) {
|
|
|
1038
1039
|
const agent = payload?.agent;
|
|
1039
1040
|
if (!agent?.session) return decision;
|
|
1040
1041
|
try {
|
|
1041
|
-
const acceptsImage = await currentModelAcceptsImage(agent, getConfig(), ctx.get("llm"));
|
|
1042
|
+
const acceptsImage = await currentModelAcceptsImage(agent, getConfig(), ctx.get("llm"), payload.modelSelection);
|
|
1042
1043
|
warnAutoPromotion(ctx.logger);
|
|
1043
1044
|
if (!acceptsImage) {
|
|
1044
1045
|
let repaired = repairedBySession.get(agent.session.id);
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "dsh-tool-vision",
|
|
3
|
-
"version": "0.10.
|
|
3
|
+
"version": "0.10.6",
|
|
4
4
|
"description": "DeepSeek Harness 外置视觉模型插件:inspect_image 把本地图片或 http(s) 图片 URL 发给任意 OpenAI 兼容端点,视觉模型看图的文字回答直接带回对话;附 Web UI 设置栏。",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "index.js",
|
|
@@ -22,7 +22,7 @@
|
|
|
22
22
|
"platform": "web"
|
|
23
23
|
},
|
|
24
24
|
"compatibility": {
|
|
25
|
-
"dsh": ">=0.2.0-rc.2 <0.3.0 || 0.2.1-rc.1 || 0.2.1-rc.2 || 0.2.1-rc.3",
|
|
25
|
+
"dsh": ">=0.2.0-rc.2 <0.3.0 || 0.2.1-rc.1 || 0.2.1-rc.2 || 0.2.1-rc.3 || 0.2.1-rc.4 || 0.2.1-rc.5 || 0.2.2-rc.1",
|
|
26
26
|
"dshReleases": {
|
|
27
27
|
"0.1.5-rc.2": "incompatible",
|
|
28
28
|
"0.1.5-rc.3": "incompatible",
|
|
@@ -35,7 +35,10 @@
|
|
|
35
35
|
"0.2.0-rc.2": "compatible",
|
|
36
36
|
"0.2.1-rc.1": "compatible",
|
|
37
37
|
"0.2.1-rc.2": "compatible",
|
|
38
|
-
"0.2.1-rc.3": "compatible"
|
|
38
|
+
"0.2.1-rc.3": "compatible",
|
|
39
|
+
"0.2.1-rc.4": "compatible",
|
|
40
|
+
"0.2.1-rc.5": "compatible",
|
|
41
|
+
"0.2.2-rc.1": "compatible"
|
|
39
42
|
},
|
|
40
43
|
"node": ">=20.18",
|
|
41
44
|
"profiles": [
|
|
@@ -71,7 +74,7 @@
|
|
|
71
74
|
"license": "MIT",
|
|
72
75
|
"engines": {
|
|
73
76
|
"node": ">=20.18",
|
|
74
|
-
"dsh": ">=0.2.0-rc.2 <0.3.0 || 0.2.1-rc.1 || 0.2.1-rc.2 || 0.2.1-rc.3"
|
|
77
|
+
"dsh": ">=0.2.0-rc.2 <0.3.0 || 0.2.1-rc.1 || 0.2.1-rc.2 || 0.2.1-rc.3 || 0.2.1-rc.4 || 0.2.1-rc.5 || 0.2.2-rc.1"
|
|
75
78
|
},
|
|
76
79
|
"repository": {
|
|
77
80
|
"type": "git",
|
|
@@ -84,8 +87,8 @@
|
|
|
84
87
|
"@deepseek-ai/schemastery": "^3.18.4"
|
|
85
88
|
},
|
|
86
89
|
"peerDependencies": {
|
|
87
|
-
"@deepseek-ai/dsh-settings": ">=0.2.0-rc.2 <0.3.0 || 0.2.1-rc.1 || 0.2.1-rc.2 || 0.2.1-rc.3",
|
|
88
|
-
"@deepseek-ai/dsh-tools": ">=0.2.0-rc.2 <0.3.0 || 0.2.1-rc.1 || 0.2.1-rc.2 || 0.2.1-rc.3",
|
|
90
|
+
"@deepseek-ai/dsh-settings": ">=0.2.0-rc.2 <0.3.0 || 0.2.1-rc.1 || 0.2.1-rc.2 || 0.2.1-rc.3 || 0.2.1-rc.4 || 0.2.1-rc.5 || 0.2.2-rc.1",
|
|
91
|
+
"@deepseek-ai/dsh-tools": ">=0.2.0-rc.2 <0.3.0 || 0.2.1-rc.1 || 0.2.1-rc.2 || 0.2.1-rc.3 || 0.2.1-rc.4 || 0.2.1-rc.5 || 0.2.2-rc.1",
|
|
89
92
|
"react": "^18.2.0"
|
|
90
93
|
},
|
|
91
94
|
"optionalDependencies": {
|
|
@@ -94,9 +97,9 @@
|
|
|
94
97
|
"sharp": ">=0.35.3 <1"
|
|
95
98
|
},
|
|
96
99
|
"devDependencies": {
|
|
97
|
-
"@deepseek-ai/dsh": "0.2.
|
|
98
|
-
"@deepseek-ai/dsh-settings": "0.2.
|
|
99
|
-
"@deepseek-ai/dsh-tools": "0.2.
|
|
100
|
+
"@deepseek-ai/dsh": "0.2.2-rc.1",
|
|
101
|
+
"@deepseek-ai/dsh-settings": "0.2.2-rc.1",
|
|
102
|
+
"@deepseek-ai/dsh-tools": "0.2.2-rc.1",
|
|
100
103
|
"jsdom": "^30.0.1",
|
|
101
104
|
"react": "^18.3.1",
|
|
102
105
|
"react-dom": "^18.3.1"
|