dsh-vision-router 1.4.5 → 1.5.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -6,7 +6,7 @@
6
6
 
7
7
  <p align="center"><strong>Paste an image and it just works — eyes for text-only agents on DeepSeek Harness. Free out of the box, no key, no Python, one command.</strong></p>
8
8
 
9
- <p align="center">DeepSeek keeps thinking; the built-in free vision chain and eleven pixel-level tools do the seeing. Image turns behave like ordinary tool-calling turns — grounded, measurable, repeatable.</p>
9
+ <p align="center">DeepSeek keeps thinking; the built-in free vision chain and fourteen deep tools do the seeing. Image turns behave like ordinary tool-calling turns — grounded, measurable, repeatable.</p>
10
10
 
11
11
  <p align="center">
12
12
  <a href="https://awesome-dsh-plugin.com"><img src="https://awesome-dsh-plugin.com/badge.svg" alt="awesome · DSH plugin" /></a>
@@ -15,8 +15,8 @@
15
15
  </p>
16
16
 
17
17
  <p align="center">
18
- <a href="https://github.com/ysr666/dsh-vision-router/releases/tag/v1.4.5"><img src="https://img.shields.io/badge/release-v1.4.5-5B4CF0?style=flat-square" alt="Release v1.4.5" /></a>
19
- <a href="tests"><img src="https://img.shields.io/badge/verified-270%20tests-2EA44F?style=flat-square" alt="Verified: 270 tests" /></a>
18
+ <a href="https://github.com/ysr666/dsh-vision-router/releases/tag/v1.5.1"><img src="https://img.shields.io/badge/release-v1.5.1-5B4CF0?style=flat-square" alt="Release v1.5.1" /></a>
19
+ <a href="tests"><img src="https://img.shields.io/badge/verified-354%20tests-2EA44F?style=flat-square" alt="Verified: 354 tests" /></a>
20
20
  <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-2EA44F?style=flat-square" alt="License: MIT" /></a>
21
21
  <a href="package.json"><img src="https://img.shields.io/badge/Node.js-%3E%3D22-339933?style=flat-square&amp;logo=nodedotjs&amp;logoColor=white" alt="Node.js >=22" /></a>
22
22
  <img src="https://img.shields.io/badge/runtime-no%20Python-8A2BE2?style=flat-square" alt="No Python" />
@@ -28,9 +28,9 @@
28
28
  <p align="center">💬 <strong>QQ community group: 1105463028</strong></p>
29
29
 
30
30
  > [!WARNING]
31
- > 📌 **Announcement (v1.4.5)**
31
+ > 📌 **Announcement (v1.5.1)**
32
32
  >
33
- > **v1.4.5:** Adds optional structured 1+x vision pre-scan, refreshes image settings, and fixes Windows log-folder opening.
33
+ > **v1.5.1:** Hardens update recovery, stops hidden Settings write loops, and adds offline attachment recovery.
34
34
 
35
35
  <p align="center">
36
36
  <img src="assets/vision-demo.gif" width="640" alt="Demo: paste an image, the agent locates the send button with vision_ground / vision_crop / vision_pixel_diff and answers with coordinates" />
@@ -187,10 +187,10 @@ Any of these channels can join the vision chain as an `httpProviders` entry (key
187
187
  ## Highlights
188
188
 
189
189
  - **Original pixels, real answers.** The vision chain reads the image at original resolution (auto-downscaled only to protect latency/quota); the agent's question travels with the image, so answers are about *your* question, not a generic description.
190
- - **Automatic failover with classified errors.** Region blocks, ToS filtering, 402 quota, 429 rate limits (with Retry-After backoff), context overflow, network failures — the chain walks providers one by one and only reports after all of them failed, with actionable advice.
190
+ - **Automatic failover with classified errors.** Region blocks, ToS filtering, 402 quota, 429 rate limits, context overflow, network failures — the chain walks providers one by one and only reports after all of them failed, with actionable advice. A 429 immediately advances to the next backend and opens a Retry-After-aware cooldown instead of sleeping inside the request.
191
191
  - **Image memory.** Vision answers are cached by attachment content hash; later text turns substitute the recorded description (marked as untrusted evidence), so DeepSeek genuinely remembers earlier images without re-spending vision calls.
192
192
  - **A verifiable pixel loop.** Reference → `vision_html_screenshot` → `vision_pixel_diff` (ratio + red heatmap + worst-region ranking) → fix → repeat until the mismatch converges. UI restoration becomes measurable instead of eyeballed.
193
- - **Stable tool schema.** All eleven deep tools are registered from session start by default, avoiding a mid-conversation tool-list expansion that can invalidate long-context KV/prefix caches. `progressiveTools: true` remains an advanced boot-time opt-in; only then does `vision_activate` mount the tools on demand. See [`docs/progressive-tools-cache.md`](docs/progressive-tools-cache.md).
193
+ - **Stable tool schema.** All fourteen deep tools are registered from session start by default, avoiding a mid-conversation tool-list expansion that can invalidate long-context KV/prefix caches. `progressiveTools: true` remains an advanced boot-time opt-in; only then does `vision_activate` mount the tools on demand. See [`docs/progressive-tools-cache.md`](docs/progressive-tools-cache.md).
194
194
  - **Selective proxy.** Only the configured vision provider hosts go through your local proxy; DeepSeek stays direct.
195
195
 
196
196
  ### Pixel loop in practice
@@ -211,24 +211,30 @@ The vision model is **only the eyes**; DeepSeek is **always the brain**. An imag
211
211
 
212
212
  ## Tools
213
213
 
214
- Default `progressiveTools: false`: all eleven deep tools stay registered from plugin startup, so text and image turns can call them immediately. If you explicitly set `progressiveTools: true` in the profile/composition `cordis.patch.yml`, progressive mode is restored: only `vision_activate` is exposed initially, the full tool set mounts on first use, and the `vision-tools` skill is registered. This is a boot-time switch; restart DSH after changing it. Built on sharp / potrace / tesseract / system Chrome — no Python:
214
+ Default `progressiveTools: false`: all fourteen deep tools stay registered from plugin startup, so text and image turns can call them immediately. If you explicitly set `progressiveTools: true` in the profile/composition `cordis.patch.yml`, progressive mode is restored: only `vision_activate` is exposed initially, the full tool set mounts on first use, and the `vision-tools` skill is registered. This is a boot-time switch; restart DSH after changing it. Built on sharp / potrace / tesseract / system Chrome — no Python:
215
215
 
216
216
  <p align="center">
217
- <img src="assets/vision-tools.svg" width="100%" alt="Eleven vision tools available in DSH Vision Router." />
217
+ <img src="assets/vision-tools.svg" width="100%" alt="Eleven image-processing tools available in DSH Vision Router." />
218
218
  </p>
219
219
 
220
+ The diagram covers the eleven image-processing tools. `vision_present` (durable image delivery) and `vision_bootstrap` (the optional 1+x structured first pass) bring the default deep-tool set to fourteen. Enabling the privacy-gated `vision_screenshot` at boot adds an optional fifteenth tool.
221
+
220
222
  | Tool | What it does | Artifact |
221
223
  |---|---|---|
224
+ | `vision_bootstrap` | Optional 1+x structured first visual pass; establishes task-independent evidence before at least one follow-up vision call | — |
222
225
  | `vision_describe` | Image Q&A / multi-image compare / structured-evidence JSON mode (summary + layout regions + entity inventory + verbatim transcription) | — |
226
+ | `vision_materialize` | Copy an authorized attachment into the session workspace and return a filesystem path for local OCR/parser fallbacks; no vision/network call | image copy |
223
227
  | `vision_ground` | Locate a target → **original-pixel box x1/y1/x2/y2** | annotated PNG (optional) |
224
228
  | `vision_detect` | Numbered inventory of every element of a kind (buttons/inputs/links…) with original-pixel boxes | annotated PNG with numbered boxes |
225
229
  | `vision_crop` | Crop and zoom into a pixel box | PNG |
230
+ | `vision_present` | Publish a generated or edited local image as a durable chat attachment so the user can see it | image attachment |
226
231
  | `vision_pixel_diff` | Per-pixel comparison: diff ratio + worst 8×8-grid regions | red heatmap PNG + JSON report |
227
232
  | `vision_colors` | Dominant colors (hex + share) | — |
228
233
  | `vision_ocr` | Text transcription: local tesseract (chi_sim+eng) first, vision model fallback | — |
229
234
  | `vision_trace` | SVG vectorization (potrace posterization; icons/logos) | SVG |
230
235
  | `vision_extract_foreground` | Cutout via border flood fill (uniform backgrounds) | transparent PNG |
231
236
  | `vision_html_screenshot` | Screenshot a local HTML file (headless system Chrome); `fullPage: true` captures the whole page and reports `pageHeight` | PNG |
237
+ | `vision_screenshot` | **Disabled by default; explicit opt-in required.** Capture the Windows virtual screen, macOS main display, or Linux root display. Windows uses PowerShell CopyFromScreen, macOS uses `screencapture`, and Linux requires ImageMagick `import` or `scrot`; optional `identify=true` tries enabled local recognition backends in order and returns the description with the path | PNG / + description |
232
238
  | `vision_long_screenshot_ocr` | Long-screenshot transcription: overlapping chunks, tesseract first / vision model fallback, stitched Markdown | chunk PNGs + Markdown + manifest |
233
239
 
234
240
  Formats are sniffed from magic bytes, so extensionless content-addressed attachment files work everywhere (no `.png` renaming needed).
@@ -239,6 +245,7 @@ Formats are sniffed from magic bytes, so extensionless content-addressed attachm
239
245
  vision_ground image="ref.png" target="the send button"
240
246
  vision_detect image="page.png" target="input fields"
241
247
  vision_crop image="ref.png" region="1067,841,1108,881"
248
+ vision_present path="rebuilt.png"
242
249
  vision_describe paths=["ref.png","impl.png"] question="list the differences" json=true
243
250
  vision_pixel_diff original="ref.png" rebuilt="screenshot.png"
244
251
  vision_ocr image="screenshot.png"
@@ -255,15 +262,17 @@ vision_long_screenshot_ocr image="chat-log.png" chunkHeight=1200 overlap=120
255
262
  The vision tools try backends in order and surface an error only after all of them fail:
256
263
 
257
264
  1. **User vision models**: one per settings row, top to bottom; only models under **Settings → Models** that explicitly declare image input are shown;
258
- 2. **Advanced custom HTTP vision endpoints**: legacy/advanced `httpProviders`, when present, run after the user models;
259
- 3. **Built-in anonymous OVH fallback**: always last and never exposed in a model picker. The current quality-first chain is `Qwen3.5-397B-A17B` → `Qwen2.5-VL-72B-Instruct` → `Qwen3.6-27B` → `Mistral-Small-3.2-24B-Instruct-2506` → `Qwen3.5-9B`. OVH anonymous limits are **2 requests/minute per IP per model**. The five models have independent buckets, so spreading requests across them is about **10 RPM in theory**, subject to OVH's actual rate limiting. No signup or API key is required. Want more headroom? See [Free vision key channels](#free-vision-key-channels) — a free OVH access key lifts this same endpoint to 400 requests/minute.
265
+ 2. **Local Ollama (optional, off by default)**: `localOllama.enabled` adds keyless, offline recognition through your local Ollama (for example qwen2.5vl);
266
+ 3. **Local LM Studio (optional, off by default)**: `localLmStudio.enabled` follows Ollama and requires the real model identifier shown in LM Studio Developer or returned by `/v1/models`;
267
+ 4. **Advanced custom HTTP vision endpoints**: legacy/advanced `httpProviders`, when present, run after the local backends;
268
+ 5. **Built-in anonymous OVH fallback**: always last and never exposed in a model picker. The current quality-first chain is `Qwen3.5-397B-A17B` → `Qwen2.5-VL-72B-Instruct` → `Qwen3.6-27B` → `Mistral-Small-3.2-24B-Instruct-2506` → `Qwen3.5-9B`. OVH anonymous limits are **2 requests/minute per IP per model**. The five models have independent buckets, so spreading requests across them is about **10 RPM in theory**, subject to OVH's actual rate limiting. No signup or API key is required. Want more headroom? See [Free vision key channels](#free-vision-key-channels) — a free OVH access key lifts this same endpoint to 400 requests/minute.
260
269
 
261
270
  > [!IMPORTANT]
262
271
  > This “vision chain” is the **eyes** used by Vision Router: each settings row selects one user vision model, while the lower-right chat picker selects the **brain/conversation model**. The two are deliberately separate. Text-only DeepSeek/opencode models are filtered out of the vision-backend dropdown, and the internal `Vision HTTP` transport route is no longer exposed to users.
263
272
 
264
273
  > In the legacy `routing: true` mode, the whole-turn chain walks only `provider + fallbacks` — `httpProviders` (including the free fallback) do not participate there. The default `routing: false` (tools-first) tries everything.
265
274
 
266
- Failures are classified (region / tos / quota / rate-limit / context / network) and the final error carries advice; `429` responses honor `Retry-After` once with a capped backoff. Oversized uploads are downscaled before the call (default budget 4 MP) to keep tool calls fast.
275
+ Failures are classified (region / tos / quota / rate-limit / context / network) and the final error carries advice; a `429` advances immediately to the next backend and applies a capped, `Retry-After`-aware circuit-breaker cooldown. Oversized uploads are downscaled before the call (default budget 4 MP) to keep tool calls fast.
267
276
 
268
277
  ## Stealth mode
269
278
 
@@ -302,7 +311,7 @@ The Web profile registers a **视觉路由(自动识图)** card under **Sett
302
311
  - switches for legacy whole-turn routing, vision tools, image-block rewriting and stealth mode (official DeepSeek route only);
303
312
  - timeout, wrapper/chain route names, proxy and other advanced parameters;
304
313
  - every field shows an overridden badge with one-click reset plus discard/save;
305
- - a **Test connection** button probes the first vision provider and reports latency inline;
314
+ - a **Test connection** button prioritizes an enabled local backend, verifies that its configured model appears in `/v1/models`, and otherwise probes the first usable vision provider;
306
315
  - artifact-producing tools render dedicated call cards with result facts and open-file buttons.
307
316
 
308
317
  <p align="center">
@@ -330,6 +339,12 @@ Everything is optional; defaults work out of the box. Edit via the Web card or a
330
339
  | `textProvider` | `deepseek-official` / `deepseek-v4-pro` | the model that reasons (your daily model) |
331
340
  | `tool` / `progressiveTools` / `autoActivateOnImage` | `true` / `false` / `true` | vision tools on / progressive mounting (off by default for a stable tool schema) / image-turn auto-mount when progressive mode is enabled; `progressiveTools` is boot-time config |
332
341
  | `rewriteImages` | `true` | rewrite image blocks in the model input (cached description or tool-hint marker); the UI log keeps images |
342
+ | `desktopScreenshot` | `false` | privacy opt-in for the model-callable `vision_screenshot` desktop-capture tool; checked live before every capture |
343
+ | `freeFallback` | `true` | append the anonymous OVH models after explicit local/custom HTTP backends; turning this off never disables an explicitly configured local backend |
344
+ | `localOllama` | `{ enabled: false, baseURL: 'http://127.0.0.1:11434/v1', model: 'qwen2.5vl', format: 'openai' }` | **Local vision backend (merged from dsh-vision)**: when enabled, `local-ollama` leads the HTTP vision chain; skipped automatically when Ollama is down; `format` selects `openai` (`/chat/completions`) or `anthropic` (`/messages`); optional `temperature` / `top_p` are sent only when explicitly set, otherwise the local server default is respected |
345
+ | `localLmStudio` | `{ enabled: false, baseURL: 'http://localhost:1234/v1', model: '', format: 'openai' }` | **Local LM Studio backend (merged from dsh-vision)**: follows Ollama and precedes custom/cloud HTTP backends; enabling it requires the real model identifier shown in LM Studio Developer or returned by `/v1/models`; supports the same optional sampling fields, while `format: 'anthropic'` requires LM Studio 0.4.1+ |
346
+ | `instantDescribe` | `false` | **Instant local translation (merged from dsh-vision)**: when on and at least one local backend is usable, uncached image blocks are recognized before the first model step; Ollama is tried before LM Studio with a shared timeout budget, multi-image batches run concurrently (up to 3), and failures fall back to the static tool-hint marker |
347
+ | `localDescribeStyle` | `plain` | **Local recognition output style (merged from dsh-vision)**: `plain` = flat description; `structured` = structured recognition (【初步判断】/【细节】/【空间结构】/【原图尺寸】), better for screenshot analysis |
333
348
  | `downscale` / `downscaleMaxPixels` | `true` / `4000000` | pre-call downscale and its pixel budget (latency guard) |
334
349
  | `cache` / `cacheTtlSeconds` / `cacheMaxEntries` | `true` / `3600` / `200` | vision answer cache |
335
350
  | `timeoutMs` | `120000` | per vision call deadline |
@@ -337,12 +352,57 @@ Everything is optional; defaults work out of the box. Edit via the Web card or a
337
352
  | `proxy` / `proxyHosts` | `''` / openrouter hosts | optional proxy for vision provider hosts only |
338
353
  | `catalogCorrections` | `true` | built-in catalog-routing corrections: when the installed pi-ai catalog routes a known model to the wrong wire protocol (e.g. `opencode-go/qwen3.6-plus` to OpenAI chat completions while OpenCode Go only serves it on `/v1/messages`), the plugin answers that backend directly over the corrected protocol. Each correction disarms itself once the catalog is fixed upstream |
339
354
 
355
+ ### Local Ollama vision backend (merged from dsh-vision)
356
+
357
+ > **Incremental author**: [shaoqiuyuavailable](https://github.com/shaoqiuyuavailable) (router local-vision increment)
358
+ >
359
+ > **Design credit**: the local vision backends (Ollama / LM Studio dual backends, instant recognition,
360
+ > structured output, screenshot identification, same-image memory dedup, failure-fallback placeholder,
361
+ > concurrency anti-snowball, timeout protection) inherit their design from
362
+ > [dsh-vision](https://github.com/shaoqiuyuavailable/text-llm-vision/tree/dsh-vision) —
363
+ > merged into the HTTP vision chain here, with per-level fallback and dual-protocol support added on top.
364
+
365
+ An optional keyless local-first vision path for private, free, offline recognition. It plugs into the existing HTTP vision chain as `local-ollama`; if it fails, any configured cloud backends can still be tried unless you deliberately configure a local-only chain.
366
+
367
+ **1. Install Ollama and pull a vision model**
368
+
369
+ ```sh
370
+ # https://ollama.com — then:
371
+ ollama pull qwen2.5vl
372
+ ```
373
+
374
+ **2. Enable it** — in the settings card's "Local vision" group, or via a profile patch:
375
+
376
+ ```yaml
377
+ - id: vision-router
378
+ config:
379
+ localOllama:
380
+ enabled: true
381
+ baseURL: 'http://127.0.0.1:11434/v1' # OpenAI-compatible endpoint
382
+ model: 'qwen2.5vl'
383
+ temperature: 0.5 # optional; low temperature is steadier for recognition
384
+ top_p: 0.8 # optional; unset = server default
385
+ instantDescribe: true # recognize images on the first model step
386
+ localDescribeStyle: 'structured' # 'plain' | 'structured'
387
+ ```
388
+
389
+ **3. What happens**
390
+
391
+ - When enabled, `local-ollama` heads the HTTP vision chain. For a strict local-only setup, remove cloud vision rows/custom HTTP endpoints and turn off `freeFallback`.
392
+ - **LM Studio works the same way** — enable `localLmStudio` in the same "Local vision" group with its OpenAI-compatible endpoint (default `http://localhost:1234/v1`) and enter the exact model identifier shown in Developer or `/v1/models`. It sits after `local-ollama` and before custom/cloud HTTP backends.
393
+ - Each local backend can speak **OpenAI or Anthropic format** via `format` (default `openai`). Anthropic mode routes to `/v1/messages` with `anthropic-version` and base64 image sources; `x-api-key` is sent only when a key is configured. LM Studio needs version 0.4.1 or newer for this endpoint.
394
+ - If a local backend is down or the call times out, its entry is skipped automatically and the chain falls through to the cloud backends — no call breaks.
395
+ - `instantDescribe` tries enabled local backends in order (Ollama, then LM Studio) before the first model step. Multiple uncached images run concurrently (up to 3); one failed image does not block the others, and attachment-memory hits are reused without another local request.
396
+ - `vision_screenshot` is disabled by default. After the separate Desktop screenshot opt-in, `identify=true` uses the same Ollama → LM Studio fallback.
397
+ - Verify runtime decisions with `image turn — instantDescribe=… localBackends=…` and results with `instant local describe recognized N/M uncached image(s), C cached, F failed attempts` in the log.
398
+
340
399
  ## Requirements
341
400
 
342
401
  - DeepSeek Harness Web profile. Normal installs can use `npx @deepseek-ai/dsh ...`; source checkouts use `pnpm dsh ...`. A bare `dsh ...` command only works when the CLI is already on your shell `PATH`.
343
402
  - Node ≥ 22 (host side).
344
403
  - No API key for the default free chain; a credential reference (`apiKeyEnv`) only for paid `httpProviders`.
345
- - Chrome / Chromium / Edge only for `vision_html_screenshot`; every other tool works without a browser.
404
+ - Chrome / Chromium / Edge is needed only for `vision_html_screenshot`; every other tool works without a browser.
405
+ - Desktop capture is opt-in. Windows and macOS use OS-provided capture facilities; Linux needs ImageMagick `import` or `scrot` and a capturable desktop session (Wayland support depends on the environment).
346
406
  - Tesseract is optional: `vision_ocr` falls back to the vision model when the local engine is absent.
347
407
 
348
408
  ## Install and lifecycle
package/README.zh.md CHANGED
@@ -6,7 +6,7 @@
6
6
 
7
7
  <p align="center"><strong>图片粘贴即用:给 DeepSeek Harness 的纯文本 Agent 装上“眼睛”——开箱免费、免 Key、无 Python、一条命令安装。</strong></p>
8
8
 
9
- <p align="center">DeepSeek 只负责思考,内置免费视觉链 + 11 个像素级工具负责“看”;图片轮次就像普通工具调用一样自然、可定位、可验证。</p>
9
+ <p align="center">DeepSeek 只负责思考,内置免费视觉链 + 14 个深看工具负责“看”;图片轮次就像普通工具调用一样自然、可定位、可验证。</p>
10
10
 
11
11
  <p align="center">
12
12
  <a href="https://awesome-dsh-plugin.com"><img src="https://awesome-dsh-plugin.com/badge.svg" alt="awesome · DSH plugin" /></a>
@@ -15,8 +15,8 @@
15
15
  </p>
16
16
 
17
17
  <p align="center">
18
- <a href="https://github.com/ysr666/dsh-vision-router/releases/tag/v1.4.5"><img src="https://img.shields.io/badge/release-v1.4.5-5B4CF0?style=flat-square" alt="Release v1.4.5" /></a>
19
- <a href="tests"><img src="https://img.shields.io/badge/verified-270%20tests-2EA44F?style=flat-square" alt="Verified: 270 tests" /></a>
18
+ <a href="https://github.com/ysr666/dsh-vision-router/releases/tag/v1.5.1"><img src="https://img.shields.io/badge/release-v1.5.1-5B4CF0?style=flat-square" alt="Release v1.5.1" /></a>
19
+ <a href="tests"><img src="https://img.shields.io/badge/verified-354%20tests-2EA44F?style=flat-square" alt="Verified: 354 tests" /></a>
20
20
  <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-2EA44F?style=flat-square" alt="License: MIT" /></a>
21
21
  <a href="package.json"><img src="https://img.shields.io/badge/Node.js-%3E%3D22-339933?style=flat-square&amp;logo=nodedotjs&amp;logoColor=white" alt="Node.js >=22" /></a>
22
22
  <img src="https://img.shields.io/badge/runtime-no%20Python-8A2BE2?style=flat-square" alt="No Python" />
@@ -28,9 +28,9 @@
28
28
  <p align="center">💬 <strong>QQ 用户交流群:1105463028</strong></p>
29
29
 
30
30
  > [!WARNING]
31
- > 📌 **公告(v1.4.5)**
31
+ > 📌 **公告(v1.5.1)**
32
32
  >
33
- > **v1.4.5:新增可选 1+x 结构化预识别,重做图片识别设置页,并修复 Windows 打开日志文件夹。**
33
+ > **v1.5.1:修复更新兜底、设置循环写入,并新增离线附件落地。**
34
34
 
35
35
  <p align="center">
36
36
  <img src="assets/vision-demo.gif" width="640" alt="演示:粘贴图片,Agent 用 vision_ground / vision_crop / vision_pixel_diff 定位发送按钮并给出坐标" />
@@ -186,10 +186,10 @@ opencode-go + 自动识图 ← 发图片时选这个
186
186
  ## 亮点
187
187
 
188
188
  - **原图像素,真实答案。** 视觉链按原始分辨率读图(仅为保护延迟/额度自动缩放);你的问题随图一起发送,答案围绕*你的问题*,而不是一段泛泛的描述。
189
- - **自动降级 + 分类报错。** 地区限制、ToS 风控、402 额度、429 限流(尊重 Retry-After 退避重试)、上下文超长、网络故障——链路逐供应商尝试,全部失败才报错并给出可操作的建议。
189
+ - **自动降级 + 分类报错。** 地区限制、ToS 风控、402 额度、429 限流、上下文超长、网络故障——链路逐供应商尝试,全部失败才报错并给出可操作的建议。遇到 429 会立即尝试下一后端,并按 Retry-After 开启冷却,不会在单次请求内睡眠等待。
190
190
  - **图片记忆。** 视觉答案按附件内容哈希缓存;后续文字轮用记录的描述替换历史图片(标注为不可信证据),DeepSeek 真正“记得”之前发过的图,且不重复消耗视觉调用。
191
191
  - **可验证的像素闭环。** 参照图 → `vision_html_screenshot` → `vision_pixel_diff`(差异率 + 红色热力图 + 最差区域排行)→ 修复 → 再对比,直到差异收敛。UI 还原从“目测”变成“实测”。
192
- - **稳定工具 schema。** 默认从会话开始就注册完整 11 个深看工具,避免图片轮中途扩展工具列表导致长上下文的 KV / prefix cache 失效。仍保留 `progressiveTools: true` 作为高级启动期 opt-in;开启后才使用 `vision_activate` 按需挂载。详见 [`docs/progressive-tools-cache.md`](docs/progressive-tools-cache.md)。
192
+ - **稳定工具 schema。** 默认从会话开始就注册完整 14 个深看工具,避免图片轮中途扩展工具列表导致长上下文的 KV / prefix cache 失效。仍保留 `progressiveTools: true` 作为高级启动期 opt-in;开启后才使用 `vision_activate` 按需挂载。详见 [`docs/progressive-tools-cache.md`](docs/progressive-tools-cache.md)。
193
193
  - **选择性代理。** 只有配置的视觉供应商域名走本地代理;DeepSeek 保持直连。
194
194
 
195
195
  ### 像素闭环实测
@@ -210,24 +210,30 @@ Agent 仅根据参考图复刻 UI,再用 `vision_pixel_diff` 验证最终结
210
210
 
211
211
  ## 工具
212
212
 
213
- 默认 `progressiveTools: false`:11 个深看工具从插件启动时就保持常驻,文本轮和图片轮都可直接调用。若你在 profile / composition 的 `cordis.patch.yml` 中显式开启 `progressiveTools: true`,才会恢复渐进模式:初始只暴露 `vision_activate`,首次需要时再挂载完整工具,并注册 `vision-tools` 技能。该开关是启动期配置,修改后需重启 DSH。全部工具基于 sharp / potrace / tesseract / 系统 Chrome——无 Python:
213
+ 默认 `progressiveTools: false`:14 个深看工具从插件启动时就保持常驻,文本轮和图片轮都可直接调用。若你在 profile / composition 的 `cordis.patch.yml` 中显式开启 `progressiveTools: true`,才会恢复渐进模式:初始只暴露 `vision_activate`,首次需要时再挂载完整工具,并注册 `vision-tools` 技能。该开关是启动期配置,修改后需重启 DSH。全部工具基于 sharp / potrace / tesseract / 系统 Chrome——无 Python:
214
214
 
215
215
  <p align="center">
216
- <img src="assets/vision-tools-zh.svg" width="100%" alt="DSH Vision Router 的 11 个视觉工具。" />
216
+ <img src="assets/vision-tools-zh.svg" width="100%" alt="DSH Vision Router 的 11 个图像处理工具。" />
217
217
  </p>
218
218
 
219
+ 图中展示 11 个图像处理工具;另有负责持久展示图片的 `vision_present` 与可选 1+x 结构化首遍识别的 `vision_bootstrap`,默认深看工具集共 13 个。若启动时显式开启隐私敏感的 `vision_screenshot`,则额外增加为第 14 个工具。
220
+
219
221
  | 工具 | 作用 | 产物 |
220
222
  |---|---|---|
223
+ | `vision_bootstrap` | 可选 1+x 结构化首遍视觉识别;先建立任务无关证据底图,再至少进行 1 次后续视觉调用 | — |
221
224
  | `vision_describe` | 看图问答 / 多图对比 / 结构化证据 JSON 模式(摘要 + 布局区域 + 实体清单 + 原文转写) | — |
225
+ | `vision_materialize` | 把已授权附件复制到会话工作区并返回真实文件路径,供本地 OCR/解析器降级使用;不调用视觉模型或网络 | image copy |
222
226
  | `vision_ground` | 定位目标 → **原图像素框 x1/y1/x2/y2** | 标注 PNG(可选) |
223
227
  | `vision_detect` | 盘点某类元素(按钮/输入框/链接…)→ 编号清单 + 原图像素框 | 编号标注 PNG |
224
228
  | `vision_crop` | 按像素框裁剪放大 | PNG |
229
+ | `vision_present` | 把生成或编辑后的本地图片发布为持久聊天附件,供用户查看 | 图片附件 |
225
230
  | `vision_pixel_diff` | 逐像素对比:差异率 + 最差 8×8 网格区域 | 红色热力图 PNG + JSON 报告 |
226
231
  | `vision_colors` | 主色提取(十六进制 + 占比) | — |
227
232
  | `vision_ocr` | 文字转写:本地 tesseract(中英)优先,视觉模型兜底 | — |
228
233
  | `vision_trace` | SVG 矢量化(potrace 分色;图标/logo) | SVG |
229
234
  | `vision_extract_foreground` | 边界洪泛抠图(纯色背景) | 透明 PNG |
230
235
  | `vision_html_screenshot` | 给本地 HTML 文件截图(无头系统 Chrome);`fullPage: true` 截整页并返回 `pageHeight` | PNG |
236
+ | `vision_screenshot` | **默认关闭,必须显式开启隐私开关。** 截取 Windows 虚拟屏幕、macOS 主显示器或 Linux 根窗口;Windows 使用 PowerShell CopyFromScreen,macOS 使用 `screencapture`,Linux 需安装 ImageMagick `import` 或 `scrot`;`identify=true` 可按顺序尝试已启用的本地识别后端并返回路径+识别文本 | PNG / +描述文本 |
231
237
  | `vision_long_screenshot_ocr` | 长截图转写:重叠分片,tesseract 优先 / 视觉模型回退,按序拼接 Markdown | 分片 PNG + Markdown + manifest |
232
238
 
233
239
  图片格式按**魔数识别**,无扩展名的内容寻址附件文件也能直接用(不用再复制成 `.png`)。
@@ -238,6 +244,7 @@ Agent 仅根据参考图复刻 UI,再用 `vision_pixel_diff` 验证最终结
238
244
  vision_ground image="ref.png" target="发送按钮"
239
245
  vision_detect image="page.png" target="输入框"
240
246
  vision_crop image="ref.png" region="1067,841,1108,881"
247
+ vision_present path="rebuilt.png"
241
248
  vision_describe paths=["ref.png","impl.png"] question="列出两图的差异" json=true
242
249
  vision_pixel_diff original="ref.png" rebuilt="screenshot.png"
243
250
  vision_ocr image="screenshot.png"
@@ -254,15 +261,17 @@ vision_long_screenshot_ocr image="chat-log.png" chunkHeight=1200 overlap=120
254
261
  视觉工具按顺序逐个尝试,全部失败才报错:
255
262
 
256
263
  1. **用户视觉模型**:设置卡里一行一个,从上到下;只显示 **设置 → 模型** 中明确声明支持 image 输入的模型;
257
- 2. **高级自定义 HTTP 视觉端点**:如果旧配置/高级配置中存在 `httpProviders`,在用户模型之后尝试;
258
- 3. **内置 OVH 匿名免费兜底**:固定最后尝试,不需要出现在任何模型选择器里。当前内置链按质量优先为 `Qwen3.5-397B-A17B` → `Qwen2.5-VL-72B-Instruct` → `Qwen3.6-27B` → `Mistral-Small-3.2-24B-Instruct-2506` → `Qwen3.5-9B`。OVH 匿名限额为 **每 IP、每模型 2 次/分钟**;5 个模型是独立限额,因此理论上分散请求可到约 **10 次/分钟**,实际仍以 OVH 当时的限流为准。免注册、免 Key。想提额度?详见[免费视觉 Key 渠道](#免费视觉-key-渠道)——同一个端点挂免费 access key 后是 400 次/分钟。
264
+ 2. **本地 Ollama(可选,默认关)**:`localOllama.enabled` 开启后,通过本机 Ollama 做免 Key、离线识别(例如 qwen2.5vl);
265
+ 3. **本地 LM Studio(可选,默认关)**:`localLmStudio.enabled` 排在 Ollama 之后,模型名必须填写 LM Studio Developer 页或 `/v1/models` 返回的真实标识;
266
+ 4. **高级自定义 HTTP 视觉端点**:旧配置/高级配置中的 `httpProviders` 排在本地后端之后;
267
+ 5. **内置 OVH 匿名免费兜底**:固定最后尝试,不需要出现在任何模型选择器里。当前内置链按质量优先为 `Qwen3.5-397B-A17B` → `Qwen2.5-VL-72B-Instruct` → `Qwen3.6-27B` → `Mistral-Small-3.2-24B-Instruct-2506` → `Qwen3.5-9B`。OVH 匿名限额为 **每 IP、每模型 2 次/分钟**;5 个模型是独立限额,因此理论上分散请求可到约 **10 次/分钟**,实际仍以 OVH 当时的限流为准。免注册、免 Key。想提额度?详见[免费视觉 Key 渠道](#免费视觉-key-渠道)——同一个端点挂免费 access key 后是 400 次/分钟。
259
268
 
260
269
  > [!IMPORTANT]
261
270
  > 这里的“视觉链”是 Vision Router 调用的**眼睛**:设置页里每一行只选一个用户视觉模型;聊天页右下角选择的是**脑子/会话模型**,两者完全分开。纯文本 DeepSeek / opencode 不会出现在视觉后端下拉里;内部 `Vision HTTP` 也不会再暴露给用户。
262
271
 
263
272
  > 在旧版 `routing: true` 模式下,整轮链只走 `provider + fallbacks`——`httpProviders`(含免费兜底)不参与。默认的 `routing: false`(工具优先)会尝试全部。
264
273
 
265
- 失败会分类(地区 / 风控 / 额度 / 限流 / 上下文 / 网络),最终报错附带建议;`429` 会尊重 `Retry-After` 做一次有上限的退避重试。超大上传图在调用前自动压缩(默认预算 400 万像素),保证工具调用不卡。
274
+ 失败会分类(地区 / 风控 / 额度 / 限流 / 上下文 / 网络),最终报错附带建议;遇到 `429` 会立即尝试下一后端,并按 `Retry-After` 开启有上限的熔断冷却。超大上传图在调用前自动压缩(默认预算 400 万像素),保证工具调用不卡。
266
275
 
267
276
  ## 隐身模式
268
277
 
@@ -301,7 +310,7 @@ Web 配置页在 **设置 → 插件 → 插件配置** 下注册「视觉路由
301
310
  - 开关:整轮自动路由(旧模式)、识图工具、图片块改写、隐身模式(仅官方 DeepSeek 路由);
302
311
  - 视觉请求超时、包装/链路由名、代理等高级参数;
303
312
  - 每个字段都有「已覆盖」徽标与一键恢复组合默认,以及放弃/保存;
304
- - 「测试连接」按钮探测第一个视觉提供方并内联显示延迟/失败原因;
313
+ - 「测试连接」按钮优先探测已启用的本地后端,并校验所填模型是否出现在 `/v1/models`;否则探测第一个可用视觉提供方;
305
314
  - 产出制品的工具在对话里渲染专用调用卡(关键字段 + 打开文件按钮)。
306
315
 
307
316
  <p align="center">
@@ -329,6 +338,12 @@ Web 配置页在 **设置 → 插件 → 插件配置** 下注册「视觉路由
329
338
  | `textProvider` | `deepseek-official` / `deepseek-v4-pro` | 负责思考的模型(你的日常模型) |
330
339
  | `tool` / `progressiveTools` / `autoActivateOnImage` | `true` / `false` / `true` | 视觉工具总开关 / 渐进式挂载(默认关闭以稳定工具 schema)/ 渐进模式下图片轮自动挂载;`progressiveTools` 为启动期配置 |
331
340
  | `rewriteImages` | `true` | 模型输入层改写图片块(缓存描述或工具提示标记);界面日志保留图片 |
341
+ | `desktopScreenshot` | `false` | 模型可调用的 `vision_screenshot` 桌面截屏隐私开关;每次截屏前实时检查 |
342
+ | `freeFallback` | `true` | 在显式本地/自定义 HTTP 后端之后追加匿名 OVH 模型;关闭它不会停用用户明确配置的本地后端 |
343
+ | `localOllama` | `{ enabled: false, baseURL: 'http://127.0.0.1:11434/v1', model: 'qwen2.5vl', format: 'openai' }` | **本地视觉后端(并入自 dsh-vision)**:开启后 local-ollama 排在 HTTP 视觉链最前;Ollama 未运行会自动跳过;`format` 可选 `openai`(`/chat/completions`)或 `anthropic`(`/messages`);可选的 `temperature` / `top_p` 只在显式填写时发送,留空尊重本地服务默认值 |
344
+ | `localLmStudio` | `{ enabled: false, baseURL: 'http://localhost:1234/v1', model: '', format: 'openai' }` | **本地 LM Studio 后端(并入自 dsh-vision)**:排在 Ollama 之后、自定义/云 HTTP 后端之前;开启时必须填写 LM Studio Developer 页或 `/v1/models` 返回的真实模型标识;可选采样参数同 Ollama,`format: 'anthropic'` 需 LM Studio 0.4.1+ |
345
+ | `instantDescribe` | `false` | **即时本地翻译(并入自 dsh-vision)**:开启且至少一个本地后端可用时,在第一模型步之前识别无缓存图片块;Ollama → LM Studio 共用总超时预算,多图并发上限 3,失败则回退静态工具标记 |
346
+ | `localDescribeStyle` | `plain` | **本地识别输出风格(并入自 dsh-vision)**:`plain` = 平铺描述;`structured` = 结构化识别(【初步判断】/【细节】/【空间结构】/【原图尺寸】),截图分析质量更高 |
332
347
  | `downscale` / `downscaleMaxPixels` | `true` / `4000000` | 调用前压缩及其像素预算(延迟保护) |
333
348
  | `cache` / `cacheTtlSeconds` / `cacheMaxEntries` | `true` / `3600` / `200` | 视觉答案缓存 |
334
349
  | `timeoutMs` | `120000` | 单次视觉调用超时 |
@@ -336,12 +351,53 @@ Web 配置页在 **设置 → 插件 → 插件配置** 下注册「视觉路由
336
351
  | `proxy` / `proxyHosts` | `''` / openrouter 域名 | 仅视觉供应商域名可选的本地代理 |
337
352
  | `catalogCorrections` | `true` | 内置目录纠错:当已安装的 pi-ai 目录把已知模型路由到错误协议时(例如 `opencode-go/qwen3.6-plus` 被指向 OpenAI chat completions,而 OpenCode Go 只在 `/v1/messages` 上提供该模型),插件直接按正确协议应答该后端;上游目录修复后每条纠错自动失效 |
338
353
 
354
+ ### 本地 Ollama 视觉后端(并入自 dsh-vision)
355
+
356
+ > **增量开发作者**:[shaoqiuyuavailable](https://github.com/shaoqiuyuavailable)(router 本地视觉增量)
357
+ >
358
+ > **思路来源**:本地视觉后端(Ollama / LM Studio 双后端、即时识别、结构化输出、截屏识别、同图去重记忆、失败降级占位、并发防雪崩、超时防护)的思路继承自 [dsh-vision](https://github.com/shaoqiuyuavailable/text-llm-vision/tree/dsh-vision)——本项目将其并入 HTTP 视觉链,并在此基础上扩展了逐级降级链与双协议支持。
359
+
360
+ 可选的本地优先视觉路径:不需要 Key,支持隐私、零费用、离线识别。它作为 HTTP 视觉链里的 `local-ollama` 接入;若本地识别失败,除非用户明确配置纯本地链,否则仍可能继续尝试已配置的云后端。
361
+
362
+ **1. 安装 Ollama 并拉取视觉模型**
363
+
364
+ ```sh
365
+ # https://ollama.com —— 然后:
366
+ ollama pull qwen2.5vl
367
+ ```
368
+
369
+ **2. 开启** —— 设置卡片「本地视觉」组,或 profile patch:
370
+
371
+ ```yaml
372
+ - id: vision-router
373
+ config:
374
+ localOllama:
375
+ enabled: true
376
+ baseURL: 'http://127.0.0.1:11434/v1' # OpenAI 兼容端点
377
+ model: 'qwen2.5vl'
378
+ temperature: 0.5 # 可选;识别用低温更稳
379
+ top_p: 0.8 # 可选;留空 = 服务端默认
380
+ instantDescribe: true # 图片轮第一轮即本地识别
381
+ localDescribeStyle: 'structured' # 'plain' | 'structured'
382
+ ```
383
+
384
+ **3. 行为说明**
385
+
386
+ - 开启后 `local-ollama` 排在 HTTP 视觉链最前。若要严格纯本地,请移除云视觉行/自定义 HTTP 端点,并关闭 `freeFallback`。
387
+ - **LM Studio 同理**——同一「本地视觉」组里开启 `localLmStudio`,填 OpenAI 兼容端点(默认 `http://localhost:1234/v1`),并使用 Developer 页或 `/v1/models` 返回的真实模型标识。它排在 `local-ollama` 之后、自定义/云 HTTP 后端之前。
388
+ - 每个本地后端可通过 `format` 选择 **OpenAI 或 Anthropic 格式**(默认 `openai`)。Anthropic 模式走 `/v1/messages`,带 `anthropic-version` 并把图片转为 base64 source;只有配置了 Key 才发送 `x-api-key`。LM Studio 需 0.4.1 或更高版本才提供该端点。
389
+ - 任一本地后端未运行或调用超时时自动跳过,继续降级到云链——任何调用都不受影响。
390
+ - `instantDescribe` 会在第一模型步之前按 Ollama → LM Studio 的顺序尝试已启用本地后端。多张无缓存图片并发识别(上限 3),单张失败不影响其余;命中附件记忆的图片不会再次请求本地服务。
391
+ - `vision_screenshot` 默认关闭。单独开启「桌面截屏」隐私开关后,`identify=true` 使用同样的 Ollama → LM Studio 降级顺序。
392
+ - 日志中的 `image turn — instantDescribe=… localBackends=…` 显示实时决策;`instant local describe recognized N/M uncached image(s), C cached, F failed attempts` 显示本轮结果。
393
+
339
394
  ## 环境要求
340
395
 
341
396
  - DeepSeek Harness 的 Web profile。普通安装可用 `npx @deepseek-ai/dsh ...`;从源码仓库运行时用 `pnpm dsh ...`。只有 CLI 已经进入系统 `PATH` 时才能直接写 `dsh ...`。
342
397
  - Node ≥ 22(宿主侧)。
343
398
  - 默认免费链路无需 API Key;付费 `httpProviders` 只需一个凭据引用(`apiKeyEnv`)。
344
- - `vision_html_screenshot` 才需要 Chrome / Chromium / Edge;其余工具无浏览器也能用。
399
+ - 只有 `vision_html_screenshot` 需要 Chrome / Chromium / Edge;其余工具无浏览器也能用。
400
+ - 桌面截屏必须显式开启。Windows/macOS 使用系统截屏能力;Linux 需安装 ImageMagick `import` 或 `scrot`,且必须处于可截取的桌面会话(Wayland 支持取决于环境)。
345
401
  - tesseract 可选:本地引擎缺失时 `vision_ocr` 自动退回视觉模型。
346
402
 
347
403
  ## 安装与生命周期
package/entry.js CHANGED
@@ -10,6 +10,7 @@ import z from '@deepseek-ai/schemastery'
10
10
  import * as core from './index.js'
11
11
  import { installVisionRouterFileLogging } from './lib/file-logger.js'
12
12
  import { contextWithDelegatedReplay } from './lib/replay-delegation.js'
13
+ import { installLocalVisionStabilizer } from './lib/local-vision-stabilizer.js'
13
14
 
14
15
  // Schemastery object schemas expose set() as the supported way to replace a
15
16
  // field schema. This mutates the Config object that index.js itself later uses
@@ -28,9 +29,32 @@ export const Config = core.Config
28
29
  export function apply(ctx, config = {}) {
29
30
  const logging = installVisionRouterFileLogging(ctx)
30
31
  const runtimeCtx = contextWithDelegatedReplay(logging.ctx)
32
+ // #141 stabilization boundary: keep the recently merged local-vision
33
+ // behavior isolated from main's existing provider/router semantics. It
34
+ // normalizes only the local settings/runtime seams before core.apply sees
35
+ // the context (desktop screenshot exposure, instant-local budget/one-pass,
36
+ // local vision-http transport and connection-probe fallback).
37
+ const { ctx: stabilizedCtx, bootConfig } = installLocalVisionStabilizer(runtimeCtx, config, core)
38
+ // 启动诊断摘要只描述 composition/apply 的基础配置。设置服务可能稍后
39
+ // 覆盖这些值;每个图片轮还会记录 current() 的实时决策,避免把这个
40
+ // 启动快照误当成最终设置状态。
31
41
  try {
32
- const result = core.apply(runtimeCtx, {
33
- ...config,
42
+ const c = config && typeof config === 'object' ? config : {}
43
+ const local = c.localOllama && typeof c.localOllama === 'object' ? c.localOllama : {}
44
+ const lms = c.localLmStudio && typeof c.localLmStudio === 'object' ? c.localLmStudio : {}
45
+ logging.logger.info(
46
+ 'vision-router: base config summary — instantDescribe=%s localDescribeStyle=%s localOllama=%s localLmStudio=%s',
47
+ c.instantDescribe === true ? 'on' : 'off',
48
+ c.localDescribeStyle === 'structured' ? 'structured' : 'plain',
49
+ local.enabled === true ? 'on' : 'off',
50
+ lms.enabled === true ? 'on' : 'off',
51
+ )
52
+ } catch {
53
+ /* diagnostics must never break apply */
54
+ }
55
+ try {
56
+ const result = core.apply(stabilizedCtx, {
57
+ ...bootConfig,
34
58
  progressiveTools: config.progressiveTools === true,
35
59
  })
36
60
  if (result && typeof result.then === 'function') {