dsh-vision-router 1.4.4 → 1.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +91 -17
- package/README.zh.md +87 -15
- package/entry.js +26 -2
- package/index.js +1253 -76
- package/lib/catalog-corrections.js +11 -3
- package/lib/client.js +577 -188
- package/lib/file-logger.js +100 -15
- package/lib/local-vision-stabilizer.js +571 -0
- package/lib/structured-bootstrap.js +94 -0
- package/package.json +3 -3
package/README.md
CHANGED
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
|
|
7
7
|
<p align="center"><strong>Paste an image and it just works — eyes for text-only agents on DeepSeek Harness. Free out of the box, no key, no Python, one command.</strong></p>
|
|
8
8
|
|
|
9
|
-
<p align="center">DeepSeek keeps thinking; the built-in free vision chain and
|
|
9
|
+
<p align="center">DeepSeek keeps thinking; the built-in free vision chain and thirteen deep tools do the seeing. Image turns behave like ordinary tool-calling turns — grounded, measurable, repeatable.</p>
|
|
10
10
|
|
|
11
11
|
<p align="center">
|
|
12
12
|
<a href="https://awesome-dsh-plugin.com"><img src="https://awesome-dsh-plugin.com/badge.svg" alt="awesome · DSH plugin" /></a>
|
|
@@ -15,8 +15,8 @@
|
|
|
15
15
|
</p>
|
|
16
16
|
|
|
17
17
|
<p align="center">
|
|
18
|
-
<a href="https://github.com/ysr666/dsh-vision-router/releases/tag/v1.
|
|
19
|
-
<a href="tests"><img src="https://img.shields.io/badge/verified-
|
|
18
|
+
<a href="https://github.com/ysr666/dsh-vision-router/releases/tag/v1.5.0"><img src="https://img.shields.io/badge/release-v1.5.0-5B4CF0?style=flat-square" alt="Release v1.5.0" /></a>
|
|
19
|
+
<a href="tests"><img src="https://img.shields.io/badge/verified-348%20tests-2EA44F?style=flat-square" alt="Verified: 348 tests" /></a>
|
|
20
20
|
<a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-2EA44F?style=flat-square" alt="License: MIT" /></a>
|
|
21
21
|
<a href="package.json"><img src="https://img.shields.io/badge/Node.js-%3E%3D22-339933?style=flat-square&logo=nodedotjs&logoColor=white" alt="Node.js >=22" /></a>
|
|
22
22
|
<img src="https://img.shields.io/badge/runtime-no%20Python-8A2BE2?style=flat-square" alt="No Python" />
|
|
@@ -28,9 +28,9 @@
|
|
|
28
28
|
<p align="center">💬 <strong>QQ community group: 1105463028</strong></p>
|
|
29
29
|
|
|
30
30
|
> [!WARNING]
|
|
31
|
-
> 📌 **Announcement (v1.
|
|
31
|
+
> 📌 **Announcement (v1.5.0)**
|
|
32
32
|
>
|
|
33
|
-
> **v1.
|
|
33
|
+
> **v1.5.0:** Adds local Ollama / LM Studio vision and desktop screenshots, with stronger Settings and lifecycle reliability.
|
|
34
34
|
|
|
35
35
|
<p align="center">
|
|
36
36
|
<img src="assets/vision-demo.gif" width="640" alt="Demo: paste an image, the agent locates the send button with vision_ground / vision_crop / vision_pixel_diff and answers with coordinates" />
|
|
@@ -40,6 +40,7 @@
|
|
|
40
40
|
|
|
41
41
|
- [Why this exists](#why-this-exists)
|
|
42
42
|
- [How it compares](#how-it-compares)
|
|
43
|
+
- [Design lineage](#design-lineage)
|
|
43
44
|
- [Acknowledgements](#acknowledgements)
|
|
44
45
|
- [Quick start](#quick-start)
|
|
45
46
|
- [Free vision key channels](#free-vision-key-channels)
|
|
@@ -88,6 +89,14 @@ Most DSH vision plugins bridge images to DeepSeek as *text descriptions* — los
|
|
|
88
89
|
| [dsh-vision-toolkit](https://github.com/Anionex/dsh-vision-toolkit) | Ten intent-aware visual tools (Q&A/OCR/pixel verification/UI restoration), called explicitly on demand | Broader tool set; this plugin adds whole-turn auto-routing and a keyless free fallback |
|
|
89
90
|
| [dsh-tool-vision](https://github.com/Scorp1o117/dsh-tool-vision) | An `inspect_image` tool plus an `agent/pre-step` waterfall bridge (pasted images become tool hints before entering the log) | Similar waterfall bridge; this plugin adds turn routing, fallback chains, caching and the free endpoint |
|
|
90
91
|
|
|
92
|
+
## Design lineage
|
|
93
|
+
|
|
94
|
+
The deep-vision tool layer and UI-restoration workflow in this project were informed by [Anionex/agent-vision-toolkit](https://github.com/Anionex/agent-vision-toolkit) and its native DSH implementation [Anionex/dsh-vision-toolkit](https://github.com/Anionex/dsh-vision-toolkit). In particular, this project drew on their intent-driven tool selection, progressive tool exposure, pixel-diff verification loop, and parts of the visual-tool decomposition and naming, including long-screenshot OCR, foreground extraction, and HTML screenshot tooling.
|
|
95
|
+
|
|
96
|
+
All code in `dsh-vision-router` is independently implemented. On top of those referenced design ideas, this project independently developed its turn-level/tools-first vision routing, DSH admission/wrapper integration, multi-backend provider and failure fallback chains, built-in free vision chain, attachment/image-memory handling, caching, and related runtime resilience mechanisms.
|
|
97
|
+
|
|
98
|
+
We appreciate Anionex's prior work and the broader DSH community. Clear attribution and independent iteration can coexist; both help keep the DSH ecosystem open, collaborative, and healthy.
|
|
99
|
+
|
|
91
100
|
## Acknowledgements
|
|
92
101
|
|
|
93
102
|
This project borrows ideas from all of the above — especially the keyless free-endpoint
|
|
@@ -127,7 +136,6 @@ If you already installed the DSH CLI globally and `dsh` is on `PATH`, the shorte
|
|
|
127
136
|
### 2. Switch to a “+ Auto Vision” model group in chat
|
|
128
137
|
|
|
129
138
|
Once loaded, the plugin discovers the model groups enabled under **Settings → Models** and creates same-name auto-vision entries. For example:
|
|
130
|
-
|
|
131
139
|
```text
|
|
132
140
|
opencode-go ← original model group, unchanged
|
|
133
141
|
opencode-go + Auto Vision ← choose this when sending images
|
|
@@ -179,10 +187,10 @@ Any of these channels can join the vision chain as an `httpProviders` entry (key
|
|
|
179
187
|
## Highlights
|
|
180
188
|
|
|
181
189
|
- **Original pixels, real answers.** The vision chain reads the image at original resolution (auto-downscaled only to protect latency/quota); the agent's question travels with the image, so answers are about *your* question, not a generic description.
|
|
182
|
-
- **Automatic failover with classified errors.** Region blocks, ToS filtering, 402 quota, 429 rate limits
|
|
190
|
+
- **Automatic failover with classified errors.** Region blocks, ToS filtering, 402 quota, 429 rate limits, context overflow, network failures — the chain walks providers one by one and only reports after all of them failed, with actionable advice. A 429 immediately advances to the next backend and opens a Retry-After-aware cooldown instead of sleeping inside the request.
|
|
183
191
|
- **Image memory.** Vision answers are cached by attachment content hash; later text turns substitute the recorded description (marked as untrusted evidence), so DeepSeek genuinely remembers earlier images without re-spending vision calls.
|
|
184
192
|
- **A verifiable pixel loop.** Reference → `vision_html_screenshot` → `vision_pixel_diff` (ratio + red heatmap + worst-region ranking) → fix → repeat until the mismatch converges. UI restoration becomes measurable instead of eyeballed.
|
|
185
|
-
- **Stable tool schema.** All
|
|
193
|
+
- **Stable tool schema.** All thirteen deep tools are registered from session start by default, avoiding a mid-conversation tool-list expansion that can invalidate long-context KV/prefix caches. `progressiveTools: true` remains an advanced boot-time opt-in; only then does `vision_activate` mount the tools on demand. See [`docs/progressive-tools-cache.md`](docs/progressive-tools-cache.md).
|
|
186
194
|
- **Selective proxy.** Only the configured vision provider hosts go through your local proxy; DeepSeek stays direct.
|
|
187
195
|
|
|
188
196
|
### Pixel loop in practice
|
|
@@ -203,24 +211,29 @@ The vision model is **only the eyes**; DeepSeek is **always the brain**. An imag
|
|
|
203
211
|
|
|
204
212
|
## Tools
|
|
205
213
|
|
|
206
|
-
Default `progressiveTools: false`: all
|
|
214
|
+
Default `progressiveTools: false`: all thirteen deep tools stay registered from plugin startup, so text and image turns can call them immediately. If you explicitly set `progressiveTools: true` in the profile/composition `cordis.patch.yml`, progressive mode is restored: only `vision_activate` is exposed initially, the full tool set mounts on first use, and the `vision-tools` skill is registered. This is a boot-time switch; restart DSH after changing it. Built on sharp / potrace / tesseract / system Chrome — no Python:
|
|
207
215
|
|
|
208
216
|
<p align="center">
|
|
209
|
-
<img src="assets/vision-tools.svg" width="100%" alt="Eleven
|
|
217
|
+
<img src="assets/vision-tools.svg" width="100%" alt="Eleven image-processing tools available in DSH Vision Router." />
|
|
210
218
|
</p>
|
|
211
219
|
|
|
220
|
+
The diagram covers the eleven image-processing tools. `vision_present` (durable image delivery) and `vision_bootstrap` (the optional 1+x structured first pass) bring the default deep-tool set to thirteen. Enabling the privacy-gated `vision_screenshot` at boot adds an optional fourteenth tool.
|
|
221
|
+
|
|
212
222
|
| Tool | What it does | Artifact |
|
|
213
223
|
|---|---|---|
|
|
224
|
+
| `vision_bootstrap` | Optional 1+x structured first visual pass; establishes task-independent evidence before at least one follow-up vision call | — |
|
|
214
225
|
| `vision_describe` | Image Q&A / multi-image compare / structured-evidence JSON mode (summary + layout regions + entity inventory + verbatim transcription) | — |
|
|
215
226
|
| `vision_ground` | Locate a target → **original-pixel box x1/y1/x2/y2** | annotated PNG (optional) |
|
|
216
227
|
| `vision_detect` | Numbered inventory of every element of a kind (buttons/inputs/links…) with original-pixel boxes | annotated PNG with numbered boxes |
|
|
217
228
|
| `vision_crop` | Crop and zoom into a pixel box | PNG |
|
|
229
|
+
| `vision_present` | Publish a generated or edited local image as a durable chat attachment so the user can see it | image attachment |
|
|
218
230
|
| `vision_pixel_diff` | Per-pixel comparison: diff ratio + worst 8×8-grid regions | red heatmap PNG + JSON report |
|
|
219
231
|
| `vision_colors` | Dominant colors (hex + share) | — |
|
|
220
232
|
| `vision_ocr` | Text transcription: local tesseract (chi_sim+eng) first, vision model fallback | — |
|
|
221
233
|
| `vision_trace` | SVG vectorization (potrace posterization; icons/logos) | SVG |
|
|
222
234
|
| `vision_extract_foreground` | Cutout via border flood fill (uniform backgrounds) | transparent PNG |
|
|
223
235
|
| `vision_html_screenshot` | Screenshot a local HTML file (headless system Chrome); `fullPage: true` captures the whole page and reports `pageHeight` | PNG |
|
|
236
|
+
| `vision_screenshot` | **Disabled by default; explicit opt-in required.** Capture the Windows virtual screen, macOS main display, or Linux root display. Windows uses PowerShell CopyFromScreen, macOS uses `screencapture`, and Linux requires ImageMagick `import` or `scrot`; optional `identify=true` tries enabled local recognition backends in order and returns the description with the path | PNG / + description |
|
|
224
237
|
| `vision_long_screenshot_ocr` | Long-screenshot transcription: overlapping chunks, tesseract first / vision model fallback, stitched Markdown | chunk PNGs + Markdown + manifest |
|
|
225
238
|
|
|
226
239
|
Formats are sniffed from magic bytes, so extensionless content-addressed attachment files work everywhere (no `.png` renaming needed).
|
|
@@ -231,6 +244,7 @@ Formats are sniffed from magic bytes, so extensionless content-addressed attachm
|
|
|
231
244
|
vision_ground image="ref.png" target="the send button"
|
|
232
245
|
vision_detect image="page.png" target="input fields"
|
|
233
246
|
vision_crop image="ref.png" region="1067,841,1108,881"
|
|
247
|
+
vision_present path="rebuilt.png"
|
|
234
248
|
vision_describe paths=["ref.png","impl.png"] question="list the differences" json=true
|
|
235
249
|
vision_pixel_diff original="ref.png" rebuilt="screenshot.png"
|
|
236
250
|
vision_ocr image="screenshot.png"
|
|
@@ -247,15 +261,17 @@ vision_long_screenshot_ocr image="chat-log.png" chunkHeight=1200 overlap=120
|
|
|
247
261
|
The vision tools try backends in order and surface an error only after all of them fail:
|
|
248
262
|
|
|
249
263
|
1. **User vision models**: one per settings row, top to bottom; only models under **Settings → Models** that explicitly declare image input are shown;
|
|
250
|
-
2. **
|
|
251
|
-
3. **
|
|
264
|
+
2. **Local Ollama (optional, off by default)**: `localOllama.enabled` adds keyless, offline recognition through your local Ollama (for example qwen2.5vl);
|
|
265
|
+
3. **Local LM Studio (optional, off by default)**: `localLmStudio.enabled` follows Ollama and requires the real model identifier shown in LM Studio Developer or returned by `/v1/models`;
|
|
266
|
+
4. **Advanced custom HTTP vision endpoints**: legacy/advanced `httpProviders`, when present, run after the local backends;
|
|
267
|
+
5. **Built-in anonymous OVH fallback**: always last and never exposed in a model picker. The current quality-first chain is `Qwen3.5-397B-A17B` → `Qwen2.5-VL-72B-Instruct` → `Qwen3.6-27B` → `Mistral-Small-3.2-24B-Instruct-2506` → `Qwen3.5-9B`. OVH anonymous limits are **2 requests/minute per IP per model**. The five models have independent buckets, so spreading requests across them is about **10 RPM in theory**, subject to OVH's actual rate limiting. No signup or API key is required. Want more headroom? See [Free vision key channels](#free-vision-key-channels) — a free OVH access key lifts this same endpoint to 400 requests/minute.
|
|
252
268
|
|
|
253
269
|
> [!IMPORTANT]
|
|
254
270
|
> This “vision chain” is the **eyes** used by Vision Router: each settings row selects one user vision model, while the lower-right chat picker selects the **brain/conversation model**. The two are deliberately separate. Text-only DeepSeek/opencode models are filtered out of the vision-backend dropdown, and the internal `Vision HTTP` transport route is no longer exposed to users.
|
|
255
271
|
|
|
256
272
|
> In the legacy `routing: true` mode, the whole-turn chain walks only `provider + fallbacks` — `httpProviders` (including the free fallback) do not participate there. The default `routing: false` (tools-first) tries everything.
|
|
257
273
|
|
|
258
|
-
Failures are classified (region / tos / quota / rate-limit / context / network) and the final error carries advice; `429`
|
|
274
|
+
Failures are classified (region / tos / quota / rate-limit / context / network) and the final error carries advice; a `429` advances immediately to the next backend and applies a capped, `Retry-After`-aware circuit-breaker cooldown. Oversized uploads are downscaled before the call (default budget 4 MP) to keep tool calls fast.
|
|
259
275
|
|
|
260
276
|
## Stealth mode
|
|
261
277
|
|
|
@@ -294,7 +310,7 @@ The Web profile registers a **视觉路由(自动识图)** card under **Sett
|
|
|
294
310
|
- switches for legacy whole-turn routing, vision tools, image-block rewriting and stealth mode (official DeepSeek route only);
|
|
295
311
|
- timeout, wrapper/chain route names, proxy and other advanced parameters;
|
|
296
312
|
- every field shows an overridden badge with one-click reset plus discard/save;
|
|
297
|
-
- a **Test connection** button probes the first vision provider
|
|
313
|
+
- a **Test connection** button prioritizes an enabled local backend, verifies that its configured model appears in `/v1/models`, and otherwise probes the first usable vision provider;
|
|
298
314
|
- artifact-producing tools render dedicated call cards with result facts and open-file buttons.
|
|
299
315
|
|
|
300
316
|
<p align="center">
|
|
@@ -322,6 +338,12 @@ Everything is optional; defaults work out of the box. Edit via the Web card or a
|
|
|
322
338
|
| `textProvider` | `deepseek-official` / `deepseek-v4-pro` | the model that reasons (your daily model) |
|
|
323
339
|
| `tool` / `progressiveTools` / `autoActivateOnImage` | `true` / `false` / `true` | vision tools on / progressive mounting (off by default for a stable tool schema) / image-turn auto-mount when progressive mode is enabled; `progressiveTools` is boot-time config |
|
|
324
340
|
| `rewriteImages` | `true` | rewrite image blocks in the model input (cached description or tool-hint marker); the UI log keeps images |
|
|
341
|
+
| `desktopScreenshot` | `false` | privacy opt-in for the model-callable `vision_screenshot` desktop-capture tool; checked live before every capture |
|
|
342
|
+
| `freeFallback` | `true` | append the anonymous OVH models after explicit local/custom HTTP backends; turning this off never disables an explicitly configured local backend |
|
|
343
|
+
| `localOllama` | `{ enabled: false, baseURL: 'http://127.0.0.1:11434/v1', model: 'qwen2.5vl', format: 'openai' }` | **Local vision backend (merged from dsh-vision)**: when enabled, `local-ollama` leads the HTTP vision chain; skipped automatically when Ollama is down; `format` selects `openai` (`/chat/completions`) or `anthropic` (`/messages`); optional `temperature` / `top_p` are sent only when explicitly set, otherwise the local server default is respected |
|
|
344
|
+
| `localLmStudio` | `{ enabled: false, baseURL: 'http://localhost:1234/v1', model: '', format: 'openai' }` | **Local LM Studio backend (merged from dsh-vision)**: follows Ollama and precedes custom/cloud HTTP backends; enabling it requires the real model identifier shown in LM Studio Developer or returned by `/v1/models`; supports the same optional sampling fields, while `format: 'anthropic'` requires LM Studio 0.4.1+ |
|
|
345
|
+
| `instantDescribe` | `false` | **Instant local translation (merged from dsh-vision)**: when on and at least one local backend is usable, uncached image blocks are recognized before the first model step; Ollama is tried before LM Studio with a shared timeout budget, multi-image batches run concurrently (up to 3), and failures fall back to the static tool-hint marker |
|
|
346
|
+
| `localDescribeStyle` | `plain` | **Local recognition output style (merged from dsh-vision)**: `plain` = flat description; `structured` = structured recognition (【初步判断】/【细节】/【空间结构】/【原图尺寸】), better for screenshot analysis |
|
|
325
347
|
| `downscale` / `downscaleMaxPixels` | `true` / `4000000` | pre-call downscale and its pixel budget (latency guard) |
|
|
326
348
|
| `cache` / `cacheTtlSeconds` / `cacheMaxEntries` | `true` / `3600` / `200` | vision answer cache |
|
|
327
349
|
| `timeoutMs` | `120000` | per vision call deadline |
|
|
@@ -329,12 +351,57 @@ Everything is optional; defaults work out of the box. Edit via the Web card or a
|
|
|
329
351
|
| `proxy` / `proxyHosts` | `''` / openrouter hosts | optional proxy for vision provider hosts only |
|
|
330
352
|
| `catalogCorrections` | `true` | built-in catalog-routing corrections: when the installed pi-ai catalog routes a known model to the wrong wire protocol (e.g. `opencode-go/qwen3.6-plus` to OpenAI chat completions while OpenCode Go only serves it on `/v1/messages`), the plugin answers that backend directly over the corrected protocol. Each correction disarms itself once the catalog is fixed upstream |
|
|
331
353
|
|
|
354
|
+
### Local Ollama vision backend (merged from dsh-vision)
|
|
355
|
+
|
|
356
|
+
> **Incremental author**: [shaoqiuyuavailable](https://github.com/shaoqiuyuavailable) (router local-vision increment)
|
|
357
|
+
>
|
|
358
|
+
> **Design credit**: the local vision backends (Ollama / LM Studio dual backends, instant recognition,
|
|
359
|
+
> structured output, screenshot identification, same-image memory dedup, failure-fallback placeholder,
|
|
360
|
+
> concurrency anti-snowball, timeout protection) inherit their design from
|
|
361
|
+
> [dsh-vision](https://github.com/shaoqiuyuavailable/text-llm-vision/tree/dsh-vision) —
|
|
362
|
+
> merged into the HTTP vision chain here, with per-level fallback and dual-protocol support added on top.
|
|
363
|
+
|
|
364
|
+
An optional keyless local-first vision path for private, free, offline recognition. It plugs into the existing HTTP vision chain as `local-ollama`; if it fails, any configured cloud backends can still be tried unless you deliberately configure a local-only chain.
|
|
365
|
+
|
|
366
|
+
**1. Install Ollama and pull a vision model**
|
|
367
|
+
|
|
368
|
+
```sh
|
|
369
|
+
# https://ollama.com — then:
|
|
370
|
+
ollama pull qwen2.5vl
|
|
371
|
+
```
|
|
372
|
+
|
|
373
|
+
**2. Enable it** — in the settings card's "Local vision" group, or via a profile patch:
|
|
374
|
+
|
|
375
|
+
```yaml
|
|
376
|
+
- id: vision-router
|
|
377
|
+
config:
|
|
378
|
+
localOllama:
|
|
379
|
+
enabled: true
|
|
380
|
+
baseURL: 'http://127.0.0.1:11434/v1' # OpenAI-compatible endpoint
|
|
381
|
+
model: 'qwen2.5vl'
|
|
382
|
+
temperature: 0.5 # optional; low temperature is steadier for recognition
|
|
383
|
+
top_p: 0.8 # optional; unset = server default
|
|
384
|
+
instantDescribe: true # recognize images on the first model step
|
|
385
|
+
localDescribeStyle: 'structured' # 'plain' | 'structured'
|
|
386
|
+
```
|
|
387
|
+
|
|
388
|
+
**3. What happens**
|
|
389
|
+
|
|
390
|
+
- When enabled, `local-ollama` heads the HTTP vision chain. For a strict local-only setup, remove cloud vision rows/custom HTTP endpoints and turn off `freeFallback`.
|
|
391
|
+
- **LM Studio works the same way** — enable `localLmStudio` in the same "Local vision" group with its OpenAI-compatible endpoint (default `http://localhost:1234/v1`) and enter the exact model identifier shown in Developer or `/v1/models`. It sits after `local-ollama` and before custom/cloud HTTP backends.
|
|
392
|
+
- Each local backend can speak **OpenAI or Anthropic format** via `format` (default `openai`). Anthropic mode routes to `/v1/messages` with `anthropic-version` and base64 image sources; `x-api-key` is sent only when a key is configured. LM Studio needs version 0.4.1 or newer for this endpoint.
|
|
393
|
+
- If a local backend is down or the call times out, its entry is skipped automatically and the chain falls through to the cloud backends — no call breaks.
|
|
394
|
+
- `instantDescribe` tries enabled local backends in order (Ollama, then LM Studio) before the first model step. Multiple uncached images run concurrently (up to 3); one failed image does not block the others, and attachment-memory hits are reused without another local request.
|
|
395
|
+
- `vision_screenshot` is disabled by default. After the separate Desktop screenshot opt-in, `identify=true` uses the same Ollama → LM Studio fallback.
|
|
396
|
+
- Verify runtime decisions with `image turn — instantDescribe=… localBackends=…` and results with `instant local describe recognized N/M uncached image(s), C cached, F failed attempts` in the log.
|
|
397
|
+
|
|
332
398
|
## Requirements
|
|
333
399
|
|
|
334
400
|
- DeepSeek Harness Web profile. Normal installs can use `npx @deepseek-ai/dsh ...`; source checkouts use `pnpm dsh ...`. A bare `dsh ...` command only works when the CLI is already on your shell `PATH`.
|
|
335
401
|
- Node ≥ 22 (host side).
|
|
336
402
|
- No API key for the default free chain; a credential reference (`apiKeyEnv`) only for paid `httpProviders`.
|
|
337
|
-
- Chrome / Chromium / Edge only for `vision_html_screenshot`; every other tool works without a browser.
|
|
403
|
+
- Chrome / Chromium / Edge is needed only for `vision_html_screenshot`; every other tool works without a browser.
|
|
404
|
+
- Desktop capture is opt-in. Windows and macOS use OS-provided capture facilities; Linux needs ImageMagick `import` or `scrot` and a capturable desktop session (Wayland support depends on the environment).
|
|
338
405
|
- Tesseract is optional: `vision_ocr` falls back to the vision model when the local engine is absent.
|
|
339
406
|
|
|
340
407
|
## Install and lifecycle
|
|
@@ -387,7 +454,6 @@ Oh-DSH Desktop's built-in plugin marketplace (search → prepare → isolated pr
|
|
|
387
454
|
- id: vision-router
|
|
388
455
|
disabled: true
|
|
389
456
|
```
|
|
390
|
-
|
|
391
457
|
Set it back to `false` to re-enable. Unloading removes the wrapper routes, tools, skill and settings card; cached artifact files remain.
|
|
392
458
|
|
|
393
459
|
### Upgrade
|
|
@@ -449,6 +515,14 @@ This removes the dependency and the bundle layer. If you disabled the stock Deep
|
|
|
449
515
|
|
|
450
516
|
## Troubleshooting
|
|
451
517
|
|
|
518
|
+
### Using dsh-web-ui / dsh-web-ui-all together
|
|
519
|
+
|
|
520
|
+
If `dsh-web-ui` / `@linxin666/dsh-web-ui-all` is installed alongside Vision Router, its `dsh-tool-describe-image` send hook can rewrite image uploads into `describe-image` references before downstream vision plugins receive the original image block.
|
|
521
|
+
|
|
522
|
+
`dsh-web-ui` now provides an explicit compatibility switch. Go to **Settings → Plugin config → Image understanding** and turn off **“Rewrite images to describe-image references on send”**, or set `interceptImageSend: false`. Image sends will then pass through unchanged so `dsh-vision-router` can receive the original image block. The switch is read on every send, so no hook reinstall or DSH restart is required.
|
|
523
|
+
|
|
524
|
+
See [dsh-web-ui#301](https://github.com/zhu1090093659/dsh-web-ui/issues/301) for the upstream compatibility change.
|
|
525
|
+
|
|
452
526
|
### Startup fails with `Unexpected token ... is not valid JSON` (UTF-8 BOM)
|
|
453
527
|
|
|
454
528
|
**Symptom:** `dsh web` / `pnpm dsh web` exits immediately at startup:
|
|
@@ -498,4 +572,4 @@ Manual fallback: in VS Code, use “Save with Encoding” → `UTF-8` (without B
|
|
|
498
572
|
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/ysr666/dsh-vision-router/star-history/assets/star-history/star-history-dark.svg">
|
|
499
573
|
<img alt="Star history chart" src="https://raw.githubusercontent.com/ysr666/dsh-vision-router/star-history/assets/star-history/star-history-light.svg" width="100%">
|
|
500
574
|
</picture>
|
|
501
|
-
</p>
|
|
575
|
+
</p>
|
package/README.zh.md
CHANGED
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
|
|
7
7
|
<p align="center"><strong>图片粘贴即用:给 DeepSeek Harness 的纯文本 Agent 装上“眼睛”——开箱免费、免 Key、无 Python、一条命令安装。</strong></p>
|
|
8
8
|
|
|
9
|
-
<p align="center">DeepSeek 只负责思考,内置免费视觉链 +
|
|
9
|
+
<p align="center">DeepSeek 只负责思考,内置免费视觉链 + 13 个深看工具负责“看”;图片轮次就像普通工具调用一样自然、可定位、可验证。</p>
|
|
10
10
|
|
|
11
11
|
<p align="center">
|
|
12
12
|
<a href="https://awesome-dsh-plugin.com"><img src="https://awesome-dsh-plugin.com/badge.svg" alt="awesome · DSH plugin" /></a>
|
|
@@ -15,8 +15,8 @@
|
|
|
15
15
|
</p>
|
|
16
16
|
|
|
17
17
|
<p align="center">
|
|
18
|
-
<a href="https://github.com/ysr666/dsh-vision-router/releases/tag/v1.
|
|
19
|
-
<a href="tests"><img src="https://img.shields.io/badge/verified-
|
|
18
|
+
<a href="https://github.com/ysr666/dsh-vision-router/releases/tag/v1.5.0"><img src="https://img.shields.io/badge/release-v1.5.0-5B4CF0?style=flat-square" alt="Release v1.5.0" /></a>
|
|
19
|
+
<a href="tests"><img src="https://img.shields.io/badge/verified-348%20tests-2EA44F?style=flat-square" alt="Verified: 348 tests" /></a>
|
|
20
20
|
<a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-2EA44F?style=flat-square" alt="License: MIT" /></a>
|
|
21
21
|
<a href="package.json"><img src="https://img.shields.io/badge/Node.js-%3E%3D22-339933?style=flat-square&logo=nodedotjs&logoColor=white" alt="Node.js >=22" /></a>
|
|
22
22
|
<img src="https://img.shields.io/badge/runtime-no%20Python-8A2BE2?style=flat-square" alt="No Python" />
|
|
@@ -28,9 +28,9 @@
|
|
|
28
28
|
<p align="center">💬 <strong>QQ 用户交流群:1105463028</strong></p>
|
|
29
29
|
|
|
30
30
|
> [!WARNING]
|
|
31
|
-
> 📌 **公告(v1.
|
|
31
|
+
> 📌 **公告(v1.5.0)**
|
|
32
32
|
>
|
|
33
|
-
> **v1.
|
|
33
|
+
> **v1.5.0:新增 Ollama / LM Studio 本地视觉与桌面截图,强化设置和生命周期稳定性。**
|
|
34
34
|
|
|
35
35
|
<p align="center">
|
|
36
36
|
<img src="assets/vision-demo.gif" width="640" alt="演示:粘贴图片,Agent 用 vision_ground / vision_crop / vision_pixel_diff 定位发送按钮并给出坐标" />
|
|
@@ -40,6 +40,7 @@
|
|
|
40
40
|
|
|
41
41
|
- [为什么做这个](#为什么做这个)
|
|
42
42
|
- [对比同类插件](#对比同类插件)
|
|
43
|
+
- [设计来源](#设计来源)
|
|
43
44
|
- [致谢](#致谢)
|
|
44
45
|
- [快速开始](#快速开始)
|
|
45
46
|
- [免费视觉 Key 渠道](#免费视觉-key-渠道)
|
|
@@ -87,6 +88,14 @@
|
|
|
87
88
|
| [dsh-vision-toolkit](https://github.com/Anionex/dsh-vision-toolkit) | 10 个意图化视觉工具(Q&A/OCR/像素校验/UI 还原),按需显式调用 | 工具集更全;本插件多出整轮自动路由与免 Key 免费兜底 |
|
|
88
89
|
| [dsh-tool-vision](https://github.com/Scorp1o117/dsh-tool-vision) | `inspect_image` 工具 + `agent/pre-step` 瀑布图片桥(粘贴图入日志前转成工具提示) | 瀑布桥思路相近;本插件多出轮次路由、降级链、缓存与免费端点 |
|
|
89
90
|
|
|
91
|
+
## 设计来源
|
|
92
|
+
|
|
93
|
+
本项目的深度视觉工具层与 UI restoration 工作流参考并受到 [Anionex/agent-vision-toolkit](https://github.com/Anionex/agent-vision-toolkit) 及其 DSH 原生实现 [Anionex/dsh-vision-toolkit](https://github.com/Anionex/dsh-vision-toolkit) 的设计影响。具体包括意图驱动的工具选择、渐进式工具暴露、pixel-diff 验证闭环,以及部分视觉工具的职责划分与命名,包括长截图 OCR、前景提取和 HTML screenshot 等设计。
|
|
94
|
+
|
|
95
|
+
`dsh-vision-router` 中相关代码均为独立实现。在这些设计参考基础上,本项目独立发展了 turn-level/tools-first vision routing、DSH 准入/包装集成、多视觉后端与故障 fallback chain、内置免费视觉模型链、附件/图片记忆、缓存与相关运行时容错机制。
|
|
96
|
+
|
|
97
|
+
感谢 Anionex 的先行工作以及整个 DSH 社区的探索。清晰的设计归因与独立迭代并不冲突;二者都有助于维护开放、协作、健康的 DSH 生态。
|
|
98
|
+
|
|
90
99
|
## 致谢
|
|
91
100
|
|
|
92
101
|
本插件借鉴了以上全部社区项目的思路,特别是 [dsh-vision-sidecar](https://github.com/121103qwq/dsh-vision-sidecar)
|
|
@@ -177,10 +186,10 @@ opencode-go + 自动识图 ← 发图片时选这个
|
|
|
177
186
|
## 亮点
|
|
178
187
|
|
|
179
188
|
- **原图像素,真实答案。** 视觉链按原始分辨率读图(仅为保护延迟/额度自动缩放);你的问题随图一起发送,答案围绕*你的问题*,而不是一段泛泛的描述。
|
|
180
|
-
- **自动降级 + 分类报错。** 地区限制、ToS 风控、402 额度、429
|
|
189
|
+
- **自动降级 + 分类报错。** 地区限制、ToS 风控、402 额度、429 限流、上下文超长、网络故障——链路逐供应商尝试,全部失败才报错并给出可操作的建议。遇到 429 会立即尝试下一后端,并按 Retry-After 开启冷却,不会在单次请求内睡眠等待。
|
|
181
190
|
- **图片记忆。** 视觉答案按附件内容哈希缓存;后续文字轮用记录的描述替换历史图片(标注为不可信证据),DeepSeek 真正“记得”之前发过的图,且不重复消耗视觉调用。
|
|
182
191
|
- **可验证的像素闭环。** 参照图 → `vision_html_screenshot` → `vision_pixel_diff`(差异率 + 红色热力图 + 最差区域排行)→ 修复 → 再对比,直到差异收敛。UI 还原从“目测”变成“实测”。
|
|
183
|
-
- **稳定工具 schema。** 默认从会话开始就注册完整
|
|
192
|
+
- **稳定工具 schema。** 默认从会话开始就注册完整 13 个深看工具,避免图片轮中途扩展工具列表导致长上下文的 KV / prefix cache 失效。仍保留 `progressiveTools: true` 作为高级启动期 opt-in;开启后才使用 `vision_activate` 按需挂载。详见 [`docs/progressive-tools-cache.md`](docs/progressive-tools-cache.md)。
|
|
184
193
|
- **选择性代理。** 只有配置的视觉供应商域名走本地代理;DeepSeek 保持直连。
|
|
185
194
|
|
|
186
195
|
### 像素闭环实测
|
|
@@ -201,24 +210,29 @@ Agent 仅根据参考图复刻 UI,再用 `vision_pixel_diff` 验证最终结
|
|
|
201
210
|
|
|
202
211
|
## 工具
|
|
203
212
|
|
|
204
|
-
默认 `progressiveTools: false`:
|
|
213
|
+
默认 `progressiveTools: false`:13 个深看工具从插件启动时就保持常驻,文本轮和图片轮都可直接调用。若你在 profile / composition 的 `cordis.patch.yml` 中显式开启 `progressiveTools: true`,才会恢复渐进模式:初始只暴露 `vision_activate`,首次需要时再挂载完整工具,并注册 `vision-tools` 技能。该开关是启动期配置,修改后需重启 DSH。全部工具基于 sharp / potrace / tesseract / 系统 Chrome——无 Python:
|
|
205
214
|
|
|
206
215
|
<p align="center">
|
|
207
|
-
<img src="assets/vision-tools-zh.svg" width="100%" alt="DSH Vision Router 的 11
|
|
216
|
+
<img src="assets/vision-tools-zh.svg" width="100%" alt="DSH Vision Router 的 11 个图像处理工具。" />
|
|
208
217
|
</p>
|
|
209
218
|
|
|
219
|
+
图中展示 11 个图像处理工具;另有负责持久展示图片的 `vision_present` 与可选 1+x 结构化首遍识别的 `vision_bootstrap`,默认深看工具集共 13 个。若启动时显式开启隐私敏感的 `vision_screenshot`,则额外增加为第 14 个工具。
|
|
220
|
+
|
|
210
221
|
| 工具 | 作用 | 产物 |
|
|
211
222
|
|---|---|---|
|
|
223
|
+
| `vision_bootstrap` | 可选 1+x 结构化首遍视觉识别;先建立任务无关证据底图,再至少进行 1 次后续视觉调用 | — |
|
|
212
224
|
| `vision_describe` | 看图问答 / 多图对比 / 结构化证据 JSON 模式(摘要 + 布局区域 + 实体清单 + 原文转写) | — |
|
|
213
225
|
| `vision_ground` | 定位目标 → **原图像素框 x1/y1/x2/y2** | 标注 PNG(可选) |
|
|
214
226
|
| `vision_detect` | 盘点某类元素(按钮/输入框/链接…)→ 编号清单 + 原图像素框 | 编号标注 PNG |
|
|
215
227
|
| `vision_crop` | 按像素框裁剪放大 | PNG |
|
|
228
|
+
| `vision_present` | 把生成或编辑后的本地图片发布为持久聊天附件,供用户查看 | 图片附件 |
|
|
216
229
|
| `vision_pixel_diff` | 逐像素对比:差异率 + 最差 8×8 网格区域 | 红色热力图 PNG + JSON 报告 |
|
|
217
230
|
| `vision_colors` | 主色提取(十六进制 + 占比) | — |
|
|
218
231
|
| `vision_ocr` | 文字转写:本地 tesseract(中英)优先,视觉模型兜底 | — |
|
|
219
232
|
| `vision_trace` | SVG 矢量化(potrace 分色;图标/logo) | SVG |
|
|
220
233
|
| `vision_extract_foreground` | 边界洪泛抠图(纯色背景) | 透明 PNG |
|
|
221
234
|
| `vision_html_screenshot` | 给本地 HTML 文件截图(无头系统 Chrome);`fullPage: true` 截整页并返回 `pageHeight` | PNG |
|
|
235
|
+
| `vision_screenshot` | **默认关闭,必须显式开启隐私开关。** 截取 Windows 虚拟屏幕、macOS 主显示器或 Linux 根窗口;Windows 使用 PowerShell CopyFromScreen,macOS 使用 `screencapture`,Linux 需安装 ImageMagick `import` 或 `scrot`;`identify=true` 可按顺序尝试已启用的本地识别后端并返回路径+识别文本 | PNG / +描述文本 |
|
|
222
236
|
| `vision_long_screenshot_ocr` | 长截图转写:重叠分片,tesseract 优先 / 视觉模型回退,按序拼接 Markdown | 分片 PNG + Markdown + manifest |
|
|
223
237
|
|
|
224
238
|
图片格式按**魔数识别**,无扩展名的内容寻址附件文件也能直接用(不用再复制成 `.png`)。
|
|
@@ -229,6 +243,7 @@ Agent 仅根据参考图复刻 UI,再用 `vision_pixel_diff` 验证最终结
|
|
|
229
243
|
vision_ground image="ref.png" target="发送按钮"
|
|
230
244
|
vision_detect image="page.png" target="输入框"
|
|
231
245
|
vision_crop image="ref.png" region="1067,841,1108,881"
|
|
246
|
+
vision_present path="rebuilt.png"
|
|
232
247
|
vision_describe paths=["ref.png","impl.png"] question="列出两图的差异" json=true
|
|
233
248
|
vision_pixel_diff original="ref.png" rebuilt="screenshot.png"
|
|
234
249
|
vision_ocr image="screenshot.png"
|
|
@@ -245,15 +260,17 @@ vision_long_screenshot_ocr image="chat-log.png" chunkHeight=1200 overlap=120
|
|
|
245
260
|
视觉工具按顺序逐个尝试,全部失败才报错:
|
|
246
261
|
|
|
247
262
|
1. **用户视觉模型**:设置卡里一行一个,从上到下;只显示 **设置 → 模型** 中明确声明支持 image 输入的模型;
|
|
248
|
-
2.
|
|
249
|
-
3.
|
|
263
|
+
2. **本地 Ollama(可选,默认关)**:`localOllama.enabled` 开启后,通过本机 Ollama 做免 Key、离线识别(例如 qwen2.5vl);
|
|
264
|
+
3. **本地 LM Studio(可选,默认关)**:`localLmStudio.enabled` 排在 Ollama 之后,模型名必须填写 LM Studio Developer 页或 `/v1/models` 返回的真实标识;
|
|
265
|
+
4. **高级自定义 HTTP 视觉端点**:旧配置/高级配置中的 `httpProviders` 排在本地后端之后;
|
|
266
|
+
5. **内置 OVH 匿名免费兜底**:固定最后尝试,不需要出现在任何模型选择器里。当前内置链按质量优先为 `Qwen3.5-397B-A17B` → `Qwen2.5-VL-72B-Instruct` → `Qwen3.6-27B` → `Mistral-Small-3.2-24B-Instruct-2506` → `Qwen3.5-9B`。OVH 匿名限额为 **每 IP、每模型 2 次/分钟**;5 个模型是独立限额,因此理论上分散请求可到约 **10 次/分钟**,实际仍以 OVH 当时的限流为准。免注册、免 Key。想提额度?详见[免费视觉 Key 渠道](#免费视觉-key-渠道)——同一个端点挂免费 access key 后是 400 次/分钟。
|
|
250
267
|
|
|
251
268
|
> [!IMPORTANT]
|
|
252
269
|
> 这里的“视觉链”是 Vision Router 调用的**眼睛**:设置页里每一行只选一个用户视觉模型;聊天页右下角选择的是**脑子/会话模型**,两者完全分开。纯文本 DeepSeek / opencode 不会出现在视觉后端下拉里;内部 `Vision HTTP` 也不会再暴露给用户。
|
|
253
270
|
|
|
254
271
|
> 在旧版 `routing: true` 模式下,整轮链只走 `provider + fallbacks`——`httpProviders`(含免费兜底)不参与。默认的 `routing: false`(工具优先)会尝试全部。
|
|
255
272
|
|
|
256
|
-
失败会分类(地区 / 风控 / 额度 / 限流 / 上下文 /
|
|
273
|
+
失败会分类(地区 / 风控 / 额度 / 限流 / 上下文 / 网络),最终报错附带建议;遇到 `429` 会立即尝试下一后端,并按 `Retry-After` 开启有上限的熔断冷却。超大上传图在调用前自动压缩(默认预算 400 万像素),保证工具调用不卡。
|
|
257
274
|
|
|
258
275
|
## 隐身模式
|
|
259
276
|
|
|
@@ -292,7 +309,7 @@ Web 配置页在 **设置 → 插件 → 插件配置** 下注册「视觉路由
|
|
|
292
309
|
- 开关:整轮自动路由(旧模式)、识图工具、图片块改写、隐身模式(仅官方 DeepSeek 路由);
|
|
293
310
|
- 视觉请求超时、包装/链路由名、代理等高级参数;
|
|
294
311
|
- 每个字段都有「已覆盖」徽标与一键恢复组合默认,以及放弃/保存;
|
|
295
|
-
-
|
|
312
|
+
- 「测试连接」按钮优先探测已启用的本地后端,并校验所填模型是否出现在 `/v1/models`;否则探测第一个可用视觉提供方;
|
|
296
313
|
- 产出制品的工具在对话里渲染专用调用卡(关键字段 + 打开文件按钮)。
|
|
297
314
|
|
|
298
315
|
<p align="center">
|
|
@@ -320,6 +337,12 @@ Web 配置页在 **设置 → 插件 → 插件配置** 下注册「视觉路由
|
|
|
320
337
|
| `textProvider` | `deepseek-official` / `deepseek-v4-pro` | 负责思考的模型(你的日常模型) |
|
|
321
338
|
| `tool` / `progressiveTools` / `autoActivateOnImage` | `true` / `false` / `true` | 视觉工具总开关 / 渐进式挂载(默认关闭以稳定工具 schema)/ 渐进模式下图片轮自动挂载;`progressiveTools` 为启动期配置 |
|
|
322
339
|
| `rewriteImages` | `true` | 模型输入层改写图片块(缓存描述或工具提示标记);界面日志保留图片 |
|
|
340
|
+
| `desktopScreenshot` | `false` | 模型可调用的 `vision_screenshot` 桌面截屏隐私开关;每次截屏前实时检查 |
|
|
341
|
+
| `freeFallback` | `true` | 在显式本地/自定义 HTTP 后端之后追加匿名 OVH 模型;关闭它不会停用用户明确配置的本地后端 |
|
|
342
|
+
| `localOllama` | `{ enabled: false, baseURL: 'http://127.0.0.1:11434/v1', model: 'qwen2.5vl', format: 'openai' }` | **本地视觉后端(并入自 dsh-vision)**:开启后 local-ollama 排在 HTTP 视觉链最前;Ollama 未运行会自动跳过;`format` 可选 `openai`(`/chat/completions`)或 `anthropic`(`/messages`);可选的 `temperature` / `top_p` 只在显式填写时发送,留空尊重本地服务默认值 |
|
|
343
|
+
| `localLmStudio` | `{ enabled: false, baseURL: 'http://localhost:1234/v1', model: '', format: 'openai' }` | **本地 LM Studio 后端(并入自 dsh-vision)**:排在 Ollama 之后、自定义/云 HTTP 后端之前;开启时必须填写 LM Studio Developer 页或 `/v1/models` 返回的真实模型标识;可选采样参数同 Ollama,`format: 'anthropic'` 需 LM Studio 0.4.1+ |
|
|
344
|
+
| `instantDescribe` | `false` | **即时本地翻译(并入自 dsh-vision)**:开启且至少一个本地后端可用时,在第一模型步之前识别无缓存图片块;Ollama → LM Studio 共用总超时预算,多图并发上限 3,失败则回退静态工具标记 |
|
|
345
|
+
| `localDescribeStyle` | `plain` | **本地识别输出风格(并入自 dsh-vision)**:`plain` = 平铺描述;`structured` = 结构化识别(【初步判断】/【细节】/【空间结构】/【原图尺寸】),截图分析质量更高 |
|
|
323
346
|
| `downscale` / `downscaleMaxPixels` | `true` / `4000000` | 调用前压缩及其像素预算(延迟保护) |
|
|
324
347
|
| `cache` / `cacheTtlSeconds` / `cacheMaxEntries` | `true` / `3600` / `200` | 视觉答案缓存 |
|
|
325
348
|
| `timeoutMs` | `120000` | 单次视觉调用超时 |
|
|
@@ -327,12 +350,53 @@ Web 配置页在 **设置 → 插件 → 插件配置** 下注册「视觉路由
|
|
|
327
350
|
| `proxy` / `proxyHosts` | `''` / openrouter 域名 | 仅视觉供应商域名可选的本地代理 |
|
|
328
351
|
| `catalogCorrections` | `true` | 内置目录纠错:当已安装的 pi-ai 目录把已知模型路由到错误协议时(例如 `opencode-go/qwen3.6-plus` 被指向 OpenAI chat completions,而 OpenCode Go 只在 `/v1/messages` 上提供该模型),插件直接按正确协议应答该后端;上游目录修复后每条纠错自动失效 |
|
|
329
352
|
|
|
353
|
+
### 本地 Ollama 视觉后端(并入自 dsh-vision)
|
|
354
|
+
|
|
355
|
+
> **增量开发作者**:[shaoqiuyuavailable](https://github.com/shaoqiuyuavailable)(router 本地视觉增量)
|
|
356
|
+
>
|
|
357
|
+
> **思路来源**:本地视觉后端(Ollama / LM Studio 双后端、即时识别、结构化输出、截屏识别、同图去重记忆、失败降级占位、并发防雪崩、超时防护)的思路继承自 [dsh-vision](https://github.com/shaoqiuyuavailable/text-llm-vision/tree/dsh-vision)——本项目将其并入 HTTP 视觉链,并在此基础上扩展了逐级降级链与双协议支持。
|
|
358
|
+
|
|
359
|
+
可选的本地优先视觉路径:不需要 Key,支持隐私、零费用、离线识别。它作为 HTTP 视觉链里的 `local-ollama` 接入;若本地识别失败,除非用户明确配置纯本地链,否则仍可能继续尝试已配置的云后端。
|
|
360
|
+
|
|
361
|
+
**1. 安装 Ollama 并拉取视觉模型**
|
|
362
|
+
|
|
363
|
+
```sh
|
|
364
|
+
# https://ollama.com —— 然后:
|
|
365
|
+
ollama pull qwen2.5vl
|
|
366
|
+
```
|
|
367
|
+
|
|
368
|
+
**2. 开启** —— 设置卡片「本地视觉」组,或 profile patch:
|
|
369
|
+
|
|
370
|
+
```yaml
|
|
371
|
+
- id: vision-router
|
|
372
|
+
config:
|
|
373
|
+
localOllama:
|
|
374
|
+
enabled: true
|
|
375
|
+
baseURL: 'http://127.0.0.1:11434/v1' # OpenAI 兼容端点
|
|
376
|
+
model: 'qwen2.5vl'
|
|
377
|
+
temperature: 0.5 # 可选;识别用低温更稳
|
|
378
|
+
top_p: 0.8 # 可选;留空 = 服务端默认
|
|
379
|
+
instantDescribe: true # 图片轮第一轮即本地识别
|
|
380
|
+
localDescribeStyle: 'structured' # 'plain' | 'structured'
|
|
381
|
+
```
|
|
382
|
+
|
|
383
|
+
**3. 行为说明**
|
|
384
|
+
|
|
385
|
+
- 开启后 `local-ollama` 排在 HTTP 视觉链最前。若要严格纯本地,请移除云视觉行/自定义 HTTP 端点,并关闭 `freeFallback`。
|
|
386
|
+
- **LM Studio 同理**——同一「本地视觉」组里开启 `localLmStudio`,填 OpenAI 兼容端点(默认 `http://localhost:1234/v1`),并使用 Developer 页或 `/v1/models` 返回的真实模型标识。它排在 `local-ollama` 之后、自定义/云 HTTP 后端之前。
|
|
387
|
+
- 每个本地后端可通过 `format` 选择 **OpenAI 或 Anthropic 格式**(默认 `openai`)。Anthropic 模式走 `/v1/messages`,带 `anthropic-version` 并把图片转为 base64 source;只有配置了 Key 才发送 `x-api-key`。LM Studio 需 0.4.1 或更高版本才提供该端点。
|
|
388
|
+
- 任一本地后端未运行或调用超时时自动跳过,继续降级到云链——任何调用都不受影响。
|
|
389
|
+
- `instantDescribe` 会在第一模型步之前按 Ollama → LM Studio 的顺序尝试已启用本地后端。多张无缓存图片并发识别(上限 3),单张失败不影响其余;命中附件记忆的图片不会再次请求本地服务。
|
|
390
|
+
- `vision_screenshot` 默认关闭。单独开启「桌面截屏」隐私开关后,`identify=true` 使用同样的 Ollama → LM Studio 降级顺序。
|
|
391
|
+
- 日志中的 `image turn — instantDescribe=… localBackends=…` 显示实时决策;`instant local describe recognized N/M uncached image(s), C cached, F failed attempts` 显示本轮结果。
|
|
392
|
+
|
|
330
393
|
## 环境要求
|
|
331
394
|
|
|
332
395
|
- DeepSeek Harness 的 Web profile。普通安装可用 `npx @deepseek-ai/dsh ...`;从源码仓库运行时用 `pnpm dsh ...`。只有 CLI 已经进入系统 `PATH` 时才能直接写 `dsh ...`。
|
|
333
396
|
- Node ≥ 22(宿主侧)。
|
|
334
397
|
- 默认免费链路无需 API Key;付费 `httpProviders` 只需一个凭据引用(`apiKeyEnv`)。
|
|
335
|
-
- `vision_html_screenshot`
|
|
398
|
+
- 只有 `vision_html_screenshot` 需要 Chrome / Chromium / Edge;其余工具无浏览器也能用。
|
|
399
|
+
- 桌面截屏必须显式开启。Windows/macOS 使用系统截屏能力;Linux 需安装 ImageMagick `import` 或 `scrot`,且必须处于可截取的桌面会话(Wayland 支持取决于环境)。
|
|
336
400
|
- tesseract 可选:本地引擎缺失时 `vision_ocr` 自动退回视觉模型。
|
|
337
401
|
|
|
338
402
|
## 安装与生命周期
|
|
@@ -444,6 +508,14 @@ pnpm dsh plugin --profile web remove dsh-vision-router
|
|
|
444
508
|
|
|
445
509
|
## 故障排查
|
|
446
510
|
|
|
511
|
+
### 与 dsh-web-ui / dsh-web-ui-all 共存
|
|
512
|
+
|
|
513
|
+
如果同时安装了 `dsh-web-ui` / `@linxin666/dsh-web-ui-all`,其中的 `dsh-tool-describe-image` 发送钩子可能会在 Vision Router 拿到原始 image block 之前,先把图片改写成 `describe-image` 引用。
|
|
514
|
+
|
|
515
|
+
`dsh-web-ui` 现在已经提供显式兼容开关:进入 **设置 → 插件配置 → 图像理解**,关闭「**发送时改写图片为 describe-image 引用**」,或配置 `interceptImageSend: false`。关闭后,带图发送会原样放行,`dsh-vision-router` 就能继续收到原始 image block。该开关每次发送都会动态读取,因此无需重装/卸载 hook,也不需要重启 DSH。
|
|
516
|
+
|
|
517
|
+
上游兼容改动见 [dsh-web-ui#301](https://github.com/zhu1090093659/dsh-web-ui/issues/301)。
|
|
518
|
+
|
|
447
519
|
### 启动报错 `Unexpected token ... is not valid JSON`(UTF-8 BOM)
|
|
448
520
|
|
|
449
521
|
**现象**:`dsh web` / `pnpm dsh web` 启动时直接退出:
|
|
@@ -493,4 +565,4 @@ npx dsh-vision-router doctor --profile web
|
|
|
493
565
|
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/ysr666/dsh-vision-router/star-history/assets/star-history/star-history-dark.svg">
|
|
494
566
|
<img alt="Star 历史趋势图" src="https://raw.githubusercontent.com/ysr666/dsh-vision-router/star-history/assets/star-history/star-history-light.svg" width="100%">
|
|
495
567
|
</picture>
|
|
496
|
-
</p>
|
|
568
|
+
</p>
|
package/entry.js
CHANGED
|
@@ -10,6 +10,7 @@ import z from '@deepseek-ai/schemastery'
|
|
|
10
10
|
import * as core from './index.js'
|
|
11
11
|
import { installVisionRouterFileLogging } from './lib/file-logger.js'
|
|
12
12
|
import { contextWithDelegatedReplay } from './lib/replay-delegation.js'
|
|
13
|
+
import { installLocalVisionStabilizer } from './lib/local-vision-stabilizer.js'
|
|
13
14
|
|
|
14
15
|
// Schemastery object schemas expose set() as the supported way to replace a
|
|
15
16
|
// field schema. This mutates the Config object that index.js itself later uses
|
|
@@ -28,9 +29,32 @@ export const Config = core.Config
|
|
|
28
29
|
export function apply(ctx, config = {}) {
|
|
29
30
|
const logging = installVisionRouterFileLogging(ctx)
|
|
30
31
|
const runtimeCtx = contextWithDelegatedReplay(logging.ctx)
|
|
32
|
+
// #141 stabilization boundary: keep the recently merged local-vision
|
|
33
|
+
// behavior isolated from main's existing provider/router semantics. It
|
|
34
|
+
// normalizes only the local settings/runtime seams before core.apply sees
|
|
35
|
+
// the context (desktop screenshot exposure, instant-local budget/one-pass,
|
|
36
|
+
// local vision-http transport and connection-probe fallback).
|
|
37
|
+
const { ctx: stabilizedCtx, bootConfig } = installLocalVisionStabilizer(runtimeCtx, config, core)
|
|
38
|
+
// 启动诊断摘要只描述 composition/apply 的基础配置。设置服务可能稍后
|
|
39
|
+
// 覆盖这些值;每个图片轮还会记录 current() 的实时决策,避免把这个
|
|
40
|
+
// 启动快照误当成最终设置状态。
|
|
31
41
|
try {
|
|
32
|
-
const
|
|
33
|
-
|
|
42
|
+
const c = config && typeof config === 'object' ? config : {}
|
|
43
|
+
const local = c.localOllama && typeof c.localOllama === 'object' ? c.localOllama : {}
|
|
44
|
+
const lms = c.localLmStudio && typeof c.localLmStudio === 'object' ? c.localLmStudio : {}
|
|
45
|
+
logging.logger.info(
|
|
46
|
+
'vision-router: base config summary — instantDescribe=%s localDescribeStyle=%s localOllama=%s localLmStudio=%s',
|
|
47
|
+
c.instantDescribe === true ? 'on' : 'off',
|
|
48
|
+
c.localDescribeStyle === 'structured' ? 'structured' : 'plain',
|
|
49
|
+
local.enabled === true ? 'on' : 'off',
|
|
50
|
+
lms.enabled === true ? 'on' : 'off',
|
|
51
|
+
)
|
|
52
|
+
} catch {
|
|
53
|
+
/* diagnostics must never break apply */
|
|
54
|
+
}
|
|
55
|
+
try {
|
|
56
|
+
const result = core.apply(stabilizedCtx, {
|
|
57
|
+
...bootConfig,
|
|
34
58
|
progressiveTools: config.progressiveTools === true,
|
|
35
59
|
})
|
|
36
60
|
if (result && typeof result.then === 'function') {
|