dsh-vision-router 3.0.4 → 3.0.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +178 -160
- package/README.zh.md +176 -158
- package/docs/development/dsh-rc1-adaptation.md +29 -0
- package/docs/development/dsh-rc2-adaptation.md +24 -0
- package/docs/development/v305-real-scenario-acceptance.md +34 -0
- package/docs/releases/v3.0.5.md +31 -0
- package/lib/client-presentation-boundary-main.js +15 -4
- package/lib/client.js +7 -4
- package/lib/core-primitives.js +2 -2
- package/lib/http-compat.js +33 -2
- package/lib/session-message-source-compat.js +13 -4
- package/lib/sharp-runtime.js +0 -2
- package/lib/vision-evidence-guidance.js +7 -1
- package/package.json +2 -2
package/README.md
CHANGED
|
@@ -9,142 +9,142 @@
|
|
|
9
9
|
<p align="center">DeepSeek keeps thinking; the built-in free vision chain and fourteen deep tools do the seeing. When an image matters, enable the composer’s “👁 Vision” control and use image turns like ordinary tool-calling turns — grounded, measurable, repeatable.</p>
|
|
10
10
|
|
|
11
11
|
<p align="center">
|
|
12
|
-
<a href="https://awesome-dsh-plugin.com"><img src="https://awesome-dsh-plugin.com/badge.svg" alt="awesome · DSH plugin" /></a>
|
|
13
12
|
<a href="https://github.com/zp-home/dsh-recommend"><img src="https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fzp-home%2Fdsh-recommend%2Fmain%2Fdata%2Fbadges%2Fysr666__dsh-vision-router.certified.json&style=flat-square" alt="dsh-recommend 🏅 certified" /></a>
|
|
14
13
|
<a href="https://www.dshbase.com/plugins/dsh-vision-router/"><img src="https://img.shields.io/badge/dshbase-install--tested-2EA44F?style=flat-square" alt="dshbase install-tested" /></a>
|
|
15
14
|
<a href="https://github.com/SoberReport-AI/DeepGuard/blob/main/reports/dsh-vision-router/2.0.1/39c8f2b2d69aa398418fd6c8ab40b691a92a1a3d.json"><img src="https://img.shields.io/badge/DeepGuard-audit%20clean-2EA44F?style=flat-square" alt="DeepGuard audit clean" /></a>
|
|
16
|
-
<a href="https://whyihaveyou.github.io/dsh-suite/"><img src="https://img.shields.io/badge/featured%20on-dsh--suite-4d6bfe?style=flat-square" alt="featured on dsh-suite" /></a>
|
|
17
15
|
</p>
|
|
18
16
|
|
|
19
|
-
<p align="center">
|
|
20
|
-
<a href="https://github.com/ysr666/dsh-vision-router/releases/latest"><img src="https://img.shields.io/github/v/release/ysr666/dsh-vision-router?label=release&color=5B4CF0&style=flat-square" alt="Latest stable release" /></a>
|
|
21
|
-
<a href="tests"><img src="https://img.shields.io/badge/verified-Node%2022%20%2B%2024-2EA44F?style=flat-square" alt="Verified: Node 22 + 24" /></a>
|
|
22
|
-
<a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-2EA44F?style=flat-square" alt="License: MIT" /></a>
|
|
23
|
-
<a href="package.json"><img src="https://img.shields.io/badge/Node.js-%3E%3D22-339933?style=flat-square&logo=nodedotjs&logoColor=white" alt="Node.js >=22" /></a>
|
|
24
|
-
<img src="https://img.shields.io/badge/runtime-no%20Python-8A2BE2?style=flat-square" alt="No Python" />
|
|
25
|
-
</p>
|
|
26
|
-
|
|
27
|
-
<p align="center">
|
|
28
|
-
<sub>Ecosystem:</sub>
|
|
29
|
-
<a href="https://dshplugin.app/plugins/dsh-vision-router">DSHPlugin.app</a> ·
|
|
30
|
-
<a href="https://github.com/diegosouzapw/awesome-omni-dsh-plugins">Awesome Omni DSH Plugins</a> ·
|
|
31
|
-
<a href="https://dshpluginhub.ai/plugins/dsh-vision-router">dshpluginhub.ai</a> ·
|
|
32
|
-
<a href="https://www.dsh.plus/en/plugins/dsh-vision-router/">dsh.plus</a> ·
|
|
33
|
-
<a href="https://dshplugins.ai/">dshplugins.ai</a> ·
|
|
34
|
-
<a href="https://dshmarket.com/p/ysr666/dsh-vision-router/">dsh-market</a>
|
|
35
|
-
</p>
|
|
36
|
-
|
|
37
|
-
<p align="center">Repository: <a href="https://github.com/ysr666/dsh-vision-router">GitHub (primary · issues, PRs and releases)</a> · <a href="https://atomgit.com/ysr666/dsh-vision-router">AtomGit (China mirror)</a></p>
|
|
17
|
+
<p align="center"><sub>DeepGuard badge links to a historical v2.0.1 audit; it does not certify current releases.</sub></p>
|
|
38
18
|
|
|
39
19
|
<p align="center">English · <a href="README.zh.md">中文</a></p>
|
|
40
20
|
|
|
41
|
-
<p align="center">💬 <strong>QQ community group: 1105463028</strong></p>
|
|
42
|
-
|
|
43
21
|
> [!CAUTION]
|
|
44
22
|
> **Data leaves your machine by default.** When a cloud vision model is used, Vision Router sends the image (or a derived crop), the vision prompt and related request metadata to that provider. A fresh install has an anonymous OVHcloud vision fallback enabled, so "free" and "no key" do **not** mean offline. Do not use the cloud chain for confidential, regulated or classified material. See [Data flow and strict local-only use](#data-flow-and-strict-local-only-use).
|
|
45
23
|
|
|
46
24
|
> [!WARNING]
|
|
47
|
-
>
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
25
|
+
> **v3.0.5 release notes:** fixes Vision reply continuation, optional-peer activation and OVH request-size handling; improves the Vision toggle's transient catalog-refresh behavior. See [v3.0.5 notes](docs/releases/v3.0.5.md). For the version actually available to install, check [GitHub Releases](https://github.com/ysr666/dsh-vision-router/releases/latest) or npm; release notes are not proof that the version has been published.
|
|
26
|
+
|
|
27
|
+
## Start here
|
|
28
|
+
|
|
29
|
+
**1. Install in an existing DSH Web profile:**
|
|
30
|
+
|
|
31
|
+
```sh
|
|
32
|
+
npx @deepseek-ai/dsh plugin --profile web add dsh-vision-router
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
**2. Keep your usual chat model** and click **👁 Vision** beside the composer; check for **👁 Vision ✓**.
|
|
36
|
+
|
|
37
|
+
**3. Paste or upload an image** and ask the agent what to inspect. The built-in keyless cloud fallback is available out of the box.
|
|
38
|
+
|
|
39
|
+
Using Desktop? Change `--profile web` to `--profile desktop` and reload the app. For Oh-DSH and manual setups, follow the [complete installation guide](#quick-start).
|
|
40
|
+
|
|
41
|
+
[Full quick start](#quick-start) · [What it can do](#highlights) · [Data flow and local-only use](#data-flow-and-strict-local-only-use)
|
|
53
42
|
|
|
54
43
|
<p align="center">
|
|
55
44
|
<img src="assets/vision-demo.gif" width="640" alt="Demo: paste an image, the agent locates the send button with vision_ground / vision_crop / vision_pixel_diff and answers with coordinates" />
|
|
56
45
|
</p>
|
|
57
46
|
|
|
47
|
+
## Try it on real images
|
|
48
|
+
|
|
49
|
+
- **Understand a screenshot:** “What does this error dialog mean? Read the exact message.”
|
|
50
|
+
- **Find a UI element:** “Where is the Save button in this screenshot? Give me its position.”
|
|
51
|
+
- **Compare two designs:** “Compare my page to this reference. What should I fix next?” Use `vision_pixel_diff` to measure changes after each iteration.
|
|
52
|
+
- **Read and inspect a long image:** “Extract the relevant text, then crop the unclear part to double-check it.”
|
|
53
|
+
|
|
54
|
+
These are ordinary chat requests: your normal model reasons through the task and calls vision tools as needed. [See the tool list](#tools) or [configure your own/local vision backend](#configuration).
|
|
55
|
+
|
|
58
56
|
## Contents
|
|
59
57
|
|
|
60
|
-
- [
|
|
61
|
-
- [Why this exists](#why-this-exists)
|
|
62
|
-
- [How it compares](#how-it-compares)
|
|
63
|
-
- [Design lineage](#design-lineage)
|
|
64
|
-
- [Acknowledgements](#acknowledgements)
|
|
65
|
-
- [Quick start](#quick-start)
|
|
66
|
-
- [Free vision key channels](#free-vision-key-channels)
|
|
58
|
+
- [Try it on real images](#try-it-on-real-images)
|
|
67
59
|
- [Highlights](#highlights)
|
|
68
60
|
- [How it works](#how-it-works)
|
|
69
61
|
- [Tools](#tools)
|
|
62
|
+
- [Quick start](#quick-start)
|
|
63
|
+
- [Data flow and strict local-only use](#data-flow-and-strict-local-only-use)
|
|
64
|
+
- [Free vision key channels](#free-vision-key-channels)
|
|
65
|
+
- [Why this exists](#why-this-exists)
|
|
66
|
+
- [How it compares](#how-it-compares)
|
|
70
67
|
- [Configuration](#configuration)
|
|
71
68
|
- [Install and lifecycle](#install-and-lifecycle)
|
|
72
69
|
- [Troubleshooting](#troubleshooting)
|
|
70
|
+
- [Design lineage](#design-lineage)
|
|
71
|
+
- [Acknowledgements](#acknowledgements)
|
|
72
|
+
- [Release notes and community](#release-notes-and-community)
|
|
73
73
|
|
|
74
|
-
##
|
|
75
|
-
|
|
76
|
-
Vision Router does not operate fully offline in its default configuration. The exact boundary depends on the operation and the selected models:
|
|
77
|
-
|
|
78
|
-
| Operation | Where data goes |
|
|
79
|
-
|---|---|
|
|
80
|
-
| Local pixel tools such as crop, pixel diff, palette, SVG trace, cutout, materialize and HTML screenshot | Processed on the DSH machine; these operations do not call a vision model. |
|
|
81
|
-
| `vision_ocr` with Tesseract | Processed on the DSH machine. With the default `auto` engine, an unavailable or empty local OCR result may fall back to a vision model. |
|
|
82
|
-
| Image-turn routing and vision-model tools such as describe, detect, ground or vision OCR | The image or derived crop, prompt and relevant context are sent to the selected vision provider. |
|
|
83
|
-
| Built-in free fallback | Sent anonymously (no API key) to OVHcloud AI Endpoints at `oai.endpoints.kepler.ai.cloud.ovh.net`; the service still receives the request payload and network metadata such as the source IP. |
|
|
84
|
-
| User-configured cloud model / HTTP provider | Sent to that provider's configured endpoint under its own retention and privacy terms. |
|
|
85
|
-
| Local Ollama / LM Studio | Image pixels are sent to the configured local endpoint. The resulting text still returns to the current chat model, which may itself be remote. |
|
|
74
|
+
## Highlights
|
|
86
75
|
|
|
87
|
-
|
|
76
|
+
- **Capability-aware Auto routing.** Keep configured order for deterministic control, or explicitly enable Auto to prioritize already-configured models using measured capability evidence. Auto never infers capability from model names, and enabling Auto alone does not start benchmarks.
|
|
77
|
+
- **Verifiable model profiling.** Exact Test Vision sends one request to one exact model; Quick and Full benchmark OCR / general / structured / document / grounding capabilities. Background profiling is separately authorized and yields to real foreground vision work.
|
|
78
|
+
- **Original pixels, real answers.** The vision chain reads the image at original resolution (auto-downscaled only to protect latency/quota); the agent's question travels with the image, so answers are about *your* question, not a generic description.
|
|
79
|
+
- **Automatic failover with classified errors.** Region blocks, ToS filtering, 402 quota, 429 rate limits, context overflow, network failures — the chain walks providers one by one and only reports after all of them failed, with actionable advice. A 429 immediately advances to the next backend and opens a Retry-After-aware cooldown instead of sleeping inside the request.
|
|
80
|
+
- **Image memory.** Vision answers are cached by attachment content hash; later text turns substitute the recorded description (marked as untrusted evidence), so DeepSeek genuinely remembers earlier images without re-spending vision calls.
|
|
81
|
+
- **A verifiable pixel loop.** Reference → `vision_html_screenshot` → `vision_pixel_diff` (ratio + worst 8×8-grid regions) → fix → repeat until the mismatch converges. UI restoration becomes measurable instead of eyeballed.
|
|
82
|
+
- **Stable tool schema.** All fourteen deep tools are registered from session start by default, avoiding a mid-conversation tool-list expansion that can invalidate long-context KV/prefix caches. `progressiveTools: true` remains an advanced boot-time opt-in; only then does `vision_activate` mount the tools on demand. See [`docs/progressive-tools-cache.md`](docs/progressive-tools-cache.md).
|
|
83
|
+
- **Selective proxy.** Only the configured vision provider hosts go through your local proxy; DeepSeek stays direct.
|
|
88
84
|
|
|
89
|
-
|
|
85
|
+
### Pixel loop in practice
|
|
90
86
|
|
|
91
|
-
|
|
87
|
+
[](https://raw.githubusercontent.com/ysr666/dsh-vision-router/main/assets/pixel-loop.png)
|
|
92
88
|
|
|
93
|
-
|
|
94
|
-
- **Free by default.** Vision tools end with a five-model OVHcloud anonymous fallback: no account, no key, 2 requests/minute per IP per model, roughly 10 RPM in theory across independent buckets. User-provided vision models run first.
|
|
95
|
-
- **No Python.** The whole pipeline — downscale, grounding, crop, pixel diff, palette, OCR, SVG trace, cutout, HTML screenshot — runs on sharp / potrace / tesseract / system Chrome.
|
|
96
|
-
- **Continuous multi-step image work.** An image turn is a text turn that calls tools: `vision_ground` → `vision_crop` → `vision_describe` → `vision_pixel_diff` → fix → screenshot again. The agent keeps iterating until the work is done.
|
|
97
|
-
- **DeepSeek stays the brain.** Text turns are untouched in model, cost and context. The vision model is only the eyes, called on demand; answers are cached by image content.
|
|
98
|
-
- **Transparent to the user.** Uploaded images keep rendering as images in the conversation UI; the rewrite that points the model at the vision tools happens only inside the model call, never in the session log.
|
|
89
|
+
<p align="center"><sub>Click the image to open the full-resolution original.</sub></p>
|
|
99
90
|
|
|
100
|
-
|
|
91
|
+
The agent rebuilt the UI from the reference image, then verified the final result with `vision_pixel_diff`: **2.54% final diff** (32,939 / 1,296,000 differing pixels, threshold 16/channel).
|
|
101
92
|
|
|
102
|
-
|
|
103
|
-
(description bridge — lossy); this plugin hands the image turn *straight to a vision model*
|
|
104
|
-
(routing bridge — pixel-level), with a built-in keyless free fallback.
|
|
93
|
+
## How it works
|
|
105
94
|
|
|
106
|
-
>
|
|
107
|
-
|
|
95
|
+
<p align="center">
|
|
96
|
+
<img src="assets/how-it-works.svg" width="100%" alt="How DSH Vision Router keeps DeepSeek as the brain and vision tools as the eyes." />
|
|
97
|
+
</p>
|
|
108
98
|
|
|
109
|
-
|
|
110
|
-
|---|---|---|---|
|
|
111
|
-
| Image pixels | ✅ available (when switched) | ❌ text description only | ✅ Host-canonical raster, on the image turn |
|
|
112
|
-
| Automatic | ❌ | ✅ | ✅ |
|
|
113
|
-
| Daily model untouched | ❌ (whole session swapped) | ✅ | ✅ |
|
|
114
|
-
| Provider failure recovery | ❌ | ❌ | ✅ fallback chains |
|
|
115
|
-
| Reusable structured queries | — | partial | ✅ JSON mode + caching |
|
|
116
|
-
| Free out-of-the-box | ❌ | ❌ | ✅ built-in keyless endpoint |
|
|
117
|
-
| Fits dsh composition | — | external server | ✅ one plugin row |
|
|
99
|
+
The vision model is **only the eyes**; DeepSeek is **always the brain**. An image turn is never hijacked by a one-shot vision answer — the agent drives the tools itself and can keep operating on the image across as many steps as the task needs.
|
|
118
100
|
|
|
119
|
-
|
|
101
|
+
## Tools
|
|
120
102
|
|
|
121
|
-
|
|
122
|
-
|---|---|---|
|
|
123
|
-
| [dsh-vision-sidecar](https://github.com/121103qwq/dsh-vision-sidecar) | Pre-describes images with an external VLM; the description joins the session as a message to DeepSeek; LLM7.io anonymous endpoint by default (OVHcloud listed as a no-key alternative) | Description bridge; this plugin adds image routing, with `vision_describe` covering descriptions on demand |
|
|
124
|
-
| [dsh-vision-proxy](https://github.com/Flyvhidbwo/dsh-vision-proxy) | Wraps a provider route and transcribes images into text in the request stream | Transcription bridge; this plugin wraps no provider — it rewrites routing through `agent/request` waterfalls |
|
|
125
|
-
| [dsh-vision-provider](https://github.com/libinyam/dsh-vision-provider) | Registers `DeepSeek + Vision` combined routes: images are described by the chosen vision model before reaching DeepSeek | Two-model bridge idea; this plugin adds automatic routing, fallback chains and tools on top |
|
|
126
|
-
| [modlens](https://github.com/liustack/modlens) | The first dsh vision plugin; reuses local Claude Code/Codex/OpenCode/Pi logins as vision engines | Engine-reuse idea; this plugin ships its own provider chain and depends on no other local CLI |
|
|
127
|
-
| [dsh-vision-toolkit](https://github.com/Anionex/dsh-vision-toolkit) | Ten intent-aware visual tools (Q&A/OCR/pixel verification/UI restoration), called explicitly on demand | Broader tool set; this plugin adds whole-turn auto-routing and a keyless free fallback |
|
|
128
|
-
| [dsh-tool-vision](https://github.com/Scorp1o117/dsh-tool-vision) | An `inspect_image` tool plus an `agent/pre-step` waterfall bridge (pasted images become tool hints before entering the log) | Similar waterfall bridge; this plugin adds turn routing, fallback chains, caching and the free endpoint |
|
|
103
|
+
Default `progressiveTools: false`: all fourteen deep tools stay registered from plugin startup, so text and image turns can call them immediately. If you explicitly set `progressiveTools: true` in the profile/composition `cordis.patch.yml`, progressive mode is restored: only `vision_activate` is exposed initially, the full tool set mounts on first use, and the `vision-tools` skill is registered. This is a boot-time switch; restart DSH after changing it. Built on sharp / potrace / tesseract / system Chrome — no Python:
|
|
129
104
|
|
|
130
|
-
|
|
105
|
+
<p align="center">
|
|
106
|
+
<img src="assets/vision-tools.svg" width="100%" alt="Eleven image-processing tools available in DSH Vision Router." />
|
|
107
|
+
</p>
|
|
131
108
|
|
|
132
|
-
The
|
|
109
|
+
The diagram covers the eleven image-processing tools. `vision_present` (durable image delivery) and `vision_bootstrap` (the optional 1+x structured first pass) bring the default deep-tool set to fourteen. Enabling the privacy-gated `vision_screenshot` at boot adds an optional fifteenth tool.
|
|
133
110
|
|
|
134
|
-
|
|
111
|
+
| Tool | What it does | Artifact |
|
|
112
|
+
|---|---|---|
|
|
113
|
+
| `vision_bootstrap` | Optional 1+x structured first visual pass; establishes task-independent evidence before at least one follow-up vision call | — |
|
|
114
|
+
| `vision_describe` | Image Q&A / multi-image compare / structured-evidence JSON mode (summary + layout regions + entity inventory + verbatim transcription) | — |
|
|
115
|
+
| `vision_materialize` | Copy an authorized attachment into a stable content-addressed workspace handoff file and return absolute + short workspace-relative paths for local OCR/parser fallbacks; no vision/network call | image copy |
|
|
116
|
+
| `vision_ground` | Locate a target → **original-pixel box x1/y1/x2/y2** | annotated PNG (optional) |
|
|
117
|
+
| `vision_detect` | Numbered inventory of every element of a kind (buttons/inputs/links…) with original-pixel boxes | annotated PNG with numbered boxes |
|
|
118
|
+
| `vision_crop` | Crop and zoom into a pixel box | PNG |
|
|
119
|
+
| `vision_present` | Publish a generated or edited local image as a durable chat attachment so the user can see it | image attachment |
|
|
120
|
+
| `vision_pixel_diff` | Per-pixel comparison: diff ratio + worst 8×8-grid regions | red heatmap PNG + JSON report |
|
|
121
|
+
| `vision_colors` | Dominant colors (hex + share) | — |
|
|
122
|
+
| `vision_ocr` | Text transcription: configurable default engine (`auto` / local Tesseract / vision model); an explicit per-call engine still wins | — |
|
|
123
|
+
| `vision_trace` | SVG vectorization (potrace posterization; icons/logos) | SVG |
|
|
124
|
+
| `vision_extract_foreground` | Cutout via border flood fill (uniform backgrounds) | transparent PNG |
|
|
125
|
+
| `vision_html_screenshot` | Screenshot a local HTML file (headless system Chrome); `fullPage: true` captures the whole page and reports `pageHeight` | PNG |
|
|
126
|
+
| `vision_screenshot` | **Disabled by default; explicit opt-in required.** Capture the Windows virtual screen, macOS main display, or Linux root display. Windows uses PowerShell CopyFromScreen, macOS uses `screencapture`, and Linux requires ImageMagick `import` or `scrot`; optional `identify=true` tries enabled local recognition backends in order and returns the description with the path | PNG / + description |
|
|
127
|
+
| `vision_long_screenshot_ocr` | Long-screenshot transcription: overlapping chunks, tesseract first / vision model fallback, stitched Markdown | chunk PNGs + Markdown + manifest |
|
|
135
128
|
|
|
136
|
-
|
|
129
|
+
Formats are sniffed from magic bytes, so extensionless content-addressed attachment files work everywhere (no `.png` renaming needed).
|
|
137
130
|
|
|
138
|
-
|
|
131
|
+
**Common workflows**
|
|
139
132
|
|
|
140
|
-
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
[
|
|
146
|
-
|
|
147
|
-
|
|
133
|
+
```text
|
|
134
|
+
vision_ground image="ref.png" target="the send button"
|
|
135
|
+
vision_detect image="page.png" target="input fields"
|
|
136
|
+
vision_crop image="ref.png" region="1067,841,1108,881"
|
|
137
|
+
vision_present path="rebuilt.png"
|
|
138
|
+
vision_describe paths=["ref.png","impl.png"] question="list the differences" json=true
|
|
139
|
+
vision_pixel_diff original="ref.png" rebuilt="screenshot.png"
|
|
140
|
+
vision_ocr image="screenshot.png"
|
|
141
|
+
vision_colors image="ref.png" top=8
|
|
142
|
+
vision_trace image="icon.png" steps=4
|
|
143
|
+
vision_extract_foreground image="logo.png"
|
|
144
|
+
vision_html_screenshot source="page.html" width=1200 height=720
|
|
145
|
+
vision_html_screenshot source="page.html" width=1200 height=720 fullPage=true
|
|
146
|
+
vision_long_screenshot_ocr image="chat-log.png" chunkHeight=1200 overlap=120
|
|
147
|
+
```
|
|
148
148
|
|
|
149
149
|
## Quick start
|
|
150
150
|
|
|
@@ -217,6 +217,21 @@ The built-in anonymous OVH vision fallback is already configured, so normal imag
|
|
|
217
217
|
<img src="assets/dsh-conversation-image-qa-result.png" width="49%" alt="The agent's structured answer describing the image content." />
|
|
218
218
|
</p>
|
|
219
219
|
|
|
220
|
+
## Data flow and strict local-only use
|
|
221
|
+
|
|
222
|
+
Vision Router does not operate fully offline in its default configuration. The exact boundary depends on the operation and the selected models:
|
|
223
|
+
|
|
224
|
+
| Operation | Where data goes |
|
|
225
|
+
|---|---|
|
|
226
|
+
| Local pixel tools such as crop, pixel diff, palette, SVG trace, cutout, materialize and HTML screenshot | Processed on the DSH machine; these operations do not call a vision model. |
|
|
227
|
+
| `vision_ocr` with Tesseract | Processed on the DSH machine. With the default `auto` engine, an unavailable or empty local OCR result may fall back to a vision model. |
|
|
228
|
+
| Image-turn routing and vision-model tools such as describe, detect, ground or vision OCR | The image or derived crop, prompt and relevant context are sent to the selected vision provider. |
|
|
229
|
+
| Built-in free fallback | Sent anonymously (no API key) to OVHcloud AI Endpoints at `oai.endpoints.kepler.ai.cloud.ovh.net`; the service still receives the request payload and network metadata such as the source IP. |
|
|
230
|
+
| User-configured cloud model / HTTP provider | Sent to that provider's configured endpoint under its own retention and privacy terms. |
|
|
231
|
+
| Local Ollama / LM Studio | Image pixels are sent to the configured local endpoint. The resulting text still returns to the current chat model, which may itself be remote. |
|
|
232
|
+
|
|
233
|
+
To stop **Vision Router itself** from sending image data to remote vision endpoints, enable **Settings → Vision Router → General → Local-only vision**. This is a runtime policy, not a destructive rewrite: saved cloud rows and `freeFallback` stay configured but cannot execute while the switch is on. Only loopback visual endpoints (`localhost`, `127.0.0.0/8`, `::1`) are eligible, including Ollama, LM Studio, and custom local HTTP backends. For a **strict local-only workflow**, also use a local chat model because vision result text still returns to the current chat model. Test the final network boundary in your own environment; Vision Router cannot make a remote chat model, proxy or Host integration local.
|
|
234
|
+
|
|
220
235
|
## Free vision key channels
|
|
221
236
|
|
|
222
237
|
The built-in OVH fallback is anonymous by design, and OVH caps anonymous use at **2 requests/minute per IP per model**. If that feels tight, every channel below offers **free vision models with much higher quotas** — all of them are free to register, and none charges for the free tier. Free policies rotate often; treat this table as an August 2026 snapshot and double-check each provider's console before relying on it.
|
|
@@ -238,80 +253,46 @@ Any of these channels can join the vision chain as an `httpProviders` entry (key
|
|
|
238
253
|
> [!NOTE]
|
|
239
254
|
> Free-tier policies change without notice — Cerebras retired its free tier in July 2026 (now a one-time $5 credit), SambaNova's free tier is down to 20 requests/day, and Hugging Face's is $0.10/month. Third-party “`:free` relay” aggregators are deliberately not listed: they rotate quickly, lack SLAs, and some resell quota in ways that violate upstream terms.
|
|
240
255
|
|
|
241
|
-
##
|
|
242
|
-
|
|
243
|
-
- **Capability-aware Auto routing.** Keep configured order for deterministic control, or explicitly enable Auto to prioritize already-configured models using measured capability evidence. Auto never infers capability from model names, and enabling Auto alone does not start benchmarks.
|
|
244
|
-
- **Verifiable model profiling.** Exact Test Vision sends one request to one exact model; Quick and Full benchmark OCR / general / structured / document / grounding capabilities. Background profiling is separately authorized and yields to real foreground vision work.
|
|
245
|
-
- **Original pixels, real answers.** The vision chain reads the image at original resolution (auto-downscaled only to protect latency/quota); the agent's question travels with the image, so answers are about *your* question, not a generic description.
|
|
246
|
-
- **Automatic failover with classified errors.** Region blocks, ToS filtering, 402 quota, 429 rate limits, context overflow, network failures — the chain walks providers one by one and only reports after all of them failed, with actionable advice. A 429 immediately advances to the next backend and opens a Retry-After-aware cooldown instead of sleeping inside the request.
|
|
247
|
-
- **Image memory.** Vision answers are cached by attachment content hash; later text turns substitute the recorded description (marked as untrusted evidence), so DeepSeek genuinely remembers earlier images without re-spending vision calls.
|
|
248
|
-
- **A verifiable pixel loop.** Reference → `vision_html_screenshot` → `vision_pixel_diff` (ratio + worst 8×8-grid regions) → fix → repeat until the mismatch converges. UI restoration becomes measurable instead of eyeballed.
|
|
249
|
-
- **Stable tool schema.** All fourteen deep tools are registered from session start by default, avoiding a mid-conversation tool-list expansion that can invalidate long-context KV/prefix caches. `progressiveTools: true` remains an advanced boot-time opt-in; only then does `vision_activate` mount the tools on demand. See [`docs/progressive-tools-cache.md`](docs/progressive-tools-cache.md).
|
|
250
|
-
- **Selective proxy.** Only the configured vision provider hosts go through your local proxy; DeepSeek stays direct.
|
|
251
|
-
|
|
252
|
-
### Pixel loop in practice
|
|
253
|
-
|
|
254
|
-
[](https://raw.githubusercontent.com/ysr666/dsh-vision-router/main/assets/pixel-loop.png)
|
|
255
|
-
|
|
256
|
-
<p align="center"><sub>Click the image to open the full-resolution original.</sub></p>
|
|
257
|
-
|
|
258
|
-
The agent rebuilt the UI from the reference image, then verified the final result with `vision_pixel_diff`: **2.54% final diff** (32,939 / 1,296,000 differing pixels, threshold 16/channel).
|
|
256
|
+
## Why this exists
|
|
259
257
|
|
|
260
|
-
|
|
258
|
+
Most DSH vision plugins bridge images to DeepSeek as *text descriptions* — lossy, one-shot, and blind to pixels. This plugin keeps the **Host-canonical image pixels on the vision model's side** and DeepSeek on the reasoning side, and makes looking at an image an **ordinary tool call**:
|
|
261
259
|
|
|
262
|
-
|
|
263
|
-
|
|
264
|
-
|
|
260
|
+
- **One command install.** The package ships its own composition patch (`dsh.bundle.patch`): `dsh plugin add` wires the row, the admission wrapper and the attachment limits automatically — zero manual file edits. Taking over the official DeepSeek route is an optional setting (stealth mode, off by default).
|
|
261
|
+
- **Free by default.** Vision tools end with a four-model OVHcloud anonymous fallback: no account, no key, 2 requests/minute per IP per model, roughly 8 RPM in theory across independent buckets. User-provided vision models run first.
|
|
262
|
+
- **No Python.** The whole pipeline — downscale, grounding, crop, pixel diff, palette, OCR, SVG trace, cutout, HTML screenshot — runs on sharp / potrace / tesseract / system Chrome.
|
|
263
|
+
- **Continuous multi-step image work.** An image turn is a text turn that calls tools: `vision_ground` → `vision_crop` → `vision_describe` → `vision_pixel_diff` → fix → screenshot again. The agent keeps iterating until the work is done.
|
|
264
|
+
- **DeepSeek stays the brain.** Text turns are untouched in model, cost and context. The vision model is only the eyes, called on demand; answers are cached by image content.
|
|
265
|
+
- **Transparent to the user.** Uploaded images keep rendering as images in the conversation UI; the rewrite that points the model at the vision tools happens only inside the model call, never in the session log.
|
|
265
266
|
|
|
266
|
-
|
|
267
|
+
## How it compares
|
|
267
268
|
|
|
268
|
-
|
|
269
|
+
**One-line take**: most dsh vision plugins turn images into *text descriptions* for DeepSeek
|
|
270
|
+
(description bridge — lossy); this plugin hands the image turn *straight to a vision model*
|
|
271
|
+
(routing bridge — pixel-level), with a built-in keyless free fallback.
|
|
269
272
|
|
|
270
|
-
|
|
273
|
+
> [!NOTE]
|
|
274
|
+
> On DSH 0.1.2-alpha.1+, attachments remain Host-owned. Vision Router consumes the Host-persisted canonical image: clean single-frame 8-bit sRGB/sRGBA images inside the configured normalization limits can pass through byte-identically, while images that need orientation, color-space, metadata, animation, or size normalization may be re-encoded. Pixel tools therefore promise the Host-canonical raster, not preservation of the uploader's original encoded bytes.
|
|
271
275
|
|
|
272
|
-
|
|
273
|
-
|
|
274
|
-
|
|
276
|
+
| | Manual model switching | MCP vision bridge | dsh-vision-router |
|
|
277
|
+
|---|---|---|---|
|
|
278
|
+
| Image pixels | ✅ available (when switched) | ❌ text description only | ✅ Host-canonical raster, on the image turn |
|
|
279
|
+
| Automatic | ❌ | ✅ | ✅ |
|
|
280
|
+
| Daily model untouched | ❌ (whole session swapped) | ✅ | ✅ |
|
|
281
|
+
| Provider failure recovery | ❌ | ❌ | ✅ fallback chains |
|
|
282
|
+
| Reusable structured queries | — | partial | ✅ JSON mode + caching |
|
|
283
|
+
| Free out-of-the-box | ❌ | ❌ | ✅ built-in keyless endpoint |
|
|
284
|
+
| Fits dsh composition | — | external server | ✅ one plugin row |
|
|
275
285
|
|
|
276
|
-
|
|
286
|
+
**Difference from existing dsh community projects** (all excellent, each with its own focus; descriptions reflect their READMEs as of 2026-08):
|
|
277
287
|
|
|
278
|
-
|
|
|
288
|
+
| Project | Approach | What this plugin adds |
|
|
279
289
|
|---|---|---|
|
|
280
|
-
|
|
|
281
|
-
|
|
|
282
|
-
|
|
|
283
|
-
|
|
|
284
|
-
|
|
|
285
|
-
| `
|
|
286
|
-
| `vision_present` | Publish a generated or edited local image as a durable chat attachment so the user can see it | image attachment |
|
|
287
|
-
| `vision_pixel_diff` | Per-pixel comparison: diff ratio + worst 8×8-grid regions | red heatmap PNG + JSON report |
|
|
288
|
-
| `vision_colors` | Dominant colors (hex + share) | — |
|
|
289
|
-
| `vision_ocr` | Text transcription: configurable default engine (`auto` / local Tesseract / vision model); an explicit per-call engine still wins | — |
|
|
290
|
-
| `vision_trace` | SVG vectorization (potrace posterization; icons/logos) | SVG |
|
|
291
|
-
| `vision_extract_foreground` | Cutout via border flood fill (uniform backgrounds) | transparent PNG |
|
|
292
|
-
| `vision_html_screenshot` | Screenshot a local HTML file (headless system Chrome); `fullPage: true` captures the whole page and reports `pageHeight` | PNG |
|
|
293
|
-
| `vision_screenshot` | **Disabled by default; explicit opt-in required.** Capture the Windows virtual screen, macOS main display, or Linux root display. Windows uses PowerShell CopyFromScreen, macOS uses `screencapture`, and Linux requires ImageMagick `import` or `scrot`; optional `identify=true` tries enabled local recognition backends in order and returns the description with the path | PNG / + description |
|
|
294
|
-
| `vision_long_screenshot_ocr` | Long-screenshot transcription: overlapping chunks, tesseract first / vision model fallback, stitched Markdown | chunk PNGs + Markdown + manifest |
|
|
295
|
-
|
|
296
|
-
Formats are sniffed from magic bytes, so extensionless content-addressed attachment files work everywhere (no `.png` renaming needed).
|
|
297
|
-
|
|
298
|
-
**Common workflows**
|
|
299
|
-
|
|
300
|
-
```text
|
|
301
|
-
vision_ground image="ref.png" target="the send button"
|
|
302
|
-
vision_detect image="page.png" target="input fields"
|
|
303
|
-
vision_crop image="ref.png" region="1067,841,1108,881"
|
|
304
|
-
vision_present path="rebuilt.png"
|
|
305
|
-
vision_describe paths=["ref.png","impl.png"] question="list the differences" json=true
|
|
306
|
-
vision_pixel_diff original="ref.png" rebuilt="screenshot.png"
|
|
307
|
-
vision_ocr image="screenshot.png"
|
|
308
|
-
vision_colors image="ref.png" top=8
|
|
309
|
-
vision_trace image="icon.png" steps=4
|
|
310
|
-
vision_extract_foreground image="logo.png"
|
|
311
|
-
vision_html_screenshot source="page.html" width=1200 height=720
|
|
312
|
-
vision_html_screenshot source="page.html" width=1200 height=720 fullPage=true
|
|
313
|
-
vision_long_screenshot_ocr image="chat-log.png" chunkHeight=1200 overlap=120
|
|
314
|
-
```
|
|
290
|
+
| [dsh-vision-sidecar](https://github.com/121103qwq/dsh-vision-sidecar) | Pre-describes images with an external VLM; the description joins the session as a message to DeepSeek; LLM7.io anonymous endpoint by default (OVHcloud listed as a no-key alternative) | Description bridge; this plugin adds image routing, with `vision_describe` covering descriptions on demand |
|
|
291
|
+
| [dsh-vision-proxy](https://github.com/Flyvhidbwo/dsh-vision-proxy) | Wraps a provider route and transcribes images into text in the request stream | Transcription bridge; this plugin wraps no provider — it rewrites routing through `agent/request` waterfalls |
|
|
292
|
+
| [dsh-vision-provider](https://github.com/libinyam/dsh-vision-provider) | Registers `DeepSeek + Vision` combined routes: images are described by the chosen vision model before reaching DeepSeek | Two-model bridge idea; this plugin adds automatic routing, fallback chains and tools on top |
|
|
293
|
+
| [modlens](https://github.com/liustack/modlens) | The first dsh vision plugin; reuses local Claude Code/Codex/OpenCode/Pi logins as vision engines | Engine-reuse idea; this plugin ships its own provider chain and depends on no other local CLI |
|
|
294
|
+
| [dsh-vision-toolkit](https://github.com/Anionex/dsh-vision-toolkit) | Ten intent-aware visual tools (Q&A/OCR/pixel verification/UI restoration), called explicitly on demand | Broader tool set; this plugin adds whole-turn auto-routing and a keyless free fallback |
|
|
295
|
+
| [dsh-tool-vision](https://github.com/Scorp1o117/dsh-tool-vision) | An `inspect_image` tool plus an `agent/pre-step` waterfall bridge (pasted images become tool hints before entering the log) | Similar waterfall bridge; this plugin adds turn routing, fallback chains, caching and the free endpoint |
|
|
315
296
|
|
|
316
297
|
## Provider fallback chain
|
|
317
298
|
|
|
@@ -321,7 +302,7 @@ The vision tools try backends in order and surface an error only after all of th
|
|
|
321
302
|
2. **Local Ollama (optional, off by default)**: `localOllama.enabled` adds keyless, offline recognition through your local Ollama (for example qwen2.5vl);
|
|
322
303
|
3. **Local LM Studio (optional, off by default)**: `localLmStudio.enabled` follows Ollama and requires the real model identifier shown in LM Studio Developer or returned by `/v1/models`;
|
|
323
304
|
4. **Advanced custom HTTP vision endpoints**: legacy/advanced `httpProviders`, when present, run after the local backends;
|
|
324
|
-
5. **Built-in anonymous OVH fallback**: always last and never exposed in a model picker. The current quality-first chain is `Qwen3.5-397B-A17B` → `Qwen2.5-VL-72B-Instruct` → `Qwen3.6-27B` → `
|
|
305
|
+
5. **Built-in anonymous OVH fallback**: always last and never exposed in a model picker. The current quality-first chain is `Qwen3.5-397B-A17B` → `Qwen2.5-VL-72B-Instruct` → `Qwen3.6-27B` → `Qwen3.5-9B`. OVH anonymous limits are **2 requests/minute per IP per model**. The four models have independent buckets, so spreading requests across them is about **8 RPM in theory**, subject to OVH's actual rate limiting. No signup or API key is required. OVH has scheduled the previously bundled Mistral Small 3.2 model for decommission on November 4, 2026; it is no longer part of the automatic fallback, but user-managed rows are not changed. Want more headroom? See [Free vision key channels](#free-vision-key-channels) — a free OVH access key lifts this same endpoint to 400 requests/minute.
|
|
325
306
|
|
|
326
307
|
> [!IMPORTANT]
|
|
327
308
|
> This “vision chain” is the **eyes** used by Vision Router: each settings row selects one user vision model, while the lower-right chat picker selects the **brain/conversation model**. The two are deliberately separate. Text-only DeepSeek/opencode models are filtered out of the vision-backend dropdown, and the internal `Vision HTTP` transport route is no longer exposed to users.
|
|
@@ -620,6 +601,43 @@ Manual fallback: in VS Code, use “Save with Encoding” → `UTF-8` (without B
|
|
|
620
601
|
- Secrets never travel: `apiKeyEnv` names a DSH credential reference; the value is resolved per call and never logged.
|
|
621
602
|
- The settings write path goes through the settings service (schema-validated, revision-checked) — a stale or invalid save is rejected, not partially applied.
|
|
622
603
|
|
|
604
|
+
## Design lineage
|
|
605
|
+
|
|
606
|
+
The deep-vision tool layer and UI-restoration workflow in this project were informed by [Anionex/agent-vision-toolkit](https://github.com/Anionex/agent-vision-toolkit) and its native DSH implementation [Anionex/dsh-vision-toolkit](https://github.com/Anionex/dsh-vision-toolkit). In particular, this project drew on their intent-driven tool selection, progressive tool exposure, pixel-diff verification loop, and parts of the visual-tool decomposition and naming, including long-screenshot OCR, foreground extraction, and HTML screenshot tooling.
|
|
607
|
+
|
|
608
|
+
All code in `dsh-vision-router` is independently implemented. On top of those referenced design ideas, this project independently developed its turn-level/tools-first vision routing, DSH admission/wrapper integration, multi-backend provider and failure fallback chains, built-in free vision chain, attachment/image-memory handling, caching, and related runtime resilience mechanisms.
|
|
609
|
+
|
|
610
|
+
We appreciate Anionex's prior work and the broader DSH community. Clear attribution and independent iteration can coexist; both help keep the DSH ecosystem open, collaborative, and healthy.
|
|
611
|
+
|
|
612
|
+
## Acknowledgements
|
|
613
|
+
|
|
614
|
+
This project borrows ideas from all of the above — especially the keyless free-endpoint
|
|
615
|
+
exploration (LLM7.io and OVHcloud anonymous tiers) by
|
|
616
|
+
[dsh-vision-sidecar](https://github.com/121103qwq/dsh-vision-sidecar). Thanks to the authors of
|
|
617
|
+
[dsh-vision-proxy](https://github.com/Flyvhidbwo/dsh-vision-proxy),
|
|
618
|
+
[dsh-vision-provider](https://github.com/libinyam/dsh-vision-provider),
|
|
619
|
+
[modlens](https://github.com/liustack/modlens),
|
|
620
|
+
[dsh-vision-toolkit](https://github.com/Anionex/dsh-vision-toolkit), and
|
|
621
|
+
[dsh-tool-vision](https://github.com/Scorp1o117/dsh-tool-vision).
|
|
622
|
+
|
|
623
|
+
## Release notes and community
|
|
624
|
+
|
|
625
|
+
Release details: [v3.0.5 release notes](docs/releases/v3.0.5.md). Check GitHub Releases or npm for the published version.
|
|
626
|
+
|
|
627
|
+
<p align="center">
|
|
628
|
+
<sub>Ecosystem:</sub>
|
|
629
|
+
<a href="https://dshplugin.app/plugins/dsh-vision-router">DSHPlugin.app</a> ·
|
|
630
|
+
<a href="https://github.com/diegosouzapw/awesome-omni-dsh-plugins">Awesome Omni DSH Plugins</a> ·
|
|
631
|
+
<a href="https://dshpluginhub.ai/plugins/dsh-vision-router">dshpluginhub.ai</a> ·
|
|
632
|
+
<a href="https://www.dsh.plus/en/plugins/dsh-vision-router/">dsh.plus</a> ·
|
|
633
|
+
<a href="https://dshplugins.ai/">dshplugins.ai</a> ·
|
|
634
|
+
<a href="https://dshmarket.com/p/ysr666/dsh-vision-router/">dsh-market</a>
|
|
635
|
+
</p>
|
|
636
|
+
|
|
637
|
+
<p align="center">Repository: <a href="https://github.com/ysr666/dsh-vision-router">GitHub (primary · issues, PRs and releases)</a> · <a href="https://atomgit.com/ysr666/dsh-vision-router">AtomGit (China mirror)</a></p>
|
|
638
|
+
|
|
639
|
+
<p align="center">💬 <strong>QQ community group: 1105463028</strong></p>
|
|
640
|
+
|
|
623
641
|
## License
|
|
624
642
|
|
|
625
643
|
[MIT](LICENSE)
|